topic Re: Load a CSV file into Hive Parquet table in Talend Studio

Load a CSV file into Hive Parquet table

Anonymous — Sat, 16 Nov 2024 06:56:33 GMT

Hello,

I have a CSV file with raw data and I'm trying to load it into Hive table that uses the Parquet format. I found a way to do this but I was wondering if there is an easier way to do it which would only require 1 single job.

Here's how I did it:

- a Big Data Batch job which reads the CSV file from HDFS (tFileInputDelimited) and outputs it as a Parquet file (tFileOutputParquet)

- a Standard job with just the tHiveLoad component which reads the Parquet file and loads it into the Hive table

My question is: is there a way to do this in 1 single job?

Many thanks,

Axel

Re: Load a CSV file into Hive Parquet table

vapukov — Thu, 03 Jan 2019 22:07:50 GMT

Hi Axel

what wrong with tHiveOutput ?

regards, Vlad

Re: Load a CSV file into Hive Parquet table

Anonymous — Thu, 03 Jan 2019 23:16:28 GMT

Hi Vlad, thanks for your reply. Are you saying that it should work fine if I connect tFileInputDelimited to tHiveOutput if I want the Hive table in Parquet format? Sorry, I'm fairly new to Talend.

Re: Load a CSV file into Hive Parquet table

vapukov — Thu, 03 Jan 2019 23:39:25 GMT

Why just not test?
It support parquet format

Re: Load a CSV file into Hive Parquet table

Anonymous — Fri, 04 Jan 2019 00:55:27 GMT

I tested it but I get an error "PartialGroupNameException Does not support partial group name resolution on Windows. Incorrect command line arguments."

Any clue what this means?

Re: Load a CSV file into Hive Parquet table

chabou19 — Fri, 28 Feb 2025 08:31:52 GMT

Hi,

If your Hive setup uses Kerberos authentication ... you must ensure it is correctly configured in Talend.

Best Regards.