topic Re: Big Data Spark Job - Load data into Hive in Talend Studio

Big Data Spark Job - Load data into Hive

Anonymous — Sat, 16 Nov 2024 09:47:07 GMT

I am creating big data spark job and want to load data into dynamic partitioned hive tables.

Which component I can use to load data into hive and what would be the workflow?

Anonymous — Mon, 15 May 2017 08:25:48 GMT

tHDFSConnection--> HIVE-->tHiveRow--->tFileInputDelimiter---->tHDFSOutput

Hive load data

tHiveRow configuration

tHiveRow1

tHiveRow2

tFileInput Delimiter

tHDFSoutput

1.tHDFSConnection set your connection to your Hadoop cluster

2.set connection to hive in HIVE component

3.tHiveRow drop the table if present

4.tHiveRow create external table table

5. tFileInutDelimiter load the data to HDFS external table location

6. tHDFSoutput to load data to external table path

Anonymous — Tue, 16 May 2017 06:43:26 GMT

Which version you are using? With 6.2.1, all these tHive component is not available. Only tHiveConfiguration, tHiveInput and tHiveOutput is there.

Data is loaded using spark job but facing problem when the Hive table is dynamically partitioned.

Thanks.

Anonymous — Wed, 17 May 2017 13:13:58 GMT

I am using Talend open studio for Big data Version TOS_BD-20150508_1414-V5.6.2 . try to Download those components from https://exchange.talend.com/.