topic Re: tHiveCreateTable in Talend Studio

tHiveCreateTable

csapparapu — Sat, 16 Nov 2024 03:57:13 GMT

Hello,

Why is tHiveCreateTable not a Big Data batch job component?

I find this component in Standard job design.

What can I do to see it in my Palette for a big data batch job?

Thanks,

Chandana

Re: tHiveCreateTable

Anonymous — Tue, 03 Dec 2019 07:38:50 GMT

Hello,

We have tHiveRow and tHiveCreateTable components in Standard Job, do you want to get supporting hive partition and bucketing in Big Data Batch Spark job?

So far, starting from V 6.4.1, we have already supported writing partitioned tables.

Best regards

Sabrina

Re: tHiveCreateTable

csapparapu — Tue, 03 Dec 2019 15:26:49 GMT

@xdshithanks for your reply. I cannot create a hive table in a Big Data Batch job in Talend Big Data Platform 7.X.

Yes, I would like to create a partitioned table, but right now, I cannot create ANY table in Big Data.

I am concerned with Big Data only.

Since Hive is a Big Data framework, my question is, why is it available in standard jobs, but not Big Data jobs.

Can somebody tell me what I can do to make this available in Big Data?

Re: tHiveCreateTable

Anonymous — Wed, 04 Dec 2019 03:06:10 GMT

Hello,

The bucketing feature is not yet supported by Spark 2.1 itself.
The Apache Spark jira (https://issues.apache.org/jira/browse/SPARK-19256) is in progress with some pull requests opened but not yet targeted for 2.2.
On the Talend side, we won't be able to work this until it's integrated into the Spark core.

If you use Parquet components (format I strongly advise to use behind Hive), you can already partition with Spark and easily combined with Hive DI createTable (declaring it as an external table_).

Best regards

Sabrina

Re: tHiveCreateTable

csapparapu — Wed, 04 Dec 2019 15:44:19 GMT

@xdshicould you please tell me how to use tHiveCreateTable, I can think about advanced features later?

I am looking for a tHiveCreateTable in Talend Big Data Platform, Big Data Batch job.

Re: tHiveCreateTable

csapparapu — Mon, 16 Dec 2019 15:03:51 GMT

@xdshi, could you please tell me how to use tHiveCreateTable, I can think about advanced features later?

Re: tHiveCreateTable

Anonymous — Tue, 17 Dec 2019 07:04:58 GMT

Hello,

Are you able to use tHiveCreateTable component to create the hive external table in a DI standard job? And you can call a child spark job by tRunJob component.

Best regards

Sabrina