topic Dynamic Schema in Talend Studio

Dynamic Schema

Anonymous — Sat, 16 Nov 2024 04:06:06 GMT

I am evaluating Talnd Big Data edition for data consumption job.
I have a set of 5 tables , my requirements is join these five tables and generate subset of columns as o/p in a hive table.
Number of Output columns are not fixed , that is why I need to generate schema dynamically.
I am using tHiveInput to read the data and tsqrow to join these five tables.
Questions -
1. Can tSqlRow generate schema dynamically ?
2. I can use tjava but the problem with t Java is it can't take more than one input.
3. I can't find tSetDynamic component in talend big data studio.
Thanks

Re: Dynamic Schema

Anonymous — Tue, 19 Nov 2019 03:21:14 GMT

Hi,

I would recommend to go through the below blogs.

https://www.talend.com/blog/2019/10/31/magically-shuffle-columns-talend-dynamic-schema/

https://www.talend.com/blog/2019/11/11/migrate-data-between-databases-one-job-using-dynamic-schema/

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: Dynamic Schema

Anonymous — Tue, 19 Nov 2019 22:02:35 GMT

Thanks for the response.

However the links are not applicable to Talend Big Data Jobs.

Talend Big Data Job does not support dynamic schema which is a major limitation. Is there a workaround of this limitation ?

if we cant overcome this limitatatin we cant really use talend , in that case we need to write spark code in pure java

Thanks