topic Re: Dynamically generate Talend packages in Talend Studio

Dynamically generate Talend packages

Anonymous — Sat, 16 Nov 2024 11:39:13 GMT

Guys,
I recently started looking at Talend Open Studio for Big data and I like most of the options available.
But i have questions with Dynamic column type and it is not supported in Big Data components, is there any options available to accomplish this.
Like SSIS(Microsoft ETL tool) can i extend the functionality, for example changing the schema on the fly or creating an api to generate talend packages programmatically.
Thanks for your help.
Anand.

Re: Dynamically generate Talend packages

Anonymous — Mon, 12 May 2014 03:55:38 GMT

Hi,
Are you looking for "Dynamic Schema" which allows you to design schema with an unknown column structure (unknown name and number of columns)? TalendHelpCenter:How to process changing data structure?.
The dynamic schema functionality is only available in Talend enterprise subscription version not open source.
Best regards
Sabrina

Re: Dynamically generate Talend packages

Anonymous — Mon, 12 May 2014 10:07:32 GMT

Thanks Sabrina.
But Dynamic schema was not supported in Big Data version, in our use cases we want to reuse the packages.
Again is there a way change the schema at run time in Big Data/ regular enterprise version?
Or can we extend Talend API to achieve this?
Thanks,
Anand.

Re: Dynamically generate Talend packages

Anonymous — Mon, 12 May 2014 10:44:34 GMT

Hi,

I recently started looking at Talend Open Studio for Big data and I like most of the options available.

The dynamic schema functionality is only available in Talend enterprise subscription version.(Talend Enterprise big data.. not open source).
So far, schemas must be defined during design, not run time.
Please have a look at talend KB article https://help.talend.com/pages/viewpage.action?pageId=5671106.
Best regards
Sabrina

Re: Dynamically generate Talend packages

Anonymous — Mon, 12 May 2014 19:30:04 GMT

Thanks for the quick reply.
Yes I am evaluating Talend Big Data Enterprise edition and it does not support Dynamic data type for Big Data components, one example i have tested so far tHiveInput?
Do you have any sample job for Hive Input to File Output or Vice Versa with Dynamic Column Data Type?
Thanks,
Anand.

Re: Dynamically generate Talend packages

Anonymous — Tue, 13 May 2014 09:11:02 GMT

Hi,
So far, thivexxx components don't support for "dynamic schema".
You can go to /plugins/org.talend.core.tis_x.x.x.rxxxxx.jar.
unzip this jar to a directory, and you will see a file call supportDynamic.txt under resources folder, the file lists all the components support the dynamic schema in the current version.
Best regards
Sabrina

Re: Dynamically generate Talend packages

Anonymous — Tue, 13 May 2014 19:37:44 GMT

Thanks Sabrina.

Re: Dynamically generate Talend packages

Anonymous — Wed, 14 May 2014 05:25:36 GMT

Hi,
It's welcome to open a feature jira issue of DI project on Talend Bug Tracker.
Best regards
Sabrina