Guys, I recently started looking at Talend Open Studio for Big data and I like most of the options available. But i have questions with Dynamic column type and it is not supported in Big Data components, is there any options available to accomplish this. Like SSIS(Microsoft ETL tool) can i extend the functionality, for example changing the schema on the fly or creating an api to generate talend packages programmatically. Thanks for your help. Anand.
Hi,
Are you looking for "Dynamic Schema" which allows you to design schema with an unknown column structure (unknown name and number of columns)?
TalendHelpCenter:How to process changing data structure?.
The dynamic schema functionality is only available in Talend enterprise subscription version not open source.
Best regards
Sabrina
Thanks Sabrina.
But Dynamic schema was not supported in Big Data version, in our use cases we want to reuse the packages.
Again is there a way change the schema at run time in Big Data/ regular enterprise version?
Or can we extend Talend API to achieve this?
Thanks,
Anand.
I recently started looking at Talend Open Studio for Big data and I like most of the options available.
The dynamic schema functionality is only available in Talend enterprise subscription version.(Talend Enterprise big data.. not open source).
So far, schemas must be defined during design, not run time.
Please have a look at talend KB article
https://help.talend.com/pages/viewpage.action?pageId=5671106.
Best regards
Sabrina
Thanks for the quick reply.
Yes I am evaluating Talend Big Data Enterprise edition and it does not support Dynamic data type for Big Data components, one example i have tested so far tHiveInput?
Do you have any sample job for Hive Input to File Output or Vice Versa with Dynamic Column Data Type?
Thanks,
Anand.
Hi, So far, thivexxx components don't support for "dynamic schema". You can go to /plugins/org.talend.core.tis_x.x.x.rxxxxx.jar. unzip this jar to a directory, and you will see a file call supportDynamic.txt under resources folder, the file lists all the components support the dynamic schema in the current version. Best regards Sabrina
