How to create a generic job in talend

[Anonymous]

Hi Team,
Could you please confirm if we can create a generic job in talend which performs different operations on different sets of data.
Basically, we want to process multiple source files (having different schema) with different transformations but the overall processing flow in the job is same for all source files. (Eg a simple Job Flow : FileInputDelimited -> TMap -> tFileOutputDelimited).
If we can create such a generic job, then please help on below points :
1) How to save and pass different schema definition  for different source files via parameter/file
2) How to pass key field via parameter/file
3) How to save and pass different transformations  via parameter/file
Thanks
Ankit Gupta

[Anonymous]

Hi 
It is impossible to do that, the schema must be defined before the job execution.
Regards
Shong

[Anonymous]

Hi,

This is a work around, which would work only for specific functions. Also, it will be tedious task to do so.

But you can create above given generic job.

Using tfileinput > Tjava > tfileoutput

In tfileinput use schema as dynamic.

In tjava ,use dynamic datatype and meta functions provided with dynamic type( like getColumnName, getColumnType ) to perform required functions. 

In tfileoutput use dynamic schema to store file data.

[Anonymous]

Hi,

    You might have to do custom coding but dynamic schema will help you to make it within a single job. A simple example can be referred from below Blog.

https://www.talend.com/blog/2019/10/31/magically-shuffle-columns-talend-dynamic-schema/

    But you will have to make necessary changes based on your use case.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved