topic Passing multi-row dataset to subjob in Talend Studio

Passing multi-row dataset to subjob

Anonymous — Wed, 30 Sep 2015 22:14:32 GMT

Folks,
I have a sub job that takes input, two flat files, parses them out and joins them based on a common key - the final resultset is returned back to the parent job. I now want to pass this dataset to another subjob to handle some data conversion. I'm trying to find a good example of how to pass a multi-row dataset to a sub job. I've searched the forums and the help guides with little success. Can anyone either point me in the right direction or give me an example of how it is done?
Currently I am using a tFlowToIterate to pass one row at a time from the parent to the sub job but I find this mechanism less than ideal.
thanks!

Re: Passing multi-row dataset to subjob

Anonymous — Thu, 01 Oct 2015 01:04:52 GMT

Hi dpower, this can done using a bit of Java as in this tutorial ( http://www.rilhia.com/tutorials/talend-connect-example - this is actually for something else, but uses a method to pass datasets to subjobs that works for me). You can also do it by using this component ( tDataRowToContext - Found in Talend Exchange) that I put together for this very purpose.

Passing large datasets to child jobs can use a lot of memory, so beware of heap space issues.

Re: Passing multi-row dataset to subjob

Anonymous — Thu, 01 Oct 2015 15:58:58 GMT

Thanks for the reply! I will take a look at the two approaches you've outlined...
D.