topic Re: Merge records from multiple input links and output as one output link in Talend Studio

Merge records from multiple input links and output as one output link

Anonymous — Fri, 17 Jul 2015 03:45:58 GMT

Is there any component that allows records from multiple input links to merge (union) and pass out as one output link ? I tried using tUnite component but it does not accept links from loop flows. I also do not want to put it into memory or temp file as it affects performances. See below.
Input Link1
id , Name
1,david
2,John

Input Link 2
id , Name
3,lester
4,Jane

Output Link
id , Name
1,david
2,John
3,lester
4,Jane

Re: Merge records from multiple input links and output as one output link

Anonymous — Fri, 17 Jul 2015 05:13:10 GMT

Hi,

I tried using tUnite component but it does not accept links from loop flows.

What's kind of loop flow you are using? From your description, you can design your work flow as: tfilelist-->tfileinputdelimited-->tfileoutdelimited.
Please take a look at my screenshots

Best regards
Sabrina

Re: Merge records from multiple input links and output as one output link

Anonymous — Fri, 17 Jul 2015 13:03:10 GMT

I wanted to merge multiple outputs from a tmap component with the same schema for example. But how can i do that ? After transformation from tmap, I wanted my records to be together again for further processing.

Re: Merge records from multiple input links and output as one output link

Anonymous — Fri, 17 Jul 2015 14:11:16 GMT

This is a very strange requirement. If you want all of your outputs from a tMap to be merged together again, why separate them in the tMap? You say they have the same schema, so I can only imagine it is the content that is different per output. If you are immediately wanting to merge these again, why separate them at all?
I am sure I either have misunderstood or there is a reason I have missed. But to merge them together again, you can use the tUnite. All you need to do is output from the tMap to several tHashOutput components. Then, in another subjob, use tHashInput components for each of the outputs and use a tUnite to merge these.
tHashInput1\
tHashInput2-tUnite-->
tHashInput3/
Basically, once you have split a flow in a subjob, you cannot re-merge it again. So in a tMap, if you split the flow into several flows, the only way to remerge them is to start a new subjob and do it from there. The tHash components are very useful for doing this in memory.

Re: Merge records from multiple input links and output as one output link

Anonymous — Sat, 18 Jul 2015 14:24:30 GMT

This is a very strange requirement. If you want all of your outputs from a tMap to be merged together again, why separate them in the tMap? You say they have the same schema, so I can only imagine it is the content that is different per output. If you are immediately wanting to merge these again, why separate them at all?
I am sure I either have misunderstood or there is a reason I have missed. But to merge them together again, you can use the tUnite. All you need to do is output from the tMap to several tHashOutput components. Then, in another subjob, use tHashInput components for each of the outputs and use a tUnite to merge these.
tHashInput1\
tHashInput2-tUnite-->
tHashInput3/
Basically, once you have split a flow in a subjob, you cannot re-merge it again. So in a tMap, if you split the flow into several flows, the only way to remerge them is to start a new subjob and do it from there. The tHash components are very useful for doing this in memory.

In tmap i did multiple validations and output in different links .. these are records rejects which includes my own reject reasons. After output from tmap i need to combine the links together again so that i can output to one single file concurrently. So i need something to merge all output links together, so that only one link writes to one file using tdelimitedfile component.

Re: Merge records from multiple input links and output as one output link

Anonymous — Mon, 20 Jul 2015 08:23:49 GMT

Hi feige82,

In tmap i did multiple validations and output in different links .. these are records rejects which includes my own reject reasons. After output from tmap i need to combine the links together again so that i can output to one single file concurrently. So i need something to merge all output links together, so that only one link writes to one file using tdelimitedfile component.

Would you mind sharing your current job design screenshot with us?
Best regards
Sabrina

Re: Merge records from multiple input links and output as one output link

Anonymous — Mon, 20 Jul 2015 09:05:43 GMT

You can do this with the method I suggested. Output to tHashOutput components in your subjob with the tMap. Then in the next subjob use a tUnite to merge the records back together from linked tHashInput components. All you have to do is ensure the schemas are the same.
Question to Talend: Why am I being told....?
" Your post is considerate as SPAM. Please verify the content and try again"