topic How to create multiple rows out from a row input in Talend Studio

How to create multiple rows out from a row input

Anonymous — Sat, 16 Nov 2024 13:20:14 GMT

Hi,
I have found a way to create multiple rows - something I was unable to do as simply as shown below -, but I do not like this "hack" that much. I hope someone will share a better way to do it.

Explanation
The trick is to send the output from one componen to a LogRow in order to have the ouput flow defined and then send the iterate connection to a tJavaFlex that will do the actual replication

Good Luck !
Alberto

Re: How to create multiple rows out from a row input

alevy — Tue, 27 Jul 2010 02:08:47 GMT

How about this?

Re: How to create multiple rows out from a row input

Anonymous — Tue, 27 Jul 2010 07:08:23 GMT

Thanks for your example. You do not show the setup for tLoop_4 but it seems that you have a For Loop from 1 to 10.
If I understand correctly, we create a Lookup table of 10 records with a field - named iteration - that has a value equal to its record number. Then we create the Cartesian product of the Lookup against the input data and filter out the records which have a RowCount field larger than iteration.
It is very interesting, I have problems understanding the iteration components so your example helps on my understanding. The thing I do not like is that I have certain rows that have to be duplicated once or twice while others have to be duplicated over a thousand times. Thus, I would have to create a loop of a 1,000 for each input row. It might be a bit slow.
Thanks for teaching me how ot use the Iteration components.
Regards,
Alberto

Re: How to create multiple rows out from a row input

alevy — Tue, 27 Jul 2010 07:43:57 GMT

Yes, my example is a loop from 1 to 10.
I'm not sure how tMap actually works behind the scenes but I don't think that this approach will create a Cartesian product and then filter. It's intelligent enough to apply the filter as a join condition. (It would be different if the filter were performed after the tMap.)
You also should be aware that the lookup flow is created (including the iteration) once only before any incoming rows (even if there is an iteration on the main flow).
Experimenting with 30,000 input rows being duplicated 900 or more times each (for a total of 28.5 million output rows) and a lookup flow of 10,000 rows, the job ran to a file in 35 seconds.

Re: How to create multiple rows out from a row input

Anonymous — Tue, 27 Jul 2010 17:56:07 GMT

Hi Alevy,
Thanks again for your help. I will give it a try. If I find anything interesting I will let you know.
Thanks again !