topic Re: result-set splitter in Talend Studio

result-set splitter

Lorenzo5 — Fri, 09 Jun 2017 13:50:48 GMT

Hi,

I have a large bunch of rows (read from a file) and I'm managing it with some elaboration.

The elaboration takes a time not proportional to the number of row, but exponential somehow. The first 10000 rows are elaborated in 2 minutes (let's say), the second 10000 in 5-6 minutes ... and so on.

My job is designed to write on DB the result, on a SubJobOk link, that is followed when all rows parsed (from file) and elaborated.

How could I split a result set (let say 40k rows) in blocks of 10k rows, like a loop?

Ideally, I would need a result-set splitter, a component that takes in input a row link and supply in output 4 times (consecutively) 10k rows.

Any idea how to achieve this task?

Thank you,

Lorenzo

Re: result-set splitter

cterenzi — Fri, 09 Jun 2017 14:43:20 GMT

You can leverage the Header field of file input components to skip numbers of rows. A tLoop component can execute a subjob a number of times, and exposes a CURRENT_VALUE variable that you can reference in other components.

Re: result-set splitter

TRF — Fri, 09 Jun 2017 20:31:21 GMT

Hi,
An other option is to read the whole input file a 1st time and write the content to a tFileOutputDelimited with the split option with a number of records per setted to 10k for example.
Then using a tFileList, iterate over the generated files to process the content by 10k rows at a time.