I have a large bunch of rows (read from a file) and I'm managing it with some elaboration.
The elaboration takes a time not proportional to the number of row, but exponential somehow. The first 10000 rows are elaborated in 2 minutes (let's say), the second 10000 in 5-6 minutes ... and so on.
My job is designed to write on DB the result, on a SubJobOk link, that is followed when all rows parsed (from file) and elaborated.
How could I split a result set (let say 40k rows) in blocks of 10k rows, like a loop?
Ideally, I would need a result-set splitter, a component that takes in input a row link and supply in output 4 times (consecutively) 10k rows.
You can leverage the Header field of file input components to skip numbers of rows. A tLoop component can execute a subjob a number of times, and exposes a CURRENT_VALUE variable that you can reference in other components.
You can leverage the Header field of file input components to skip numbers of rows. A tLoop component can execute a subjob a number of times, and exposes a CURRENT_VALUE variable that you can reference in other components.
Hi,
An other option is to read the whole input file a 1st time and write the content to a tFileOutputDelimited with the split option with a number of records per setted to 10k for example.
Then using a tFileList, iterate over the generated files to process the content by 10k rows at a time.
