How to execute on every 100 rows of data from DB?

[Anonymous]

Hi,

I am pretty sure talend should be able to do this task relatively easily, but I am not sure the best way to go about it. I have 100,000 rows of data, but an tSOAP I am calling can only take 100 rows of data per tSOAP call. I would like to execute an tSOAP call on 100 rows each time until I have looped through the full 100,000 row data set. Any advice/recommended components on going about this is much appreciated. 

Thanks,

Pratap.

[Anonymous]

Any Update please

[JR1 · Accepted solution]

Hi

Not really sure what you have as an input but here I will assume it is a csv file. Set your job up like this (assuming you have a header in your input file):

tFileInputDelimited (pointing at your file just to count the records - no outputs) 
-- OnSubjobOK -->
tLoop (From: 0 To: ((Integer)globalMap.get("tFileInputDelimited_1_NB_LINE")), Step: 100)
-- Iterate --> 
tFileInputDelimited (pointing at your file again but this time with Header: 1 + ((Integer)globalMap.get("tLoop_1_CURRENT_VALUE")) and Limit: 100)
-- Main --> 
tSOAP (or whatever you want to do after this)

This will split up your input into batches of 100. If you are using a different kind of input (e.g. an Oracle table) the counting of the records and querying the table will look different of course.

Let us know if this helps.

[Anonymous]

Thanks
It seems like, working but i have 220 custoemrs in .csv file, but when every do the iteration with 100, then 300 customers are processed, i hope, duplicate records also processing? How we can fix this issue

[JR1]

Hi

Have you tried it? If you set the limit to 100 and there are only 20 records left from where you start (by setting the header value), it will only process those 20 records. I do not really know what you mean with duplicates.

[Anonymous]

Yes, i tried, but getting some of customer coming again into flow ,
Below implementation where i did,
tFileInputDelimited (pointing at your file just to count the records - no outputs) - Just file location given, i didn't given any header and limit here, is it do any thing other then file
-- OnSubjobOK -->
tLoop (From: 0 To: ((Integer)globalMap.get("tFileInputDelimited_1_NB_LINE")), Step: 100)
-- Iterate -->
tFileInputDelimited (pointing at your file again but this time with Header: 1 + ((Integer)globalMap.get("tLoop_1_CURRENT_VALUE")) and Limit: 100)- same i did, i didn't done anything other then above
-- Main -->
tSOAP (or whatever you want to do after this)-used

[JR1]

Now I get what you mean with duplicates. You will have to look into the calculations to make sure that a single record is not processed in another batch again. I do not know if your file has any headers or footers in it. Please try to understand the calculations done here and try to adapt them to your needs. Unfortunately, I cannot present you with a perfect solution here.

[Anonymous]

here what is happening, i have file with values
123451
123456
124578
124574
125485
till 1000 rows
Ourput: 123451,123456,124578,124574,125485.. 100rows
123456,124578,124574,125485.. 100rows
124578,124574,125485.. 100rows
124574,125485.. 100rows
like that displaying, next row start with previous row 2nd id/customer, so its sending duplicate ids
Suggest if have any solution.

[JR1]

There is something wrong with the "header" value in the second file component. Looking at this, I would assume that you have chosen ((Integer)globalMap.get("tLoop_1_CURRENT_ITERATION"))) instead of ((Integer)globalMap.get("tLoop_1_CURRENT_VALUE"))). Correct?

[Anonymous]

Yes, Exactly, its working fine now.Thanks