Hey,
I am pretty sure talend should be able to do this task relatively easily, but I am not sure the best way to go about it.
I have 100,000 rows of data, but an API I am calling can only take 100 rows of data per API call.
I would like to execute an API call on 100 rows each time until I have looped through the full 100,000 row data set.
Any advice/recommended components on going about this is much appreciated.
Thanks,
Brian
If you need to filter also on other places in your job (even child or parent jobs) take care you give the sequence different names because they are static.
Ahh i see what youre saying, but Im not sure this solves my problem. I can only get it to process every 100th record alone, not all in between?
I'm trying to create an XML document to be processed when its 100 element large, once it hits that send the data off to the external service then create the next document with another 100 rows in until there arent anymore rows to process. Is there a way to achieve this?
I have an idea, please check if you can implement this.
- Create a sequence (rowID) for incoming rows using sequence generator
- Create a filter to filter records having rowID > 100 and write it to file
- Create subjob and process the file like first step above
- Once the file is processed, move the file and push to archive folder
- and repeat the process again till you have a file
Thanks
Vaibhav
@Vaibhav,
thanks for the response. I was really trying / hoping to stay away from saving files (as i know there is a split functionality in tAdvancedFileOutputXML), do you know if there are any other ways to keep it all in short term memory without saving to disk?
Okay, i could try. Ive only used these (tBufferOutput) before for passing data up to parent jobs though, how would i use this to pass data at the right time within the same job?
I have had a look here, I have tried something similar but couldnt get it to work correctly. I have already pulled all of the data out of the database, I want to loop through each of those records and on record 100 do something. Id rather not make a call to the DB every time I loop. Im really struggling to understand how to do this in Talend without it being really complicated 😞
