topic Re: TRestClient and batch execution in Talend Studio

TRestClient and batch execution

Anonymous — Wed, 25 Jan 2017 14:25:43 GMT

Hello guys! That's my first interaction here and if I did something wrong, just let me know, ok?
I started to work with Talend one month ago I have some good jobs that I developed and are working well, but now I have a situation that I still don't figure out how to solve.
My need is to call a RESTful service and send a defined amount of records to the API, that'll treat the data in the database. The point is that some execution I have a high number of records and the API isn't able to handle this amount, so I need to break the API call in parts, like a batch execution. E.G. Call the API each 500 rows, get the response and call it again until I finish all the data that must be integrated.
I attached an image of my current test job which I'm generating a X number of rows, converting each row to JSON and join all together in one string and send it to tRestClient. The first part of the job is just to request authorization to login into the API. (tRestClient_1 = tExtractJSONFields_1 = tJavaRow)
Any further information, just let me know
Thanks in advance

Re: TRestClient and batch execution

TRF — Wed, 25 Jan 2017 14:39:56 GMT

Can't see your job.
What's your question?
What do you expect?
Regards,
TRF

Re: TRestClient and batch execution

Anonymous — Wed, 25 Jan 2017 14:50:36 GMT

I paste the image here, but nothing happens in the post, I can see it when I'm writing, but not when It's posted.
My question is how to develop this job to break the REST calls in "parts", like batch execution.

Basically my process is:
TRowGenerator == TMap == TWriteJSONFields == tRestClient

Re: TRestClient and batch execution

TRF — Wed, 25 Jan 2017 15:07:35 GMT

Simple way,
tRowGenerator > tMap > tFileOutputDelimited complete "File Name" field (for ex. C:/blabla/yourfile_.csv will generate yourfile_0.csv, yourfile_1.csv and so on) + tab Advanced settings, check "Split output in several" files then 500 in "Rows in each output file" field + check "Custom the flush buffer size" then 1 in "Row number" field
On subjob Ok (from tRowGenerator) > tFileList (C:/blabla/yourfile_*.csv) > tFileInputDelimited > tWriteJSONFields > tRestClient
Hope this helps,
TRF

Re: TRestClient and batch execution

Anonymous — Wed, 25 Jan 2017 15:38:07 GMT

I think I understand what you need to do. When you call the service and it has to return more records than one response can handle, does it return a new URL for you to call to get the next 500? So does it work in this sort of scenario.....
1) Send initial request
2) Receive response of first 500 records + a URL for next 500 records
3) Call new URL
4) goto 2
If so, I used this method in a tutorial I posted a while ago. The Facebook functionality may have changed, but the idea will work. The tutorial is here: https://www.rilhia.com/tutorials/using-talend-get-your-spotify-listening-history-facebook

Re: TRestClient and batch execution

TRF — Wed, 25 Jan 2017 16:00:51 GMT

Seems much more sophisticated than the approach I proposed.
Should be better.
TRF

Re: TRestClient and batch execution

Anonymous — Wed, 25 Jan 2017 16:16:28 GMT

Simple way,
tRowGenerator > tMap > tFileOutputDelimited complete "File Name" field (for ex. C:/blabla/yourfile_.csv will generate yourfile_0.csv, yourfile_1.csv and so on) + tab Advanced settings, check "Split output in several" files then 500 in "Rows in each output file" field + check "Custom the flush buffer size" then 1 in "Row number" field
On subjob Ok (from tRowGenerator) > tFileList (C:/blabla/yourfile_*.csv) > tFileInputDelimited > tWriteJSONFields > tRestClient
Hope this helps,
TRF

Thanks, I think that this approach will make me get in.