topic Re: Talend component to loop through API to download multiple files in Talend Studio

Talend component to loop through API to download multiple files

ManjunathBhat — Fri, 15 Nov 2024 22:00:19 GMT

Hi Team,

new to Talend and we have a contract expiring. Need help.

Issue: We have an application who hosts our files

They have provided us with API credentials using OAuth 2.0

one API URL gives a list of all available documents.

https://api.icims.com/customers/{customerId}/forms/list

The other one gives option to download the file as PDF but have to pass formname and formdataID

https://api.icims.com/customers/{customerId}/forms/{formname}/data/{formDataId}.pdf

I am guessing I have to first read through first URL to get all names and ID and then dynamically loop and pass them to second URL to download files?

Can anyone help me what components to use, authenticate and download forms?

Re: Talend component to loop through API to download multiple files

Anonymous — Sun, 12 Mar 2023 01:36:42 GMT

First of all, try the first URL using tRest or tHttpRequest component to see if you are able to call the API successfully, set OAuth 2.0 bearer token as one of header.

Second, if step1 works, check what data does the API contains in the response? use a tExtractXMLFields to extract the required data from the response string.

Third, iterate the extracted data and call the second URL. The job looks like:

tHttpRequest --main--tExtractXMLFields--main--tFlowToIterate--iterate--tHttpRequest

Regards

Shong

Re: Talend component to loop through API to download multiple files

ManjunathBhat — Sun, 12 Mar 2023 05:54:26 GMT

Hello,

Thank you for your response. I followed your steps, please forgive my mistakes as I am very new to this and to API as well. I setup as in below screenshot. The vendor has provided me just these details client ID and client secret. When I run the job I am getting second screenshot error

Sorry I think I might be doing something wrong while setting the Outh 2.0 Bearer token?

Re: Talend component to loop through API to download multiple files

Anonymous — Mon, 13 Mar 2023 07:08:38 GMT

Using a tLogRow to print the reponse the console. The job looks like

tHttpRequest--main--tLogRow

If you fill in the header, you have to set the key and value.

I think you need to check the 'Need authencation' box and fill in the client id and client secret.

Regards

Shong

Re: Talend component to loop through API to download multiple files

ManjunathBhat — Wed, 15 Mar 2023 17:59:22 GMT

Hi Shong,

Thank you.

I was able to get the credentials from vendor and use basic authentication. I am also now able to connect to API.

There is an API "https://api.icims.com/customers/{customerId}/forms/{formDataID}.pdf"

The minimum value for formdataID is 11 and the maximum value around 600000

I used tFileFetch for sample number and I was able to download the pdf file.

Now problem is I have to iterate from 11 to 600000 and for each run pass the incremented value dynamically so that job picks up the next value pass that as parameters to the URI and fetch next file. how can i do this? sorry again new to talend

like in below example for each run the number 2289 need to increment to next value

Re: Talend component to loop through API to download multiple files

Anonymous — Wed, 15 Mar 2023 23:05:02 GMT

You can use tLoop component to iterate each formdataID, eg:

tLoop--iterate--tFileFetch.

on tLoop: select 'for' type, start 11 to 600000.

on tFileFetch: set a dynamic URI:

"https://server/xxxx/forms/"+((Integer)globalMap.get("tLoop_1_CURRENT_VALUE"))+"pdf"

Hope it helps!

Regards

Shong

Re: Talend component to loop through API to download multiple files

ManjunathBhat — Thu, 16 Mar 2023 14:49:02 GMT

Hi Shong,

Thank you so much, this really helps. I got started and now able to pass the value from tloop to tfilefetch.

One issue I ran into is from number 11 to 600000 in between there are few numbers where the form does not exist

ex: 11.pdf works, and all way from 12 to 48 it does not exists and again starts from 49 goes till 120 and stops and starts back again at 170

Everytime the tfilefetch does not find a file it fails.

below is the error response

{

"errors": [

{

"errorMessage": "A form with the id of 12 does not exist.",

"errorCode": 2

}

]

}

How can I catch the response from API check if the file exists(response is ok) then go to filefetch if not go back to loop increment and check again ex below

tLoop-iterate-checkAPIforError-NoErrorgotoTfilefetch

tloop-iterate-checkAPIforerror-iferrorgobacktoloop-tloop-iterate-checkAPIforError-NoErrorgotoTfilefetch

Re: Talend component to loop through API to download multiple files

Anonymous — Fri, 17 Mar 2023 01:53:16 GMT

About checkAPIforError, you need to check with vendor whether there exist an API available for checking the file if it exists or not. If no, need to skip this step, create the job as below:

main job:

tLoop--iterate--tRunJob

on tRunJob, call the child job, pass the current iteration value to child job using context variable, please see this documentation page about how to do it, uncheck the 'die on error' box.

child job:

tFileFetch

on tFileFetch: set a dynamic URI:

"https://server/xxxx/forms/"+context.varName+"pdf"

//context.varName is a context variable used to receive the current iteration value from main job.

if you need to catch the error message, use a tLogCatcher component in child job, and output the error to a log file.

tFileFetch

tLogCatcher --main--tFileOutputDelimited

on tFileOutputDelimited: check the 'append' box to append the error message to the same file.

Regards

Shong

Re: Talend component to loop through API to download multiple files

ManjunathBhat — Fri, 17 Mar 2023 05:18:11 GMT

Hi Shong,

You rock, thank you so much, one last question.

Everything now works as expected and even able to catch error and append to log.

One modification here is it throws standard "java.lang.Exception:Method failed: HTTP/1.1 404 ;1"

Instead of this can I write the contextvariable value so that I can add to file which filenumber does not exist?

Like example when child job runs for 12 it fails and logs error and along with this how can I append the current context.varName value to erroroutput file?

Many thanks again

Re: Talend component to loop through API to download multiple files

Anonymous — Fri, 17 Mar 2023 07:13:46 GMT

tLogCatcher has a predefined schema, however, you can add extra columns on tMap, eg:

tLogCatcher--main--tMap--tFileOutputDelimited

on tMap, add a new output, drag and drop all the columns from input data flow, add a new column called "invalidFormDataId" for example, set its value expression as: context.varName

this will output the invalid formdataID to the log file.

Re: Talend component to loop through API to download multiple files

ManjunathBhat — Mon, 20 Mar 2023 17:36:52 GMT

Hi Shong,

Thank you so much. Everything is now working as expected.

One query I have is, the tfilefetch download the file to my local desktop, instead is there a way Instead of downloading files to my local desktop, Can i send/upload these files to S3 Bucket?

Regards

Re: Talend component to loop through API to download multiple files

Anonymous — Tue, 21 Mar 2023 01:16:48 GMT

Check the 'Use cache to save resource' box to read the file as inputstream object, then, set the File/Stream field of tS3Put component as:

((java.io.InputStream)globalMap.get("tFileFetch_1_INPUT_STREAM"))

tFileFetch_1--oncomponentok--tS3Put_1

Please try and let me know if it works.

Regards

Shong

Re: Talend component to loop through API to download multiple files

ManjunathBhat — Tue, 21 Mar 2023 05:06:36 GMT

Hi Shong,

I tried it but somehow its still writing to my local desktop. I even deactivated and added new tfilefetch but still its writing to my local desktop

Re: Talend component to loop through API to download multiple files

Anonymous — Tue, 21 Mar 2023 06:31:15 GMT

I tested with v8.0.1 and it works well on my side. when you check 'Use cache to save resource' box, the destination directory field disappear, you have no option to set up a directory path, so I don't understand you said the file is still downloaded to local desktop.