Talend component to loop through API to download multiple files

[ManjunathBhat]

Hi Team,

new to Talend and we have a contract expiring. Need help.

Issue: We have an application who hosts our files

They have provided us with API credentials using OAuth 2.0

one API URL gives a list of all available documents.

https://api.icims.com/customers/{customerId}/forms/list

The other one gives option to download the file as PDF but have to pass formname and formdataID

https://api.icims.com/customers/{customerId}/forms/{formname}/data/{formDataId}.pdf

I am guessing I have to first read through first URL to get all names and ID and then dynamically loop and pass them to second URL to download files?

Can anyone help me what components to use, authenticate and download forms?

[ManjunathBhat]

Hi Shong,

Thank you so much. Everything is now working as expected.

One query I have is, the tfilefetch download the file to my local desktop, instead is there a way Instead of downloading files to my local desktop, Can i send/upload these files to S3 Bucket?

Regards

[Anonymous]

Hi

Check the 'Use cache to save resource' box to read the file as inputstream object, then, set the File/Stream field of tS3Put component as:

((java.io.InputStream)globalMap.get("tFileFetch_1_INPUT_STREAM"))

tFileFetch_1--oncomponentok--tS3Put_1

Please try and let me know if it works.

Regards

Shong

[ManjunathBhat]

Hi Shong,

I tried it but somehow its still writing to my local desktop. I even deactivated and added new tfilefetch but still its writing to my local desktop

[Anonymous]

I tested with v8.0.1 and it works well on my side. when you check 'Use cache to save resource' box, the destination directory field disappear, you have no option to set up a directory path, so I don't understand you said the file is still downloaded to local desktop.