Not able to Download all files from AWS S3 bucket to Local

[Anonymous]

Hello,

 I want to download all the files from AWS S3 bucket to my local directory.

I am using Talend flow as:

tS3connection->tS3List->(Iterate)->tS3Get->tS3Close.

The issue is only last file from S3 bucket is getting downloaded. All files are not downloaded.

As per the community discussions, this should work, but it is not working for me.

tS3List,tS3Get settings details are attached.

Can anyone help me out with the issue.

Thank you

[Anonymous]

Hi,

   Could you please refer a similar issue below? I believe your current key value is allowing to pick only one file.

https://community.talend.com/t5/Design-and-Development/Unable-to-put-multiple-files-in-Amazon-S3/td-...

   Could you please chnage the values to key filed and try to print the current file value in a tjava and see the results?

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[Anonymous]

Hi Nikhil,

 Thank you for your inputs.

I tried changing the key now.

1st file got downloaded, but got error after that.

Error is:

Exception in component tS3Get_1 (Job_try_out)
java.lang.IllegalStateException: Connection pool shut down

Any idea why this error is coming after downloading one file from AWS S3?

[iamabhishek · Accepted solution]

The issue is not with the key but with the job design. After your iteration you have "On Component ok" to close the S3 connection as a result once your first iteration is done it would call the tS3Close and the connection to S3 would be closed which would give you the error - "Connection pool shut down". Instead close your connections using tPostJob and likewise open all your connections using tPrejob.

[Anonymous]

Hi Abhishek,

 Thanks for your help. I did not recognize that.

I am able to download all files from S3 now.

thank you