Issue with aws s3 components(ts3get)

[Anonymous]

Hi ,

   I am trying to download all files in my s3 bucket in one go.

  I am able to list all files using ts3list but not able to download all files .

  ts3get is only downloading the last file.

  Please guide me through this or this function is not applicable in talend.

Please go the below link , I am getting same issue.

https://community.talend.com/t5/Design-and-Development/Not-able-to-get-all-the-files-from-S3-bucket-...

[cterenzi]

If you followed the job design in the link you provided, the cause of the issue is the trigger link between tS3List and tS3Get. To get all the files found by tS3List you need to connect it to tS3Get with an Iterate link.

[Sid3]

did you understand the solution provide by cterenzi?

Please post an update we can help you if need further help

[Anonymous]

I have written  a python code to download all files from my s3 bucket and running my code from tsystem component.

[Vinlogs]

Python script to extract s3 files seem to be a great idea! share with us how you did it or personal message me if possible, Iterating the output was all you need if you are trying to download s3 files using Talend components,don't hesitate to get back to me on this, if you are still looking for any help on this as I have already developed a similar job in our project. Thanks.

[NMangaba_TAP]

Hi Vineet,

Please could you provide an example of how you overcame this issue as I am current facing the same issue. 

I am trying to retrieve all files from a bucket with different names (key prefix), can this be done irrespective of the file name ? 

Regards,

Naledi

ts3get example.PNG

[NMangaba_TAP]

Hi Rohitthapliyal, 

Hi Vineet,

Please could you provide an example of how you overcame this issue with the python code, as I am currently facing the same issue. 

I am trying to retrieve all files from a bucket with different names (key prefix), can this be done irrespective of the file name ? 

Regards,

Naledi

ts3get example.PNG

[Anonymous]

Hi Naledi,

     It is possible if you add Key Prefix in your tS3list as blank. Then i will pick all the files from the bucket.

Also make sure that Key and local file name is having current key value from the previous list.

I tried to move multiple files and all of them got downloaded successfully.In your screenshot, you have given static file name for the local path. So the data must be getting overwritten.

Please let me know whether this method has resolved your issue.

Warm Regards,

Nikhil Thampi