Copy multiple files with pattern from S3 bucket to local unix server

Anonymous — Sat, 16 Nov 2024 06:54:45 GMT

Hi All,

Need suggestion for below:

I need to copy multiple files from S3 bucket to local unix server. I am using TS3List to iterate each file and passing tS3list current key to tS3Get. My files are in subfolders under S3 bucket ("inbound_bucket/DW/Vendor"). If i am giving this whole bucket in S3list bucket, am getting error as "tS3List_1 - Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler".

I tried below 2 options:

1. If am putting my files in top bucket itself -"inbound_bucket", then am able to copy into local unix server.

2. If am putting top bucket only - "inbound_bucket" in bucket and putting subfolders along with file pattern ("DW/Vendor/filepattern"), then files are getting copy in local along with S3 subfolders. This is what i get in local:

Lets say i have given unix local path in S3Get is /user/data. Then under this one new subfolder-/DW is getting created then another subfolder-/Vendor, then multiple files are getting copy-filepattern1.txt/filepattern2.txt.

Please suggest

Re: Copy multiple files with pattern from S3 bucket to local unix server

dipanjan93 — Tue, 08 Jan 2019 06:48:49 GMT

Could you please show your job design as well as component settings?

Re: Copy multiple files with pattern from S3 bucket to local unix server

Anonymous — Tue, 08 Jan 2019 06:55:14 GMT

Am sorry but i can't show job design.

What am mentioning in bucket and key prefix i have mentioned in query itself

topic Re: Copy multiple files with pattern from S3 bucket to local unix server in Talend Studio

Copy multiple files with pattern from S3 bucket to local unix server

Re: Copy multiple files with pattern from S3 bucket to local unix server

Re: Copy multiple files with pattern from S3 bucket to local unix server