topic Re: Migrating multiple S3 files from one Bucket to Another Bucket in Talend Studio

Migrating multiple S3 files from one Bucket to Another Bucket

Karuetl — Sat, 16 Nov 2024 05:37:19 GMT

Under the source bucket, there are multiple folders and in each folder, there are multiple files

As the example below

/folder/export=2019-06-01/
/folder/export=2019-06-02/
/folder/export=2019-06-03/
/folder/export=2019-06-04/

I want to move each file from each folder from the source bucket to the Target Bucket as below, Basically, files should be under the same date as in source bucket

/abc/2019-06-01/
/abc/2019-06-02/
/abc/2019-06-03/
/abc/2019-06-04/

Re: Migrating multiple S3 files from one Bucket to Another Bucket

Anonymous — Tue, 11 Jun 2019 17:57:57 GMT

Hi,

The high level steps will be as below.

a) Use S3List to iterate the objects for a folder

b) Parse the date part of the folder from the source

b) Add the parsed date part in the target bucket key while using tS3Copy to copy the object from source to target

In this way, you can move the components from source to target.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: Migrating multiple S3 files from one Bucket to Another Bucket

Karuetl — Tue, 11 Jun 2019 19:06:40 GMT

Should i get the key prefix part as below in the tS3List object ? So that it goes through each subfolder ?

/folder/

Re: Migrating multiple S3 files from one Bucket to Another Bucket

Karuetl — Tue, 11 Jun 2019 19:17:12 GMT

Also, need help in Java how to extract date part

abc/ex_date=20190610/

Re: Migrating multiple S3 files from one Bucket to Another Bucket

Anonymous — Tue, 11 Jun 2019 22:38:01 GMT

Hi,

Here you go!

row1.data.replaceAll("\\D+","")

You can do the same stuff in other java components in Talend too.

For your earlier query, you can pick the records inside folder.

Hope I have answered your query. Please spare a second to mark the topic as resolved 🙂

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂