Migrating multiple S3 files from one Bucket to Another Bucket

[Karuetl]

Under the  source bucket, there are multiple folders and in each folder, there are multiple files

As the example below 

/folder/export=2019-06-01/
/folder/export=2019-06-02/
/folder/export=2019-06-03/
/folder/export=2019-06-04/

I want to move each file from each folder from the source bucket to the Target Bucket as below, Basically, files should be under the same date as in source bucket  

/abc/2019-06-01/
/abc/2019-06-02/
/abc/2019-06-03/
/abc/2019-06-04/

[Anonymous]

Hi,

    The high level steps will be as below.

a) Use S3List to iterate the objects for a folder

b) Parse the date part of the folder from the source 

b)  Add the parsed date part in the target bucket key while using tS3Copy to copy the object from source to target

    In this way, you can move the components from source to target.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[Karuetl]

Should i get the key prefix part as below in the tS3List object ?  So that it goes through each subfolder ?

/folder/

[Karuetl]

Also, need help in Java how to extract date part 

abc/ex_date=20190610/

[Anonymous · Accepted solution]

Hi,

    Here you go!

row1.data.replaceAll("\\D+","") 

You can do the same stuff in other java components in Talend too.

For your earlier query, you can pick the records inside folder.

Hope I have answered your query. Please spare a second to mark the topic as resolved 🙂

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂