topic Re: Archiving multiple files in S3 in Talend Studio

Archiving multiple files in S3

Anonymous — Tue, 21 May 2019 11:05:35 GMT

Hi,

I am trying to archive the files present in S3 to a new folder appended with current date.

I need to create a new folder in the same bucket and move all the files starting with XXX to that folder.

Could anyone please suggest an approach to achieve this.

Thanks in advance!

Alekhya.

Re: Archiving multiple files in S3

Anonymous — Tue, 21 May 2019 22:12:20 GMT

Hi,

If a folder is not present, it will be automatically created during Put event using tS3Put command.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: Archiving multiple files in S3

Anonymous — Wed, 22 May 2019 08:25:10 GMT

Thanks for the reply.

But, my requirement is to move the files(already present in S3 in a Source folder) to a new folder(to be created from Talend whose name should be appended with sysdate).

Re: Archiving multiple files in S3

Anonymous — Wed, 22 May 2019 15:32:59 GMT

Hi,

There are no components right now which can move the data directly within S3. You will have to use Get and Put combination.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: Archiving multiple files in S3

Anonymous — Thu, 23 May 2019 08:33:00 GMT

Thank You Nikhil.

But, can I move multiple files(placed in a folder) at once?

Thank You,

Alekhya.

Re: Archiving multiple files in S3

Anonymous — Thu, 23 May 2019 15:17:03 GMT

Hi,

You will have to do it in iterative fashion. First you will have to list the components and pick the files which you need to move (Or have the list handy by any other method). Then iteratively, you can move the files one by one using Get and put commands.

Ideally you should target the final resting place in S3 when you are moving the files from source itself since each transfer will increase your cost in AWS. So I would consider rewriting the first loading program itself to arrange the files in systematic way so that later file movements are not required.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: Archiving multiple files in S3

Anonymous — Mon, 27 May 2019 17:45:01 GMT

@alekhyak

Hi,

Hope I have answered your queries related to this post. If you do not have any outstanding query, could you please mark the topic as resolved? 🙂

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂