topic Re: Dynamically create S3 folder in Talend Studio

Dynamically create S3 folder

Parikhharshal — Sat, 16 Nov 2024 02:51:44 GMT

Hi experts

I have scenario where based on local file properties I want to create S3 folder dynamically. Once I get date/time from fileproperties from files (received on local) I want to create Q1,2,3,4 folder for each year in S3 bucket.

Is it possible to create such a dynamically folder in S3 in Talend?

Thanks

Harshal.

Re: Dynamically create S3 folder

Anonymous — Fri, 27 Mar 2020 08:08:21 GMT

Hi Harshal

Do you want to create the s3 folder dynamically and put the file to corresponding folder or only create the empty folder?

Regards
Shong

Re: Dynamically create S3 folder

Parikhharshal — Fri, 27 Mar 2020 08:51:32 GMT

@shong I want to put the corresponding files into those Quarter folders too based on date/time.

Re: Dynamically create S3 folder

Anonymous — Fri, 27 Mar 2020 10:52:20 GMT

You can use tS3Put to put the files to S3 folder, set the dynamic key path based on the date/time you get and parse it from the file properties, the component wil create the folder if it does not exist.
eg:
"folder_path/"+context.quarter+"/"+context.filename
Read the component documentation and learn tS3Put component, let me know if you have any questions.

Regards
Shong

Re: Dynamically create S3 folder

Parikhharshal — Fri, 27 Mar 2020 11:22:31 GMT

@shong Thanks for your reply.

I get that but how do I dynamically calculate quarter based on date and store in context?

Re: Dynamically create S3 folder

Anonymous — Fri, 27 Mar 2020 18:11:15 GMT

Hi
Use tFileProperties to get the file properties including m_time, convert the string date to a Date, and then hard code to get the quarter of the given date, see
https://stackoverflow.com/questions/302658/how-do-i-discover-the-quarter-of-a-given-date

Regards
Shong