topic Writing the data to S3 sub-folders in Talend Studio

Writing the data to S3 sub-folders

_AnonymousUser — Sat, 16 Nov 2024 11:02:49 GMT

Hi All,
We are currently working on a requirement which would write files to the S3 bucket.
Under the main bucket there are many sub-folders and some of them need to be created on the fly.
For example we need to create folders on a daily basis for each hour under the main bucket.
Has anyone worked on similar requirement before?If yes,could you suggest a good approach?
One option which I can think of is to replicate the folder structure in the location from where I need to upload to S3,then put this filepath as a dynamic key in the ts3put bucket.but there is no option to create folders if it doesn't exist.not sure this works similar to the aws s3:cp command with recursive option.Currently I am waiting to get write access on the bucket,before I can actually try this out
Regards,
Sunil

Re: Writing the data to S3 sub-folders

Anonymous — Wed, 16 Sep 2015 08:24:01 GMT

hi,
did you already have a look at component around s3 from Talend help ?
there is a list of some video/tutos that can help you I guess ...
Working with Talend's Amazon Simple Storage Service %28S3%29 components - Talend v5.4 features
https://help.talend.com/search/all?query=tS3Put&content-lang=en
and so on
hope it helps
regards
laurent

Re: Writing the data to S3 sub-folders

_AnonymousUser — Wed, 16 Sep 2015 08:43:32 GMT

Hi Laurent,
Thanks.
I already checked the Talend videos . I have been able to get the files recursively.However I am looking at an option where the copy can be done recursively to the S3 bucket.
Regards,
Sunil