topic s3List recursive in Talend Studio

s3List recursive

Anonymous — Thu, 08 Dec 2016 21:48:10 GMT

Hello,
I am using TOS DI 6.3.
Is there a way to do use the Talend tS3List component to do a recursive list of objects in an Amazon S3 bucket?
I am able to get a list of object under a defined prefix in the component, but if objects are added into additional deeper levels, those new objects are not being detected.
For example:
bucket/folder1/folder2/filename.text
-> this is fine when listing to the folder2 prefix.
bucket/folder1/folder2/random_folder/newfile.txt
-> the random folder and files therein are not being detected by ts3List.
I would prefer not to have to develop using the s3 CLI --recursive option to get a real-time list.
As I never know what the "random_folder" may be, I'd like to emulate the --recursive function in Talend to detect new files in s3.
I do not see any --recursive option for tS3List in Talend. Any suggestions?
Thank you.

Re: s3List recursive

Anonymous — Fri, 09 Dec 2016 05:56:52 GMT

Hi,
There is no recursive function in tS3List component.
Have you tried to check out "List all buckets objects" option and enter the prefix of files to be listed? In this way, tS3list will list all the files on the S3 server.
Best regards
Sabrina

Re: s3List recursive

Anonymous — Fri, 09 Dec 2016 17:48:43 GMT

Hi Sabrina,
Thank you for the quick reply. I'll make use of the "List all buckets objects" and change the job design to apply the filters I was using from context since I don't want to poll all prefixes in the bucket. I'll be able to achieve the desired result with just a few extra steps.
Thank you.