topic Re: List all files from S3 bucket in Talend Studio

List all files from S3 bucket

Parikhharshal — Sat, 16 Nov 2024 03:45:35 GMT

I need to read all files from Following S3 bucket:

prod/input/dwh_input_files/Showcause_Atrisk_Data/Delta

Files in this bucket are:

Showcause 2.xlsx

Showcause 1.xlsx

Risk.xlsx

What I know is we can't use '*' to read all files. We need to have proper combination of key and file name.

My job is designed like this:

Properties of ts3List is as below:

Titeratetoflow properties are as below:

Not sure what I am doing wrong. It's only printing last file for showcause. This is issue 1 and another issue is: Is it possible to list all files together ie Showcause_* and Risk_* related files?

Thanks

Harshal.

Re: List all files from S3 bucket

TRF — Fri, 20 Dec 2019 06:51:07 GMT

You didn't show what you actually get as the result. However, as tJava component starts 1rst and only once for the subjob, replace it by tJavaRow or tJavaFlex in order to print all incoming values.
For the 2nd point, did you try to add lines in the array bucket names/keys.

Re: List all files from S3 bucket

Anonymous — Fri, 20 Dec 2019 06:53:45 GMT

Hi,

In the key prefix, provide only the path till the folder you want to list out the files from or till root folder in case you want to list out each and every file.

Now, to print the filenames, you can use tJava component and type: System.out.println(((String)globalMap.get("tS3List_1_CURRENT_KEY")))

This will list out all the keys from tS3List.

Components:

tS3List-----(iterate)---->tJava

Hope this helps.

Regards,

Bhagwat

------------------------------------------------------------------------------------------------------

Don't forget to give kudos and accept the reply as solution if it helps.

Re: List all files from S3 bucket

Parikhharshal — Fri, 20 Dec 2019 08:04:30 GMT

Hi @TRF

Tjava only prints Showcause2.xlsx which is second file of showcause.

Does not print anything else. Yes you are correct. When I try tlogrow it prints 2 objects which is good but the problem exists at tS3get where I am getting only second file for Showcause. Not sure what's wrong in there.

This is the config inside tS3get compoent:

For the second ques: No I did not try using array bucket names. How can you do that?

Re: List all files from S3 bucket

Anonymous — Fri, 20 Dec 2019 08:40:50 GMT

Hi,

Why don't you try using tS3List component? And then proceed further with the steps I mentioned.

Try this and let me know if it works.

Regards,

Bhagwat

Re: List all files from S3 bucket

Parikhharshal — Fri, 20 Dec 2019 09:08:19 GMT

Hi @bhagwatn

Yes that gives me all file names which I require. Which is awesome. Now I want to get these files locally. So what should be config of tS3get and should I use titeratetoflow?

Thanks for your help so far!

Thanks

Harshal.

Re: List all files from S3 bucket

Parikhharshal — Fri, 20 Dec 2019 18:03:35 GMT

Hi @bhagwatn

How do I get the files into local now?

Can you please guide?

Thanks
Harshal.

Re: List all files from S3 bucket

Parikhharshal — Fri, 20 Dec 2019 19:34:56 GMT

I resolved this by designing the flow like this:

tS3filelist ---> (Iterate) tS3get.