topic Ingest folder names from s3 bucket and load to postgres table in Talend Studio

Ingest folder names from s3 bucket and load to postgres table

Vijay_1823 — Mon, 16 Sep 2024 15:18:07 GMT

I am trying to ingest multiple s3 folder names into a table.

eg: from folder A i am trying to process one specific folder instead of going in all folders

here are the list of components ts3list--->iterate--->tjava,--->ts3get i am using ts3list component to fetch all the files from folder. i would like to use a component before the ts3list that would tell ts3list what folder it should process.

Regards

Re: Ingest folder names from s3 bucket and load to postgres table

Dave_Simo — Mon, 16 Sep 2024 16:11:55 GMT

Hello,

You ca use something like this

In tForeach_2

In tS3List

Hope this can help you

Best Regards,

Re: Ingest folder names from s3 bucket and load to postgres table

Vijay_1823 — Tue, 17 Sep 2024 10:09:08 GMT

Hi Dave_Simo,

Thanks for your quick reply,

Let me explain what i want to achieve here, there is one s3 bucket named ABCD and in that bucket there are multiple objects or folders named AA, AB, AC, AD, AE and in that objects there are again multiple folders/objects named as AAA, ABA, ACA, ADA but here i want ts3list to go in only folder named ACA and load the data into the table, also this process should go in every folder of AA, AB, AC, AD, AE and go in only ACA folder and load the data.(because ACA folder contains required data we need to load).How i can achieve this in talend ? Thanks !

Regards,

Re: Ingest folder names from s3 bucket and load to postgres table

Dave_Simo — Tue, 17 Sep 2024 12:06:52 GMT

Hello,

Not having an AWS environment to test I think the link below could help you.

https://community.qlik.com/t5/Design-and-Development/S3-bucket-naming-from-talend/td-p/2270460

Best Regards,