How to extract data from s3 folder by folder in a single query

Anonymous — Sat, 16 Nov 2024 04:20:45 GMT

Hi Team,

I have a requirement like this to extract data from s3 bucket to PostgreSQL table

For Ex:

In s3 bucket - i have folder like

PI/Data/2019-10-18/ .json files

/2019-18-19/ .json files

/2019-18-20/ .json files

How can i achieve this ? Once it finish loading the 2019-10-18 .json files to PostgreSQL tables , it should automatically go to the next date folder and send those as well to PostgreSQL tables.

Thanks,

Prabuj

Re: How to extract data from s3 folder by folder in a single query

Anonymous — Fri, 18 Oct 2019 08:28:41 GMT

Hi @prabuj27 ,

You can try using tS3Filelist to get all the files from S3 and use the key prefix as "*.json" and load into PostgreSQL table.

topic Re: How to extract data from s3 folder by folder in a single query in Talend Studio

How to extract data from s3 folder by folder in a single query

Re: How to extract data from s3 folder by folder in a single query