topic Accessing the data from S3 to database table in Talend Studio

Accessing the data from S3 to database table

sushantk19 — Fri, 15 Nov 2024 23:42:46 GMT

Hello,

I am trying to access one of the S3 buckets with bucket name, access key and secret key. I want to extract the data from this s3 bucket and load to database table. Can someone share sample ETL job design along with configurations if someone has done this before? what components do we use?

Re: Accessing the data from S3 to database table

manodwhb — Wed, 22 Sep 2021 01:15:53 GMT

@Sushant Kapoor , I can not able to share the job screenshot. but below are the components I used.

tPreJob-->tS3Connection -->On Component OK -->tDBConnection

main job

tS3Get-->On Component OK --> tFileinputDelimitted-->tDBOutput

tPostJob-->tS3Close-->tDBClose

Thanks,

Manohar

Re: Accessing the data from S3 to database table

manodwhb — Wed, 22 Sep 2021 02:54:46 GMT

@Sushant Kapoor ,check the below link will help you.

https://www.nuwavesolutions.com/using-talend-to-extract-data-from-an-aws-s3-bucket-to-postgres/

Thanks,

Manohar

Re: Accessing the data from S3 to database table

sushantk19 — Wed, 22 Sep 2021 06:46:30 GMT

@Manohar B : Thanks. I will check and let you know.

Re: Accessing the data from S3 to database table

sushantk19 — Wed, 22 Sep 2021 08:09:05 GMT

@Manohar B : do you know if we can access/copy all the files from a folder. they are gzip files

in "Key" field what do we give to access multiple files.please see the screenshot.

If i give *.*, it gives me an error.

Re: Accessing the data from S3 to database table

sushantk19 — Wed, 22 Sep 2021 09:31:06 GMT

@manodwhb: how can we access(get) multiple files from S3 bucket to our local folder as i have to unzip the files also.