topic Re: moving data from Amazon S3 bucket to postgres DB on RDS in Talend Studio

moving data from Amazon S3 bucket to postgres DB on RDS

Anonymous — Sat, 16 Nov 2024 04:43:09 GMT

My current pipeline consists of tS3_Connection --> tS3_Get --> tFileInputDelimited --> tMap. When I run the job, the data is successfully pulled from my S3 bucket and stored in a local file on my computer; however, when I try to use tFileInputDelimited to pull the data into tMap for transformation, it is all empty. Do I need to use a different component for this process? I've checked all of my settings and they appear to be in order.

Re: moving data from Amazon S3 bucket to postgres DB on RDS

Anonymous — Tue, 10 Sep 2019 16:57:10 GMT

Please post some screenshots of your job / component config

Re: moving data from Amazon S3 bucket to postgres DB on RDS

Anonymous — Tue, 10 Sep 2019 17:04:07 GMT

Updated my post

Re: moving data from Amazon S3 bucket to postgres DB on RDS

manodwhb — Wed, 11 Sep 2019 12:05:41 GMT

@SAI_Chief ,what type of file is it ?

Re: moving data from Amazon S3 bucket to postgres DB on RDS

manodwhb — Wed, 11 Sep 2019 12:15:38 GMT

@SAI_Chief ,have you defined,schema in the tFileInputDelimitted? i think you have not defined

Re: moving data from Amazon S3 bucket to postgres DB on RDS

Anonymous — Wed, 11 Sep 2019 21:37:17 GMT

It is a csv file. Is there a way to define the schema without having to manually type in all of the column names? Especially with csv files with 40-50 columns?

Re: moving data from Amazon S3 bucket to postgres DB on RDS

manodwhb — Thu, 12 Sep 2019 07:13:08 GMT

@SAI_Chief ,if your using Licensed Talend ,you can try with Dynamic data type. If your using Open stiudio,you need to define schema.