topic Load TSV from S3 to Snowflake in Talend Studio

Load TSV from S3 to Snowflake

g7XneJXR_jxs — Sat, 16 Nov 2024 05:00:09 GMT

Hi, need help in loading tsv files into snowflake. the files are stored in s3.

Re: Load TSV from S3 to Snowflake

iamabhishek — Mon, 12 Aug 2019 14:41:31 GMT

From Talend 7.1.1 version there is the component (tSnowflakeBulkExec) from which you can load data directly from S3 to Snowflake.This component mimics the copy command of Snowflake. There will not be any on-premise files involved.
For Storage you have to select S3 and select an Amazon S3 bucket to load data from. And mention rest of the properties as per configuration - Region, Access Key and Secret Key, Bucket, Folder.
Link - here.
But if you are on lower version then you could use a tJDBCRow component to connect to Snowflake via JDBC and execute a COPY command to load the data into a table in Snowflake. This method leverages the bulk-loading capability of Snowflake without requiring the data to be pulled into Talend.

Re: Load TSV from S3 to Snowflake

Anonymous — Mon, 12 Aug 2019 14:58:28 GMT

Hi,

You have to first get the file from S3 using tS3Get. Once the file is in your local, you can read them using tfileinputdelimited and write to Snowflake using tDBOutput(Snowflake).

If the file is big, I would advise you to use the Snowflake Bulk components.

Another approach (which is better) is as shown in previous post. I didn't see the comment while typing the reply.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved