topic Re: AWS S3 to Snowflake Bulk load taking time using Talend in Talend Studio

AWS S3 to Snowflake Bulk load taking time using Talend

Anonymous — Sat, 16 Nov 2024 04:47:24 GMT

Hello,

I have designed a job to Load multiple files from AWS S3 to Snowflake table using Bulk Load components.

My Flow is:

1)tPrejob->tS3Connection

2)tS3list->tS3Get->tFileinputdelimited->tDBOutputBulk->tDBBulkExec->tDBROW

3)TPostJob->tS3Close

Where:

tDBOutputBulk has storage as "Internal" stage.

tDBROW has "Commit" command

There are total 2 files 450MB each on S3(total around 1GB data i.e 20 million records with 6 columns)

To load 1GB data, it is taking 25 min. I want to improve performance of my job.

Can anyone help in improving performance?

Also how to handle restartability in case of failure here?

thank you.

Anonymous — Wed, 04 Sep 2019 15:37:43 GMT

For this you can do this:

Regards,

Anonymous — Wed, 04 Sep 2019 17:55:40 GMT

Hello,

I want to load data into snowflake using Talend Bulk components.

Any performance tips on my existing job design or any modifications?

Please let me know