AWS S3 to Snowflake Bulk load taking time using Talend

[Anonymous]

Hello,

  I have designed a job to Load multiple files from AWS S3 to Snowflake table using Bulk Load components.

My Flow is:

1)tPrejob->tS3Connection

2)tS3list->tS3Get->tFileinputdelimited->tDBOutputBulk->tDBBulkExec->tDBROW

3)TPostJob->tS3Close

Where:

tDBOutputBulk has storage as "Internal" stage.

tDBROW has "Commit" command

There are total 2 files 450MB each on S3(total around 1GB data i.e 20 million records with 6 columns)

To load 1GB data, it is taking 25 min. I want to improve performance of my job.

Can anyone help in improving performance?

Also how to handle restartability in case of failure here?

thank you.

[Anonymous]

For this you can do this:

1. Create a named file formats that clearly describe your data files. 
2. Then, create name stage objects.
3. Load the data of S3 bucket into Snowflake tables
4. Data files error resolution.

Regards, 

192.168.0.254 192.168.2.1 192.168.1.2  192.168.178.1

[Anonymous]

Hello,

  I want to load data into snowflake using Talend Bulk components.

Any performance tips on my existing job design or any modifications?

Please let me know