topic Re: How to load 500+ Million Records from SQL Server to Snowflake in Talend Studio

How to load 500+ Million Records from SQL Server to Snowflake

AKushnapalli — Sat, 16 Nov 2024 00:41:01 GMT

Hi..

How to load 500+ Million Records from SQL Server to Snowflake using Talend.

Currently using tSnowflakeOutputBulkExec component to store the data locally, but it's struckked loading into files after fetching 14+ Million records from SQL Server.

Please see Attached Screenshots:

Talend Job Design - Includes Basic Settings of
tDBOutputBulkExec
tDBOutputBulkExec Advanced setting

Any help is greatly appreciated.

Thank you.

Anil

Re: How to load 500+ Million Records from SQL Server to Snowflake

David_Beaty — Sun, 07 Feb 2021 12:48:47 GMT

Hi,

I would find someway of splitting the incoming data into chunks and loading up in manageable chunks, using some kind of partitioning key (a date is a good example). This would also allow you to build in some recoverability into the process if, as you've already found, the job hangs part way through.

Thanks

David

If you find these answers helpful, don't forget to Like and/or set as the answer