topic REST API to Snowflake data load in Talend Studio

REST API to Snowflake data load

Anonymous — Tue, 05 Feb 2019 13:44:55 GMT

I am trying to load data from REST API to Snowflake using Talend Open Studio. I am counting the number of objects in the JSON and running the loop for all the objects and reading them in a loop and finally loading them in a Snowflake table. For loading 813 records, it took 4:15 hrs which is not acceptable in real time scenario. Does anyone know if there are any settings that need to be changed to load the data faster. Earlier, I had loaded data from Oracle 11g to Snowflake and it did not take much time.

Re: REST API to Snowflake data load

vapukov — Tue, 05 Feb 2019 23:14:55 GMT

Hi,

if number of iterations (rows) not huge - store all in hash, then insert from hash to tSnowFlake as single subjob

if number of rows is huge - always need to check about bulk load ( https://docs.snowflake.net/manuals/user-guide/data-load-s3.html )

Re: REST API to Snowflake data load

Anonymous — Mon, 11 Feb 2019 06:09:30 GMT

Hi,

As my number of iterations were not huge, I tried using tHashInput and tHashOutput components and it got completed in 15 minutes. Seems like it worked fine for less number of records.

Thank you so much.