REST API to Snowflake data load

[Anonymous]

I am trying to load data from REST API to Snowflake using Talend Open Studio. I am counting the number of objects in the JSON and running the loop for all the objects and reading them in a loop and finally loading them in a Snowflake table. For loading 813 records, it took 4:15 hrs which is not acceptable in real time scenario. Does anyone know if there are any settings that need to be changed to load the data faster. Earlier, I had loaded data from Oracle 11g to Snowflake and it did not take much time.

[vapukov · Accepted solution]

Hi,

if number of iterations (rows) not huge - store all in hash, then insert from hash to tSnowFlake as single subjob

if number of rows is huge - always need to check about bulk load ( https://docs.snowflake.net/manuals/user-guide/data-load-s3.html )

[Anonymous]

Hi,

As my number of iterations were not huge, I tried using tHashInput and tHashOutput components and it got completed in 15 minutes. Seems like it worked fine for less number of records.

Thank you so much.