Export data from Sql Server to csv and archived

[rafaaaaa]

I need to know what is the best practice to export data fro SQLServer based on a condition to CSV then archive it to be loaded to Snowflake through SnowSQL.

My current implementation is as the following:

check attached image

export data to csv into splitted files by number of rows 300K.

I wait tell the files finished to start archiving and then after finish archiving i took the archived filed to be loaded to SNowflake.

How can I archive the written files in parallel not to be series process

[JohnRMK]

Hello,

Snowflake by default loads the data in parallel on condition of giving it in parameters, the zip file with all the files in it.

If you have the enterprise version, you can click on the iteration link and then activate the parallelization mode, and increase the level

[rafaaaaa]

Sorry I forgot to mention that I am working on Talend OpenStudio 7.3, SO do I have any alternative solution.

[rafaaaaa]

I found it in the TOS 7.3 is there any way to run both process in parallel extracting and archiving

[JohnRMK]

Hello,

I'm afraid it has nothing to do with it.

If you had the enterprise version, there are paradigms and components but with the free version, it remains limited.

However, if you try to extract several tables, you can parallelize the processing.

(with multiple tRunJob with direrent table)

Or if you try to extract data from a single table with an id, you can build a routine that splits your query

(by adding automatic conditions where id> ... and id <...) and you can have a seed and several extraction and archiving processes