topic Re: Export data from Sql Server to csv and archived in Talend Studio

Export data from Sql Server to csv and archived

rafaaaaa — Sat, 16 Nov 2024 01:54:17 GMT

I need to know what is the best practice to export data fro SQLServer based on a condition to CSV then archive it to be loaded to Snowflake through SnowSQL.

My current implementation is as the following:

check attached image

export data to csv into splitted files by number of rows 300K.

I wait tell the files finished to start archiving and then after finish archiving i took the archived filed to be loaded to SNowflake.

How can I archive the written files in parallel not to be series process

Re: Export data from Sql Server to csv and archived

JohnRMK — Wed, 15 Jul 2020 10:57:14 GMT

Hello,

Snowflake by default loads the data in parallel on condition of giving it in parameters, the zip file with all the files in it.

If you have the enterprise version, you can click on the iteration link and then activate the parallelization mode, and increase the level

Re: Export data from Sql Server to csv and archived

rafaaaaa — Thu, 16 Jul 2020 05:02:41 GMT

Sorry I forgot to mention that I am working on Talend OpenStudio 7.3, SO do I have any alternative solution.

Re: Export data from Sql Server to csv and archived

rafaaaaa — Sat, 18 Jul 2020 09:56:20 GMT

I found it in the TOS 7.3 is there any way to run both process in parallel extracting and archiving

Re: Export data from Sql Server to csv and archived

JohnRMK — Mon, 20 Jul 2020 12:32:48 GMT

Hello,

I'm afraid it has nothing to do with it.

If you had the enterprise version, there are paradigms and components but with the free version, it remains limited.

However, if you try to extract several tables, you can parallelize the processing.

(with multiple tRunJob with direrent table)

Or if you try to extract data from a single table with an id, you can build a routine that splits your query

(by adding automatic conditions where id> ... and id <...) and you can have a seed and several extraction and archiving processes