idea Re: Replicate Synapse / ADLS target - using GZIP files - Status changed to: Open - Collecting Feedback in Suggest an Idea

Replicate Synapse / ADLS target - using GZIP files

John_Roberts — Thu, 01 Apr 2021 13:28:02 GMT

One of the key pieces in our performance story is the transfer rate between on prem to azure.

Currently the data flow goes:

Netezza db (data is highly compressed)

QLIK replicate server load files (data is not compressed)

Network pipe (on-prem to Azure datacentre)

ADLS (data is not compressed)

Synapse tables (data is compressed)

Has your engineering team considered gzipping the QLIK replicate server load files ?

So that the data flow would be :

Netezza db (data is highly compressed)

QLIK replicate server load files (gzipped)

Network pipe (on-prem to Azure datacentre)

ADLS (gzipped)

Synapse tables (data is compressed).

Based on my reading, the Polybase engine supports loading .gz files directly into Synapse.

I realize this may affect the parallelization of the run, but our experience is that the Synapse load process takes 20-35 seconds per Gb file, and the network move takes 100x that, so it might be a compromise worth making, that would lead to a significant throughput boost.

Please ask your team if they have explored this idea,

It would be super easy to implement / prototype to see if yields performance improvements.

Re: Replicate Synapse / ADLS target - using GZIP files - Status changed to: Open - Collecting Feedback

Shelley_Brennan — Wed, 07 Apr 2021 15:01:35 GMT

Thank you for the suggestion. We have considered for Polybase but are also looking to implement COPY instead of Polybase and will probably support optional compression with the COPY method in a future release.

From now on, please track this idea from the Ideation por...

Meghann_MacDonald — Wed, 02 Aug 2023 16:19:25 GMT

From now on, please track this idea from the Ideation portal.

Link to new idea

Meghann

NOTE: Upon clicking this link 2 tabs may open - please feel free to close the one with a login page. If you only see 1 tab with the login page, please try clicking this link first: Authenticate me! then try the link above again. Ensure pop-up blocker is off.

Re: Replicate Synapse / ADLS target - using GZIP files - Status changed to: Closed - Archived

Ideation — Wed, 02 Aug 2023 16:19:27 GMT