topic Re: Optimization on Change data capture (200 million records or so in 5-6 transactions) in Qlik Replicate

Optimization on Change data capture (200 million records or so in 5-6 transactions)

Syra1359 — Wed, 12 Aug 2026 15:47:54 GMT

The amount of change data that is being captured from source is very large. 200 million records or so for about 20 tables. And this is causing a latency of about 4 hrs or 5 hrs to commit this to target. Is there any way we can speed up this CDC commits? I am attaching a photo of the current monitor screen and my change capture settings. Please let me know of any tuning that can be performed. Source is SQL server (MS-CDC) and target is Snowflake.

Re: Optimization on Change data capture (200 million records or so in 5-6 transactions)

Syra1359 — Wed, 12 Aug 2026 15:49:22 GMT

Latency of about 30 mins or so is okay, but 4-5 hours is too much for my case.

Re: Optimization on Change data capture (200 million records or so in 5-6 transactions)

Anil_Babu_Samineni — Thu, 13 Aug 2026 09:34:05 GMT

the CDC should be only in Incremental concepts, did you follow the same?

Re: Optimization on Change data capture (200 million records or so in 5-6 transactions)

Syra1359 — Thu, 13 Aug 2026 09:51:20 GMT

There are batch jobs running on top of the source databases. And the commit rates and other aspects related to those are not in my control. We don't have access to those systems at all. I know we can't reduce the latency completely, because there will be the latency caused by source side commit delays. But I am trying to improve the target writes, so that we can save as much time as possible.

Re: Optimization on Change data capture (200 million records or so in 5-6 transactions)

Anil_Babu_Samineni — Thu, 13 Aug 2026 10:02:12 GMT

I guess question might draft in other forum and not to Qlik community. I just started understanding of snowflake, do you wish to answers for me -

What is the current batch size being used for commits?

How are changes staged before loading into Snowflake?

Are you using MERGE INTO or row‑level updates?

What is the Snowflake warehouse size during CDC loads?

Is parallelism applied across tables or partitions?

How frequently are CDC batches committed?

Are you using Snowpipe or Streams/Tasks?

How is network throughput between SQL Server and Snowflake?

What is the retry/error handling strategy?

Are indexes or constraints slowing down merges?

Based on input, I can check internally and post suggestions here.

In short, the target architecture flow would look like this: SQL Server CDC → Cloud Storage Staging → Snow pipe → Snowflake Streams → MERGE Tasks → Analytics Warehouse. This design reduces latency from hours to minutes, improves scalability, and ensures that your Snowflake environment remains responsive even under heavy CDC loads.

Re: Optimization on Change data capture (200 million records or so in 5-6 transactions)

-mjl- — Thu, 13 Aug 2026 13:03:44 GMT

Is the size of the Snowflake warehouse causing the bottleneck? Also, are there particular tables that are busier than the others and can you be more selective with the data?

BTW, you can trace the journey of the data in the Replicate logs in TARGET_APPLY which is going to be: loading data to stage, loading that into a table and then apply, so by it's nature you'll struggle with a lot of transactions.

Snowpipe can be considered too but you'll need to take care of the apply part outside of Replicate.

Cheers!