Skip to main content
Announcements
Introducing Qlik Answers: A plug-and-play, Generative AI powered RAG solution. READ ALL ABOUT IT!
cancel
Showing results for 
Search instead for 
Did you mean: 
gkaur
Contributor III
Contributor III

Qlik Replicate Latecny

I have a task in which 98 tables are being replicated from Postgresql to Snowflake. However, the task latency sometimes shows more than 20 minutes. My question is, how can I determine which table among the 98 is running behind and by how much?

I want to identify if there is a particular table or set of tables that is always causing the slowness.

If so, I plan to remove them from the big list and have them run under a separate task. I'm not sure if this is a recommended option, so I would appreciate expert advice on this matter.

Labels (1)
1 Reply
Heinvandenheuvel
Specialist III
Specialist III

Your best bet might be to directly go to Snowflake and ask for its long running, or frequent running queries for the username Replicate is using. This will not only identify but likely give you input as to how to fix it.

To investigate through Replicate you need to enable LOGGING for TARGET_APPLY to TRACE (DEBUG)

Now run for a few hours until you 'caught' a slowdown window.

Lower Logging level, Roll the log, start investigating. What you are looking for is the lines with  "from seq" which tells you Replicate is about to do the action on the target. On the left is the THREAD ID and 'start time'. Look for the next occurrence of that thread id  which will be the end time, after the action. Subtract start from end to get a duration on the action.

Now this can become tedious so years ago i scripted this using PERL that this is readily available on Linux as well as Windows. Similar scripts can of course be done in Python or Powershell.but for me the Perl works so why bother.

Anyway, script attached run with -h for help and typically with -v and/or -t for verbose, per table outputs

Hein