Skip to main content
Announcements
UPGRADE ADVISORY for Qlik Replicate 2024.5: Read More
cancel
Showing results for 
Search instead for 
Did you mean: 
feal
Creator
Creator

Resume processing and CDC only modifications of table filters

Hello

 

We have a Qlik CDC only task and we performed a modification on a filer transformation on a table.

We stopped that task , perform Import task with the desired modifications and perform (for mistake) resume processing of the same task.

The filer was loaded correctly (and unexpectedly !)

We remember in the past documentation that , if we have to perform a filter modification, we are aware that we will perform stop+import of the task (or perform modification by Qlik Enterprise Manager GUI and save the currently running task) and perform reload target

Can you explain us what happened ? in what case resume processing is acceptable for filter modifications scanarios ?

Lt us know

Regards

Alessandro Ferrario

Labels (1)
1 Solution

Accepted Solutions
Heinvandenheuvel
Specialist III
Specialist III

You always need to STOP followed by some sort of start to make tasks or endpoint changes active, the only exception being LOGGING LEVELS which take effect immediatly (Just as well! :-).

It doesn't matter whether the (re)start is done with Replicate UI, AEM, AEM-API or REPCTL.

For a CDC only task a reload is not needed - nor possible.

For a FULL + CDC task a reload is forced and required to make sure the target data matches the rules.

As soon as you IMPORT (or SAVE) any table with any change will be in 'queued' state and reloaded on task activation. Using the GUI - SAVE you'll get some relatively clear messages (see below). Using task json import there will be no message, but the table(s) will be in the queued pile and will reload.

The only way to stop a table reload in a FULL+CDC task is to restart with RUN ADVANCED - TABLES ALREADY LOADED + TIMESTAMP. This may well be justified if you happen to know that say a filter change is appropriate for the target - but there is no way Replicate could make that decision as it does not understand business rules. And for bigger changes, like adding a column, resuming with tables already loaded would clearly be wrong and not give you that new column and similar without further manual interventions.

Hein.

 

Heinvandenheuvel_0-1713450436167.png

 

Heinvandenheuvel_1-1713450458673.png

 

View solution in original post

9 Replies
john_wang
Support
Support

Hello @feal ,

Thanks for reaching out to Qlik Community!

We'd like to confirm the behavior for you, if you can share the source and target endpoint type then we may provide more accurate information.

Regards,

John.

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
ShaunaB1
Contributor
Contributor

Hi,

I read this and I got some ideas related to it. You can consider it as a suggestion if you want. Here it's: 

If you change a filter in a Qlik CDC task and just resume it, it might still work because the task picks up changes from where it stopped. It’s like pausing and playing a video. But if you need to start over or from a specific point, you should reload the target. Always check Qlik’s guidelines for the best steps.

I hope my suggestion will help you to solve this issue.

Best regard,
Doglikesbest
feal
Creator
Creator
Author

Hi Joghn

i attacehd JSON definition (Export with endpoints,) ,  log after import/resume propcessing  and log generated during the stop before modifications

 

Let me know

Regards

AF

john_wang
Support
Support

Hello @feal ,

Thank you for sharing the details.

I conducted a quick test utilizing Oracle as the source and SQL Server as the target endpoint. I observed that the updated filter expression takes effect following the RESUME task operation. Here are the steps I followed:

  1. Applied a filter with "ID > 10".
  2. Stop the task.
  3. Modified the filter from "ID > 10" to "ID > 100", save the changes.
  4. RESUME the task, confirming that the new filter expression became active.

To summarize, altering the filter expression does not necessitate the export/import of task JSON files unless you specifically aim to adjust task settings without utilizing the Replicate or QEM console GUI.

Additionally, I noticed an issue with your task as indicated in the log file at line #9:

2024-04-09T11:54:34 [TASK_MANAGER ]I: Task 'R1_HP_PERSON_STRUCTURE_CHANGES' running CDC only in resume mode after recoverable error, retry #5

I suggest opening a support ticket and attaching these files. Our support team will assist you in resolving the abnormal stopping issue and addressing any concerns regarding filter behavior. Additional information may be requested through the support ticket.

By the way, I'm deleting the log files in the community as they may contain sensitive information.

Regards,

John.

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
feal
Creator
Creator
Author

Hi John

 

thank you for your quick answer.

By the way ;. this time we perform stop+import+resume processing NOT by Qlik GUI but with REST API commands : can this difference may be relevent in tabel filter transformation behaviour ?

 

Let me know

regards

AF

john_wang
Support
Support

Hello @feal ,

I do not think so. If you can provide your script in support case then I'd like to double-confirm for you.

Regards,

John.

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
feal
Creator
Creator
Author

Hi John

so you can confirm that this operation/way

 

- stop task

- import of a task

- resume processing of the task 

is always possible , also in he case  of task is CDC only  and with modifications of table  filters?

So reload target is not mandatory for this scenario ?

?

Accordingly to current documentation there is no caveat or different specification on task re-start   

Adding or removing filter ranges #Adding or removing filter ranges | Qlik Replicate Help

Let us know 

regards

AF

 

 

Heinvandenheuvel
Specialist III
Specialist III

You always need to STOP followed by some sort of start to make tasks or endpoint changes active, the only exception being LOGGING LEVELS which take effect immediatly (Just as well! :-).

It doesn't matter whether the (re)start is done with Replicate UI, AEM, AEM-API or REPCTL.

For a CDC only task a reload is not needed - nor possible.

For a FULL + CDC task a reload is forced and required to make sure the target data matches the rules.

As soon as you IMPORT (or SAVE) any table with any change will be in 'queued' state and reloaded on task activation. Using the GUI - SAVE you'll get some relatively clear messages (see below). Using task json import there will be no message, but the table(s) will be in the queued pile and will reload.

The only way to stop a table reload in a FULL+CDC task is to restart with RUN ADVANCED - TABLES ALREADY LOADED + TIMESTAMP. This may well be justified if you happen to know that say a filter change is appropriate for the target - but there is no way Replicate could make that decision as it does not understand business rules. And for bigger changes, like adding a column, resuming with tables already loaded would clearly be wrong and not give you that new column and similar without further manual interventions.

Hein.

 

Heinvandenheuvel_0-1713450436167.png

 

Heinvandenheuvel_1-1713450458673.png

 

feal
Creator
Creator
Author

OK Thanks and regards !!

AF