topic Re: Full Load with CDC vs CDC in Qlik Replicate

Full Load with CDC vs CDC

user12345AA — Mon, 15 Apr 2024 05:59:22 GMT

HI All,

I'm beginner and exploring the things.

My scenario : From Source I have to pull all data and load into target, then it has to pull the CDC data daily from Source to target based on the schedule.

I bit confused with the behaviour of Full load with Applied changes and Applied Changes alone.

Full Load and Applied Changes:

When I create a task with both Full load and Applied Changes and upon scheduling the task, each time it is identifying the records under Full load and loading into Target table. Suppose If I keep my Task as in running status, then the CDC records being captured.

Applied Changes:

When I create a task with Applied changes and schedule to run, it is not identifying the CDC data, unless the task is in running.

Please suggest me , on how I should create the task in order to work for my scenario.

Re: Full Load with CDC vs CDC

sureshkumar — Mon, 15 Apr 2024 06:35:52 GMT

Hello @user12345AA

Kindly refer below article

An Introduction to Qlik Replicate Tasks: Full Load... - Qlik Community - 2019167

Regards,

Suresh

Re: Full Load with CDC vs CDC

DesmondWOO — Mon, 15 Apr 2024 06:37:06 GMT

Hi @user12345AA ,

For your scenario, you should choose "Full Load and Applied Changes".

"Applied Changes" alone does not perform data loading from the source table to target table. Instead of, Replicate reads changes from the transaction log (depends on the source endpoints) and applies changes to the target database.

Regards,
Desmond

Re: Full Load with CDC vs CDC

john_wang — Mon, 15 Apr 2024 06:38:41 GMT

Hello @user12345AA ,

Thanks for reaching out to Qlik Community!

In Qlik Replicate, there are 2 stage of a task: Full Load , and Change Processing. The 2 stages can be enabled both, or either one.

Full Load (with Change Processing disabled)
In Full Load stage, Qlik Replicate verifies and detects the target table (depending on the task "Target Table Preparation" setting), re-create, or truncate the target table before loading the data.
In this stage, Qlik Replicate initialize the existing history data rows, and replicate them from source endpoint DB to target endpoint DB. Please take note in this stage Qlik Replicate retrieve data from source DB tables.
CDC Change Processing
In Change Processing stage, Qlik Replicate gets changes from source endpoint transaction log and applies the change to target endpoint DB. Please take note in this stage Qlik Replicate retrieve changing data from source transaction log (eg Redo Log in Oracle, TLOG in SQL Server , OPLOG in MongoDB etc).
Full Load + Change Processing
If both of them are enabled, whenever Qlik Replicate task is starting up, it will startup Change Processing thread first and cache all the changes during the Full Load stage; IN the meanwhile, the Full Load stage will initialize the history data rows. after the existing Full Load stage done, the CDC Change Processing will follow up smoothly. This is automatically switch oever.

Hope this helps.

John.

Re: Full Load with CDC vs CDC

user12345AA — Mon, 15 Apr 2024 08:47:44 GMT

Thanks for the reply @ John_wang

Will the above stages works with same behaviour,even when it is scheduled because I have seen if my task is in running status then only the both stages CDC and Fullload+CDC started processing CDC data. When I schedule the CDC stage not loading the data and Fullload+CDC going on with history data everytime.

Re: Full Load with CDC vs CDC

john_wang — Mon, 15 Apr 2024 08:58:01 GMT

Hello @user12345AA ,

If both Full Load and Change Processing (see below sample) are set to ON:

Then after the task startup, the task will keep in running because CDC processing is a continuous progress - unless the task is stopped manually , or by scheduler.

So far for your question, there is no difference between the task startup/stop manually (eg via Qlik Replicate, or QEM console), or via scheduler.

Feel to let me know if you need any additional information.

Regards,

John.

Re: Full Load with CDC vs CDC

john_wang — Mon, 15 Apr 2024 08:58:03 GMT

Hello @user12345AA ,

If both Full Load and Change Processing (see below sample) are set to ON:

Then after the task startup, the task will keep in running because CDC processing is a continuous progress - unless the task is stopped manually , or by scheduler.

So far for your question, there is no difference between the task startup/stop manually (eg via Qlik Replicate, or QEM console), or via scheduler.

Feel to let me know if you need any additional information.

Regards,

John.

Re: Full Load with CDC vs CDC

user12345AA — Mon, 15 Apr 2024 10:05:48 GMT

When I schedule a task , after the completion it gets stopped, in that case you mean we cannot get the CDC records from source. Ideally the task should be in running status to get the CDC records. Is my understanding correct?