topic Re: Full load task in Qlik Replicate

Full load task

Preethi13 — Tue, 23 Apr 2024 18:49:00 GMT

Hi I had a quick question about full load, let's say if I started my task and its running for 2 hours, meanwhile any new changes that are coming on that DB2 table - will those also be replicated?

Thanks

Re: Full load task

Dana_Baldwin — Tue, 23 Apr 2024 19:11:26 GMT

Hi @Preethi13

Assuming that your task is set to both full load and apply changes, yes - when you start a full load, the first thing that happens is we start reading the source log files and cache those changes. When the full load completes, those changes will be applied to the target tables.

Please note - any pending transactions at the time you start the full load that do not commit before the value set for "Transaction consistency timeout" (Task Settings \ Full Load \ Full Load Tuning) will not be replicated to the target. Transactions can only be recognized by Replicate when they begin - pending/in flight transactions cannot be processed when a full load starts.

Thanks,

Dana

Re: Full load task

Preethi13 — Tue, 23 Apr 2024 19:18:07 GMT

Hi Dana, thank you so much for your response. How would it behave if it is a full load only task. The CDC is not enabled in this case. Only the source has cdc happening. Please help me understand.

Re: Full load task

Dana_Baldwin — Tue, 23 Apr 2024 21:01:16 GMT

Hi @Preethi13

If only full load is running, the task will not read from the source transaction logs, it will only run a "select *" query against all the tables and move the committed data to the target as of the time the query ran. By default tasks process 5 tables at a time, so the target table data will not necessarily be consistent as of the same point in time on the source. The number of tables handled at a time can be adjusted on the same task settings screen I referenced earlier. Of course, the more tables processed concurrently, the more source & network resources will be consumed, as well as memory & CPU on the Replicate server.

Thanks,

Dana

Re: Full load task

SushilKumar — Wed, 01 May 2024 07:36:35 GMT

hello @Preethi13

To add more to Dana's answer. as you query raised regarding "started my task and its running for 2 hours, meanwhile any new changes that are coming on that DB2 table. "

then you would see those changes in Cached changes in monitor.

Such behavior depends upon the table nature categories as (Static and dynamic table)

It aways recommend running initial task FL+CDC. reason behind it.

when you start loading your table record count was 100 and when got 4 changes in between. if it's a full load then only 100 changes pushed to target end.

to apply 4 changes to need to start the CDC when. full load completes and it's a manual work which sometime cause data mismatch.

so better use FL+CDC . if you don't want to keep FL then Stop task disable FL and resume

Regards,

Sushil Kumar

Re: Full load task

Preethi13 — Wed, 01 May 2024 23:26:47 GMT

Thank you for all your input!

Re: Full load task

DesmondWOO — Thu, 02 May 2024 01:24:31 GMT

Hi @Preethi13 ,

What is your source, DB2 LUW, DB2 iSeries or DB2 z/OS?

Regards,
Desmond