topic Re: Full Load Check questions in Qlik Replicate

Full Load Check questions

MoeE — Mon, 18 Dec 2023 01:29:44 GMT

Hi,

I hope you guys are doing well. I'd like to learn more about how "full load checks" work. My understanding is that a full load check runs on the target database in these scenarios. An initial full load, a reload, a start from timestamp. My understanding is that it checks if the target tables exist and that the records from the source are consistent with the target records. Is this correct?

(1) What exactly does a full load check do? How does it work?

(2) What happens if the records from the source and inconsistent with the target?

(a) I have a scenario where only CDC is enabled while full load is disabled, what happens if the source and target records differ?

(b) Will that be fixed in CDC? How will it be fixed?

(3) Looking through some logs of when a full load check occurs. What exactly does 0 rows received. 0 rows skipped. Volume transferred mean?

Please share your great wisdom with me. Any help with this would be greatly appreciated. Thank you.

Kind regards,

Mohammed

Re: Full Load Check questions

Heinvandenheuvel — Mon, 18 Dec 2023 03:04:29 GMT

Where exactly do you see this "full load checks"?

I do not recall seeing this anywhere in the UI, and cannot find it in the 2023-11 documentation. The closest I find is "Check for changes" in the salesforce an (csv)file source endpoint.

Hein.

Re: Full Load Check questions

MoeE — Mon, 18 Dec 2023 04:05:11 GMT

Hi Hein,

Doesn't Replicate check to see if the source tables exist on the target?

Regards,

Mohammed

Re: Full Load Check questions

Heinvandenheuvel — Mon, 18 Dec 2023 04:41:04 GMT

ok, the way you wrote it made me think you found an option/setting labeled "full load checks" but that does not seem to be the case.

>>> My understanding is that it checks if the target tables exist

Correct, and it creates it if it does not exists. If it does exists, based on task settings it leaves it alone, truncates or drop+recreate (default)

>>>> and that the records from the source are consistent with the target records. Is this correct?

Noop. It just starts a SELECT * FROM TABLE and starts moving data. No checks, and just a point in time for full-load only. Now if you enable CDC than that will be activated BEFORE the SELECT such that any changes during the transfer can be cached and applied after the transfer for maximum consistency.

>>> (a) I have a scenario where only CDC is enabled while full load is disabled, what happens if the source and target records differ?

Nothing. You'd have to enable full-load, or use a second task for the full-load if they were out of sync. You can also for a 'benign' update to all rows in the tables with error handler 'create if target row does not exist' to fake out updates to each row getting deletes+(re)-inserts.

>>> (3) Looking through some logs of when a full load check occurs. What exactly does 0 rows received. 0 rows skipped. Volume transferred mean?

That just means that the whole table, or in this case a partition for the table was empty.

Hein.

Re: Full Load Check questions

john_wang — Mon, 18 Dec 2023 05:24:11 GMT

Hello @MoeE ,

Besides @Heinvandenheuvel comment, I think you are confused by the message:

00027832 2023-07-11T16.31Q6.747865 [TARGET_LOAD ]I: Load finished for segment #143 of segmented table ''ASSET_ODS.'RM_UE'(ld = 22). 0 rows received. 0 rows skipped. Volume transferred 0 (streamcomponent.c:4078)

This is expected message. Glad to see you pay attention to it: Qlik Replicate does a 'dummy' Full Load while the task is in RESUME startup mode - it does check all thing as same as on a real Full Load or on 'first' startup (after task design completion) but no data rows was transferred. The dummy full load occurs in below scenarios:

1- Full Load is off (CDC is ON)

2- The task is in RESUME startup mode (does not matter Full Load is ON or OFF)

3- Metadata ONLY mode

hope this helps.

Regards,

John.