Target table columns become null during CDC

[desmondchew]

We are live with a QDI task in production. The source endpoint is MySQL RDS AWS while the target endpoint is Snowflake AWS. After reloading successfully, CDC run fine. After a few days, we noticed that for a table called "bookings" we have some columns fields got updated as null in the target endpoint. However it is not null in the source endpoint. 

Can we know what is happening? How can we troubleshoot further?

Thank you.
Desmond

[john_wang]

Hello Desmond, @desmondchew 

Thanks for reaching out to Qlik Community!

I'd like suggest you troubleshooting the issue by:

1- set SOURCE_CAPTURE/TARGET_APPLY to Verbose

2- in target endpoint, add internal parameters keepCSVFiles & keepErrorFiles, set them values to True

3- Recreate the behavior then check the task log files, and the CSV files to understand the reason.

4- You may need to decrypt the verbose task log files.

Hope this helps.

John.

[desmondchew]

Hi John,

Under Task Settings, will "Apply batched changes to multiple tables concurrently" cause this behavior? Will I be able to workaround without this batched changes? I have ensure the "error handling" of applying conflicts set to "INSERT-UPDATE" records. 

Thanks
Desmond

[john_wang]

Hello Desmond, @desmondchew 

I do not think "Apply batched changes to multiple tables concurrently" option leads to the data problem. Let's get the CSV files to understand why the column values become NULL.

Regards,

John.