Didn't notice CDC turned back after successful completion of full load
Did anyone face an issue for Qlik Replicate to turn back CDC immediately after successful completion of full load? It was working as expected a week back, but since yesterday we didn't see that anymore. Can anyone please share how to address this step by step? This is a new product for us and want to validate this issue in our lower environment before roll-out to production. Thank you for your help in advance ...
While the system is designed to seamlessly transition from Full Load to CDC, technical bottlenecks and configurations can cause delays or failures, often resulting in high latency or CDC not applying changes.
Here are the common causes and solutions found in Qlik Community discussions and support articles:
Common Causes & Scenarios
Target Performance Bottleneck: CDC may start, but the task shows high latency because it is busy applying cached changes (changes that occurred during the full load) before it can start processing real-time changes.
Table Name/Column Name Length: For Oracle targets, if table or column names exceed 30 characters, they may not be properly registered in the logs, causing CDC to fail after a successful full load.
Internal Parameter Misconfiguration: Specifically, the skipValidationLongNames setting might cause issues, or require a GoldenGate license with Oracle.
Premature Task Stop: If you stop the task before the full load officially completes (i.e., before the task transitions to 'running' for CDC), the task may default back to full load when resumed.
Missing Supplemental Logging (Oracle): If not enabled, Oracle does not write all necessary log data, causing updates to fail.
Key Troubleshooting & Solutions
Check Task Status & Logs: Monitor the "Full Load" and "CDC" tabs in the console to see if the task is in a "Caching Changes" state, which is normal behavior.
Verify Oracle Column Names: Ensure column names do not exceed 30 characters or that the appropriate database parameter is set.
Use "Store changes" (CDC) + "Full Load": Ensure both are checked in task settings for a seamless transition, allowing Replicate to capture changes during the load.
Check Oracle Supplemental Logging: Ensure it is enabled for all necessary tables.
Upgrade Replicate: A known issue (RECOB-9924) causing data truncation in special characters during the transition was fixed in Qlik Replicate 2024.11 Build 620 or higher.
If the issue persists, the recommended action is to set SOURCE_CAPTURE to VERBOSE in the task settings to analyze the logs for the specific moment of failure.
Thanks Greg. Our source is SQL Server and we are loading the data in ADLS in JSON format. From ADLS, we are consuming the data into Snowflake using Snowpipe. Everything was working till the beginning of last week. We are configuring our pipelines using Qlik Cloud. Little different approach than regular source to target in table format. But this is our configuration.
Investigation is going on. Will post what we've found. Working with Qlik PS and Support. But thought to post this in the community if anyone else faces this issue with the similar configuration.
Qlik Pipeline is a replication project to land data in data lake from SQL Server Gateway. The update method is CDC means Landing always begins with a full load followed by CDC.
For this setting, the CDC is turned back on after successful completion of full load. Depending on the volume, definitely there will be some delay but in dev environment, we noticed earlier that CDC is back in the monitor UI immediately as the full load completed within 90 minutes. This was last week. But not at present.
