topic Re: Qlik Replicate : Postgres to Snowflake often chockes in Qlik Replicate

Qlik Replicate : Postgres to Snowflake often chockes

gkaur — Sun, 20 Oct 2024 16:12:59 GMT

I have raised this concern many time but no solid explanation...

Now trying to interpret following log message - Is it saying that at 11:59 AM, task is stil reading log generated at 4:31 AM ?

03082376: 2024-10-20T11:59:46:907929 [SOURCE_CAPTURE ]T:

Throughput monitor: Last DB time scanned: 2024-10-20 04:31:02.161301-04. Last LSN scanned: 00019C5E/597EACB8. #scanned events: 0. (postgres_endpoint_wal_engine.c:430)

---

Behavior of my task : It says running with zero latency but the WAL on the system keeps growing, on number of occasions it reaches 200GB without capturing anything for the entire day and ultimately dies when DB script remove the log file.

Turning on log in verbose give no clue.. prints no table, no indicating of LSN progressing so.. kind of stuck and thinking if this product works for postgres as source ?

Re: Qlik Replicate : Postgres to Snowflake often chockes

Kent_Feng — Mon, 21 Oct 2024 00:03:18 GMT

Hi @gkaur

11:59:46 is the time when this log entry was recorded. 04:31:02.is the last time the database's WAL was scanned. The log tells that Replicate has scanned WAL between 04:31:02.and 11:59:46 but found no events related to the selected tables of the task.

Your WAL is growing not it may be related to the tables selected in the current task. Do you see any errors or warnings in the task log? You may need to open a support case so our TSE can look into the issue for you.

Thanks

Kent

Re: Qlik Replicate : Postgres to Snowflake often chockes

SachinB — Mon, 21 Oct 2024 01:20:51 GMT

Hello @gkaur ,

Yes, the log message you shared indicates that at 11:59 AM (2024-10-20T11:59:46), the task is still scanning the WAL logs from 4:31 AM (2024-10-20 04:31:02). This suggests that the task is not keeping up with the changes being written to the WAL (Write-Ahead Log) in real-time.

The fact that the task shows zero latency but WAL is growing suggests that Qlik Replicate may not be processing the captured events properly. It could be an issue with reading or processing the WAL logs.

If the WAL logs keep growing to 200GB without any events being captured, it's likely that Replicate is unable to process or move forward from a particular log segment. This can lead to the accumulation of logs, eventually filling up disk space.

We would request you the create support case to debug further on this.

Regards,

Sachin B

Re: Qlik Replicate : Postgres to Snowflake often chockes

DesmondWOO — Mon, 21 Oct 2024 07:32:25 GMT

Hi @gkaur ,

The WAL size keeps growing, which may be caused by other replication slots. Please run

SELECT * FROM pg_replication_slots;

to check for any additional replication slots.

Regards,
Desmond

Re: Qlik Replicate : Postgres to Snowflake often chockes

john_wang — Tue, 22 Oct 2024 13:30:10 GMT

Hello @gkaur ,

If a PostgreSQL CDC task that is running but not capturing changes (due to the interesting source tables inactivity) will continue to occupy the LSN position in its replication slot, thereby preventing truncation of the WAL. Please enable the WAL heartbeat, When this option is enabled, the PostgreSQL source endpoint mimics task activity by periodically committing pseudo transactions, the sample:

Hope this helps.

John.