AWS RDS for Postgres as source - supported versions

[andile]

Hi Qlik Community

Replicating from a Postgres read replica is merely impossible on the currently supported versions. However, Postgres 16.1 does allow read/write on read replicas which enables setting the wal level to 'logical' on the read replica. This is significant progress as Teams opt to use read replicas instead of replicating directly from the primary instance.

That said, there's limitations like enabling 'Create DDL events' and 'WAL heartbeats'. We'd like to capture the DDL events, and also benefit from keep the slots alive. But these require a 'create statement being run, and Postgres 16.1 is also not supported. Currently we don't have a solution for reading from a Postgres read replica if we want to replicate changes - 'cause full load works but not viable for large databases.

Can I ask if there's plans to include Postgres 16.1 on the supported source endpoints. If so, when? And if we'd be able to replicate from a read replica, plus benefit from 'Create DDL events' and 'WAL heartbeats'.

Regards.

[DesmondWOO]

Hi @andile ,

Thank you for reaching out to the Qlik Community.

The inclusion of support for PostgreSQL 16.1 is already planned in the roadmap. However, regarding other features such as WAL heartbeats, I don't have details on hand. You may submit your questions to the Qlik Ideation.

Regards,
Desmond

[john_wang]

Hello @andile ,
Not sure what's the replication between the primary/standby node and "read replica" node, is it streaming replication, or logical replication? Take note the DDL commands are not replicated in Logical Replication even in PostgreSQL v16.2.
Would you share a sample configuration file "postgresql.conf" (of the "read replica"), we'd like to understand the request further.

thanks,
John.

[SushilKumar]

hello @andile 

Can I ask if there's plans to include Postgres 16.1 on the supported source endpoints. If so, when? And if we'd be able to replicate from a read replica, plus benefit from 'Create DDL events' and 'WAL heartbeats'.

Request you follow communty link it may be possible in upcoming product release notes. you will find the feature or support with postgre endpoint.

Regards,

Sushil Kumar 

[andile]

Thanks for prompt responses.

@DesmondWOO I've doing more reading on WAL heartbeats. I'll be testing some of the theories to verify them.

@john_wang I'm unable to get the config file on the spot, but I've noted your point. Once I get the config file, I will attach it on the future reply.

@SushilKumar  I'll check the release notes to confirm this.

I'm looking forward to the 2024.5 release as this will be a huge benefit for Postgres sources we're replicating.

Regards

Andile.

[john_wang]

Hello @andile ,

Thank you so much for the update.

Yes, this is the exactly concern.

While primary node and replica node are connected by steaming replication, the DDL will be replicated within the 2 nodes in PostgreSQL. However the PG Replica's WAL_LEVEL cannot set to "logical" so far Qlik Replicate cannot work with PostgreSQL Replica node.

While primary node and replica node are connected by logical replication, the PG Replica's WAL_LEVEL can be set to "logical" however with this configuration the DDL cannot be replicated between the 2 nodes in PostgreSQL, this is PG itself behavior. We cannot expect Qlik Replicate to get DDLs changes from PG replica's node either.

BTW, I've confirmed Qlik Replicate 2023.11 works with PG v16.2 pub/sub configuration, the sub node acts as Qlik Replicate source endpoint, the sanity test passed. Hope v16 will be supported in the next major versions.

Best Regards,

John.

[john_wang · Accepted solution]

Hello @andile ,

Qlik Replicate 2024.5 was formal released, it can downloaded from Qlik Download Center.  Qlik Replicate 2024.5 supports PostgreSQL 16.x as source endpoint. 

Regards,

John.

[JWade_Qlik]

Hi @andile we have a customer that has submitted a similar enhancement request.  Please see Ideation #420176 titled "Ability to use Postgres RDS Read Replica for change data capture" and like/comment if it will be useful. https://community.qlik.com/t5/Ideation/ct-p/qlik-product-insight 

[DesmondWOO]

Hi @andile ,

Qlik Replicate v2024 May supports PostgreSQL 16.x. Please find the use guide for details.

Regards,
Desmond