Skip to main content
Announcements
UPGRADE ADVISORY for Qlik Replicate 2024.5: Read More
cancel
Showing results for 
Search instead for 
Did you mean: 
andile
Contributor II
Contributor II

AWS RDS for Postgres as source - supported versions

Hi Qlik Community

Replicating from a Postgres read replica is merely impossible on the currently supported versions. However, Postgres 16.1 does allow read/write on read replicas which enables setting the wal level to 'logical' on the read replica. This is significant progress as Teams opt to use read replicas instead of replicating directly from the primary instance.

That said, there's limitations like enabling 'Create DDL events' and 'WAL heartbeats'. We'd like to capture the DDL events, and also benefit from keep the slots alive. But these require a 'create statement being run, and Postgres 16.1 is also not supported. Currently we don't have a solution for reading from a Postgres read replica if we want to replicate changes - 'cause full load works but not viable for large databases.

Can I ask if there's plans to include Postgres 16.1 on the supported source endpoints. If so, when? And if we'd be able to replicate from a read replica, plus benefit from 'Create DDL events' and 'WAL heartbeats'.

Regards.

Labels (3)
1 Solution

Accepted Solutions
john_wang
Support
Support

Hello @andile ,

Qlik Replicate 2024.5 was formal released, it can downloaded from Qlik Download Center.  Qlik Replicate 2024.5 supports PostgreSQL 16.x as source endpoint

Regards,

John.

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!

View solution in original post

8 Replies
DesmondWOO
Support
Support

Hi @andile ,

Thank you for reaching out to the Qlik Community.

The inclusion of support for PostgreSQL 16.1 is already planned in the roadmap. However, regarding other features such as WAL heartbeats, I don't have details on hand. You may submit your questions to the Qlik Ideation.

Regards,
Desmond


 

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
john_wang
Support
Support

Hello @andile ,
Not sure what's the replication between the primary/standby node and "read replica" node, is it streaming replication, or logical replication? Take note the DDL commands are not replicated in Logical Replication even in PostgreSQL v16.2.
Would you share a sample configuration file "postgresql.conf" (of the "read replica"), we'd like to understand the request further.

thanks,
John.

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
SushilKumar
Support
Support

hello @andile 

Can I ask if there's plans to include Postgres 16.1 on the supported source endpoints. If so, when? And if we'd be able to replicate from a read replica, plus benefit from 'Create DDL events' and 'WAL heartbeats'.

Request you follow communty link it may be possible in upcoming product release notes. you will find the feature or support with postgre endpoint.

Regards,

Sushil Kumar 

andile
Contributor II
Contributor II
Author

Thanks for prompt responses.

@DesmondWOO I've doing more reading on WAL heartbeats. I'll be testing some of the theories to verify them.

@john_wang I'm unable to get the config file on the spot, but I've noted your point. Once I get the config file, I will attach it on the future reply.

@SushilKumar  I'll check the release notes to confirm this.

 

I'm looking forward to the 2024.5 release as this will be a huge benefit for Postgres sources we're replicating.

 

Regards

Andile.

john_wang
Support
Support

Hello @andile ,

Thank you so much for the update.

Yes, this is the exactly concern.

While primary node and replica node are connected by steaming replication, the DDL will be replicated within the 2 nodes in PostgreSQL. However the PG Replica's WAL_LEVEL cannot set to "logical" so far Qlik Replicate cannot work with PostgreSQL Replica node.

While primary node and replica node are connected by logical replication, the PG Replica's WAL_LEVEL can be set to "logical" however with this configuration the DDL cannot be replicated between the 2 nodes in PostgreSQL, this is PG itself behavior. We cannot expect Qlik Replicate to get DDLs changes from PG replica's node either.

BTW, I've confirmed Qlik Replicate 2023.11 works with PG v16.2 pub/sub configuration, the sub node acts as Qlik Replicate source endpoint, the sanity test passed. Hope v16 will be supported in the next major versions.

Best Regards,

John.

 

 

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
john_wang
Support
Support

Hello @andile ,

Qlik Replicate 2024.5 was formal released, it can downloaded from Qlik Download Center.  Qlik Replicate 2024.5 supports PostgreSQL 16.x as source endpoint

Regards,

John.

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!
JWade_Qlik
Employee
Employee

Hi @andile we have a customer that has submitted a similar enhancement request.  Please see Ideation #420176 titled "Ability to use Postgres RDS Read Replica for change data capture" and like/comment if it will be useful. https://community.qlik.com/t5/Ideation/ct-p/qlik-product-insight 

DesmondWOO
Support
Support

Hi @andile ,

Qlik Replicate v2024 May supports PostgreSQL 16.x. Please find the use guide for details.

Regards,
Desmond

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!