Inquiry About Using Aurora Read Replica for Replication to Snowflake

[Ayman1]

We are currently planning to set up data replication from our Aurora PostgreSQL & Aurora MySQL databases to Snowflake using Qlik Replicate (Attunity).

However, we have a specific requirement and constraint:

• Requirement: Replication from a read replica.
• Constraint: Replicating directly from the master database is not an option.

We understand that binary logs (binlogs) are necessary for change data capture (CDC) to function correctly. Given our constraints, we would like to know if it is possible to achieve replication using a read replica of some sort (either for PostgreSQL or MySQL) instead of the master database.

Specifically, we would like to understand:

1. Can Qlik Replicate use a read replica for CDC purposes?
2. Are there any specific configurations or setups required to enable this functionality using a read replica?
3. Are there any known limitations or considerations we should be aware of when using a read replica for this purpose?

Thanx

[john_wang]

Hello @Ayman1 ,

Thanks for reaching out to Qlik Community!

Qlik Replicate is a log-based Change Data Capture (CDC) product that retrieves changes from database transaction logs (binlog in MySQL, WAL in PostgreSQL). If the necessary conditions are met, it can obtain changes from a secondary replica node in both PostgreSQL and MySQL. However, there are some limitations, and the feasibility depends on the specific configurations of PostgreSQL and/or MySQL.

PostgreSQL: Postgres 16.x does allow read/write on read replicas which enables setting the WAL level to 'logical' on the read replica. Qlik Replicate 2024.5 supports PostgreSQL 16.x as source endpoint. We have a discussion about this topic. The limitation is that, DDL cannot be replicated from PostgreSQL master node to PostgreSQL replica node hence DDL replication is impossible being supported by Qlik Replicate, DML operations are supported.

MySQL: If the slave/replica node enabled "Log_slave_update" then the binlog is available in the replica node, Qlik Replicate could get changes from binlog of the replica node, especially if the master slave replication setup is GTID (Global Transaction ID) then it's much easier.

The above conclusion is based on technical analysis. We strongly recommend engaging Professional Services (PS) to conduct a thorough acceptance test before implementing it in the production system.

Hope this helps.

John.