How does your company handle replication and CDC?

[doketwaj]

My team has been struggling for months with finding a good replication and CDC solution. We are trying to replicate data from an app database and a few third party databases to BigQuery (or blob storage then eventually BQ). When I joined we had just signed a license with Fivetran which has worked seamlessly for the third party DBs but is extremely costly and prone to failure when using it for our application database.

Does anyone have a good solution for replication and CDC from Postgres to GCP? Our team is currently evaluating Confluent + Kafka. Ideally we want a solution that can handle a wide variety of types, handle schema drift, and won’t break the bank.

[SushilKumar]

Hello @doketwaj 

Our Qlik Replicate Have this Functionality request you to Get in touch with Sales team and move with POC.

Sharing a Link about Your Specific endpoint Querry.

https://help.qlik.com/en-US/replicate/May2024/Content/Replicate/Main/Support%20Matrix/supported_sour...

Source :- 

PostgreSQL

12.x, 13.x, 14.x, 15.x, and 16.x

Yes

Yes

https://help.qlik.com/en-US/replicate/May2024/Content/Replicate/Main/Support%20Matrix/supported_targ...

GCP :- 

Regards,

Sushil Kumar

[john_wang]

Hello @doketwaj ,

Welcome to Qlik Community forum and thanks for reaching out here!

Please allow me to introduce Qlik Replicate briefly:

1. Qlik Replicate is a powerful and easy-to-implement solution that provides seamless replication between various endpoints. With Qlik Replicate, you can set up a new task in just a few minutes.
2. Qlik Replicate can execute replication tasks between a large number of enterprise endpoints including Oracle, PostgreSQL, Microsoft SQL Server, Kafka, GQB, Hadoop, and IBM DB2. refer to the complete supported source endpoints and target endpoints list.
3. Qlik Replicate integrates Full Load and Change Data Capture (CDC) stages seamlessly and automatically. It also supports automatic Data Definition Language (DDL) changes and the creation of schemas and objects (tables, files).
4. As an in-memory processing product, Qlik Replicate generates no trial files.
5. Qlik Replicate is zero-footprint for most of the source and target systems thereby reducing the intrusion in the databases and data systems. There is no need to install anything on the Postgres database and Google BigQuery platform. Only the database ODBC clients need to be installed on the Qlik Replicate server.
6. Qlik Replicate retrieves change records from source endpoint transaction logs (e.g., TLOG in SQL Server, Redo Log in Oracle, Journal in DB2i, WAL in PostgreSQL, BinLog in MySQL, OPLOG in MongoDB) and applies these changes to the target endpoint database.

In summary, Qlik Replicate supports a wide variety of source and target types, making it a versatile and user-friendly solution that you'll love using.

Hope this helps,

John.