topic Re: Data quality checks for replication in Databricks in Qlik Replicate

Data quality checks for replication in Databricks

maiyaa — Mon, 09 May 2022 10:20:40 GMT

Is there any tooling for validating data that has been mirrored from source tables to Databricks? I’m using AWS DMS for CDC but running into lots of issues with updates being reflected incorrectly in Databricks. Running checks across thousands of tables is needed.

Re: Data quality checks for replication in Databricks

Michael_Litz — Mon, 09 May 2022 17:11:17 GMT

Hi Maiyaa,

Replicate does not have any data validation routines built in.

Thanks,
Michael

Re: Data quality checks for replication in Databricks

SwathiPulagam — Mon, 09 May 2022 19:56:58 GMT

Hi @maiyaa ,

You can try Querysurge.

It is not a free tool but it is a low-cost validation tool.

Thanks,

Swathi