topic Re: Cannot able to read _ct tables from databricks in Qlik Replicate

Cannot able to read _ct tables from databricks

chandraprakash_j_volvo — Mon, 11 Dec 2023 09:11:08 GMT

source: Oracle Database

target: Databricks Lakehouse Delta

Sample Table: Location

We enabled Store Changes. We could see addition folder created in our storage account and location_ct table in databricks. I can preview the table. But I cant open the table from the notebook.

And I want to store before image and after image in target table aswell. Is there a way to do it

Re: Cannot able to read _ct tables from databricks

SachinB — Mon, 11 Dec 2023 09:32:21 GMT

Hello @chandraprakash_j_volvo ,

Thank you for reaching out to the Qlik community !

Yes, we have do this option in store change tunning (As below)

On UPDATE: Select one of the following options to determine how to store UPDATEs to the source tables:
- Store before and after image - To store both the pre-UPDATE data and the post-UPDATE data.
- Store after image only - To store only the post-UPDATE data.

You can view the at target endpoint by querying on it.

e.g select * from table1__ct;

Hope this is clears you!!

Regards,

Sachin B

Re: Cannot able to read _ct tables from databricks

sureshkumar — Mon, 11 Dec 2023 09:32:32 GMT

Hello @chandraprakash_j_volvo

Can you please elaborate when you mean by "I cant open the table from the notebook"

In Store changes we have options to enable both before image and after image as well
Go to Task Settings-->Change Processing-->Store Changes Settings-->On UPDATE: Store before and after image

Regards,
Suresh

Re: Cannot able to read _ct tables from databricks

vinayakct — Tue, 02 Jul 2024 09:56:44 GMT

Hi @chandraprakash_j_volvo ,

Just wanted to understand how you are reading the stored changes along with the original table.
Is there a dynamic framework or script that you have written in Databricks to merge them.

Context: From Qlik we are sending the data into AWS S3 using full load + store changes.

How do you suggest we read the data in databricks from S3 from both tables / merge them into a single one for further analytics.
If you can suggest a dynamic approach that would be best, or a script which we can develop.

Thanks in advance! 🙂