Cannot able to read _ct tables from databricks

[chandraprakash_j_volvo]

source: Oracle Database

target: Databricks Lakehouse Delta

Sample Table: Location

We enabled Store Changes. We could see addition folder created in our storage account and location_ct table in databricks. I can preview the table. But I cant open the table from the notebook. 

And I want to store before image and after image in target table  aswell. Is there a way to do it

[SachinB · Accepted solution]

Hello @chandraprakash_j_volvo ,

Thank you for reaching out to the Qlik community !

Yes, we have do this option in store change tunning (As below)

• On UPDATE: Select one of the following options to determine how to store UPDATEs to the source tables:

  • Store before and after image - To store both the pre-UPDATE data and the post-UPDATE data.
  • Store after image only - To store only the post-UPDATE data.

You can view the at target endpoint by querying on it.

e.g  select * from table1__ct;

Hope this is clears you!!

Regards,

Sachin B

[sureshkumar]

Hello @chandraprakash_j_volvo 

Can you please elaborate when you mean by "I cant open the table from the notebook"

In Store changes we have options to enable both before image and after image as well
Go to Task Settings-->Change Processing-->Store Changes Settings-->On UPDATE: Store before and after image

Regards,
Suresh

[vinayakct]

Hi @chandraprakash_j_volvo ,

Just wanted to understand how you are reading the stored changes along with the original table.
Is there a dynamic framework or script that you have written in Databricks to merge them.

Context: From Qlik we are sending the data into AWS S3 using full load + store changes.

How do you suggest we read the data in databricks from S3 from both tables / merge them into a single one for further analytics. 
If you can suggest a dynamic approach that would be best, or a script which we can develop.

Thanks in advance! 🙂