Qlik Replicate and SAP HANA Trigger based CDC

[shashi_holla]

This is an attempt to provide more details on how the Trigger based CDC method works for SAP HANA source end points.

 

Environment

Qlik Replicate 

When we work with SAP HANA source endpoint, we have two options (Advanced tab):

1. Log Based CDC
2. Trigger Based CDC 

Log Based CDC is the recommended approach and it reads the  data from Online Logs/Backup Logs or combination of both giving preference to Backup logs. But in some cases this can be non-beneficial if the redo log size is huge (depending on the DB size and number of tables) and Replicate task do not have most of the tables from DB. In these scenarios sorter operation in Replicate has to read and store all data from redo before it can figure out the required data and process those.

To avoid this drawback we have the other method which is Trigger based CDC and it focuses only on the tables mentioned in the task and doesn't consider everything else that happens in the Database.

This is how it works:

1. Create three triggers (INSERT, UPDATE and DELETE) on the source tables.
2. The triggers are defined to write any of those three operation data to attrep_cdc_changes table
3. Then the task reads from the attrep_cdc_changes and process the records.
4. We can also choose to have Log table as per the settings, which adds one more table attrep_cdc_log
5. When the Use log table option in the SAP HANA endpoint's Advanced tab is enabled, changes are copied from the attrep_cdc_changes table to the attrep_cdc_log table. During the task, Replicate reads the changes from the attrep_cdc_log table and deletes the old data according to the cleanup settings (specified in the endpoint's Advanced tab).

User guide which provides details on the required permissions to enable the option: Permissions

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

[WalterSchmidt]

How to use "Log Based CDC" when the SAP hoster is using the SAP-“Backint interface”.
As far as I know, this means that access to the log files is not possible.

Does Qlik already have a solution for this problem, as this type of backup creation is currently "state of the art"?
So the only solution is probably to use the redo logs - although this is not recommended for a productive SAP system.

[Dana_Baldwin]

Hi @WalterSchmidt 

Please check our documentation on the different ways to work with an SAP source endpoint. Some use the log files, some do not:

Adding and managing source endpoints | Qlik Replicate Help

Thanks,

Dana

[WalterSchmidt]

Hi @Dana_Baldwin 
sorry but that's not helpful.

I wouldn't ask if I don't have to use the log-based cdc (for example S4-Hana table ACDOCA).

Is it possible to address this problem do the development team?
This backup procedure will prevent many customers from using log-based cdc, which means that Replicate will lose its unique selling point. It should therefore be in Qlik's interest to analyze this.

[Sonja_Bauernfeind]

Hello @WalterSchmidt 

If you have identified a feature you're interested in seeing in our products, our Ideation section is the best way to address our product teams directly. The suggestions brought there by our customers are reviewed by our product managers and we appreciate any feedback sent to us this way.

You also have the possibility of raising the question in our Qlik Replicate forum, where our active engineers and your knowledgeable Qlik peers can help you with more specific use cases and questions; and, of course, we're always here to assist you more directly through Support (you can start chatting with us to create a case in the bottom right corner of this screen). 

All the best,
Sonja