Handle replication after database clones

[chris_s]

Does anybody have experience with replication and DB cloning?

We have 3 stages:

Production, Acceptance, Development

On a regular basis the data on Acceptance and Development is deleted and replaced by the Data from Production. We call this process cloning.

All three databases are replicated with Qlik replicate into Kafka (and some tables into BigQuery and Postgres).

On Production this cloning process has no impact. But on Acceptance and Development this is causing some problems because during this cloning process we have to stop all tasks and after the cloning we have to reload all tasks. And because the involved data is around 3 TB the whole reload takes several days. During this time the data on our Development and Acceptance stage is not available and therefore other teams are affected by this cloning process.

Does anybody face a similar situation? Is it possible to do such a reload without doing a full reload?

Involved Source Database: Oracle

Involved Target System: Kafka (mainly), BigQuery, Postgres

Occurrence: monthly

[john_wang]

Hello @chris_s ,

Thanks for reaching out to Qlik Community!

If the other teams downstream apps were impacted by the Full Load stage, and you are interesting the change data only, then you may turn off Full Load and turn on Change Processing ONLY. With this task setting, whenever the task is startup again, it will process Change Data ONLY. Certainly there is a dummy Full Load stage however the stage only verify the metadata, it will be completed in seconds. Sample:

Hope this helps.

John.

[john_wang]

Hello @chris_s ,

Please take note that the above settings can be used in Kafka target tasks.

And for BigQuery, Postgres target tasks,

1- No need Full Load if the task set to Store Changes 

If the task is set to Apply Changes then there are 2 different scenarios, depends on the task settings and your project needs:

2.1 - Need the Full Load still if task set to default Apply Conflicts Handling Policy. Because the subsequence UPDATE/DELETE will impact the existing row(s).

2.2 - No need the Full Load if task set to UPSERT mode.

Hope this helps.

John.

[chris_s]

Hi @john_wang 
Thanks a lot for the quick response.

I guess with this we would not get the changes.

Let's say we have the following timeline on a prd table:
1) Insert Record A 
2) Insert Record B
3) Insert Record C
4) Delete Record B
---
This would result in the following content of the PRD table: A + C

Lets say on ACC we have the following State: A

Then the clone happens: 
1. Truncate 
2a Insert Record A
2b Insert Record C

Now it would be really convenient if we only detect and propagate the following change with Qlik replicate: 
Insert Record C.

Is this somehow possible?

[SushilKumar]

Hello team,

If our response has been helpful, please consider clicking "Accept as Solution". This will assist other users in easily finding the answer.

Regards,

Sushil Kumar

[DesmondWOO]

Hi @chris_s ,

I am not sure I understand your question. What is your source database? Why does the TRUNCATE operation occur in the ACC environment? 

Let's say an Oracle database. If you know the SCN of the Record C (BEGIN TRANSACTION), You may resume your task from that stream position.

Regards,
Desmond