Qlik Replicate: Starting Log Stream Tasks with Timestamp

[Michael_Litz]

Content:

• Restart with a timestamp for both log stream staging (parent) and replication (child) tasks
• Restart with a timestamp for replication task only
• Configuration tips:
• Related Content

Restart with a timestamp for both log stream staging (parent) and replication (child) tasks

Sometimes a log stream task pair will stop processing CDC data and you may need to restart the tasks with a timestamp in order to get the CDC flowing again.

If you are going to restart a log stream task with a timestamp, the same timestamp must be used for each task, the staging task and the replication task.

NOTE: If you start the staging task from a timestamp you must start all associated replication tasks with the same timestamp.

Restart with a timestamp for replication task only

If you want to just restart with a timestamp on the replication task and you do not know the correct time stamp you can find it following these 3 steps.

1. Restart the replication task with any timestamp you want - this will generate a log entry.
2. Use the generated timestamp in the above log line "last source timestamp 1640589670000000" - this number should be converted using this link: https://www.epochconverter.com/ . This timestamp should then be adjust to the time zone of the Replicate console (if they differ).
3. Use this timestamp to restart the replication task.

Configuration tips:

For an Oracle source log stream task

1. What are the chances of data duplication if we start the staging task with a past SCN number?
   
   Answer: Chances are good there will be issues. The following setting on Error Handling / Apply Conflicts will mitigate them.
   
   In the Error Handling / Apply Conflicts select Ignore record, for all 3 settings. Alternatively, if the target DB handles deletes okay you could set the above to Upsert / Merge mode.
   
   Reference: Qlik Replicate: Implementing UPSERT and MERGE modes by applying a Conflicts Handling Policy
   
   
2. There are a few options in apply conflicts policy, what exact options we should use to avoid data duplication after starting the parent task with the SCN number?
   
   Answer: Replicate will process CDC again, events for inserts, deletes and updates will be replayed. The above apply conflicts will suffice.
   
   
3. How do we get the exact SCN number from Oracle Source?
   
   Answer: There are two options to get the correct timestamp:
   
   
   1. On the same line in the log file, from which the SCN was taken, you can see an indication of the equivalent source timestamp, for example - "last source timestamp 1640589670000000" - this number should be converted using this link: https://www.epochconverter.com/ . This timestamp should then be adjusted to the time zone of the Qlik Replicate console (if they differ).
      
      
   2. Use Oracle command - SELECT SCN_TO_TIMESTAMP(ORA_SCN) FROM DUAL; instead of ORA_SCN put the SCN in decimal format, that was used for the LS task.
      
      The timestamp can be used for both the log stream staging and the replicate tasks (instead of using the SCN)

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

Related Content 

Qlik Replicate: Implementing UPSERT and MERGE modes by applying a Conflicts Handling Policy

Environment

• Qlik Replicate 
• Log Stream