Use last seen LSN instead of last used LSN

[mwj]

Hello,

We are trying to set up Qlik Replicate with a db2 for luw database as source. The database uses commvault as its primary archive method. However we are seeing that Qlik will loop through logs from the last used LSN to the latest LSN.

Is there anyway to specify in Qlik Replicate that it should look for changes from the latest/last seen LSN instead of looking for changes all the way back to the last used LSN (last time there was a update in the selected tables)?

We use Qlik Enterprise Manager to configure Qlik Replicate.

/mwj

[Barb_Fill21]

@mwj   Hello!

Please confirm your Issue: reading the logs from the last used LSN to the latest LSN.

If YES,  this is normal  behavior - to read from where it was last processing... and then it continues to read up to the current LSN  - This is correct  - 

Please confirm - 

many thanks!

Barb

[SushilKumar]

Hello team,

If our response has been helpful, please consider clicking "Accept as Solution". This will assist other users in easily finding the answer.

Regards,

Sushil Kumar

[mwj]

Hello @Barb_Fill21. Yes I can confirm this is what I'm asking. However we are experiencing that it will loop in this read from last used LSN to latest LSN if there are no relevant changes to the selected tables which makes me wonder why does it need to read fra last used LSN to latest LSN and not just check the latest LSN for changes?

[SushilKumar]

Hello Team,

It is the default behaviour of any replication tool. As they Scan changes from the transaction log to capture the changes for the participating table.  

there are few condition replicate will work. 

1. let say it was stopped when reading lSN  123 and committed the data to target. If it was graceful stopping of task then it will keep the info about till where or which LSN it processed the data.  once it resumes it will start from LSN 123 Which we refer as  latest LSN.

2. db is up and running and put the task to hold for 2 hours then your latest LSN will be 154. Then it will read from last used LSN 123 up to latest LSN 154. 

3. let say you initiate a dml operation when LSN was 121 and Replicate is reading latest LSN 154. it will keep all the information in Sorter or temp area until the commit happened. assume due to XYZ reason QR task stopped and once resume it will require to read from the LSN 121 not from LSN 154 . because its scanning the changes for participating table although it didn't find commit/rollback for ongoing DML

hope this helps.

Reagrds,

Sushil Kumar

[mwj]

Hello @SushilKumar,

Thank you for clarifying. However I will try to describe what we are experiencing. 

1. I have a replication task of two db2 tables. Which on the Qlik Enterprise Manager looks as it should.
2. The DBA of db2 however sees that when we start CDC we will look for LSN back in time. This is fine as if changes are made on one of the tables Qlik Replicate stops retrieving logs back in time and starts looking at the newest LSN - lets call it LSN123.
3. Now Qlik Replicate starts to look at LSN123 but meanwhile there are changes to other stuff on the database and LSN124 is generated. However no changes to the two selected tables in LSN124, but Qlik Replicate still looks "from" LSN123. So it will look through LSN123 and LSN124. Even though it know LSN124 is the newest or last seen.
4. Let's say there is generated LSN125, LSN126, LSN127 and there are still no changes to the two tables Qlik Replicate will look through LSN123, LSN124, LSN125, LSN,126, LSN127 and then find no changes. A few moments later when it will scan for changes again it will loop over the same LSN's again. But could have just looked at LSN127..

I will point out this might not be a problem for many, but when using commvault it takes a lot of process power to read through multiple LSN.

I think you described the technology just fine, but my question is still why does it need to read over all LSNs from the last used to the last seen instead of just reading the last seen?

Best regards,

Malthe

[SushilKumar]

Hello Team,

To Answer your querry "but my question is still why does it need to read over all LSNs from the last used to the last seen instead of just reading the last seen?" Request you to enagge technical Support via a case and provide Diagnostics package .

Please refer to Replicate https://community.qlik.com/t5/Knowledge/How-to-collect-Diagnostics-Package-from-Qlik-Replicate/ta-p/...

Regards,
Sushil Kumar

[mwj]

Hello @SushilKumar,

Why cannot this question be answered here? 🙂 We are not yet commercial users of Qlik Replicate and therefore do not have a support agreement.

[Dana_Baldwin]

Hi @mwj 

Are you seeing this behavior while the task is running, or only when you stop and resume the task?

Generally, we request a support case to be opened when log files need to be checked or we suspect that we might need input from our internal support team / R & D in the event there is a software issue, or to help explain the behavior. We don't have a mechanism to elevate support questions from our Community posts. Another way to get help might be to ask your account manager for help from our technical pre-sales team.

Thanks,

Dana

[mwj]

Hi @Dana_Baldwin,

It is while the task is running it keeps looping over det same logs files over and over again.

/Malthe