IBM DB2 for iSeries(Log Stream Task)

[Pranita123]

Hello Community,

We've encountered a significant issue with our Qlik Replicate log stream task connected to our source database, IBM DB2 for iSeries. A large number of CDC (Change Data Capture) events were received in just a few transactions, causing a bottleneck in processing. These events were stuck on disk until the source committed them, resulting in a spike in latency that we couldn't resolve.

Upon investigation, we found that huge sorter files were generated for these changes, further exacerbating the problem.

We're now seeking advice on how to address this issue. Any guidance or recommendations would be greatly appreciated.

Regard,

Pranita

[Dana_Baldwin]

Hi @Pranita123 

Large and/or long running transactions on the source that do not commit right away are cached to disk in order to avoid exhausting memory resources on the server. This is by design. We only move committed transactions to the target so we are dependent upon, and impacted by, the behavior of the transactions on the source. The latency shown is not avoidable due to this. If this type of transaction profile is normal for your source you may need to allocate additional disk space to accommodate the cached data required when this happens.

Thanks,

Dana

[Bill_Steinagle]

@Pranita123 

Thank you for the post to the QDI Forums. For the Latency on the DB2i Source you can also check with the DBA for any Maint activities on the environment along with any Reorg's to clean up RRN's as needed. As noted by @Dana_Baldwin noted the uncommited transactions are stored in the sorter folder until the commit is captured. You can also check for large batch processing on the Source as well which would impact the Latency of the Task and the Tables in the Task the commit rates as well from the Application team. You can also confirm with the DBA on the generation of the Receivers for this environment as well as this would also be good indication of batch process being run on the Source. Hope this is also helpful. 

Regards,

Bill