Best Practices for Handling Quarterly Source Archival Activities

[shyamkatika]

Hi Team,

We have one question regarding the quarterly DB2 zos source archival activities.

Previously, during a similar archival activity, we experienced very high latency issues due to huge transaction volumes generated in the source logs. Additionally, a few Qlik tasks encountered data missing issues because of resource constraints during heavy processing.

To mitigate the issue, we later increased the infrastructure resources by doubling the disk space, CPU, and RAM capacity.

Could you please help us understand the best practices or recommended approach for handling these kinds of large-volume archival activities? We would like to know the recommended planning, tuning, or operational strategies to minimize latency and avoid data loss/issues in Qlik replication tasks during such events.

Your guidance and recommendations would be very helpful for us.

Thanks,
Shyam Sundar.

[john_wang]

Hello Shyam Sundar, @shyamkatika 

Thanks for reaching out.

To ensure we address this effectively, we recommend engaging our Professional Services (PS) team. Given that this requires a holistic understanding of your entire environment—including hardware, software, and data volumes—their expert assistance will ensure a successful implementation.

Hope this helps.

John.

[Rahul_Kale · Accepted solution]

Hello shyamkatika,

Thank you for reaching out to the Qlik community.

You will need to split the archival into small commits and avoid huge transactions, causing latency spikes. 

Use Batch Optimised Apply (not row-by-row) to improve throughput. document.

Tune batch settings (memory + time) for large volumes.

Ensure enough CPU, RAM, and a fast disk (sorter) to prevent spills to the disk. document.

Isolate heavy tables/tasks or reload them separately.

Schedule archival during the low activity window.

Monitor latency + backlog queues continuously. document.