Qlik Replicate and how to Replicate very large table having greater than 600 million records

[shashi_holla]

This article is based on the suggestions provided to replicate a very large table that had more than 600 million records in Oracle. The table full load took around 15 hours but was able to complete successfully and did not fail with the original error ORA-01555: snapshot too old: rollback segment number XX with name ""XX"" too small.

 

Environment

• Qlik Replicate 

Resolution

Each business environment is different and there is no definitive method that works for all. Here is one such example which worked in this case and I'm pretty sure this would work for other similar cases as well. Please tune the Full Load settings to the following values.

• Transaction Consistency Timeout = 1000
• Commit Rate during Full load = 50000
• Enable Internal parameter bulkArraySize  on the target endpoint and set it to 5000

Cause 

This was more of an environment-specific and we had to work with the resources we had at that time. Since the table had a lot of varchar columns the internal memory consumption was high. To avoid this we had to tune the Full Load parameters after a few iterations were able to arrive at the right settings. 

[bryce_leinan]

@Sonja_Bauernfeind - we experienced a 20% performance increase on one of our tasks doing the  bulkArraySize  change. We also set our commit rate to 250000.