By default, Qlik Replicate keeps data for replication in the memory. However, in some cases, when replication data exceeds memory space, data for replication will be kept in swap files on disk. Below are the scenarios in which data for replication will be kept on disk:
- When target endpoint is temporarily not available
When replicating very big transactions:
- If a transaction is too large to be kept in memory or
- If the transaction does not commit for a long time
In these cases, Qlik Replicate will offload the transaction changes and will store them on disk, saving them in files under the task sorter directory with the suffix .tswp. Every transaction will be saved in a separate file.
This way, Qlik Replicate can continue reading CDC changes from the source without the need to stop due to lack of memory.
Once the transaction is committed, the transaction changes will be sent by Qlik Replicate to the target endpoint via the outgoing stream queue. This is done by the Qlik Replicate sorter and once the target endpoint acknowledges it was received the transaction and committed it successfully to the target db, the sorter will remove the transaction from memory or from disk (if was saved on disk).
The following error may be seen in the Replicate log:
"Outgoing stream is full. Forwarding events to target is postponed. "
This problem can be caused when scenario 2 or 3 mentioned above took place, and this means that changes made by the source and processed by Qlik Replicate are too rapid for the target. As a result, the task may also show target latency in which case a further investigation and troubleshooting is required to find out where is the bottleneck.
To increase the outgoing stream buffer size (in case the problem is caused by reason 2 mentioned above) you should perform the following steps:
- Stop the task.
- Export the task & and edit it
- In the task json locate the section: common_settings
Under this section add the following:
"stream_buffers_number" : xx,
"stream_buffer_size" : yy,
For example:
"common_settings":{
"stream_buffers_number" : 10,
"stream_buffer_size" : 40,
"change_table_settings": {
(Note: in the example above 10 buffers with a buffer size of 40MB will be allocated)
The default values are:
"stream_buffers_number" : 3
"stream_buffer_size" : 8 (MB)
- Save the json file and import the task back to your Replicate server.
- Resume the task
NOTES:
1. this stream buffer apply to all endpoints.
2. The default is 3 4MB buffers and should be sufficient for most situations. If you have very large lobs in a bust system is about the only time you want to manipulate. The max recommended settings are 5 1200MB buffers.
**Remember that the out going stream buffer full is not indicative of an issue with the buffers but usually an issue with the speed of the target applies.**
3. When Outgoing stream is full, this is more of target latency, and need to investigate more before adjusting the stream buffer.
