Qlik Replicate: Outgoing stream is full. Forwarding events to target is postponed

[OritA]

By default, Qlik Replicate stores replicated data in memory. However, in some cases, when replication data exceeds memory space, data for replication will be kept in swap files on disk.

Below are the scenarios in which data for replication will be kept on disk:

1. When the target endpoint is temporarily not available
2. When replicating large transactions:
   • If a transaction is too large to be kept in memory or
   • If the transaction does not commit for a long time

In these cases, Qlik Replicate will offload the transaction changes and will store them on disk, saving them in files under the task sorter directory with the suffix .tswp. Every transaction will be saved in a separate file.

This way, Qlik Replicate can continue reading CDC changes from the source without the need to stop due to a lack of memory.

Once the transaction is committed, the transaction changes will be sent by Qlik Replicate to the target endpoint via the outgoing stream queue. This is done by the Qlik Replicate sorter and once the target endpoint acknowledges it was received the transaction and committed it successfully to the target db, the sorter will remove the transaction from memory or from disk (if was saved on disk). 

The following error may be seen in the Qlik Replicate log: 

Outgoing stream is full. Forwarding events to target is postponed.

This problem can be caused when the scenarios outlined above take place, and this means that changes made by the source and processed by Qlik Replicate are too rapid for the target. As a result, the task may also show target latency in which case a further investigation and troubleshooting is required to find out where the bottleneck is. 

How to increase the outgoing stream buffer size

Latest versions of Qlik Replicte

New Qlik Replicate versions allow for the streamBufferSize and streamBuffersNumber to be set in the task settings.

1. Go to your Task Settings
2. Go to More Options
3. Add the streamBufferSize and streamBuffersNumber
   
   

For previous versions of Qlik Replicate

1. Stop the task.
2. Export the task & and edit it
3. In the task json locate the section:  common_settings
   
    Below the common_settings add the following:
   

   "stream_buffers_number" : xx,
   "stream_buffer_size" : yy,​

   
   For example:
   

   "common_settings":{
   "stream_buffers_number" : 10,
   "stream_buffer_size" : 40,                                                                           
   "change_table_settings":     {

   (Note: In the example above, 10 buffers with a buffer size of  40MB will be allocated.
   
   The default values are:

   "stream_buffers_number" : 3
   "stream_buffer_size" : 8 (MB)

4. Save the JSON file and import the task back to your Qlik Replicate server.
5. Resume the task.

Notes

• 1This stream buffer applies to all endpoints.
• The default is three 4MB buffers and should be sufficient for most situations. If you have very large lobs in a bust system is about the only time you want to manipulate. The max recommended settings are 5 1200MB buffers.

  Remember that the outgoing stream buffer full is not indicative of an issue with the buffers, but usually an issue with the speed of the target applies.

• When the Outgoing stream is full, this is more of a target latency, and we need to investigate more before adjusting the stream buffer.

[Ashs]

Hi,

What I understand is that buffer size only works for source side latency. Is there any way to handle target side latency? I have a process in which target runs some queries before inserting data and during high volume I see latency.

Thanks, Ash 

[OritA]

Hi Ash, 

Not exactly: the stream buffer impact target enpoint and not source endpoint . i.e.:  The amount of outgoing stream buffer controls the buffers that are used by the sorter to pass information to the target. If there are not enough stream buffers to handle the task load, then in SORTER TRACE mode, you'll see the message 'Outgoing stream is full. Forwarding events to target is postponed' which means that the sorter has a information to pass to the target but it lacks buffers to pass it. Therefore, in this case it may see target latency. 

Regards,

Orit

Opens in new window

PDF Download

Word Download

Excel Download

PowerPoint Download

Document Download

[Ashs]

Thanks Orit for more insight!!