Qlik Sense Repository or Proxy unable to perform operation due to buffer space TCP Port Exhaustion

[Andre_Sostizzo]

When the Repository is unable to perform an operation but is not able to assign an internal port to handle the operation, then it will output this error in the Repository and/or Proxy > Trace > System_Repository log:

Unexpected error in ExecuteGet    An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full
Unable to read data from the transport connection: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full. The underlying provider failed on Open.

At the same time, the following warning will be found in Windows Event System Log: 

Type of entry: Warning
Source: Tcpip
Message: A request to allocate an ephemeral port number from the global TCP port space has failed due to all such ports being in use.

Type of entry: Warning
Source: Tcpip
Message: TCP/IP failed to establish an outgoing connection because the selected local endpoint was recently used to connect to the same remote endpoint. This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection. To minimize the risk of data corruption, the TCP/IP standard requires a minimum time period to elapse between successive connections from a given local endpoint to a given remote endpoint. 

Environment

• Qlik Sense Enterprise - All versions

Cause

This signifies that the Repository was unable to perform an operation due to the lack of an internal port on the server, which it uses to handle the routing of different requests.

By adjusting the MaxUserPort registry key (https://technet.microsoft.com/en-us/library/cc938196.aspx), we are allocating more internal ports available for the OS.

If further issues present, then the system may need to have an adjusted value for the TcpTimedWaitDelay registry key (https://technet.microsoft.com/en-us/library/cc938217.aspx), which may also be effective in dropping old connections which are taking up internal TCP ports.

Resolution

1. Open regedit
2. Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip\Parameters
3. Either:
   1. Edit the existing MaxUserPort entry to a higher value. The default value is 5,000 and the maximum value is 65,534.
   2. Create a DWORD Entry, edit the entry, and enter a decimal value:
      

      

   3. In addition, add the following values:

      [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
      "TcpTimedWaitDelay"=dword:0000001e
      "TcpNumConnections"=dword:00fffffe
      "TcpMaxDataRetransmissions"=dword:00000005

   4. Reboot the server for registry changes to take effect.

The settings must be applied to all nodes.

To verify the port range increase:

Open a command prompt and enter the following two commands:

c:\netsh int ipv4 show dynamicport tcp
c:\netsh int ipv6 show dynamicport tcp

To check how many ports are actively established:

c:\netstat -an | find /c "ESTABLISHED"

These examples will not show all dormant ports.

If you suspect this is the problem, you can measure the port consumption using this batch file.

The example is set to capture once a minute and runs until terminated. The text output can be loaded into Qlik Sense app for quick analysis

:loop
set /p=%date%, %time%, %computername%, <nul >>"<your file system >\ports.log"
netstat -an |find /c "ESTABLISHED" >> "<your file system >\ports.log"
timeout 60
goto loop