Hello everyone.
This video aims at providing some guidance in determining Qlik Sense Hub disconnections point of failures.
We will go over two of the most common messages displayed when the connection to Qlik Sense web interface is terminated, as of the September 2020 release.
The first is usually simply the result of a connection that is terminated in a controlled fashion due to the client's authentication session inactivity timeout.
The indication that the authentication session has been deleted can be found in the Proxy Audit Trace logs.
This timeout period can be increases via QMC in the Qlik Sense Virtual Proxy configuration under Session inactivity timeout.
For example increasing to 480 will allow for a full 8 hour day session. With this in place there should be no disconnections unless it occurs due to a networking related event.
Disconnections related to a network issue usually lead to the following error message. It may be observed persistently immediately after accessing the Qlik Sense Hub, as seen here.
In this case, a common cause is WebSocket protocol being disabled or not being transmitted properly to the Qlik Sense server.
Possible network devices playing a role in the issue may include but not limited to load balancers, firewalls, or reverse web proxies.
The browser's Developer Tool may indicate under Console tab a problem establishing WebSocket connections.
Here FireFox indicates a problem with the wss protocol which stands for WebSocket secured.
With Google Chrome Dev. Tool, the error is more descriptive as seen here indicating an incorrect header value which may also be missing.
Under the Network tab, clicking on the WS (for Websocket) filter, will list websocket packets where the response header may be further inspected for clues.
Here we see that the response header mentioned in the Console was in fact not processed so it is not displayed.
At this point it is still unclear if this was a server or client related failure. Administrators can also use network sniffer tools to capture network traffic between the Qlik Sense server and the client.
In this case we use Wireshark with the Capture Filter displayed running on the client's computer. The IP address is the one referenced by the Qlik Sense DNS record at the client's location.
We will refresh the web browser in in order to generate the traffic over tcp port 443.
Notice that under protocol the connection is negotiated to TLS due to secured https, so Wireshark is not able to decode Websocket.
However, we can see that an encrypted connection is terminated unexpectedly by the client with a FIN ACK tcp packet.
This can still be misleading, since we cannot see any Websocket protocol data which is what the Browser's Dev tool indicated a problem with prior.
So in order to help further troubleshoot, we can enabled plain HTTP in the Qlik Sense proxy configuration.
Note that this may require opening the configured http port in the network infrastructure.
We'll change the port in the capture filter to port 80 in this case.
The capture of the unsecured HTTP connection shows details such as identified Websocket packets as well as the nature of the terminated connection that was seen on the previous https traffic capture.
Here we see the same FIN ACK packet, and use the Follow feature to filter out only the tcp stream related with this packet.
This reveals the Web Socket handshake response header which can also be visualized when selecting the response packet in the filtered traffic.
Now we need to compare it to a working websocket connection upgrade response for when connecting directly to the Qlik Sense server bypassing all network infrastructure.
This next capture will be run on the Qlik Sense server itself and will be used as a baseline.
Note that you may need to use the loopback address 127.0.0.1 in the the URL instead of the server name, and will need to select the interface "Adapter for loopback traffic capture".
The capture filter should reflect the IP used and port. We can save the capture and bring the file to the same computer for comparing.
The comparison in this case reveals a missing header "Connection: Upgrade" when the connection fails. This finally warrants further investigation on the Web Proxy side as websockets is likely disabled or misconfigured.
Note that if there were no identifiable differences, investigating potential problems with websockets on the client side would also be a suitable next step.
Other common reasons for the connection lost error are network instability, and/or the absence of sufficient data transfer activity at the TCP session level.
Next we will look at the same error when observed some time after accessing and actively working via the Qlik Sense Hub.
We troubleshoot the error by running captures on both Qlik Sense server and Client computer.
In the capture filter on the Qlik Sense server we have the client's IP (10.20.48.208) and the default https port 443,
while on the capture run on the client's machine the Capture filter states the Qlik Sense server's IP (10.76.201.169), and port 443.
The error occurs when actively working on an App and we see the connection being terminated with a reset tcp packet which is sent by the client.
In this case we know the connection was not reset by an intermediary network device, or by the server.
In this case a brief network connectivity issue was experienced on the client's computer. Had the reset packet or even a FIN ACK connection tear down request come from the server, or another network device, this would lead to different routes in the investigation.
Next we will look at the problem when it occurs only after a period of inactivity. This inactive period is shorter than what is configured under the mentioned Virtual Proxy setting "Session inactivity timeout", so the problem is also most likely network related.
Network devices may terminate the connection if there is no data being transferred for a specified period.
Web browsers work around this issue by sending TCP keep-alive packets for the TCP connections established.
Here's an example of a Google Chrome capture where keep-alive packets are sent every 45 seconds.
If a network device such as a stateful firewall for example, has a connection timeout of less then 45 seconds, or a number of keep-alive packets are dropped somewhere else in the network before reaching the firewall, the device may close that tcp session.
This behavior can also be investigated with network captures run on both ends simultaneously.
Here we see network captures being ran simultaneously on the client machine, and on a reverse web proxy server.
Notice the keepalive messages are captured on both ends as they should in a working environment.
We know when captures are referencing the same packet on both ends by checking the combination of details such as the ephemeral source tcp port used by the tcp session, timestamps, as well as other TCP header details, and packet payload if necessary.
Note however that the keepalive messages sent by the client may not always make it to the Qlik Sense server in the presence of certain network devices or unstable network environments.
Let's say a specific reverse web proxy does not relay the keep-alive packets sent by the client's web browser, and a stateful firewall is found between the proxy and Qlik Sense servers. The connection may get reset by that firewall due to tcp session inactivity, again leading to the Lost Connection error.
One way to resolve or alleviate the problem is to enable TCP Keep Alive on the Qlik Sense server Proxy service.
It is done in the Proxy.exe.config file by setting the value for WebSocketPingInterval to the number of seconds between keep-alive packets. Then restarting the Qlik Proxy service manually.
The Qlik Proxy service will then send tcp traffic towards the client endpoint if no keep-alive packets arrive at the Qlik Sense server within that configured interval.
Here is an example where keep-alive packets from the Qlik Sense server are sent to a reverse proxy every 30 seconds when the a FireFox browser client is inactive.
The same websocket keepalive setting is commonly found in network devices in order to proactively send keep-alive messages to the client.
As an example, here is where the server to client websocket keep-alive interval is set in a Windows reverse proxy server, also known as an ARR.
Notice that the websocket protocol can also be disabled in the same location.
Lastly, it is important to mention that a network device may or may not notify the connection endpoints that the tcp session was terminated by sending a packet with the RST or FIN flag set. This may sometimes done in order to prevent a denial of service condition.
In cases where there is no notification of a session drop, the browser may continue to send a number of keep-alive messages before the "Connection Lost" error is displayed to the user.
Microsoft Edge browser for example, will send about 10 keep-alive message with no response before terminating the connection.
This means that an inactive user will only find out that the connection was lost after about 7 and a half minutes.
Also as a second browser example, lets consider Firefox keep-alive default behavior.
It will send keep-alive message every 10 seconds for just under 2 minutes (115 sec). After that it will only send them every 10 minutes.
So again in this scenario, many network appliances will close inactive tcp sessions long before 10 minutes is reached, so the connection may be lost long before the user receives the Connection Lost error.
The FireFox keep-alive defaults can be visualized by typing "about:config" in the address bar, and searching for the key word "alive".
If you’d like more information,
search for answers using the unified search tool on the Support Portal.
It searches across the support knowledge base, Qlik Community, Qlik Help site, and Qlik YouTube channels.
Take advantage of the expertise of peers, product experts, and technical support engineers by asking a question in a Qlik Product Forum on Qlik Community.
And don’t forget to subscribe to the Support Updates Blog.
Thanks for watching!
