Connect to Engine API from Python script

[Solvica]

Hi there 🙂

We are trying to connect to the Engine API of our Qlik Sense Enterprise (on Windows) installation from a Python script, using a websocket. Qlik is running in a virtual machine in Azure.

After following these instructions on how to call Qlik Sense Engine API with Python (thanks @Damien_V) I get this response:

{"jsonrpc":"2.0","method":"OnAuthenticationInformation","params":{"loginUri":"https://server.domain.com/internal_forms_authentication/?targetId={someID}","mustAuthenticate":true}}

I understand from other posts that this means that there is a problem at the authentication stage. I have checked our certificates many times and everything seems to be normal there.

In the other hand, we have managed to connect to the QRS from a Python script with "requests".  But the authentication through certificates failed again. We could only make it through by using Windows authentication. Which is an option not available to connect to the Engine API using a websocket, if I am not mistaken.

So, it seems that there is indeed a problem with the certificates, although Qlik is working flawlessly (if this helps, I am running the scripts from the same server where Qlik is installed, which is already whitelisted in Qlik).

So the question is: where should I start troubleshooting this issue? The virtual proxies? Security rules? Any kind of additional validation of the certificates I must do in Qlik?

Any orientation will be hugely appreciated.

Best,

Ginés - Solvica

[Øystein_Kolsrud · Accepted solution]

If you are connecting using certificates, then you need to add the port of the engine to your URL. Have you done that? I'm asking because the message seems to indicate that you are connecting via the proxy.

The standard port for the engine is 4747, so if you use the URL "https://my.server.com", then simply try this URL instead: "https://my.server.com:4747".

Beware that, depending on your settings, firewalls might be blocking that port.

[Solvica]

Hi @Øystein_Kolsrud ,

Thanks for your prompt reaction and apologies for my late one.

I have actually tried both, proxy and certificate. Both unsuccessfully.

Your comment regarding the firewalls was relevant indeed and I had not open some of them. This is now solved but I am still facing rejections problems to the certificates.

After such changes and revisiting this post (hoping that I will get a different result now), I am using the following code (this is all of it, no previous code before the chunk I am copying here):

import websocket, json, ssl
ssl.match_hostname = lambda cert, hostname: True
websocket.enableTrace(True)
privateKeyPath = 'MyRoute\\QlikCertificates\\'
url = 'wss://MyServer:4747/app/'
certs = ({"ca_certs": privateKeyPath + "root.pem", 
          "certfile": privateKeyPath + "client.pem", 
          "keyfile": privateKeyPath + "client_key.pem", 
          "cert_reqs": ssl.CERT_REQUIRED, 
          "server_side": False})

ws = websocket.create_connection(url, 
                                 sslopt=certs, 
                                 header={"X-Qlik-User": "UserDirectory=INTERNAL; UserId=sa_engine"})

Unfortunately, I am obtaining the following reply:

SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'MyServer'. (_ssl.c:1129)

I have just downloaded again the certificates, so it seems to me very unlikely that the certificate is not valid for our server.... Is there any other instances I could check to make sure that this request goes through?

Would I need to do a previous Proxy Authentication in the QPS? I have tried that but it did not go well either.

Thanks in advance for your guidance!

Ginés

[Damien_V · Accepted solution]

Hello @Solvica 

Does that work with the below certificate options?

certs = ({"ca_certs": privateKeyPath + "root.pem", 
          "certfile": privateKeyPath + "client.pem", 
          "keyfile": privateKeyPath + "client_key.pem", 
          "cert_reqs": ssl.CERT_NONE, 
          "check_hostname": False,
          "server_side": False})

[Solvica]

Yes!! It does now! Amazing, thanks a million @Damien_V and @Øystein_Kolsrud, both answers were super relevant to finally hack this out.Have a nice day 🙂

Ginés - Solvica

[Dan01]

Hello

Although the above approach has worked for some, in my case I am still having problems. I tried to use a similar script as suggested by  @Solvica with the same certificate settings but I still get the following errors:

- trying to access the Engine API by using the url "https://my.server.com:4747/app/" my connection is reset with a “ConnectionResetError: [Errno 104] Connection reset by peer”.

- when the url is set to "https://my.server.com/app/" I get the response below.

The script is running from a Jupyter notebook opened in a web-client that has access to the target Qlik server. Trying to access the QRS API from the same notebook with NTLM authentication works fine.

As stated above some possible causes might be:

- Network issues: firewall rules, proxy settings, dropped packets, port closed

- Incorrect SSL/TLS certificates or misconfigured security settings.

We have checked:

- Network issues: firewall opened, 4747 port opened

- Certificates: both client and server ones have been generated on the same day with no expiration date.

From what I understood NTLM or Basic Authentication is not possible for the Engine API

I am certain I am missing something and would appreciate your guidance.

Thanks a lor in advance

SERVER RESPONSE:

--- request header ---

GET /app/ HTTP/1.1

Upgrade: websocket

Host: my.server.com

Origin: https://my.server.com

Sec-WebSocket-Key: xxxxxxxxxxxxxx==

Sec-WebSocket-Version: 13

Connection: Upgrade

X-Qlik-User: UserDirectory=INTERNAL; UserId=sa_engine

-----------------------

--- response header ---

HTTP/1.1 101 Web Socket Protocol Handshake

Upgrade: WebSocket

Connection: Upgrade

Sec-WebSocket-Accept: xxxxxxxxxxxxxx=

-----------------------

++Rcv raw: […]{"jsonrpc":"2.0","method":"OnAuthenticationInformation","params":{"loginUri":"https://login.microsoftonline.com/[…]","mustAuthenticate":true}}'

++Rcv decoded: fin=1 opcode=1 data=b'{"jsonrpc":"2.0","method":"OnAuthenticationInformation","params":{"loginUri":"https://login.microsoftonline.com/[…]","mustAuthenticate":true}}'

++Sent raw: [...]'

++Sent decoded: [...]'

close status: 1001

SCRIPT:

import websocket

import ssl

privateKeyPath = '/CertsPath/'

url = 'wss://my.server.com/app/'

certs = {

"ca_certs": privateKeyPath + "root.pem",

"certfile": privateKeyPath + "server.pem",

"keyfile": privateKeyPath + "server_key.pem",

"cert_reqs": ssl.CERT_NONE,

"check_hostname": False,

"server_side": False

}

try:

ws = websocket.create_connection(url, sslopt=certs, header={"X-Qlik-User": "UserDirectory=INTERNAL; UserId=sa_engine"})

ws.recv()

except ConnectionResetError as e:

print(f"ConnectionResetError: {e}")

ws.close()