Can't perform a simple Engine API call to an app using python

[virilo_tejedor]

Update 2023.08.08 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Solved, Thanks to Øystein Kolsrud! (accepted solution)

Here you can find the python working code to perform a simple WSS call to Engine API

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I'm trying to perform a very simple Engine API method call using python websockets-client.

My setup:

- Qlik Sense Server Enterprise 2022

- Virtual proxy:

- Annonymous access mode: no

- Authentication method: Header authentication dynamic user directory

- Header authentication header name: header_user

- Header authentication dynamic user directory: $ud\\$id

- has secure attribute (https): checked

- SameSite attribute (https): None

- Hosts white list: my QlikSense server domain (using just one QS server)

- I created cert/key/root files exported from the QMC/Certificates 

- Certificates were converted to .pem using openssl.   "client.pem" file contains both: a private key and a certificate

- I'm using this certificate sucesfully for a QPS API call

- Engine\Settings.ini configured with:

EnableTTL=1

SessionTTL=30

(and extra CR-LF line)

I'd like to perform a simple call (already tested via Engine API explorer) to method EvaluateEx in order it to perform a simple expression evaluation: =127

I receive three JSON messages, and then it gets stucked  eternally waiting for a third message (never ends the while loop):

{'jsonrpc': '2.0', 'method': 'OnAuthenticationInformation', 'params': {'userId': 'xxx...', 'userDirectory': 'xxx...', 'logoutUri': 'https://(domain)/(proxy-prefix)/qps/user', 'serverNodeId': 'e13.....-....-....-....-.........b68', 'mustAuthenticate': False}}

result JSON 1 {'jsonrpc': '2.0', 'method': 'OnConnected', 'params': {'qSessionState': 'SESSION_CREATED'}}

result JSON 2 {'jsonrpc': '2.0', 'id': 1, 'error': {'code': -32602, 'parameter': 'Invalid handle', 'message': 'Invalid Params'}}

I'm using the next code:

QLIK_SENSE_SERVER_DOMAIN=...

USER_DIRECTORY = ...
USER_ID = ...  # having Root Admin role

# QlikSense Server: using app-id hash that I get from the app URL:
# https://.../sense/app/6749d6...981c/overview
APP_ID='6749d6..-....-....-...-........981c' 

PROXY_PREFIX = ...  # Your virtual proxy prefix

ENGINE_API_URL = f"wss://{QLIK_SENSE_SERVER_DOMAIN}/{PROXY_PREFIX}/app/{APP_ID}/"

import websocket
import json
import ssl


# QlikSense server trust this certificate, tested with another API:
cert_path = 'cert_and_key.pem' # this file contains both, the private key and the certificate

# Create a custom ssl context
ssl_context = ssl.create_default_context(purpose=ssl.Purpose.CLIENT_AUTH)
ssl_context.load_cert_chain(certfile=cert_path, keyfile=cert_path)

header_user = {
    'header_user': f'{USER_DIRECTORY}\{USER_ID}', 
    'X-Qlik-user': f'UserDirectory={USER_DIRECTORY};UserId={USER_ID}'
    }

ws = websocket.create_connection(ENGINE_API_URL, 
                                 sslopt={"cert_reqs": ssl.CERT_NONE, "ssl_context": ssl_context},
                                 header=header_user)

message={
    "handle": 1,
    "method": "EvaluateEx",
    "params": {
        "qExpression": "=127"
    },
    "id": 1,
    "outKey": -1,
}

ws.send(json.dumps(message))


result = ws.recv()
print(json.loads(result))

i=1
while result:
    result=ws.recv()
    # print(f"result {i}", result)
    y = json.loads(result) if result is not None and result!="" else "None"
    print(f"result JSON {i}", y)
    i+=1


ws.close()

What means the Invalid handle / Invalid Params with error code = -32602?

I can't see any parameter apparently being invalid in the "message" variable.

I tried both: adding two extra params "id" and "outKey" and also not adding anything extra,  yet the result is the same

Why does it get stucked?  It seems like if the server wasn't ending the web socket properly

[Øystein_Kolsrud · Accepted solution]

You need to open the app first. So before you call "EvaluateEx", send this message and wait for its response:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "OpenDoc",
  "handle": -1,
  "params": [
    "<AppId>"
  ]
}

That tells the engine to load the app into memory and create a session for you. After that you can run the method "EvaluateEx" given the handle that is returned by your OpenDoc call (which will almost always be 1).

If you'd like to learn more about the low level workings of the engine API, then this series of blog posts might be of interest to you:

https://community.qlik.com/t5/Design/Dissecting-the-Engine-API-Part-5-Multiple-Hypercube-Dimensions/...

In particular, part two of that series discusses handles: https://community.qlik.com/t5/Design/Dissecting-the-Engine-API-Part-5-Multiple-Hypercube-Dimensions/...

[virilo_tejedor]

Thanks a lot @Øystein_Kolsrud for helping me and for sharing your fabulous "Let's Dissect the Qlik Engine API" blog posts.

After reading it, I was able to have my first conversation with Engine API using websockets.

For those who it may help, this is the working code:

# -*- coding: utf-8 -*-
"""
Links (must read):
    
    
    "Qlik Sense: call Qlik Sense Engine API with Python" by Damien Villaret
    
        https://community.qlik.com/t5/Official-Support-Articles/Qlik-Sense-call-Qlik-Sense-Engine-API-with-Python/ta-p/1716089
    
    
    "Let's Dissect the Qlik Engine API" by Øystein Kolsrud:
        
        1. RPC Basics: https://community.qlik.com/t5/Qlik-Design-Blog/Let-s-Dissect-the-Qlik-Engine-API-Part-1-RPC-Basics/ba-p/1734116
        2. Handles: https://community.qlik.com/t5/Qlik-Design-Blog/Let-s-Dissect-the-Qlik-Engine-API-Part-2-Handles/ba-p/1737186
        3. Generic Objects: https://community.qlik.com/t5/Qlik-Design-Blog/Let-s-Dissect-the-Qlik-Engine-API-Part-3-Generic-Objects/ba-p/1761962
        4. Hypercubes: https://community.qlik.com/t5/Qlik-Design-Blog/Let-s-Dissect-the-Qlik-Engine-API-Part-4-Hypercubes/ba-p/1778450
        5. Multiple-Hypercube-Dimensions: https://community.qlik.com/t5/Design/Dissecting-the-Engine-API-Part-5-Multiple-Hypercube-Dimensions/ba-p/1841618

"""

import websocket # pip install websocket-client 
import ssl
import json


'''
QLIK_GOBLAL_CONTEXT ia a special handle that identifies what is called the "Global" context
which is a  context that is associated with the Qlik Sense system it self 

See: https://community.qlik.com/t5/Design/Let-s-Dissect-the-Qlik-Engine-API-Part-2-Handles/ba-p/1737186
'''
QLIK_GOBLAL_CONTEXT = -1 # entry point for base functionality like "OpenDoc"


QLIK_SENSE_SERVER_DOMAIN='myqlikserver.mycompany.com'

USER_DIRECTORY = ...
USER_ID = ...  # Root Admin user

# USER_ID needs permissions granted to access the app APP_ID
APP_ID='6749d6..-....-....-...-........981c' 

PROXY_PREFIX = ...  # Your virtual proxy prefix

ENGINE_API_URL = f"wss://{QLIK_SENSE_SERVER_DOMAIN}/{PROXY_PREFIX}/app/{APP_ID}/"


# Use pem format.  In my case, both key and cert are stored in the same file
cert_path = 'cert_and_key.pem'
key_path= cert_path

# Create a custom ssl context
ssl_context = ssl.create_default_context(purpose=ssl.Purpose.CLIENT_AUTH)
ssl_context.load_cert_chain(certfile=cert_path, keyfile=key_path)

header_user = {
    'header_user': f'{USER_DIRECTORY}\{USER_ID}', # you've to declare 'header_user' literal in the virtual-proxy setting called "header authentication header name"
    }


ws = websocket.create_connection(ENGINE_API_URL, 
                                 sslopt={"cert_reqs": ssl.CERT_NONE, "ssl_context": ssl_context},
                                 header=header_user)




msg_dummy_first = {
   "jsonrpc": "2.0",
   "id": 0,
   "method": "QTProduct",
   "handle": QLIK_GOBLAL_CONTEXT,
   "params": []
}

msg_open_doc = {
  "jsonrpc": "2.0",
  "id": 1,
  "method": "OpenDoc",
  "handle": QLIK_GOBLAL_CONTEXT,
  "params": [
    APP_ID
  ]
}

msg_evaluate_expression={
 	"handle": 1,
 	"method": "EvaluateEx",
 	"params": {
		"qExpression": "=127"
 	},
    "id": 2,
    "outKey": -1,
}

print('dummy call (this is necessary since, for some authentication methods, the "OnAuthenticationInformation" message will be sent by the server only after a valid message has been received from the client. )\n')
ws.send(json.dumps(msg_dummy_first))

result = json.loads(ws.recv())
print(result)
print()

if 'params' in result and 'mustAuthenticate' in result['params']:
    if result['params']['mustAuthenticate']==False:
        print("correctly authenticated")
    else:
        print("authentication error:")
else:
    print("WTF! no authentication info!!!")


print("\nopening the doc:", APP_ID, "\n")
ws.send(json.dumps(msg_open_doc))
open_doc_id=msg_open_doc['id']

result = json.loads(ws.recv())
print(result, "\n")


i=1
doc_handle = False
while result and not doc_handle:
    result=ws.recv()
    parsed_result = json.loads(result) if result is not None and result!="" else "None"
    print(f"result JSON {i}\n", parsed_result)
    i+=1
    
    if parsed_result is not None and parsed_result.get('id', None)==open_doc_id:
        doc_handle=parsed_result.get('result',{}).get('qReturn', {}).get('qHandle', False)

print(f"\nreceived handle {doc_handle} for app {APP_ID}\n")
msg_evaluate_expression['handle'] = doc_handle


print("simple call\n")
ws.send(json.dumps(msg_evaluate_expression))


result = ws.recv()
print(json.loads(result))
print()


ws.close()