Qlik's advanced analytics integration provides a path to making modern data science algorithms more accessible to the wider business audience. This project is an attempt to show what's possible.
This repository provides a server side extension (SSE) for Qlik Sense built using Python. The intention is to provide a set of functions for data science that can be used as expressions in Qlik.
Sample Qlik Sense apps are included and explained so that the techniques shown here can be easily replicated.
The implementation includes:
Supervised Machine Learning : Implemented using scikit-learn, the go-to machine learning library for Python. This SSE implements the full machine learning flow from data preparation, model training and evaluation, to making predictions in Qlik. In addition, models can be interpreted using Skater.
Unsupervised Machine Learning : Also implemented using scikit-learn. This provides capabilities for dimensionality reduction and clustering.
Deep Learning : Implemented using Keras and TensorFlow. This SSE implements the full flow of setting up a neural network, training and evaluating it, and using it to make predictions. Deep Learning models can be used for sequence predictions and complex timeseries forecasting.
Named Entity Recognition : Implemented using spaCy, an excellent Natural Language Processing library that comes with pre-trained neural networks. This SSE allows you to use spaCy's models for Named Entity Recognition or retrain them with your data for even better results.
Association rules : Implemented using Efficient-Apriori. Association Rules Analysis is a data mining technique to uncover how items are associated to each other. This technique is best known for Market Basket Analysis, but can be used more generally for finding interesting associations between sets of items that occur together, for example, in a transaction, a paragraph, or a diagnosis.
Clustering : Implemented using HDBSCAN, a high performance algorithm that is great for exploratory data analysis.
Time series forecasting : Implemented using Facebook Prophet, a modern library for easily generating good quality forecasts. Now with the ability to use multiple regressors as input.
Seasonality and holiday analysis : Also using Facebook Prophet.
After I had install everything and I created a Analytical Connection in the QMC, how do I tell if the qlik-py-tools are working? I could not start the Qlik-Py service.
The Qlik-Py-Init script will need to have been run successfully before you can run the start script. There was a recent bug introduced by a new version of pip that was causing the init script to fail. So please use the zip file from release 7.2: https://github.com/nabeel-oz/qlik-py-tools/releases
How do I install fbprophet and hdscan with Python already installed. Please see the error message that I am receiving. Please help.
out with exit status 1: command: 'e:\program files\qlik\sense\tools\python tool 7.2\qlik-py-tools-7.2\qlik-py-env\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\AL003234\\AppData\\Local\\Temp\\pip-install-ifethzk4\\fbprophet\\setup.py'"'"'; __file__='"'"'C:\\Users\\AL003234\\AppData\\Local\\Temp\\pip-install-ifethzk4\\fbprophet\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\AL003234\AppData\Local\Temp\pip-wheel-f6kvivvd' cwd: C:\Users\AL003234\AppData\Local\Temp\pip-install-ifethzk4\fbprophet\ Complete output (11 lines): running bdist_wheel running build running build_py creating build creating build\lib creating build\lib\fbprophet creating build\lib\fbprophet\stan_model INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_db8febf10f23d7fa27a62fbcfd7f17b2 NOW. e:\program files\qlik\sense\tools\python tool 7.2\qlik-py-tools-7.2\qlik-py-env\lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\Users\AL003234\AppData\Local\Temp\tmpsc854stt\stanfit4anon_model_db8febf10f23d7fa27a62fbcfd7f17b2_5477640984157076311.pyx tree = Parsing.p_module(s, pxd, full_module_name) error: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/
HDscan error listed below.
out with exit status 1: command: 'e:\program files\qlik\sense\tools\python tool 7.2\qlik-py-tools-7.2\qlik-py-env\scripts\python.exe' 'e:\program files\qlik\sense\tools\python tool 7.2\qlik-py-tools-7.2\qlik-py-env\lib\site-packages\pip\_vendor\pep517\_in_process.py' build_wheel 'C:\Users\AL003234\AppData\Local\Temp\tmpmsr6ayif' cwd: C:\Users\AL003234\AppData\Local\Temp\pip-install-gguiasdl\hdbscan Complete output (29 lines): running bdist_wheel running build running build_py creating build creating build\lib.win-amd64-3.6 creating build\lib.win-amd64-3.6\hdbscan copying hdbscan\hdbscan_.py -> build\lib.win-amd64-3.6\hdbscan copying hdbscan\plots.py -> build\lib.win-amd64-3.6\hdbscan copying hdbscan\prediction.py -> build\lib.win-amd64-3.6\hdbscan copying hdbscan\robust_single_linkage_.py -> build\lib.win-amd64-3.6\hdbscan copying hdbscan\validity.py -> build\lib.win-amd64-3.6\hdbscan copying hdbscan\__init__.py -> build\lib.win-amd64-3.6\hdbscan creating build\lib.win-amd64-3.6\hdbscan\tests copying hdbscan\tests\test_hdbscan.py -> build\lib.win-amd64-3.6\hdbscan\tests copying hdbscan\tests\test_rsl.py -> build\lib.win-amd64-3.6\hdbscan\tests copying hdbscan\tests\__init__.py -> build\lib.win-amd64-3.6\hdbscan\tests running build_ext skipping 'hdbscan\_hdbscan_tree.c' Cython extension (up-to-date) skipping 'hdbscan\_hdbscan_linkage.c' Cython extension (up-to-date) cythoning hdbscan/_hdbscan_boruvka.pyx to hdbscan\_hdbscan_boruvka.c cythoning hdbscan/_hdbscan_reachability.pyx to hdbscan\_hdbscan_reachability.c skipping 'hdbscan\_prediction_utils.c' Cython extension (up-to-date) skipping 'hdbscan\dist_metrics.c' Cython extension (up-to-date) building 'hdbscan._hdbscan_tree' extension C:\Users\AL003234\AppData\Local\Temp\pip-build-env-y4vm_fqy\overlay\Lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\Users\AL003234\AppData\Local\Temp\pip-install-gguiasdl\hdbscan\hdbscan\_hdbscan_boruvka.pyx tree = Parsing.p_module(s, pxd, full_module_name) C:\Users\AL003234\AppData\Local\Temp\pip-build-env-y4vm_fqy\overlay\Lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\Users\AL003234\AppData\Local\Temp\pip-install-gguiasdl\hdbscan\hdbscan\_hdbscan_reachability.pyx tree = Parsing.p_module(s, pxd, full_module_name) error: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/ ---------------------------------------- ERROR: Failed building wheel for hdbscan Failed to build hdbscan ERROR: Could not build wheels for hdbscan which use PEP 517 and cannot be installed directly
Hi @wallace0834 , at the end of the first error block you can see a message saying that MS Visual C++ is required. This is covered in the installation steps at the link below. You'll need to install Build Tools with the options shown there.
When I click on the right clicked on the Qlik Py Start to run, I receive this message below.
2020-05-28 10:50:30,704 - INFO - Logging enabled E0528 10:50:30.710000000 10444 src/core/ext/transport/chttp2/server/insecure/server_chttp2.cc:40] {"created":"@1590681030.710000000","description":"No address added out of total 1 resolved","file":"src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1590681030.710000000","description":"Failed to add port to server","file":"src/core/lib/iomgr/tcp_server_windows.cc","file_line":512,"referenced_errors":[{"created":"@1590681030.710000000","description":"OS Error","file":"src/core/lib/iomgr/tcp_server_windows.cc","file_line":203,"os_error":"Only one usage of each socket address (protocol/network address/port) is normally permitted.\r\n","syscall":"bind","wsa_error":10048}]}]} 2020-05-28 10:50:30,710 - INFO - *** Running server in insecure mode on port: 50055 ***
Does this message mean it is fully running and ready to be used?
@wallace0834, There's an error saying the port is already in use. This could be because you have the SSE running in two terminals. If not, another application is using port 50055.
If you need to change the port you can do so in the file "qlik-py-tools\qlik-py-env\core\__main__.py" by opening it with a text editor, changing the value of the _DEFAULT_PORT variable, and then saving the file. You may need to allow the new port in your firewall's inbound rules as well.
@wallace0834, this is a Server-Side Extension that provides functions which can be used from chart expressions or the load script. You won't see any new visualization extensions.
@Nabeel_Asif Thank you Nabeel for creating such a useful Data Science SSE. I have a software engineering background, but I am brand new to Qlik, so this project has been serving me as a guiding light as I create a similar SSE for a project at my work. I do have a couple of optimization questions related to the Forecasting SSE.
1 - I see that in your sample app, you have separate measurements (each with their own Qlik expression calling the backend SSE) for the forecast, lower and upper limits. Is it possible to combine all three forecasting data points in a single SSE call response, and display them as three separate measurements on the chart? Qlik documentation mentions tensor function type that can return multi-column rows back to Qlik, but I am not clear on how I can retrieve the multi-column rows through the Qlik expression and parse, access and display individual column values as separate measurements on a chart. Can you point me to any documentation or example app/code that explains this?
2 - Is there a graceful way of notifying the Qlik user about backend SSE failure/exception (for example, due to bad data)? Something like sending a "response status" and "response detail" string back to the Qlik app, so the app developers can monitor the status of the SSE call, and react accordingly if there is an exception processing the SSE request.
