Data Science algorithms implemented as a Python SSE

[Nabeel_Asif]

Project page: https://github.com/nabeel-oz/qlik-py-tools

Qlik's advanced analytics integration provides a path to making modern data science algorithms more accessible to the wider business audience. This project is an attempt to show what's possible.

This repository provides a server side extension (SSE) for Qlik Sense built using Python. The intention is to provide a set of functions for data science that can be used as expressions in Qlik.

Sample Qlik Sense apps are included and explained so that the techniques shown here can be easily replicated.

The implementation includes:

• Supervised Machine Learning : Implemented using scikit-learn, the go-to machine learning library for Python. This SSE implements the full machine learning flow from data preparation, model training and evaluation, to making predictions in Qlik. In addition, models can be interpreted using Skater.
• Unsupervised Machine Learning : Also implemented using scikit-learn. This provides capabilities for dimensionality reduction and clustering.
• Deep Learning : Implemented using Keras and TensorFlow. This SSE implements the full flow of setting up a neural network, training and evaluating it, and using it to make predictions. Deep Learning models can be used for sequence predictions and complex timeseries forecasting.
• Named Entity Recognition : Implemented using spaCy, an excellent Natural Language Processing library that comes with pre-trained neural networks. This SSE allows you to use spaCy's models for Named Entity Recognition or retrain them with your data for even better results.
• Association rules : Implemented using Efficient-Apriori. Association Rules Analysis is a data mining technique to uncover how items are associated to each other. This technique is best known for Market Basket Analysis, but can be used more generally for finding interesting associations between sets of items that occur together, for example, in a transaction, a paragraph, or a diagnosis.
• Clustering : Implemented using HDBSCAN, a high performance algorithm that is great for exploratory data analysis.
• Time series forecasting : Implemented using Facebook Prophet, a modern library for easily generating good quality forecasts. Now with the ability to use multiple regressors as input.
• Seasonality and holiday analysis : Also using Facebook Prophet.
• Linear correlations : Implemented using Pandas.

For more information refer to the project page on GitHub.

For more information on Qlik Server Side Extensions see qlik-oss.

Disclaimer: This project has been started by me in a personal capacity and is not supported by Qlik.

[stipas]

Hello @Nabeel_Asif , and thanks for publish your proyect. 

I am traing to run the (Sample -APP-Pretrained-Predict) But i see this error in qlik  when load script. 

Line of error :PyTools.Get_Features_Expression(TEMP_MODEL{Model_Name});

Connector reply error: grpc::StatusCode::UNKNOWN: 'Exception iterating responses: Model definition file not found. A YAML file with the model path, type and features needs to be placed in ../models/'

there aren't the model (HR-Attrition-v1), i am runing the docker  and the others proyect work well,  the idea is  run or call the models, thanks so much for your help. 

[Nabeel_Asif]

@stipas, the repository doesn't come with a pre-trained model by default. However, there is a complete example available at the link below which will work with the sample app: https://github.com/nabeel-oz/qlik-py-tools/blob/master/docs/Pretrained.md#complete-example

[wallace0834]

After I had install everything and I created a Analytical Connection in the QMC, how do I tell if the qlik-py-tools are working?  I could not start the Qlik-Py service.

[Nabeel_Asif]

@wallace0834, you should see the GetCapabilities call shown in step 7 here: https://github.com/nabeel-oz/qlik-py-tools#installation

The Qlik-Py-Init script will need to have been run successfully before you can run the start script. There was a recent bug introduced by a new version of pip that was causing the init script to fail. So please use the zip file from release 7.2: https://github.com/nabeel-oz/qlik-py-tools/releases

[wallace0834]

How do I install fbprophet and hdscan with Python already installed.  Please see the error message that I am receiving.  Please help.

out with exit status 1:
command: 'e:\program files\qlik\sense\tools\python tool 7.2\qlik-py-tools-7.2\qlik-py-env\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\AL003234\\AppData\\Local\\Temp\\pip-install-ifethzk4\\fbprophet\\setup.py'"'"'; __file__='"'"'C:\\Users\\AL003234\\AppData\\Local\\Temp\\pip-install-ifethzk4\\fbprophet\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\AL003234\AppData\Local\Temp\pip-wheel-f6kvivvd'
cwd: C:\Users\AL003234\AppData\Local\Temp\pip-install-ifethzk4\fbprophet\
Complete output (11 lines):
running bdist_wheel
running build
running build_py
creating build
creating build\lib
creating build\lib\fbprophet
creating build\lib\fbprophet\stan_model
INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_db8febf10f23d7fa27a62fbcfd7f17b2 NOW.
e:\program files\qlik\sense\tools\python tool 7.2\qlik-py-tools-7.2\qlik-py-env\lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\Users\AL003234\AppData\Local\Temp\tmpsc854stt\stanfit4anon_model_db8febf10f23d7fa27a62fbcfd7f17b2_5477640984157076311.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
error: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/

HDscan error listed below.

out with exit status 1:
command: 'e:\program files\qlik\sense\tools\python tool 7.2\qlik-py-tools-7.2\qlik-py-env\scripts\python.exe' 'e:\program files\qlik\sense\tools\python tool 7.2\qlik-py-tools-7.2\qlik-py-env\lib\site-packages\pip\_vendor\pep517\_in_process.py' build_wheel 'C:\Users\AL003234\AppData\Local\Temp\tmpmsr6ayif'
cwd: C:\Users\AL003234\AppData\Local\Temp\pip-install-gguiasdl\hdbscan
Complete output (29 lines):
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-3.6
creating build\lib.win-amd64-3.6\hdbscan
copying hdbscan\hdbscan_.py -> build\lib.win-amd64-3.6\hdbscan
copying hdbscan\plots.py -> build\lib.win-amd64-3.6\hdbscan
copying hdbscan\prediction.py -> build\lib.win-amd64-3.6\hdbscan
copying hdbscan\robust_single_linkage_.py -> build\lib.win-amd64-3.6\hdbscan
copying hdbscan\validity.py -> build\lib.win-amd64-3.6\hdbscan
copying hdbscan\__init__.py -> build\lib.win-amd64-3.6\hdbscan
creating build\lib.win-amd64-3.6\hdbscan\tests
copying hdbscan\tests\test_hdbscan.py -> build\lib.win-amd64-3.6\hdbscan\tests
copying hdbscan\tests\test_rsl.py -> build\lib.win-amd64-3.6\hdbscan\tests
copying hdbscan\tests\__init__.py -> build\lib.win-amd64-3.6\hdbscan\tests
running build_ext
skipping 'hdbscan\_hdbscan_tree.c' Cython extension (up-to-date)
skipping 'hdbscan\_hdbscan_linkage.c' Cython extension (up-to-date)
cythoning hdbscan/_hdbscan_boruvka.pyx to hdbscan\_hdbscan_boruvka.c
cythoning hdbscan/_hdbscan_reachability.pyx to hdbscan\_hdbscan_reachability.c
skipping 'hdbscan\_prediction_utils.c' Cython extension (up-to-date)
skipping 'hdbscan\dist_metrics.c' Cython extension (up-to-date)
building 'hdbscan._hdbscan_tree' extension
C:\Users\AL003234\AppData\Local\Temp\pip-build-env-y4vm_fqy\overlay\Lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\Users\AL003234\AppData\Local\Temp\pip-install-gguiasdl\hdbscan\hdbscan\_hdbscan_boruvka.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
C:\Users\AL003234\AppData\Local\Temp\pip-build-env-y4vm_fqy\overlay\Lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\Users\AL003234\AppData\Local\Temp\pip-install-gguiasdl\hdbscan\hdbscan\_hdbscan_reachability.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
error: Microsoft Visual C++ 14.0 is required. Get it with "Build Tools for Visual Studio": https://visualstudio.microsoft.com/downloads/
----------------------------------------
ERROR: Failed building wheel for hdbscan
Failed to build hdbscan
ERROR: Could not build wheels for hdbscan which use PEP 517 and cannot be installed directly

[Nabeel_Asif]

Hi @wallace0834 , at the end of the first error block you can see a message saying that MS Visual C++ is required. This is covered in the installation steps at the link below.  You'll need to install Build Tools with the options shown there.

https://github.com/nabeel-oz/qlik-py-tools#installation

[wallace0834]

When I click on the right clicked on the Qlik Py Start to run, I receive this message below.

2020-05-28 10:50:30,704 - INFO - Logging enabled
E0528 10:50:30.710000000 10444 src/core/ext/transport/chttp2/server/insecure/server_chttp2.cc:40] {"created":"@1590681030.710000000","description":"No address added out of total 1 resolved","file":"src/core/ext/transport/chttp2/server/chttp2_server.cc","file_line":395,"referenced_errors":[{"created":"@1590681030.710000000","description":"Failed to add port to server","file":"src/core/lib/iomgr/tcp_server_windows.cc","file_line":512,"referenced_errors":[{"created":"@1590681030.710000000","description":"OS Error","file":"src/core/lib/iomgr/tcp_server_windows.cc","file_line":203,"os_error":"Only one usage of each socket address (protocol/network address/port) is normally permitted.\r\n","syscall":"bind","wsa_error":10048}]}]}
2020-05-28 10:50:30,710 - INFO - *** Running server in insecure mode on port: 50055 ***

Does this message mean it is fully running and ready to be used?

[Nabeel_Asif]

@wallace0834, There's an error saying the port is already in use. This could be because you have the SSE running in two terminals. If not, another application is using port 50055.

If you need to change the port you can do so in the file "qlik-py-tools\qlik-py-env\core\__main__.py" by opening it with a text editor, changing the value of the _DEFAULT_PORT variable, and then saving the file. You may need to allow the new port in your firewall's inbound rules as well.

[wallace0834]

Everything see to be working correctly; however, I do not see any of the extension with Python tools in the Qlik Hub.  Anything else I might could do.

[Nabeel_Asif]

@wallace0834, this is a Server-Side Extension that provides functions which can be used from chart expressions or the load script. You won't see any new visualization extensions.

Help on using this SSE is available here: https://github.com/nabeel-oz/qlik-py-tools#usage