Skip to main content

Data Science algorithms implemented as a Python SSE

No ratings
cancel
Showing results for 
Search instead for 
Did you mean: 
Nabeel_Asif
Employee
Employee

Data Science algorithms implemented as a Python SSE

Last Update:

Apr 2, 2021 4:25:34 AM

Updated By:

Nabeel_Asif

Created date:

Jan 8, 2019 11:06:44 PM

Project page: https://github.com/nabeel-oz/qlik-py-tools

Qlik's advanced analytics integration provides a path to making modern data science algorithms more accessible to the wider business audience. This project is an attempt to show what's possible.

This repository provides a server side extension (SSE) for Qlik Sense built using Python. The intention is to provide a set of functions for data science that can be used as expressions in Qlik.

Sample Qlik Sense apps are included and explained so that the techniques shown here can be easily replicated.

The implementation includes:

  • Supervised Machine Learning : Implemented using scikit-learn, the go-to machine learning library for Python. This SSE implements the full machine learning flow from data preparation, model training and evaluation, to making predictions in Qlik. In addition, models can be interpreted using Skater.
  • Unsupervised Machine Learning : Also implemented using scikit-learn. This provides capabilities for dimensionality reduction and clustering.
  • Deep Learning : Implemented using Keras and TensorFlow. This SSE implements the full flow of setting up a neural network, training and evaluating it, and using it to make predictions. Deep Learning models can be used for sequence predictions and complex timeseries forecasting.
  • Named Entity Recognition : Implemented using spaCy, an excellent Natural Language Processing library that comes with pre-trained neural networks. This SSE allows you to use spaCy's models for Named Entity Recognition or retrain them with your data for even better results.
  • Association rules : Implemented using Efficient-Apriori. Association Rules Analysis is a data mining technique to uncover how items are associated to each other. This technique is best known for Market Basket Analysis, but can be used more generally for finding interesting associations between sets of items that occur together, for example, in a transaction, a paragraph, or a diagnosis.
  • Clustering : Implemented using HDBSCAN, a high performance algorithm that is great for exploratory data analysis.
  • Time series forecasting : Implemented using Facebook Prophet, a modern library for easily generating good quality forecasts. Now with the ability to use multiple regressors as input.
  • Seasonality and holiday analysis : Also using Facebook Prophet.
  • Linear correlations : Implemented using Pandas.

For more information refer to the project page on GitHub.

For more information on Qlik Server Side Extensions see qlik-oss.

Disclaimer: This project has been started by me in a personal capacity and is not supported by Qlik.

Comments
Nabeel_Asif
Employee
Employee

Hi @swarnendu , there's not enough information to tell for sure, but you may have a problem with your Python environment. Go through the installation instructions again and make sure you have the Python 3.6.8 64 bit version installed.

If you're still having trouble please raise a issue on the GitHub repository and include details of the errors from the Qlik-Py-Init.bat log.

https://github.com/nabeel-oz/qlik-py-tools/issues

 

0 Likes
prakashsonti
Contributor II
Contributor II

Hi @Nabeel_Asif ,

Great work in implementing and documenting SSE.  I am new to using Prophet via SSE but a long time Qlik user.  

I have been able to see good results in using PyTools.Prophet_Holidays (returning yhat) function in Charts but am unable to get the same results using this in the Qlik script (during data load).  I guess three questions:

1. What format should the HOLIDAY_NAME parameter ?  Is it similar to building ds, y and appending HOLIDAY_NAME to the dataset and then passing, ds, y and HOLIDAY_NAME into the SSE function?

2. Alternatively should the format for Holiday parameter be a concatenation of date: holiday name?

The way I have set it up, it seems to be tracking past actuals but "almost" flatlines for future dates.  I have tried this both with PyTools.Prophet and Prophet_Holidays, same results.  However this is not the case with the two functions when plotted in a graph.  In the graph, they seem to calculating future values more accurately.

I am sure I am doing something weird and am out of ideas to dive deeper.

3. Is there a way to dive deeper in what Qlik script is doing?

Any help you can provide is greatly appreciated.

0 Likes
Nabeel_Asif
Employee
Employee

Hi @prakashsonti , have you seen the documentation for using holidays on https://github.com/nabeel-oz/qlik-py-tools/blob/master/docs/Prophet.md#holidays.

To pass holidays use the Prophet_Holidays function and for calling it in the load script pass load_script=true in the arguments for the function.

The holidays field that you pass to this function should be a dimension that provides the holiday name for each date, simply passing NULL for the dates which are not holidays . For e.g.

Date                   Holiday

25/12/2018     Christmas

26/12/2018     Boxing Day

27/12/2018    

...

prakashsonti
Contributor II
Contributor II

Hi @Nabeel_Asif ,

Thanks for your quick response.  To be clear, I think you mean, simply pass, Holiday_Name by building another column along with original dataset of ds, y and pass the Holiday_Name as the 3rd parameter

ds                      y                    Holiday_Name

1/1/2018         12                 New Years

1/2/2018          15                NULL

1/3/2018           10                NULL

...

12/25/2018      15                Christmas

 

Best Regards

0 Likes
Nabeel_Asif
Employee
Employee

@prakashsonti , yes that's right.

Cheers,

Nabeel

0 Likes
Nabeel_Asif
Employee
Employee

The ability to use additional regressors with Prophet is now available with the latest release: https://github.com/nabeel-oz/qlik-py-tools/releases

0 Likes
maxsheva
Creator II
Creator II

Hi @Nabeel_Asif ,

Thanks again for this solution and latest release!

 

Earlier I have tried it in Desktop however need to check in Server as well.

The Qlik Sense Server version is 3.2 where is not possible to create an analytic connection.

Can I change settings.ini file the same way as in Desktop version?

Will that work or the only one way is to upgrade Qlik Sense server to the latest version?

Nabeel_Asif
Employee
Employee

@maxsheva  I don't remember if QSE version 3.2 supported analytics connections. I think that release itself is no longer supported as it is more than 2 years old.

I would recommend a move to the latest version but with due precautions since it will be a major jump.

wallace0834
Contributor III
Contributor III

I attempt to download the Python extension from this URL https://github.com/nabeel-oz/qlik-py-tools.  I extracted it to E:\Qlik\Sense\Tools.  I ran it as the Administrator just as the instruction indicated.  When I attempted to import the extension, I am prompted for the zip file and it's destination and the password for the zip file.

I don't know the password.  I did not get this message before I upgrade to the Qlik February release of Qlik Sense 13.62.9.  Please help.

 

When I attempt to install it, I received this error message listed below.

Activating the virtual environment...

The system cannot find the path specified.
'activate' is not recognized as an internal or external command,
operable program or batch file.

Installing required packages...

'python' is not recognized as an internal or external command,
operable program or batch file.
'pip' is not recognized as an internal or external command,
operable program or batch file.
'pip' is not recognized as an internal or external command,
operable program or batch file.
'pip' is not recognized as an internal or external command,
operable program or batch file.
'pip' is not recognized as an internal or external command,
operable program or batch file.
'pip' is not recognized as an internal or external command,
operable program or batch file.
'pip' is not recognized as an internal or external command,
operable program or batch file.
'pip' is not recognized as an internal or external command,
operable program or batch file.
'pip' is not recognized as an internal or external command,
operable program or batch file.
'pip' is not recognized as an internal or external command,
operable program or batch file.
'pip' is not recognized as an internal or external command,
operable program or batch file.
'python' is not recognized as an internal or external command,
operable program or batch file.

Creating a new firewall rule for TCP port 50055...

Ok.


Setup completed. Please check the log above for errors in red text.

Run Qlik-Py-Start.bat to start this Server Side Extension.

Press any key to continue . . .

0 Likes
Nabeel_Asif
Employee
Employee

Hi @wallace0834 , you do not need to import this extension in QMC. This is a Server-Side Extension and will run as a service that can be called from Qlik.

From your error log it looks like you don't have Python installed. Please follow the step by step instructions here to setup the pre-requisites and the extension itself: https://github.com/nabeel-oz/qlik-py-tools#installation

 

 

0 Likes
Version history
Last update:
‎2021-04-02 04:25 AM
Updated by: