article Data Science algorithms implemented as a Python SSE in Qlik Sense Documents

Data Science algorithms implemented as a Python SSE

Nabeel_Asif — Fri, 02 Apr 2021 08:25:34 GMT

Data Science algorithms for Qlik implemented as a Python Server Side Extension (SSE).

Re: Data Science algorithms implemented as a Python SSE

maxsheva — Tue, 09 Apr 2019 21:14:56 GMT

Hi, thanks for sharing this project!

I use template with time series forecasting where Implemented Facebook Prophet.

Could you please clarify how can I correctly add Product as a column on a Trend Predictions sheet?

When I add such extra column the Prophet procedure takes all possible values in y parameter and then generate same forecast per each of product.

How is it possible to add Product in a table and retrieve correct prediction for each of them?

Many thanks!

Re: Data Science algorithms implemented as a Python SSE

Nabeel_Asif — Wed, 10 Apr 2019 01:37:35 GMT

Hi @maxsheva , the Prophet function in this SSE is only designed to take in a measure and a datetime dimension. So it won't work with additional dimensions in the table or chart object.

However, the function produces the forecast in the context of any selections in the app. So if you select a product and then a region for example, the forecast will be produced in real-time for that product and region.

A function to generate forecasts by a dimension could be created, but if you have 10 products that would mean the forecasting algorithm needs to run 10 times for the chart, which would result in a long wait time for the user.

For such cases a better way would be to pre-calculate the forecasts in the load script. This needs an enhancement which I'll add to the next release of PyTools.

Re: Data Science algorithms implemented as a Python SSE

maxsheva — Wed, 10 Apr 2019 14:17:03 GMT

Hi @Nabeel_Asif ,

Thank you for information.

I would like to try such pre-calculated solution before next release as it is quite urgent case for me. I would much appreciate if you could please share a part of code I should use in a script to implement prediction with a different dimensions and then present results in UI.

Thank you!

Re: Data Science algorithms implemented as a Python SSE

Nabeel_Asif — Thu, 11 Apr 2019 23:51:30 GMT

@maxsheva I've published the new release. Get it on GitHub and refer to the documentation here: https://github.com/nabeel-oz/qlik-py-tools/blob/master/docs/Prophet.md#precalculating-forecasts-in-the-load-script

Re: Data Science algorithms implemented as a Python SSE

maxsheva — Fri, 12 Apr 2019 12:50:18 GMT

Hi @Nabeel_Asif ,

Much appreciate for so quick response and new release!

I will try this solution and get back with a feedback.

Many thanks

Yevhenii

Re: Data Science algorithms implemented as a Python SSE

maxsheva — Fri, 12 Apr 2019 19:51:23 GMT

Hi @Nabeel_Asif ,

I downloaded sample app "Sample_App_Forecasting_Simple.qvf" and created the same structure of xlsx file with Emergency department data.

It stops with the error "Field 'ds' not found"

It is on a step when Response table generating forecast.

I suppose something wrong while querying this part of code " Extension PyTools.Prophet(temp{ds, y, args});"

It is even highlighted in editor.

I have correctly installed all necessary Python libraries and it worked fine with on a fly calculated solution.

Could you please take a look at this error?

Re: Data Science algorithms implemented as a Python SSE

maxsheva — Thu, 25 Apr 2019 08:33:22 GMT

Hi @Nabeel_Asif ,

Is there any chance you may take a look at the error mentioned in my previous post?

I would be much appreciate for your help.

Re: Data Science algorithms implemented as a Python SSE

Nabeel_Asif — Sun, 28 Apr 2019 22:53:11 GMT

@maxsheva if you look at the load statement for the temp table above you'll see I have a field called Month Start which I rename to ds. If you have a different name for your date field you'd have to update the script accordingly.

Re: Data Science algorithms implemented as a Python SSE

maxsheva — Mon, 29 Apr 2019 19:30:31 GMT

Hi @Nabeel_Asif ,

In the script ds field is renamed correctly.

Script loads data from ds if I add 'Exit Script' before loading from extension PyTools.Prophet

How can I share my app with you?

Re: Data Science algorithms implemented as a Python SSE

Nabeel_Asif — Tue, 30 Apr 2019 05:54:55 GMT

@maxsheva The error doesn't have to do with the SSE so it's due to some difference in your load script or from reverse engineering the data source.

I've just added the original data source to GitHub and updated the sample app. You can just grab those and run a reload with that.

If you're using Qlik Sense Enterprise, the data is attached to the app. If using Qlik Sense Desktop you will need to create a data connection named AttachedFiles and place the data source in the folder for that connection.

Re: Data Science algorithms implemented as a Python SSE

maxsheva — Wed, 01 May 2019 19:43:53 GMT

Hi @Nabeel_Asif ,

Thanks for suggestion. I have grab both app and data file but result is with the same error.

I suppose there could be some missed or incorrectly installed Python library or other related to extension issue.

Could you please check log of Qlik-Py-Start

Re: Data Science algorithms implemented as a Python SSE

Nabeel_Asif — Thu, 02 May 2019 02:38:01 GMT

If you're still getting the error it looks like you're not using the latest version of the SSE.

The Qlik load script fails saying that there is no field called 'ds' at the point where the SSE returns the results. There is definitely a field called 'ds' returned in release 4.0 when you pass load_script=true to the Prophet function. This was not the case with release 3.9 and earlier.

Re: Data Science algorithms implemented as a Python SSE

maxsheva — Thu, 02 May 2019 20:19:57 GMT

@Nabeel_Asif, many thanks!

It works with a new version of the SSE.

Let me adapt a script for another data and I will provide a feedback.

Much appreciated!

Re: Data Science algorithms implemented as a Python SSE

maxsheva — Mon, 13 May 2019 08:33:19 GMT

Hi @Nabeel_Asif ,

I have tried to integrate own data into solution. I am able to execute and get forecast results.

However I cannot understand how 'freq' parameter is working e.g. freq=D (W,M,MS,Y)

I see yhat forecast is the best when freq=D but it is still less than 20% from real numbers. For sure I may multiply result * 1.2 but wondering whether any option to adjust it using built-in Prophet parameters?

Re: Data Science algorithms implemented as a Python SSE

dubdev — Wed, 22 May 2019 19:57:12 GMT

Hi @Nabeel_Asif , I'm new to use analitycs with Qlik and python. Are there some functions in your extension for binary classification? Like kNN or SVM/SVC and others. Is it possible to realise binary classification with stock function of this extention? I'll be gratefull for advise.

Re: Data Science algorithms implemented as a Python SSE

Nabeel_Asif — Wed, 22 May 2019 22:51:37 GMT

@dubdev , yes this SSE has functions that support both classification and regression. Most of the algorithms from the scikit-learn library are supported.

For usage information please head over to the project's GitHub repository: https://github.com/nabeel-oz/qlik-py-tools

Re: Data Science algorithms implemented as a Python SSE

Nabeel_Asif — Wed, 22 May 2019 22:57:38 GMT

@maxsheva , the Freq parameter is based on the granularity of your data so there is only one correct option for a given dataset, for e.g. D if you have daily data.

The forecast will not align perfectly with historical values as that would be overfitting the model to a sample of data. However, there are a few ways to adjust the output explained here: https://github.com/nabeel-oz/qlik-py-tools/blob/master/docs/Prophet.md

Re: Data Science algorithms implemented as a Python SSE

evanplancaster — Thu, 11 Jul 2019 21:45:26 GMT

@Nabeel_Asiffirst off, this SSE is amazing! Very well-documented, and very well-implemented. Thank you so much for it!

I have two questions:

1) Regarding Prophet forecasting, are there any plans to incorporate additional regressors into the mix? https://facebook.github.io/prophet/docs/seasonality,_holiday_effects,_and_regressors.html#additional-regressors

2) For those who might not feel comfortable enough with Qlik's load scripting to want to go to the trouble of understanding how to feature engineer, train, cross-validate, tune hyper-parameters, and select the best model all inside Qlik, is there a way to do all that "grunt work" outside of Qlik, and then, having deployed the model in the proper location, use the sklearn.Predict function you've developed to pass a dataset in Qlik to that model? This would be a great feature to have for those of us who already have models out in production that we built using other tools and who don't want to rebuild them just so we can see the results in Qlik. (And yes, I know that we could just take our results from a pre-built model and shove them in a table on a database somewhere and pull them into Qlik, but the on-the-fly capabilities of SSEs are what we're really after here.)

Again, great job on this, and I'm so thankful to see that you are actively enhancing it and helping us all get it up and running!

Re: Data Science algorithms implemented as a Python SSE

Nabeel_Asif — Fri, 12 Jul 2019 02:23:22 GMT

Hi @evanplancaster , thanks for the compliments.

I did think about implementing the additional regressors option for Prophet, but felt restricted by a current limitation of SSEs, which is that a function cannot have a variable number of arguments. I guess I could create a new SSE function that allows for just one additional regressor, or come up with a scheme for passing multiple regressors using concatenation. I'll have a think.

On your second question, the models built using the SSE have a bit more in them than a standard sklearn model. They consist of a sklearn pipeline that needs to handle pre-processing (OHE, scaling, etc.), evaluation metrics from cross-validation, and meta-data to interpret features, their data types and how they need to be pre-processed. As I type this, I realize it should be possible to take an existing sklearn pipeline and add metadata to it so the model becomes easier to use with Qlik. So you've given me two things to think about!