Technology Partners Ecosystem Documents

[Steven_Pressland]

DataRobot is a Automated Machine Learning platform used by a number of Qlik customers. This document outlines how the integration between Qlik and DataRobot can unlock insight through the power of integration using the broad APIs available.

Overview

Businesses want to not only understand how they are performing, but also look to the future and make decisions based upon predictions. Platforms such as DataRobot provide an easy way for a broad range of business professionals to make these predictions, however in isolation there can be challenges in both ingesting data and integrating the predictions into the analytic platforms used by the broader business to inform decision making.

That is where the integration outlined here comes in; seamlessly and securely selecting and sending data from Qlik Sense to DataRobot directly from an application without any manual exports required.

Once a predictive model has been created and deployed in DataRobot, making predictions against the new data being generated each day is critical. Through a seamless request in a Qlik Sense application refresh, the new predictions are associated in your data model to drive insight.

All of this is possible using the Qlik2DataRobot client and server-side extensions, which leverage the unique integration capabilities of the patented Qlik associative engine. These extensions are available now for you to download for free, under MIT open source licence permission.

Capabilities

• Send associated data securely from Qlik to DataRobot
  • Identify data for analysis directly from your Qlik Sense application using the associative difference, before sending it securely to DataRobot directly from the Qlik engine
• Automated model scoring within a Qlik application reload
  • Seamlessly score new production data against your chosen model deployments in DataRobot, all seamlessly from the Qlik reload process without the need for external triggers or data exports
• Real time scoring direct from a Qlik dashboard
  • Simulate your model predictions directly from a Qlik dashboard using the live analytic connector expressions within your charts

Download

Server Side Extension Communication between Qlik and DataRobot is performed by a Server Side Analytic Connector, for windows servers its recommended to use the supplied installer to get started (12/09/2019 - New v2.1.0 released with improved features)	Client Extension To interactively send data from a Qlik Sense application, the client extension can be used in conjunction with the server side extension. This is the easiest way to get started and integrate between Qlik Sense and DataRobot (12/09/2019 - New v2.1.0 released with improved features)
Download	Download

Documentation

The full installation instructions are available here: Download

Source Code

Both of these projects are also made available on GitHub with the full source code. If you wish to dig a bit deeper or build your own version, then you can visit the project pages:

Server Side Extension: https://github.com/AnalyticsEarth/Qlik2DataRobot

Client Side Extension:  https://github.com/AnalyticsEarth/Qlik2DataRobot-Ext

Ready for Multi-cloud?

The Qlik2DataRobot connector is also available as a docker image, allowing for seamless deployment in a container environment. Its compatible with all variants of the Associative Engine across Qlik Sense, QlikView and Qlik Core. It really comes into its own when running as part of the Qlik Sense Enterprise on Kubernetes, scaling with demand on your Kubernetes cluster

Docker Hub

Further Information

For more information on the Joint Value Proposition between Qlik and DataRobot, click here.

[Dalton_Ruer]

Very nicely done Steven.  

[skb]

Great job Steven, 
This is an fantastic extension of the Qlik Sense capabilities and a great way for Data Robot users to Visually Interact with their models in real-time. 

[alexander_nagle]

Congrats Steve and very cool to see one more example for the power of Qlik's SSE!

[huzefawit]

Thank you for giving this awesome integration example. I was wondering if there is any api request to get the 'prediction explanations' from DataRobot too. Please let me know if there is any or in the future if it will be added. 

Thank you,
Huzefa

[abdelrahman_zoh1]

Super! Thank you very much for this Steven.

I am getting an error with the last part (Consuming published deployment).

The following error occurred:

Connector reply error: grpc::StatusCode::INVALID_ARGUMENT: 'The given key was not present in the dictionary.'

Even though I updated my key, endpoint, token, and deployment ID. On the DataRobot platform, I am getting a request everything I reload the Qlik script, thus I assume the connection is working. Do I need to update this SSE connector (predictapi) based on Appendix 1: Connector Specification? If yes, where can I updated this 

Any advise would be appreciated.

Thank you in advance,

Zozo

[huzefawit]

Hi Zozo,

I was having the same problem initially, make sure you change the 'RecID' in the given example to your ID. In this context, the ID is the primary key in your dataset. For example Customer ID, Patient ID, etc. There are two places where you need to make the change, see the code example below: 
  
GetFromDataRobot:
LOAD
[RecID],            <-------Make change here
[Prediction]
EXTENSION DataRobot.ScriptEvalStr('{
"request_type":"predictapi",
"auth_config":{
"api_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"endpoint":"xxxxxxxxxxxxxxxxxxxxxxxxxxx",
"datarobot_key":"xxxxxxxxxxxxxxxxxxxxxxxxxx"
},
"deployment_id":"xxxxxxxxxxxxxxxxxxxxxxxxxxx",
"keyfield":"RecID"      < ----- Make Change here
}',BatchData);

Hope that works!

Best,
Huzefa

[evanplancaster]

This is fantastic!

I have successfully been able to send data up to DR from Qlik, and having deployed a model in DR, I can know use the load script to get back predictions in Qlik. However, I would like to see more documentation on how to implement this in chart expressions, so that I can create what-if scenarios that send data from my selections of fields to DR and get back predictions on the fly.

Does this documentation exist yet, or does anyone else have an example of this working yet?

Thank you, @Steven_Pressland! Really great work and really great timing for our company!

[abdelrahman_zoh1]

Thank you Huzefa!

The problem was as you mentioned but also I was using scrambled data and IDs, which the API call did not found in the predictions from DR. Now all works using normal data!

Regards,

Zozo

[abdelrahman_zoh1]

Another interesting question besides Evanplancaster's question is, can we do the same in QlikView? No need for an extension, just the API/JSON call from the script to DR deployment and back.

Best,

Zozo

[abdelrahman_zoh1]

@evanplancaster 

I think you can search for executing python codes in chart expressions rather than specifically DataRobot.

This Advanced Analytics stream does what you are trying to do but unfortunately you cannot edit chart to check expression.

https://west.qlikpoc.com/public/hub/stream/79949ac0-b71d-4421-bc36-3f233d4a5586

Hope that helps a bit.

Regards,

Zozo