Qlik Community

Qlik Design Blog

All about product and Qlik solutions: scripting, data modeling, visual design, extensions, best practices, etc.

Employee
Employee

An Advanced Analytics Integration Timeline

steve.pngToday's Guest Blogger is, Qlik's Steven Pressland. Steven joined Qlik in 2016 as a Solution Architect. Prior to Qlik he was responsible for BI transformation and delivery across a number of Financial Services organizations. This experience and extensive product knowledge enables some of our largest enterprise customers to succeed at unlocking business insight through Qlik. Steven leads the global Advanced Analytics team within Pre-Sales and through this role brings us this update.  In this blog, Steve recaps the last 8 months of Qlik's latest enhancement to Qlik Sense and QlikView: Advanced Analytics Integration (AAI). He also shares the details on what's new with AAI in the February 2018 release including a sample on how it works.


Take it away Steve!


8 Months of Advanced Analytics Integration


Last June when Qlik Sense was released, it contained one of the biggest changes to the engine since the second generation data indexing engine engine was released with the launch of Qlik Sense in 2015. The server-side extensions API, more commonly referred to as Advanced Analytics Integration (AAI) was made generally available, providing a way for calculations to be externalized from the engine on demand as part of the user’s session.

Advanced Analytics Integration - Qlik Sense in 60

The capability and flexibility enabled developers to explore new use cases for their Qlik applications, and most importantly business users have more access to insights than ever before.

When we launched AAI, two open source connector projects were created. One for python and one for R. Supporting these two popular languages first, provided a way for users to connect to two of the most popular advanced analytics tools of choice. Since then, the R&D team has broadened Qlik's AAI projects by adding examples for C#, C++, GO and JAVA.

Over the last 8 months we have seen customers and partners deliver solutions using the out of the box sample R connector as well as create custom calculation engines to solve other calculation problems such as decryption and live data augmentation.

UK Reported Crime Demo App.jpg

UK Reported Crime Demo App

(LIVE DEMO LINK)

The next big milestone for AAI was the November 2017 release of QlikView, which brought the same AAI capabilities launched with Qlik Sense to QlikView. This gave QlikView customers a whole new set of opportunities for both old and new applications. The underlying technology for QlikView is the same as Qlik Sense, meaning you can share all the services already built with only a simple configuration step on your desktop or server. If you have not yet looked at how you can bring new a new perspective to your QlikView applications with Advanced Analytics Integration I will be presenting a session dedicated to AAI for QlikView at Qonnections 2018 (<- Customers and Partners sign up here), I hope to see some of you there.

The Qlik Community


As with many aspects of the Qlik product line, AAI’s success is made possible by the strong community who collaborate and share. The Server Side Extensions (SSE) space is widely used and moving forward. This year we are planning to give it a facelift to make it easier to navigate and find information so more customers can make use of the technology in their businesses. If you have not yet got started with AAI or are experienced in the technology, do drop by and explore the resources available.

What’s new


This blog was written to coincide with the Qlik Sense February 2018 release, which is now generally available and includes the latest feature release for AAI which now supports whole table processing in the load script. A commonly requested feature is to be able to batch process algorithms for tasks such as model training and scoring. This was possible in a limited row by row capacity at launch, however with the new version we now support sending whole tables to and from the external calculation engine as a standard part of the application load script - made possible through a simple extension to the LOAD syntax.

So, how does this work?


When writing your application script there is now an extended syntax to the load statement, like the FROM or RESIDENT statements that have their specific uses. Qlik can now use the reserved statement, EXTENSION, to specify that the source of the loaded data table will be the result of an AAI function.

The following example calls an R script to score some input data using a table already loaded into the Qlik data model and returns a result:

Load
RowID,
num(p,'##0.##') AS Predict_Score
Extension
R.ScriptEval(' … R Script Here … ',

Rentals{RowID,Year,Month,Day,RentalCount,WeekDay,Holiday,Snow });


As you can see, the returned results are accessible to be processed using all the standard Qlik script functions providing a useful addition to the script syntax.

AAI Script Process.jpg

Where to Next?


There are lots of resources available to help you succeed with Advanced Analytics Integration, here are a few of the key links to explore:

  • The Advanced Analytics Integration Qlik Community site, with installation instructions and example exercises to get started.
    Server Side Extensions (SSE)

  • This repository provides a server-side extension (SSE) protocol based on gRPC that allows you to extend the Qlik built-in expression library with functionality from external calculation engines. You can use external calculation engines in both load scripts and charts. In Qlik Sense and QlikView, you connect to these server-side extensions (SSEs) by defining analytic connections.
    https://github.com/qlik-oss/server-side-extension
  • This repository provides a server-side extension (SSE) that allows you to extend the Qlik built-in expression library with functionality from R. You can use this in both load scripts and charts. In Qlik Sense and QlikView, you connect to this SSE R-plugin by defining an analytic connection.
    https://github.com/qlik-oss/sse-r-plugin
  • The AAI Advanced Analytics Expression Builder is a Qlik Sense extension used to create advanced analytics expressions as master items and wrap these in to a pre-built visualization. These visualizations use native chart types and can be edited either as a master item or un-linked and customized.
    http://branch.qlik.com/#!/project/596f87f186a5cf7ec72e90e9


  • Qlik Sense extension which allows you to create charts to perform advanced analytics by simple drag and drop without complex scripting. This extension works on the top of the Advanced Analytics Integration (AAI) functionality with R.
    http://branch.qlik.com/#!/project/5979da222ef8975d99132f88


Hope you find this information useful.


Regards,

Steven Pressland
Senior Solution Architect

10 Comments
adamlopuch
New Contributor

>> with the new version [February 2018] we now support sending whole tables to and from the external calculation engine as a standard part of the application load script

Many thanks, Steven P., for this clarification on the February 2018 functionality update.  The quote above (emphasis mine) is especially helpful!

I saw this referenced in Mike T.'s excellent video summary for Feb 2018, but I didn't realize that this was only for data load scripting (rather than chart scripting)... this blog definitely helps make that clear!

24 Views
Employee
Employee

Hi Adam - thanks for sharing your thoughts with us. We're happy to hear that this helped. Stay well!

0 Likes
24 Views

I tried the R integration a while back and it is well groovy.  Have not yet found a valid use case for it in my work environment, but I am sure others will in other work environments and I will also in due course.

0 Likes
24 Views
velazquez
New Contributor

Hi Steven

Thanks for the update.

How about QlikView? According to https://github.com/qlik-oss/sse-r-plugin/releases "Qlikview does not support the functionality of table load using the (Load ... Extension ...) syntax and there is not yet a plan for when to support it.".

Any update on this?

Thanks.

0 Likes
24 Views
Employee
Employee

Hi Yosdeny - I have sent Steven your question - he will reply when he is back next week. Have a great weekend.

0 Likes
24 Views
philipplandis
New Contributor

Hello Michael

Are there any news regarding this topic yet?

Thank you very much.

0 Likes
24 Views
rva_heldendaten
Contributor III

hi!

It seems the qlik community space https://community.qlik.com/community/advanced-analytics-integration does no longer exist. Does anyone have a new link?

0 Likes
24 Views
bjn
New Contributor

Hi Roland,

You should be able to find related content in the Server Side Extensions space (Server Side Extensions (SSE)).

Regards,

Byron

0 Likes
24 Views
rva_heldendaten
Contributor III

Thx Bryon!

mto‌‌: maybe you can update the link in this article!

24 Views
Employee
Employee

Thanks Roland - all done!

0 Likes
24 Views