Introduction to integrate Qlik Sense in your Software/Site (embedded analytics)

    <created together with Raymond Neves>


    See integration.qlik.com for an interactive multi tenant demo and more links to docs and videos... Make sure you checkout the new slide generator based on the Qlik Sense Engine API!


    Table of Contents

     

    Introduction to Sense integration

    Dashboard - Building blocks of an app

    Typical embedded analytics (OEM) scenario

    High level Integration overview

    Qlik Sense Topology (architecture overview)

    Governed Self Service process

    Detailed OEM use cases architecture

        Web integration

        Data integration

        Security integration

        Process integration (automatic creation of dashboards)

        Integrated architecture overview diagram

     

     

    Introduction to Sense integration

    Using Qlik Sense embedded analytics (OEM) you can show Sense charts or data inside your website/software.

     

    Start by watching the "Multi-tenant SaaS Applications with Qlik Sense (the concept)"  video from our technical OEM specialist. It will explain you what an app/dashboard is, an intro to our architecture and will introduce you to the concept on how you can distribute/deploy/provision these template apps to your customers. In other words, it will show you the power of the Qlik Sense API's for the maintainance of your SaaS platform. In the end you will see that only have to create 1 template app per business area (domain/process/section in your software), and that all other things like security or the maintenance will be done by an automatic or dynamic process (the security).


    architecture - OEM SaaS scenario.png


     

     

    Key features

    • Context sharing between host website and Qlik Sense (e.g. share a session, share variables, transfer selections made) by making use of the Authentication, Session and Mashup API
    • Fully responsive and smart vizualizations: When you include a Sense chart or complete dashboard, the charts always looks nice (e.g. it removes labels if the space is too small). You can include  single charts, sheets (a DIV tag per chart) or a complete dashboard using an IFrame.
    • merge internal and external data (and pass on selections in both directions). We use our in memory associative database combined with external data to get the best of both worlds.
    • Use our open API's and open source Qlik Branch Projects to integrate custom made extensions to include new charts or functionality all using the standards of todays web development tools: Javascript, CSS, HTML5.
    • Integrate security to provide single sign on and tranfers authorizations on the fly.
    • Integrate the management/maintenance processes by using the API's

     

     

    Dashboard - Building blocks of an app

     

    A Qlik Sense app/dashboard is a collection of reusable data items (measures, dimensions and visualizations), sheets and stories. It is a self-contained entity that includes the data to analyze in a structured data model.

    The purpose of an app is to let you and others make data discoveries and decisions using data visualizations and making selections.

    qvf.png

     

     

    Foundation

    Data load script

    You use a data load script to load data into the app. The script connects to a data source (database, Excel sheet, etc.) and retrieves and compresses the data.

    Data model

    The loaded data is structured in a data model. You edit the data load script and reload the data to build the data model you find is best suitable for your app.

    Expressions

    You can use expressions in different places in Qlik Sense. Expressions can be complex, involving functions, fields, and operators. Expressions differ from measures in that expressions have no name or descriptive data.

    Measures

    Measures are calculations and expressions for use in visualizations.

    Dimensions

    Dimensions are fields for use in visualizations.

     

    Structure and visuals

    Sheets

    Sheets include data visualizations, such as charts and tables. You create a structure in the app by grouping visualizations with different purposes on different sheets.

    Bookmarks

    Bookmarks are shortcuts to a certain set of selections on a particular sheet.

    Stories

    Stories are based on snapshots of visualizations. You present your data by creating a story that guides you to new insights by combining snapshots of data at specific times and selection states.

     

    More about apps

     

    The app makes it possible for people to create new visualizations based on any questions they might have, for example by using dimensions and measures that are defined in the app, thus further developing the app for personal use or to share with others. Whoever creates an app is automatically designated as its owner. An app can be re-used, modified and shared with others, depending on access rights. Different actions can be carried out depending on if the app is published or not.

     

    The .qvf file format is a proprietary format.

    In Qlik Sense Desktop, apps are typically stored in <user>\Documents\Qlik\Sense\Apps, as <Appname>.qvf, where <Appname> is the name of the app.

     

     

    Typical embedded analytics (OEM) scenario

     

    Goals

    • You already have a website and you would like to have better search and analytics capabilities and nicer looking vizualizations.
    • You want to have a seamless integrated website, the user should not need know that under the hood Qlik Sense is running. so no double logins or pop ups.
    • You don't want to duplicate settings like users, roles and other configuration, your software should be in the lead and Qlik Sense should be integrated with the least amount of manual labour.
    • We create a dashboard template per business process, each customer will get an automatically generated "instance/copy" of this dashboard with his own private data. Each customer can only view his own dashboards. Within each dashboard it should be possible to limit the rows of data (row level security), or use a hierarchy based authorization. (E.g. see all the sales of all individual cities within a country, if you are the country manager)

     

     

    only generation explanation.gif

    Our approach

    • Dashboards/apps: We normally create 1 dashboard per business process.
    • Content/web integration: We integrate complete dashboards, sheets or individual objects using an Iframe or Div tag. All Sense functionality "is delivered via webserver", all web based; javascript, CSS, HTML5, angular.
    • Data integration: We will create an app, and load a compressed copy all the relevant tables for this process into this app. We can schedule this reload or based on a (REST) trigger. We can do full and delta loads. We can do all the things you would normally do in a datawarehouse, since we have a data integration layer (a script) inside the dashboard. So, we can create multiple layers of data to ensure you can re-use data that has already been loaded from your source system. (Qlik Sense creates a compressed copy of your data in order to create a associative datamodel in which each datapoint is connected to every other. So if you select a country, you will immediatly see the data for this country in all charts and even when you go to a next sheet. No need to program anything.)
    • Security: We re-use your existing way to authenticate the users. We will insert a little piece of code inside your existing "security module". In this module you already know who the user is (authentication) and what his rights are (authorization). The only thing we need to do is create a trust relationship between your server and the new Qlik Sense server (via a SSL certificate exchange). And send this info about the user (userId, group membership) in a JSON object to Sense (we call this ticketing). Based on the userId and group membership streams (a way to group apps, like HR/Finance), apps, sheets (all called "resources" in our rule based security model) are shown or not. Let's say we start simple and each customer can only view their own dashboards. Since we gave each customer its own stream we can grant access if the name of the stream (your company name) equals the the name of the company we received from your website in the ticket. We can all control this by adding 1 row of security configuraton: User.group = Stream.name.
    • Process integration: Using our API's, which we also use ourselves, you can automate everything. So your software can control and configure Sense. You can even create a complete dashboards or indivual charts on the fly

     

    full integration flow.gif

    For one OEM customer I used 1 security rule to secure 400K apps. In that case we used a customerId (provided by the security ticket from the OEM host) and mapped that to the stream name. Note that using the Rule Based Security model prevents that you have to maintain a lot of groups and roles in your BI environment, which are basically copies of your source system causing a lot of rework.

     

    High level Integration overview

    This chaper discusses the high level concepts for

    • Data -
    • Web -
    • Security -
    • Process integration

     

     

    integrationTopics.png

     

    Data integration

    Data integration creates 1 associative model

    A powerful data integration capability for cleansing, transforming, and unifying multiple, disparate data sources for analysis.  A robust data integration layer creates a unified view of information from many different sources, without requiring external tools or data repositories. This includes databases, twitter, facebook or any other REST/JOSN API, HTTP get web content found in tables, and Big Data sources. So, you will get 1 model of the truth in which all sources are combined, no data left behind and it is not needed to create 1 view  as input for 1 dashboard and no separate cubes or queries anymore to maintain.

     

    ·      Qlik’s Associative Model – Part I

    ·      Qlik’s Associative Model – Part II

    <<Qlik’s Associative Model.pdf>>

     

    Direct Discovery – Provides a live "SQL" connection to ‘Big Data’ sources without requiring the data to reside in memory, seamlessly combining big data with in memory data and maintaining the associative experience across sources.

     

    http://www.qlik.com/~/media/Images/Products/product_unlock_data_v3.ashx?h=259&w=458

     

    Visual data manager

    You can add data to your app from a multitude of data sources using the Data manager. This tool means you don't need to learn a scripting language to get your data into your apps. As you load, you will also get assistance with creating data relationships based on data profiling. Data manager also lets you edit your data selections and add or remove fields easily. The visual data load editor speeds the ability to get common tasks completed easily with this click-and-go approach that requires no data scripting, while power users will still have the ability to leverage script for more complex use cases.

     

    DataMarket

    You can now add data from external sources with Qlik DataMarket. Qlik DataMarket offers an extensive collection of up-to-date and ready-to-use data from external sources accessible directly within Qlik Sense. Qlik DataMarket provides current and historical weather and demographic data, currency exchange rates, as well as economic and societal data.

     

    Qlik as raw data engine (so not as source but as data provider for other web or fat client applications)

    If you just want to receive any (reactive) dataset (field, table) based on a query you can do this using the REST, javascript or .NET SDK API's. Your client/server needs to support/import websockets. Because we use websockets your and our application will be kept in sync automatically. If you make a selection the "dataset table" will be updated, or if you select something in your client the selection will be sent to Qlik also all by making use of automatic call back functions. (see process integration for more info). Also note that all our API's use REST calls in the end, they are just wrappers around REST based API.

    Web integration

    Web Integration 

    1. Share user session/context: Passing selections between Sense and OEM.
    2. Iframe integration, show a complete dashboard inside your website
    3. Single object mode, you can easily integrate a single chart from an app in your site using this guide: http://localhost:4848/single  (ensure Sense desktop runs)
    4. Workbench - DIV tag integration, create mashups of Qlik and other content inside your side. using this guide: http://localhost:4848/workbench (ensure Sense desktop runs)
    5. Extensions, create new charts based on open standards such as HTML5, JavaScript and CSS, the new objects behave the same as "out of the box charts" with full two way interaction with Qlik datamodel and selections made. They create new ways to visualize data and enhance user input and interaction. They can integrate with other systems (such as write back selected items to source database) and visual libraries such as D3, google maps, youtube and other webservices, or you can add new selection objects (sliders, input boxes).

    Examples

    Axis group

    Qlik mashups demo page

    United nations (UN)

    Mashup using tabs

    AFAS jaarverslag

    Security integration

    Security integration

    1. Authentication: How do we provide Single Sign On using windows Active Directory,  SAML, tickets or header authentication using the Proxy API?
    2. Authorization: What are you allowed to see/do on a specific resource?

     

    Authorization check points

    Authorization steps:

    Example resources

     

    authorizationCheckPoints.png

    Stream and resource access control (QMC - Security rules)

    1. Which streams are you allow to see?
    2. Which dashboards are you allow to see?
    3. Which sheets or buttons (Edit script, edit dashboard, story telling) are you allow to see?
    4. This type of authorizaton is managed by security rules in the management console. The resource access control system in Qlik Sense is based on attributes. This means that the access is based on rules that refer to attributes connected to resources and users in Qlik Sense.

     

    The resource access control system determines the access based on the following parameters:

    • User name and user properties: The user name and user properties are supplied by the Qlik Sense Proxy Service (QPS) that authenticated the user.
    • Action: The method that the user is trying to perform on a resource (for example, create, read, or print).
    • Resource: The entity that the user is trying to perform an action on (for example, app, sheet, or object).
    • Environment: The environment is supplied by the QPS and describes, for example, time, location, protection, and the type of Qlik Sense web client used.


    Data reduction (App - Script - Section Access):

    • Answers the question: What data are you allow to see given your userId or group? (Both rows and columns/fields can be hidden)
    • The data reduction functionality is based on the concept of section access, which is a way to dynamically change which data a user can view. This makes it possible to build apps that can be consumed by many users, but with different data sets that are dynamically created based on user information.

    sectionAccess.png

    MAINTAIN SHEET

    MAINTAIN STORY

    MAINTAIN SCRIPT

    MAINTAIN CONNECTIONS

    MAINTAIN SECURITY - USERS

    MAINTAIN SECURITY - SECURITY RULES

    MAINTAIN TASKS

    MAINTAIN PROXY

    MAINTAIN STREAMS

    MAINTAIN LICENSE

     

     

     

    A typical OEM Scenario is very simple and effective:

    OEMscenarioSecurity.png

    Process integration/automation

    Process integration

    Use Our REST API and .NET SDK based API to automate all processes between host application and Sense. You want to make sure that you don't have 2 separate applications to maintain when creating streams, apps, reload tasks or adding users etc.

    .NET SDK

    The Qlik Sense .NET SDK can be used for building a Qlik Sense app from scratch or for visualizing an existing app in a custom client by embedding Qlik Sense objects.

    Getting started using the .NET SDK

    Qlik Analytics plugin for Visual Studio

    The Visual Studio plugin is a development toolbox created to simplify the process of integrating Qlik Sense functionality into other environments. Qlik Analytics for Visual Studio provides quick access to relevant information from your Qlik Sense environment, which for instance makes it easy to integrate Qlik Sense visualizations into an existing web site.

    When you have installed the tool, you will be able to:

    • Connect to Qlik Sense servers.
    • See all your apps, with related sheets and objects.
    • See a preview of the object.
    • See the properties of the server and charts.
    • Drag-and-drop objects (in different modes) directly into your code.

    Engine API

    The Qlik Engine API is as Websocket protocol that uses JSON to pass information between the Qlik Engine (also sometimes called QIX engine) and clients. This API is the most important API used by the Qlik Sense client, but works completely independent of any platform and programming language. The only requirement is that the platform/programming language support Websocket communication. More on the Qlik Engine API can be found in the Qlik Sense Developer Help > Qlik Engine API. Or see the engine API explorer for more info if your Sense desktop runs (manual).

     

    The best examples of the use of the Engine API is Qlik Branch's qSocks project or on the qlik developer help examples.

     

    QRS API

    The Qlik Sense Repository Service (QRS) contains all data and configuration information for a Qlik Sense site. The data is normally added and updated using the Qlik Management Console (QMC) or a Qlik Sense client, but it is also possible to communicate directly with the QRS using its API. This enables the automation of a range of tasks, for example:

    • Start tasks from an external scheduling tool
    • Create streams/apps
    • Export, change and import all Qlik Sense Settings so you can easily spin up a new Qlik Sense server.

    Example use cases (QRS and engine API combined)

    • Embedd our hub, app, sheet, chart or for example our search box in your site/client, so customers have 1 search for all the fields in your app.
    • Generate a chart or dashboard on the fly
    • Import a template dashboard
    • Copy this  template for each of  your customers
    • Load the template with customer data  from their own database (if you have a database per customer)
    • Create an app, an object, a hypercube, a  table with JSON data  to be used by my custom HTML control
    • Create dimensions and measures, or translate the text
    • Get a list of dimensions & measures
    • Make selections
    • Set properties of the app, a field, an object, etc.

    Sense Server Architecture

    Architecture

     

    Or watch this or this video.

     

    Scalability (CPU/RAM usage) information:

    <<DS-Qlik-Sense-Scalability.pdf>>

     

    System requirements for Qlik Sense

    Supported browsers in Qlik Sense

    Server whitelist

    Manuals and Qlik sense deployment guides

    Overview of ports

    Qlik Sense introduction videos

     

    Qlik Sense Topology (architecture overview)

     

    See this link for introduction to Qlik Sense. Or this presentation for an high level overview of Sense integration topics.

     

    See the attachment below for a complete overview in 1 page of sense apps, server environment, governance and security.

    <<QlikSense TopologiesV0_18.pdf>>

     

     

    Governed Self Service process

     

    In the past, IT would create a reporting environment and the business users would just be able to read what IT delivered. But by definition, people learn and  always want something else when they see the result. Qlik Sense has been developed with this is mind. Sense enables you to create a starting template by filling "a library of master items" with 80% of the functionality and let the business create the remaining 20% in a controlled way.

     

    Use of centrally governed libraries for data, dimensions and measures

    A library is a shared, centralized repository for data and visualization definition within a Qlik Sense app. Libraries provide a way for those responsible for managing the data to make it available to others in a way that makes it easier to use. But libraries include more than just data. Measures can be added to the library to ensure that everyone has a common definition for calculated results. And, even visualizations can be added to the library for easy re-use by others.

     

    The OEM/IT/Reporting team (the left side of the picture below) will

    • Combine all the datasets together into 1 associative Qlik Sense datamodel. This ensures all data is available for the end user.
    • Create a set of
      • dimensions,
      • measures/KPI's
      • A set of charts for the user

      The customer (the right gray block) can easily continue with the base template, and create his own design in a controlled way. This by choosing the appropriate user type (=access rights) for your customer. Also note that you can choose to ask more money from your customers for the more advanced use cases.

     

    SelfServiceBI.png

    Example screenshot  below which shows  you

    1. The base sheet: sheets that you as software company create, and cannot be changed by the customer (1 version of the truth)
    2. My sheets: sheets  that only I can see, and play around with
    3. Community: sheets that after an approval step (the publishing of the sheet) are visible for the rest of the team who has access to the app.

    communitySheets.png

     

     

     

    Detailed OEM use cases architecture

     

    In the picture below you will find all integration aspects together in 1 overview for a typical OEM architecture, in which we have

     

    Web integration

    • A webserver which serves the existing website (the web portal)
    • Qlik Sense which also acts as a webserver (it delivers JSON content via the web or API clients)

     

    Data integration

    • A database per customer
    • A security database/table in which user rights are stored
    • A  configuration database/table in which is defined
      • Which customers do we have?
        • Per customer
          • the database connection
          • The tables and fields relevant

    Security integration

    Actually we don't need to do so much on the security side. You don't create users or groups in Sense. You just supply the userID and group membership in the security ticket. Based on the info in the ticket access is provided.

     

    A typical OEM authorization concept is:

    • Each customer can only see its own data. Each customer can have 1 or more users. These users can be a member of 1 or more groups/roles which determine what they can see or do in the system. E.g. a manager can see the "whole world" and a regional manager only his "own region".
    • The security concept should be as simple as possible. In the first phase we will authorize on streams and therefore we will give each customer its own stream. (A stream is the key grouping structure of dashboards)
    • Therefore we create
      • 1 security rule to configure "resource access control" which compares the group of the customer to the name of the stream. If they match, the user gets access to the steam and its dashboards:
        • @User.environment.group= @resource.name
      • Section access per dashboard to limit what a user/role can see (data filter/reduction). We do this based on the attribute: 'group' you provided in the ticket. In this way you don't need to sync user-group relationships to Sense.

     

    Process integration (automatic creation of dashboards)

    • We would like to automate everything, so if a user needs to get access to a dashboard we would like this process to be automated.
    • We only would like to maintain the least amount of dashboards, there we create a dashboard template per business process (HR, Finance, Sales etc)
    • We create a copy for each customer
    • Publish this app the stream of the customer
    • Replace the script and reload the app.
    • (no need to do anything with security, because we already have 1 security rule for all our apps: only show the apps for which the stream name is equal to the customer name supplied in the security ticket)

     

    If the user has access to the dashboard, he then can use our Self Service approach to copy sheets, edit the things he would like. Since he can't modify "the base template sheets". So it is safe for you to update this sheets everyday if needed.

     

    Integrated architecture overview diagram

     

     

     

    integratedArchitecture.png