Process integration and automation using APIs

    Make sure you also checkout integration.qlik.com and checkout the new slide generator based on the Qlik Sense Engine API!

     

    Table of Contents

     

    • Overview of the APIs and what they do
    • Client APIs
    • Server side API's
      •     Engine API
      •     Qlik Sense Repository Service API
    • .NET developers
      •     Engine API  .NET SDK
      •     Web integration with Visual Studio
    • Useful starting templates and community projects from Qlik Branch   
      •     Wrappers and tools around our Engine and QRS APIs
      •     Generate apps on demand
      •     Tips to get started
      •     Use Chrome dev tools to sniff the REST call made by the QMC   
    •     Use Chrome Postman tool to test your REST calls

     

     

    Typical embedded analytics (OEM) scenario

     

    Goals

    • You already have a website and you would like to have better search and analytics capabilities and nicer looking vizualizations.
    • You want to have a seamless integrated website, the user should not need know that under the hood Qlik Sense is running. so no double logins or pop ups.
    • You don't want to duplicate settings like users, roles and other configuration, your software should be in the lead and Qlik Sense should be integrated with the least amount of manual labour.
    • We create a dashboard template per business process, each customer will get an automatically generated "instance/copy" of this dashboard with his own private data. Each customer can only view his own dashboards. Within each dashboard it should be possible to limit the rows of data (row level security), or use a hierarchy based authorization. (E.g. see all the sales of all individual cities within a country, if you are the country manager)

        only generation explanation.gif

     

     

    Integrated architecture

    The integration of two software platforms entails multiple considerations. Each platform has its own nuances. In our approach here , we deconstruct our integration approach into the following categories: dashboard, data, security, web, APIs and architecture.

    Integration flow summary

    The picture below outlines the key integration aspects.

    • A user logs in into your multi-tenant SaaS application platform. Your platform contains some Qlik Sense information in an IFrame or DIV tag.
    • The proxy intercepts the request and redirects the user to the authentication module of your SaaS platform. Your platform knows the identity and group membership of the user and It requests an access ticket for this user.
    • We redirect the user’s request back to the proxy, but now with the ticket number appended to the URL. Qlik Sense validates the request and will set a session cookie.
    • Now the authentication is complete and Qlik Sense will evaluate the security rules and will only present the authorized streams and apps to the user.

    Web integration

    Qlik allows integration options. You can choose your desired level of web integration according to your business requirements. On the client side, you can use either the Qlik client or create your own client.

    If you choose to use the Qlik Sense Client, you have the following integration options: integrate the hub, an app, a sheet or individual charts. In most cases you have the ability to control whether you see our menu, see a selection bar or whether or not you provide initial selections to make charts static.

    How would a successful integration of Qlik Sense into your application architecture look?

    Your goal is to create embedded analytics using the Qlik platform within a multi-tenant SaaS environment, where each of your customers would only have access to their own data. It’s possible for each customer to have a more finely tuned authorization structure.

    For example, it is possible to provide authorizations for a specific department or country within a dashboard. You’ll see a simple solution in our demonstration where both of our fictitious users, Peter and John, go to the same website but see different things based on their authorizations. Qlik Sense makes this possible because it uses a dynamic, attribute-based security approach in which we re-use all the pre-existing users and roles from your SaaS platform. This ensures Qlik Sense stays in sync automatically and avoids the overhead of double maintenance.

    The technology used

    You can integrate the Qlik Sense hub, apps, sheets or individual objects using an Iframe or DIV tag. All functionality is delivered via a webserver called the proxy (QPS). The visual parts of Qlik Sense are based on common web technology including web sockets, Javascript, CSS3, HTML5 and Angular. If you want to use our software to create apps, then you can use the APIs or you will need the Qlik Sense Client which is contained in the hub of Qlik Sense Enterprise. If you just want to display sheets or charts you can use both QAP as well as Qlik Sense Enterprise.

    Options to integrate

    There are multiple ways to integrate Qlik Sense into your website. You can design apps and charts upfront or generate them on the fly. Our development hub or Visual Studio plugin help generate the code for you. (These tools make a link between a Qlik Sense resource ID and an HTML IFrame or DIV).

    Data integration

    A powerful data integration capability for cleansing, transforming, and unifying multiple, disparate data sources for analysis. A robust data integration layer creates a unified view of information from many different sources, without requiring external tools or data repositories.In the demo we’ve assumed you have a database per customer instance and you also have a master database which contains the customers, the database connections, user rights and possibly customer specific custom tables and fields.

    Security integration

    Actually you don't need to do so much on the security side. You don't create users or groups in Sense. You just supply the userID and group membership in the security ticket. Based on the info in the ticket access is provided. Resources are protected using the security rules, data filtering inside an app is enforced using "section access".

    API - Process integration

    In the demo you will see the broker in action. But what are the activities that the SaaS broker automates in the demo? What are the steps or stages?

    • First it will create a stream for each customer.
    • Secondly it will copy the template app.
    • Next it replaces the script with a customer-specific one, or in most cases you just have to replace the database connection.
    • Using the new script, the app is reloaded. You can also use this mechanism to query big data sources like MongoDB. This is where we basically make a compressed copy of the data and create the associative model.
    • In the final stage we publish the app in the stream with the name of the customer.

     

    Use the SaaS  platform demo source code

    The Open source SaaS demo platform provides you with a wrapper around the Qlik Sense QMC, so you can send API calls to Qlik Sense by pressing certain buttons. Using these calls you can, for example, create streams and apps or complete any other activity that you’d be able to do in the QMC or app. Your SaaS platform most likely uses rest APIs already, so just copy the code and start automating. The tasks we have automated are what we refer to as the "generation.” These key steps that should be completed for each new customer are:

     

    Overview of the API's and what they do

     

     

    mindMapAPIIntegration.png

    API components overview.png

    Client side APIs

     

    Client APIs

    Use the client APIs to build visualization extensions and mashups.

    Backend API

    Extension API

    App Integration API

    Single Integration API

    Root API

    Capability APIs

    Use the capability APIs to bring capabilities into your visualization extensions and mashups.

    Root API

    App API

    Bookmark API

    Field API

    Global API

    Navigation API

    Selection API

    Table API

    Variable API

    Visualization API

     

     

    Engine API

    Engine API - This is out most advanced API which gives the developer direct access to the Qlik Engine. Using this API, a developer could do just about anything our product can do and more. The best examples of the use of the Engine API is Qlik Branch's qSocks project.

     

    Engine API Examples

    Creating an app and loading data

    Overview

    The data load script connects an app to a data source and loads data from the data source into the app.

    This section gives you examples on how to create apps, connect to data sources, create your data load script, and load your data in the app.

    Create apps

    Apps can be created from an empty structure, by copying an existing app or by importing an app.

    This section provides you with examples of how you can create apps, migrate apps, delete apps, and list the apps.

    Objects in an app can be replaced by other ones. An example on how to replace an app is also given.

    See: Create apps.

    Connect to data sources

    Data connections provide a way to save shortcuts to the data sources you commonly use: databases, local files or remote files.

    This section provides you with examples on how to create a connection, update a connection, and list the connections.

    See: Connect to data sources.

    View the data to load

    This section provides you with examples on how to view the data from your data sources before loading them.

    See: View the data to load

    Edit the data load script

    The script must be written using the Qlik Sense script syntax. There are a number of script functions that are available.

    This section provides you with examples on how to create a script, and add some breakpoints in the script. Examples on how to list the Qlik Sense script functions and the breakpoints are also provided.

    See: Edit the data load script.

    Load data

    You can run the script in order to load data in your app. You can get the progress of the load process and cancel the loading if necessary.

    This section provides you with examples on how to run the data load script, get the progress of the script or cancel the loading (if for example, it takes too much time).

    See: Load the data into the app.

    Other options


    Server side APIs

    Qlik Sense Repository Service API

    This documentation describes the Qlik Sense Repository Service (QRS) API.

    The QRS contains all data and configuration information for a Qlik Sense site. The data is normally added and updated using the Qlik Management Console (QMC) or a Qlik Sense client, but it is also possible to communicate directly with the QRS using its API. This enables the automation of a range of tasks, for example:

    • Start tasks from an external scheduling tool
    • Change license configurations
    • Generate apps on demand
    • Extract and import data about the system

      The

    QRS API is delivered as a REpresentational State Transfer (REST) web service that contains a number of paths (also referred to as endpoints) that are used to execute various functions. To communicate with these paths, HTTP calls that contain GET, POST, PUT, or DELETE requests are made.

     

    The content sent to and received from the paths is normally formatted as JSON structures and in some cases as files. For example, if a GET request is submitted to the /qrs/app path, the user receives a list of all apps in the system formatted as a JSON structure.

     

    .NET developers

    Engine API  .NET SDK

    To interact with the QIX engine you can use the NET SDK and start creating your first Windows application using this tool. The Qlik Sense .NET SDK can be used for building a Qlik Sense app from scratch or for visualizing an existing app in a custom client by embedding Qlik Sense objects. Get started with .NET SDK examples or on help.qlik.com

    Web integration with Visual Studio

    You can also use this visual studio 2015 plug-in to embed Sense in your website if  you are already using Visual studio.

     

    Useful starting templates and community projects from Qlik Branch

     

    Videos

    Introduction to API's

    Video from Jeff Koch

    OEM Qlik Sense SaaS

    Youtube playlist from Martijn Biesbroek and Raymond Neves

    Explains:

    • SaaS automation using the Qlik APIs
    • Security
    • Web integration
    • Architecture

     

    Wrappers and tools around our Engine and QRS API's

     

    Qsocks

    Javascript wrapper around the engine API using promises on

    Qlik Branch. With the engine you control everything inside a QVF.

    QRS

    Javascript wrapper around the QRS API on Qlik Branch. With the QRS you control everything outside of a QVF, like copy an app, stream, security rule, task.

    Postman browser extension

    Postman Interceptor helps you send requests which use browser cookies through the Postman app. It can also send headers which are normally restricted by Chrome but are critical for testing APIs.

    Fiddler

    Use fiddler to do something by hand in the  qmc and "sniff" the REST API calls with the tool. You can use these api calls and see the JSON being send. Also see this manual.

    How to Connect to the engine API

    With your own software:

    • Open a socket connection to ws(s)://server/(virtualproxy)/null for a transient session, or use QSOCKS javascript wrapper.

    Use our engine explorer tool which is part of the dev-hub

    QRS end points

    See old 1.1 rest end points  or this REST request on the QRS

    Service Charges - Interactive Web App by Axis Group

    Custom made website using qlik Sense as data engine. Vizualizations are all custom made.

    Create  table object on the fly in JSON

    The Generic Object will have a HyperCube definition and a ValueExpression definition.

    Creating a master dimension in a Qlik Sense app.

     

    Manage QMC in Sense via Python

     

    QlikSocial - on demand social media apps

     

     

    Generate apps on demand

    SaaS demo environmentFrom Martijn & Raymond:  In our demonstration site, we show and reveal how you can easily leverage the power of the Qlik Sense APIs within a multi-tenant SaaS application environment.

    Architeqt

    From Alex: Node.js code to generate apps from a template on

    Qlik  Branch

    Elastic, Generate Qlik Sense apps

    From Loic: Generate apps from a template and create custom reload scripts using express and REST calls.

    • REST calls via a query function to the QRS
    • Perform action inside a QVF, like setting the script via the Engine API via Qsocks.
    Generate apps on demand

    Generates an application on demand within Qlik Sense

    Simple on demand  "temp app" 

     

    On demand app generation for big data

    By Ian Crossland

     

    Tips to get started

    Some examples API calls you can make:

    Get list of apps

     

    API_getApps.png

    Get list of streams

     

    API_getStreams.png

    Log the user out

    You send a `HTTP DELETE` to `http://server/qps/user`

     

     

    Use Chrome dev tools to sniff the REST call made by the QMC

    If you want to automate something you can do the step of your work in the QMC and record the REST call in Chrome. Then you can copy paste this code into your software! (Don't forget to authenticate yourself against Qlik Sense. With windows auth. (happens automatically when you test this code, because you communicate via the central proxy, which uses windows auth. Out of the box) r in integration scenarios we use certificates most of the time.)

    • Start Chrome dev tools (F12), network tab, ensure the round recording button is on/off (left corner)

     

    • Do your action in the QMC, like delete some apps
    • In the trace below you can see the result. In this case you can select multiple resources at once using the '/Selection' path. In the body (Request payload) you supply the array of items to be deleted.
      • The type is the key categorization of things in the QMC. (resources like app, stream, systemrules)

        ChromeDevTools.png

    Publish an app

    Request    API_PublishAppRequest.png

     

     

    Response    API_PublishAppResonse.png

     

     

     

    Create a tag

    This use case goes in a couple of steps

    Step

    Method

    Request

    Response

    Create selection for a specific type

    Put: qrs/selection

     

    tagRequest.png

     

    tagRequest2.png

    Get synthetic by selection (allows multi-edit).

    Update all entities of a specific [type] in the selection set identified by {id} based on an existing synthetic object.

    Put: qrs/selection

     

    Tagselection.png

    Use this as the request body:

    {

            "latestModifiedDate": "2016-06-07T21:00:00.001Z",

            "properties":

            [

                    {

                            "name": "refList_Tag",

                            "value":

                            {

                                    "added": ["TAGGUIDGOESHERE"]

                            }

                            "valueIsModified":true

                    }

            ],

            "type": "Stream"

    }

     

    Create the lastModifiedDate via a function yourself:

    you need to have latest modified date

     

    function buildModDate()

      var d = new Date();

      return d.toISOString();

    }

     

     

     

    User logout

     

    logout1.png

    logout2.png

     

     

    Import and upload a QVF

    Using this mechanism you can create a Dev-test-Prod street in which you import and export apps between environments. Or you can use it to upload a QVF from one 1 location to another. For example, you can use this mechanism to upload a QVF with just data to a central Sense server, to sync the data. The fun part is that this uses the standard port 80! So firewalls are normally not an issue.

    Import app

    Upload app

    Help reference

    Help reference

     

     

    Export QVF

    Do this by manually

    exportQVF.png

     

     

    See this in your browser console:

     

    exportQVF2.png

    Use Chrome Postman tool to test your REST calls

     

    Postman: Get all Streams

     

    postmanGetStreams.png

    Postman: Copy an app

     

    PostManCopyApp.png

     

    Example script pieces

     

    Make a http call using a certificate to authenticate

    See this video for an introduction to certificates

    We are going to logout a user using the proxy API logout end point

    logout3.png

     

     

    Register a notification if things in Sense change

    In the data object, you specify the URL that Sense should call if things change.

    notificationCode.png

     

    Example SSO module code in your SaaS platform

    This example code runs in your SaaS platform's authentication module. (server side off course). So in this case if the Sense proxy bounces the user because he does not have a session cookie for the host/proxyName/hub URL. Sense will forward the user to this module (https://yourSaaSplatformUrl/SSO). It makes use of the QlikAuth modules

    qlikAuthCode.png

     

    Using the Qlik Sense Session API

    See this video or this example on Branch

     

    Export all Qlik Sense settings in JSON file

     

    You can use this method to create your own apps on the fly by modifying settings and then write it back to Qlik Sense.

              serializeApp.png

    See

    mindspank/serializeapp