Building an advanced Visualization extension using Qlik's Nebula.js & D3.js.

[Dipankar_Mazumdar]

In my last post, I discussed the robust capabilities of Qlik Sense(QS) APIs to build out-of-the-box visual metaphors and ways to integrate them within Qlik’s ecosystem. A natural choice for developers while building QS extensions throughout the years has been the Extension API primarily using vanilla JavaScript, jQuery and AngularJS.

The Extension API consists of methods and properties used to create custom visualization extensions.

Enter… Qlik Sense’s Open Source Solution — Nebula.js!

Nebula.js is a collection of product and framework agnostic JavaScript libraries and APIs that helps developers integrate visualizations and mashups on top of the Qlik Associative Engine in QS Desktop, QS Enterprise on Windows, and SaaS editions of Qlik Sense. This tutorial specifically applies to the QS SaaS edition. Nebula.js offers developers an alternative to the 'Capability APIs' that have historically been used to create mashups. The tutorial will focus on developing a new visualization based on a user scenario using Nebula.js and the 3rd-party visualization library D3.js. Our target is to understand how we can leverage Nebula.js to build a QS extension object and bring in out-of-the-box visualization capabilities within the SaaS platform. This tutorial does not emphasize the D3.js programming part, but the motivation behind the visualization is discussed.

User scenario: An organization using Qlik Sense has a new requirement to develop a visual representation to understand high-dimensional mutlivariate dataset for their organization. Their dataset consists of numerical values, and they want to compare multiple features together to analyze the relationships between them. Based on these requirements, their Data Visualization Engineer presents to them the ‘Parallel Coordinate plot’. 

So, what is a Parallel coordinate plot?

Parallel coordinate plots (PCP) have proved to be efficient in effectively visualizing high-dimensional multivariate datasets. In a parallel coordinate, each feature is represented as vertical bars and the values are plotted as a series of lines connected across each axis. Their advantage is that the vertical bars(features) can have their own scale, as each feature works off a different unit of measurement. PCP provides insights into specific hidden patterns in data like similarities, clusters, etc., and allows for more straightforward comparative analysis.

Based on the requirements, lets get started with building a QS extension using Nebula.js.

Prerequisites:

1. Node.js(version 10 or newer)
2. A terminal(for example, Git Bash on Windows or Terminal on Mac)
3. An IDE of your choice, for example, VS Code.
4. An existing web integration, or possibility to get one created in your tenant(this specifically applies to QS SaaS edition).
5. A Qlik Sense app with data.

Step1: Use nebula.js CLI to import the necessary packages. The command scaffolds a project into the /hello folder with the following structure:

• /src
• index.js - Main entry point of this visualization
• object-properties.js - Object properties stored in the app
• data.js - Data configuration
• /test - Integration tests
• package.json

Command:npx @nebula.js/cli create hello --picasso none

Step 2: Start the development server by running:

cd hello
npm run start

The command starts a local development server and opens up http://localhost:8080 in your browser. The benefit of having the dev server with Nebula.js is that it provides an interactive way to test and edit your extension without the need to iteratively deploy in QS every time a new change is made.

Step 3: Configure the data structure.

Visualizations in QS are based on a hypercube definition(qHyperCubeDef ). Therefore, any new visual object we want to bring into the QS ecosystem needs to have the data structure defined. With Nebula.js, we have the object-properties.js file that allows defining the structure of our object.

const properties = {
  showTitles: true,
  qHyperCubeDef: {
    qInitialDataFetch: [{ qWidth: 30, qHeight: 200 }],
  }
}

We also need to set a data target in the data.js file so we refer to the right hypercube definition(important to note in case you have multiple qHyperCubeDef objects).

export default {
  targets: [
    {
      path:'/qHyperCubeDef',
    }
  ],
};

Step 4: Developing the visualization extension using Nebula.js and D3.js.

QS Nebula.js specific code:

Now that we have everything ready, we start developing our extension with the custom visualization object using the index.js file from our project.

Note that Nebula.js and its primary package @nebula.js/stardust is built on the concept of custom hooks. This might sound familiar to people working with React.js. Hooks is a concept that emphasizes reusable, composable functions rather than classical object-oriented classes and inheritance. The primary hooks that we are dependent on for developing our extension object are described below:

The method that helps us in rendering our visualization object is the component()function. The component() function is executed every time something related to the object rendering changes, for example, theme, data model, data selections, component state, etc. This function can be compared to the paint() function in the Extension API.

To render our data, we first need to access the layout through the useLayout hook and then use it in combination with the useEffect hook. The hypercube’s qDataPages[0].qMatrix contains all the data(dimension and measures) used in the QS environment, and we will need to pass this data to our D3.js-based visualization.

component() {
      const element = useElement();
      const layout = useLayout();
      useEffect(() => {
        var qMatrix = layout.qHyperCube.qDataPages[0].qMatrix;
      }
                }

To see the data values and understand the structure of the qHyperCube, it is always a good idea to do a console.log(layout). A snippet shows values specific to our use case. Every time a new dimension or measure is added to our extension object, qDataPages[0].qMatrix is updated with those values.

The required dimension values for our chart are then extracted from the hypercube using the qText property from qDataPages[0].qMatrix like below.

var data = qMatrix.map(function (d) {
          return {
            PetalLength: d[0].qText,
            PetalWidth: d[1].qText,
            SepalLength: d[2].qText,
            SepalWidth: d[3].qText,
            Species: d[4].qText,
          };
        });

Our next step is to define the width and height of the visualization object, and capture its id. We will use this id to bind it to our element object from the useLayout hook as shown below:

var width = 1000;
var height = 400;
var id = "container_" + layout.qInfo.qId;
const elem_new = `<div id=${id}></div>`;
element.innerHTML = elem_new;

Finally, we make a call to the D3.js function from within the useEffect hook.

viz(data, width, height, id);

D3.js specific code:

The viz() function contains all of our D3.js code that allows us to draw a Parallel coordinate plot. First, we would need to append the SVG to the <div> that contains the id of our QS object, like below.

var svg = d3
          .select("#" + id)
          .append("svg")
          .attr("width", width + margin.left + margin.right)
          .attr("height", height + margin.top + margin.bottom)
          .append("g")
          .attr(
            "transform",
            "translate(" + margin.left + "," + margin.top + ")"
          );

We then get all of the dimensions except Species to build our x and y axes.

var dimensions = Object.keys(data[0]).filter(function (d) {
          return d != "Species";
        });

        var y = {};
        for (var i in dimensions) {
          var name_new = dimensions[i];
          y[name_new] = d3.scaleLinear().domain([0, 8]).range([height, 0]);
        }

        var x = d3.scalePoint().range([0, width]).domain(dimensions);

To draw the lines for our Parallel coordinate plot, we will need to build the path function that would take a row from our qHyperCube and return the x and y coordinates of the line.

function path(d) {
          return d3.line()(
            dimensions.map(function (p) {
              return [x(p), y[p](d[p])];
            })
          );
        }

And finally, we bind everything with our SVG like below:

svg
  .selectAll("myPath")
  .data(data)
  .enter()
  .append("path")
  .attr("class", function (d) {
    return "line " + d.Species;
  })
  .attr("d", path)
  .style("fill", "none")
  .style("stroke", function (d) {
    return color(d.Species);
  })
  .style("opacity", 0.5);

Step 5: Deploying the extension.

To build our project, we use the below command below to generates all QS readable files and puts them in a folder /hello-ext . This folder can then be compressed(.zip) and uploaded to the Extension section of SaaS console to be used within the QS environment.

npm run sense

Dashboard:

If you are just getting started with Nebula.js, https://qlik.dev is a great place to review the basics and drill-down on related functions.

This project’s source code is made available at: https://github.com/dipankarqlik/Nebula

[rarpecalibrate]

Hi,

Thanks for sharing this great post. I have a few comments that I think is worth highlighting.

• When using the below for qlik legacy applications there are a few steps that need to be considered.

npm run nebula sense​

• Firstly, the ext.js file shouldn't be a function eg. export default function ext() etc, but rather 

export default { ... }

• Secondly, create a meta.json file (similar to .qext). Then do the following:

nebula sense --ext ./src/ext.js --meta ./src/meta.json

• If not legacy Qlik Sense, its worth just using the js file from the nebula build, located in the dist folder. The .qext file can be created by the user.
• Lastly, one needs to add "supernova": true to the .qext file.

Many Thanks,

Ryan Arpe

[Dipankar_Mazumdar]

Thanks @rarpecalibrate for adding to this!

[talluru_sankar_naidu]

Hi Dipankar,

We are developing custom extension using Nebula JS and we are not able to find few attributes

- How to separate Group and Line under dimension.

- What is the attribute/property for Persistent color.

Can you please help.

Thanks