In any subscription-based business model, one of the primary goals is to retain its customers. Also, with the increase in market competition, it is crucial to identify ‘unhappy’ customers at an early stage so as to provide additional benefits and retain them. Churn prediction refers to determining which consumers are most likely to abandon a service or terminate their membership. To be able to predict churn gives businesses the necessary edge since getting new customers is sometimes more expensive than keeping old ones.Ref: https://www.opexengine.com/keep-customer-churn-rate-under-control-or-labor-like-sisyphus/The application of Machine learning techniques to understand & predict customer churn isn’t something new. Several ML algorithms have been used in the field of telecom, banking, insurance, etc. to detect early churn signals. However, just relying on an ML algorithm’s output to understand whether a customer will churn or not isn’t really an optimal approach anymore. To have a thorough understanding of the churn analysis process, the need to amalgamate historical data(what happened in the past?) with the predicted outcome(what will happen in the future?) is paramount.This is where Qlik Sense’s visual analytics platform serves as an effective solution. Now, using advanced analytics connectors within Qlik Sense SaaS, users can build a Machine Learning model in an AutoML platform, consume the predictions in Qlik Sense and use them along with the Qlik’s data model to take advantage of things such as:associative benefitsfaster calculationsout-of-the-box visualizationsperform what-if analysisIf you want to understand the entire workflow to integrate 3rd-party ML endpoints within Qlik Sense, I highly recommend going through this first part that I wrote a few days back. The generic workflow is depicted below.In this specific tutorial, we are going to analyze customer churn behavior for a telecom company by building an end-to-end Qlik Sense app and leveraging both historical as well as predicted data. For building the Machine Learning model and hosting the endpoint, we will use the Amazon SageMaker platform. We will keep our focus on building the Qlik Sense app and not on developing the ML model.Pre-requisite:To be able to use Analytics Endpoints the"Enable machine learning endpoints" parameter should be enabled in the Management Console.Step 1: Load sourcedataFirst, let us load the source data into Qlik Sense using the Data load editor.We will analyze our dataset in detail when we build our ‘descriptive’ dashboard but for now we know that we have 15 attributes and 3333 records that describe the profile of each customer of the telecom operator.The last attribute, Churn, is known as the target attribute–the attribute that we want our ML model to predict to know if a customer will churn or not.Step 2: Train a churn-prediction model & deploy the inference APIOur next step is to build the churn prediction model. The target is to classify each customer into either of the two categories — churn or not churn. Therefore, this is a binary classification problem. We will be leveraging SageMaker Autopilot that allows us to automatically build, train, and tune the best machine learning model based on our data without having to write much code.Credit: Amazon Web ServicesYouTubeIf you are just getting started with SageMaker Autopilot, here is a great video from AWS to help you understand the basics. I use the describe_auto_ml_job API to look up the best algorithm selected by the SageMaker Autopilot job.Finally, we will create our model based on the best candidate (automl-churn-28–18–16–29r2UGiyXI-011–5e61e1c5) & deploy it to a hosted endpoint. When the endpoint is ready, the endpoint status will change to ‘InService’ like below.To make it easier for you to learn about how I trained & deployed my model in SageMaker, I will attach my Python notebook along with this blog.Step 3: Send data from QS to the ML endpoint for predictionNow that we have the model endpoint ready for inference, we will send all the fields required by the model from QS to predict if a customer would churn or not. Please note that we will only send 14 attributes and exclude the last one(churn) since we want that prediction to be made by the model.To do so, we go to the Data load editor and create a new SageMaker connection like below. You can read more about creating a new connection here.You should now see the SageMaker_Autopilot_churn name in your list of connections. Now, click on ‘select data’ to start sending your data from QS data model to SageMaker.Click on ‘Insert script’ to get the script in the editor.Please note how I have changed the raw script that we got from our connection to include all the 14 fields to be sent to our endpoint. Like our previous use case, we use RowNo( ) here as a field to associate the source data & the returned prediction table.Here’s a peek at our data model after the data is loaded.Let’s quickly check what is returned by the ML model based on the data we sent from Qlik Sense.So, for every customer row, we have a predicted_label field that shows whether the customer will stay or leave. We also have the individual class probabilities for deeper analysis.Step 4: Building the QS analytics appOur final step is to build a Qlik Sense app so we can perform our analysis and present it to the stakeholders.We will segregate the app into 3 sheets as shown below each serving its purpose:Descriptive Analysissheet:Goal: This sheet will help us understand the historical source data & allow for detailed analysis.First, I want to understand the distribution of a couple of features and since we have 15 of them, I won’t visualize all of them but highlight the ones that my stakeholders are interested in. In terms of visualization, I will use a container object and add the distributions as histograms like below.We can see that most of the fields (Day mins, Eve mins, etc.) have a normal distribution while Cust Serv Calls appear to be positively skewed.I also wanted to highlight how the target attribute(Churn) was distributed since it is important to know the reality of how many customers can actually churn. Looks like 14.49 % of the customers did churn.Next, since our data is high-dimensional (10+ features) and I want to enable detailed analysis of individual customers, choosing a visualization that works well with multidimensional data was crucial. I decided to go with a Parallel coordinate plot extension that I built sometime back.Finally, putting everything together here’s our Descriptive dashboard.Let’s do a simple analysis. I want to compare a customer who wants to churn with one that does not. So, I randomly select two such rows.This view allows us to easily compare all the 11 numerical attributes. So, looks like both of these customers are new customers(observe acc_length=1). The orange line represents Churn=‘True’ and the cyan represents Churn=‘False’. For most of the features, we see the lines in a criss-cross form which helps us understand how these 2 customers differ. One thing that stands out is how the customer who churns makes 5 customer service calls in just a day and the other one makes 1. This gives us an indication that the orange customer might have faced some issues with the operator.Predictive Analysissheet:Goal: This sheet will help us understand the churn predictions that we made using SageMaker & our model’s performance.Let’s see how our overall predictions looks like.The predictions are almost similar to the ground truth.Next, I want to visualize the churn predictions by each state so the telecom operator can keep their focus on those ‘risky’ states.This is a great example of how using Qlik’s associative property, we were able to integrate both historical and predicted data.It is also important to understand what mistakes the ML model makes. False negatives are the most problematic because the model incorrectly predicts that a churning customer will stay. The best way to evaluate our model would be to draw a confusion matrix like below.We have 17 such cases. We can select this ‘17 block’ from our matrix and perform detailed analysis of the special cases by analyzing both descriptive & predictive sheets.Predictive dashboard:What-If Analysissheet:Our final piece is the ‘What-if’ scenario builder. Personally, I love this native capability of Qlik Sense as it allows us to look beyond traditional analysis. Also, note how easy & quick it is to build this. I have used a custom object called ‘variable input’ that allows me to include sliders, dropdown & text fields.The ‘Will the customer churn?’ is a KPI object and uses a Server side extension function ScriptAggrStr()as a chart expression which allows us to get predictions in real-time by passing the values dynamically from the input boxes. Here’s the expression -endpoints.ScriptAggrStr('{"RequestType":"endpoint", "endpoint":{"connectionname":"Amazon_SageMaker_Telco"}}',vState,vActLen,vIntPln,vVmailPln,vVmailMsg,vDayMins,vDayCalls,vEveMins,vEveCalls,vNightMins,vNightCalls,vIntMins,vIntCalls,vCustServCalls)Let us do quick & simple what-if analysis. From our Descriptive sheet, we noted that the field Cust Serv calls might be an important one. After all, a happy customer doesn’t need to call customer service. To really prove that correlation, let’s play around.And looks like the hypothesis makes sense! However, please note that this is just one factor. I tried increasing the Int’l Mins to 15 instead of 12 and even though I had a lot of customer service calls, the prediction was False.So, maybe providing more Int’l Mins to the customer would be a great idea to retain them. This kind of insight can help businesses dealing with churn to really understand the pitfalls and improvise on them even at a granular level.That brings us to the end of this exciting blog. The tutorial is a detailed one as the whole idea was to allow Qlik users to quickly adapt to these capabilities and understand the process end-to-end. Let me know what you think!~Dipankar, Qlik R&D
...View More
Let’s face it, having all the visualizations available for Qlik in a single app is a task that is set to failure from the start. Qlik Sense offers dozens of charts out of the box, each one of those comes with several features and customization options that creates a matrix of hundreds of possible outputs. Still today, we sporadically find new and smart ideas to represent data with the good old bar chart in a completely new way.So, when we started this project, we knew already that it would be forever an incomplete piece. We knew it would only show a fraction of the possible ways of using Qlik Sense Charts, but even though we decided that it was important to showcase some of the Qlik Sense offering around data visualization.For this project we handpicked some of the new visualizations such as the Org Chart or the Grid Chart, and those that have been massively improved recently as the Line Chart with the new line chart styling.When working on a Qlik Sense app I believe the theme makes an big difference for it to be successful, with the right theme in place we can make sure the data has the relevance it deserves.For the Visualization Showcase app I created a theme that focus on the charts removing some of focus from other elements such as the description text. Text and descriptions are important, but in this particular app they are secondary to the more powerful story that each chart has to tell. To achieve this effect, this theme wraps each one of the charts within a container while the text elements land flat in the background. We intentionally guide users to the "highlighted" objects in the screen where they can interact with our glorious charts.The app contains 23 sheets with examples on how to use any of the showcased charts, some of the chart pages contain multiple variations of a chart, including:New VisualizationsSankey ChartMekko ChartVariance Waterfall ChartRadar ChartFunnel ChartNetwork ChartOrg ChartWord CloudBullet ChartP&L Pivot TableHeatmap Box PlotGrid ChartVisualization ImprovementsTrellis ChartSparklineTrend Lines and IndicatorsAccumulations & ModifiersVideo PlayerBar Chart ImprovementsLine Chart StylingPie Chart StylingTable ImprovementsMulti KPIMapping & GeoAnalyticsKeep in mind that we will continue to update the app with new and exciting ways of visualizing data. It would never be complete, but we will do our best.Check the Visualization Showcase hereAnd don’t forget to check other related resources: Patrik Lundblad's blog Qlik Gallery The top 10 viz apps FT Visual vocabulary, Sense version
...View More
In recent years we have seen tremendous growth in the generation of textual data online. Analyzing such textual data can prove to be extremely beneficial for businesses in today’s digital world which would allow them to make critical decisions. For instance, for a company selling a product, it is crucial to understand the consumer’s viewpoint, and therefore analyzing the ‘reviews’ can be a great starting point. However, this also leads to an intrinsic problem — How do we analyze such constantly generating huge volume of text and interpret their meaning?Ideally, the solution would be to break down sentences and phrases into specific components and then analyze these components by using a technique such as Natural Language Processing(NLP). NLP comprises features such as sentiment analysis, entity recognition, syntax parsing, part-of-speech tagging, etc. that can help us break down unstructured data to a granular level and allow for deeper analysis. There is a handful of toolkit such as NLTK, SpaCy, Amazon Comprehend, Google Cloud NL API, etc. that allows to process and derive insights from texts.Scenario: Imagine your organization leverages Qlik Sense’s data analytics platform for visually understanding data & key metrics and Amazon Comprehend for text analysis. Your requirement is to be able to analyze some textual data, understand sentiments & most importantly be able to visualize the processed texts and extract any hidden insights. How would you achieve this using the two?This is where Qlik Sense’s augmented analytics capabilities come into play. Now, as a Qlik Sense application developer, you can integrate the Qlik data model with Amazon Comprehend using the native connector for Qlik Sense. This would allow you to send a dataset from a Qlik reload script or a chart expression to Comprehend and derive inferences back to Qlik.Now before we focus on ‘sentiment analysis’ specifically, let us get some background behind using any 3rd-party ML connectors in Qlik Sense(including Amazon Comprehend for this specific use case). This would help you understand the things running behind the hood and set the base for using any analytic connector.1. Analytic connections:The first step to start communicating with any 3rd-party Machine Learning endpoint from Qlik Sense is to establish an analytic connection. This can be created in the Data load editor and is native to the Qlik Sense client.2. Server Side Extension(SSE) functions:After a connection has been created, the next step is to integrate Qlik’s data model with the 3rd-party ML platform using the connection. This would allow for:sending data from Qlik’s data model & getting the inferences back using the load scriptperforming real-time calculations in chart expressionsTo achieve the above two use cases, we rely on the SSE functions. SSE is used to extend the built-in Qlik expression library with functionality from external calculation engines. If you are not aware of the SSE syntax, you can read more about them here. Our focus in this blog would primarily be 2 functions:ScriptEval — used with data load script. Here, you can send a single table to the ML model, and a single data table is returned. We can then use LOAD… EXTENSION statement to load the data back to Qlik.ScriptAggrStr — used with chart expression. Here, we cannot consume a table returned from the SSE function call and only the first column returned will be used by Qlik Sense.The entire workflow for any Machine Learning connection integration within Qlik Sense comprises of the below steps:the SSE functions and parameters are first processed within Qlik Sense.they are then converted into REST requests.the REST requests are sent to 3rd-party ML models for processing.ML models process the request and send inference back to Qlik.The image below depicts the entire flow.Prerequisites:Analytic connections for machine learning endpoints must be enabled by a tenant administrator in the Management Console.Need to have access to 3rd-party ML models that exposes REST-based API endpoints.Alright, now that we have an understanding of the background and things required for us to get started with text analysis, let us deep dive into the steps.For this tutorial, we will use a part of the Datafiniti’s hotel reviews dataset and our goal would be to understand the sentiments from hotel reviews.1. Load Data: First, we will load our dataset, which is in a CSV file using a folder connection. The script in the Data Load editor looks like below. Please note we also add a new field ‘RowNo( ) as RowID’ in our table [Hotel_review]. The purpose of this is to create associations between this and the tables returned by the 3rd-party ML systems(comprehend in this case) so we can take advantage of Qlik’s unique associative engine during analysis.2. Create connection: Since we plan to use Amazon Comprehend for text analysis, we will create a new connection in the Data Load editor. Amazon Comprehend provides the following 5 services for text analysis -Since in this case, our goal is to do sentiment analysis, we will select the highlighted service from the dropdown. You will also need to provide Amazon-specific details for the connection (for details read here) and finally a name (‘Amazon_Comprehend_demo’).3. Send data to Comprehend: Next, we will use the ‘Select data’ button from our connection to send a table and a fieldfrom Qlik Sense to the Amazon Comprehend system for sentiment analysis. The table should be the name of the table with source data that you have loaded into your app. In our case, the table is ‘Hotel_Review’ and the field is ‘Reviews’.4. Load returned table: After a table & field name is sent from Qlik Sense to Comprehend, the available return table will automatically appear under the ‘Tables’ section(as seen below). When selecting the table, you can select or deselect the columns to load. In our case, we will select all the 5 fields under the ‘Sentiments’ table returned by Comprehend and click ‘Insert script’.Below is how the script looks like. Similar to Step 1, we also add a ‘RowNo( ) as RowID’ field to maintain the associations and then reload our app. Note that when reload of an app occurs it will first expect to load the source data as a resident table and use this as input to the request made to Amazon Comprehend endpoints.For the purpose of simplicity, let’s breakdown the SSE function visually.Now quickly check the Data model viewer. We can see that the associations has rightly been made and everything is as expected.5. Analysis: Our final step is to build a dashboard and do some analysis so we can understand the sentiments of the hotel reviews processed by Comprehend.First, I want to know the count of each predicted sentiment category. So, I create a bar chart by using ‘Sentiment’ as a dimension and ‘Count(RowID)’ as a measure. Notice, how we can use a field returned by Comprehend system along with the available Qlik sense data to derive insights.So, looks like there are a lot of ‘mixed’ reviews for the hotels. Out of curiosity, I wanted to know why there were so many mixed reviews about these hotels. Therefore, I created a table object with detailed reviews, titles and selected only ‘Mixed’ from my bar chart as a filter. The result is below:The reason is that most of these reviews have both the ‘bad’ and ‘good’ context in the text. So, the sentiments are mixed. The ability to derive these kinds of insights using Qlik Sense is crucial so it has the right impact on your analysis.I also wanted to visualize the predicted sentiments by the original ratings of the hotels. So, I decided to create a Mekko chart that would allow me to visualize ratings for each segment of sentiment. To do so, I use ‘Sentiment’ as a dimension, ‘Ratings’ as cells, and ‘Count(RowID)’ as a measure.We can infer some things from here. For e.g. out of all the Positive sentiments, 71.3% were 5-star ratings, which aligns with our general understanding. The Negative sentiments are composed of ratings ranging between 2.5–2.9. So, they turned out to be negative.Finally, let’s build a real-time sentiment analysis sheet in our Qlik Sense app. This is very interesting as it facilitates the following:provides a user interface experience to input any text.allows inferring sentiments in real-time.To build the sheet, we will use the ‘Variable input’ object from Qlik’s dashboard bundle that can serve as a text field. We create a new variable called vText to be used with this variable input.We then drag & drop the object to our sheet and the result is below:Next, we need to pass the input text to Amazon Comprehend and get the result back in real-time. To do so, we will take advantage of SSE-based chart expression. Since we need to create an expression, we basically need an object that can show us ‘Measures’. The KPI object is a natural choice for this purpose. So, we drag & drop a KPI object and write our expression.endpoints.ScriptAggrStr('{"RequestType":"endpoint", "endpoint":{"connectionname":"Amz_comprehend_sentiment","parameters":{"languagecode":"$(vLanguage)"}}}',vText)The result can be seen below.To enhance user experience, we will also display the result as an emoji placed inside a KPI object. The expression for passing the input text & deriving sentiment remains the same. However, we use a pick-match function to get the right emoji. Below is the expression.pick(match(endpoints.ScriptAggrStr('{"RequestType":"endpoint", "endpoint":{"connectionname":"Amazon_Comprehend_Sentiment","parameters":{"languagecode":"$(vLanguage)"}}}',vText),'POSITIVE','NEUTRAL','NEGATIVE','MIXED'),'😀','😐','🙁','🙄')After putting everything together, the final result can be seen below -Here’s the dashboard in action.The idea behind this blog was to give a starting point to Qlik Sense users who plan to integrate 3rd-party ML systems and do advanced analytics seamlessly. There are also certain limitations specific to the Amazon Comprehend & the connector, which you can read about here. In the next couple of blogs, we will extend this tutorial to some more interesting use-cases using the various analytics connector available in Qlik Sense SaaS.If you wanted to check out this demo app, here is a link — https://github.com/dipankarqlik/QlikSenseAppsLet me know what you think of this amazing augmented analytics capability within Qlik Sense.~Dipankar, R&D Developer Advocate
...View More
Understanding d3-interpolateThe d3-interpolate module offers a way to compute intermediate values between two given values. The main interpolation method "d3.interpolate" takes two values as parameters (d3.interpolate(a, b)), and depending on the inferred type (numbers, colors, arrays, objects etc...), it routes them through the appropriate function that will create the relevant interpolator. This interpolator implementation is based on the type of the end value b.For instance, if two strings representing CSS colors are passed to d3.interpolate, it will call the color interpolation method:let color = d3.interpolate("blue", "red");
color(1) // will return #FF0000 the hex equivalent of red
color(0.5) // will return #800080, an purple-ish color that represents the mid color between blue and redThe same goes for interpolating between numbers, or even numbers inside strings. Take the following code for example where CSS transforms are interpolated, allowing us to make basic animations:let transform = d3.interpolate("translate(0,0) rotate(0)", "translate(500, 50) rotate(90)");
transform(0.5) // returns translate(250, 25) rotate(45)Animating a Picasso.js chartGoing back to our mashup, let's implement d3 interpolation within our nebula.js code, the goal is to animate a line chart in order to simulate a drawing effect of the line from left to right.If you are not very familiar with how to build a chart using nebula.js and picasso.js, you can check out the official docs, this blog post, orthis one which will walk you through all the building blocks to get up and running.First things first, import the functions we need from their respective d3 modules, including the ease function, timer, and the interpolate function.import { easeSinIn as ease } from 'd3-ease';
import { timer } from 'd3-timer';
import { interpolate } from 'd3-interpolate';We then declare our transition variable and a way to reset it.// Define transition & stopTransition function
let transition = null;
const stopTransition = () => {
if (transition) {
transition.stop();
transition = null;
}
};Now for the fun part, we schedule a new timer to invoke the callback that includes our Picasso.js chart instance update function repeatedly until stopped.The interpolate function takes an old and a new value and then a time argument from 0 to 1 and returns the right x and y coordinates which in our example results in the line animating. useEffect(() => {
if (!instance) {
return;
}
// reset transition
if (transition) {
stopTransition();
}
// Set duration
const duration = 5000;
// Run timer
transition = timer((elapsed) => {
// Set t
const t = Math.min(1, ease(elapsed / duration));
instance.update({
data: [
{
type: 'q',
key: 'qHyperCube',
data: layout.qHyperCube,
},
],
settings: {
scales: {
x: {
data: {
extract: {
field: 'qDimensionInfo/0',
},
},
},
y: {
data: { field: 'qMeasureInfo/0' },
invert: true,
expand: 0.1,
},
},
components: [
{
key: 'lines',
type: 'line',
data: {
extract: {
field: 'qDimensionInfo/0',
props: {
y: { field: 'qMeasureInfo/0' },
},
},
},
settings: {
coordinates: {
major: {
scale: 'x',
fn: (d) => {
const { items } = d.data;
const start = Math.max(0, d.datum.value - 1);
const end = d.datum.value;
const time = Math.min(1, (t - d.datum.value / items.length) * items.length);
return time < 0
? null
: interpolate(d.resources.scale('x')(start), d.resources.scale('x')(end))(time);
},
},
minor: {
scale: 'y',
fn: (d) => {
const { items } = d.data;
const start = items.find((item) => item.value === Math.max(0, d.datum.value - 1)).y.value;
const end = items.find((item) => item.value === d.datum.value).y.value;
const time = Math.min(1, (t - d.datum.value / items.length) * items.length);
return time < 0
? null
: interpolate(d.resources.scale('y')(start), d.resources.scale('y')(end))(time);
},
},
},
},
},
],
},
});
if (t === 1) {
stopTransition();
}
});
}, [layout, instance]);That's all! Below is how the final result looks like. Keep in mind that this concept can be extended to be used with other Picasso.js components to achieve the animations you need, for instance interpolating the size of point components in bubble charts or box components in bar charts.Please find the full code on Githubfor reference.
...View More
Nebula.js Developers, version 2.0 is officially here!Hello Qlik Community.This is Dipankar from Qlik R&D and today I am going to introduce the latest version of Nebula - version 2.0. Since the initial release of Nebula, the R&D team has been working on improving various capabilities within the library to help developers achieve more agility and robustness in terms of developing/embedding Qlik's visualization using Nebula. With the new version, we take this a step further.What is Nebula 2.0?Nebula 2.0 is a collection of the following items:updated chart packagesNebula CLINebula API featuresimproved documentationLet's talk about each of these items in some details.Updated chart packages:While nebula.js provides capabilities and APIs to build mashups and custom visualizations, it's primary purpose is often to display Qlik developed charts. These charts are available through npm under the @nebula.js scope. With the latest release, the focus is on streamlining the charts. Now, you can find an inventory of all the supported Nebula charts under the 'visualizations' library in qlik.dev -An interesting addition to the charts is the 'Package and core build' part. With Nebula 2.0, when you have multiple charts, using the "core" build can significantly reduce the overall size of your bundle, thus reducing the amount of code that needs to be downloaded every time and executed. Here is an example link that shows how to achieve this -https://github.com/qlik-oss/nebula.js-examples/tree/main/examples/simple-setup. We believe this can be very beneficial from a performance perspective. Nebula CLI:Nebula 2.0 updates the robust Nebula CLI which is the main entry point for getting started with a nebula visualization,enabling a local development server, and building extensions and mashups. This version also streamlines the commands for usage. The four commands arelisted below:nebula create - creates a new nebula visualization or mashup to get you startednebula serve - runs a nebula visualization with a local web development servernebula build - builds a nebula visualization bundlenebula sense - generates a nebula visualization to be used as an extension in Qlik SenseNebula API:The Nebula API has some new functionality -useEmbed -this hookallows access to the Nebula instance inside a chart, to for example render a chart inside another or to use the field listbox.useKeyboard - this hook gets the desired keyboard settings & status to apply when rendering a visualization. Read more about this here.You can also alter property definition for field-listbox now. See an example below that shows how to change sorting.nebbie.field('[My Field]').then((s) =>
s.mount(document.querySelector('.listbox'), {
properties: {
qListObjectDef: {
qDef: {
qSortCriterias: [
{
qSortByState: 0,
qSortByAscii: 1,
qSortByNumeric: 1,
qSortByLoadOrder: 1,
},
],
},
},
},
})
);Apart from these, Nebula 2.0 also exposes class names for embedded charts that allows for basic theming capabilities.div.njs-cell {
background: maroon;
}
p.njs-cell-footer, h6.njs-cell-title, p.njs-cell-sub-title {
color: white;
}
button.njs-cell-action {
color: white;
}Improved Documentation:Overall, Nebula 2.0 solidifies the official documentation throughout the Qlik Developer site. I particularly like the Specifics section for each of the chart that serves as a guide in getting started with understanding & developing the Nebula-based visualizations. Be sure to check it out.Hope this post helps you leverage all the amazing capabilities of Nebula 2.0!!We will curate and share a Nebula 2.0 playlist in Glitch soon for our developer community, so keep an eye out.~Dipankar
...View More
My friend,Øystein Kolsrud - Software Architect at Qlik, is back with part 5 of the Qlik Engine API fundamentals:Multiple Hypercube DimensionsPrevious articles:Part 1:Let's Dissect the Qlik Engine API - Part 1: RPC BasicsPart 2:Let's Dissect the Qlik Engine API - Part 2: HandlesPart 3:Let's Dissect the Qlik Engine API - Part 3: Generic ObjectsPart 4:Let's Dissect the Qlik Engine API - Part 4: HypercubesWith part 5 of this series Øystein will discuss some consequences of how the engine computes the data sets that hypercubes define.IntroductionThe hypercubes we have looked at so far were used to calculate multiple expressions for all values of a single field. We have seen how we in one go can compute both the sales, and the sales per month for all employees. Our next step will be to compute these statistics not only for the total set of data, but for sales per year as well. The data we have contains sales information for two different years, so if we want to calculate our expressions for a particular year, then we could simply select that year, and retrieve the data. The engine will do all the filtering for us and guarantee that only sales for the selected year is included in the hypercube computations. It is quite possible to use such a flow to iterate across the years in order to get sales information for each year, but as anyone who has ever added a table visualization to an app knows, there is a more efficient way to do this! We can simply extend the hypercube with the "Year" dimension, and have the engine do all the work for us!How the engine deals with multiple dimensionsWhen the engine encounters multiple dimensions in a hypercube, it will compute the expressions for all possible combinations of values of the dimensions. Since our data contains information about sales for two employees from the years 2019 and 2020 we get the following set of four combinations:"Amalia Craig", 2019"Amalia Craig", 2020"Amanda Honda", 2019"Amanda Honda", 2020The hypercube definition we will use for our computations needs to contain two entries in the "qDimensions" array, like this:{
"jsonrpc": "2.0",
"id": 21,
"method": "SetProperties",
"handle": 4,
"params": [
{
"qInfo": {
"qId": "3cb898dc-b7dc-44ed-ba83-c27fae5d0658",
"qType": "myCube"
},
"qHyperCubeDef": {
"qDimensions":
[ { "qDef": { "qFieldDefs": [ "SalesRep" ] } },
{ "qDef": { "qFieldDefs": [ "Year" ] } }
],
"qMeasures":
[ { "qDef": { "qDef": "Sum(Sales)" } },
{ "qDef": { "qDef": "Sum(Sales)/Count(Month)" } }
]
}
}
]
}Now that we have added a second dimension, the resulting table will contain four columns (two dimensions, two measures), so in order to get all data, we need to set the width of the page ("qWidth") to 4:{
"jsonrpc": "2.0",
"id": 22,
"method": "GetHyperCubeData",
"handle": 4,
"params": [
"/qHyperCubeDef",
[ { "qLeft": 0,
"qTop": 0,
"qWidth": 4,
"qHeight": 20
}
]
]
}The response will contain 4 rows with 4 cells each for a total of 16 cells (and I'll be truncating to reduce verbosity):{
"jsonrpc": "2.0",
"id": 22,
"result": {
"qDataPages": [
{
"qMatrix": [
[ { "qText": "Amalia Craig", ... },
{ "qText": "2019", ... },
{ "qText": "104480", ... },
{ "qText": "2749.4736842105", ... }
],
[ { "qText": "Amalia Craig", ... },
{ "qText": "2020", ... },
{ "qText": "156662", ... },
{ "qText": "2701.0689655172", ... }
],
[ { "qText": "Amanda Honda", ... },
{ "qText": "2019", ... },
{ "qText": "84396", ... },
{ "qText": "2482.2352941176", ... }
],
[ { "qText": "Amanda Honda", ... },
{ "qText": "2020", ... },
{ "qText": "169293", ... },
{ "qText": "2821.55", ... }
]
],
...
}
]
}
}And there you have it! The engine has computed our two expressions for both employees for both years!The size of hypercubesThe data rows you get for a particular cube depends not only on the number of field values of the dimensions, but also on how those values relate to each other in the data model. Say for instance that "Amanda Honda" had no sales recorded for the year 2019 (perhaps because she was hired in 2020). The loaded data will then not contain any records associating "Amanda Honda" with the year "2019". When you get the data for the hypercube in this scenario, then there will only be three rows returned as the combination of the field values "Amanda Honda" and "2019" is excluded by the data.You can check how many rows a hypercube has by looking at the property "qSize" of the hypercube layout. This is what the "GetLayout" response could look like in our case:{
"jsonrpc": "2.0",
"id": 23,
"result": {
"qLayout": {
"qInfo": {
"qId": "3cb898dc-b7dc-44ed-ba83-c27fae5d0658",
"qType": "myCube"
},
"qHyperCube": {
"qSize": {
"qcx": 4,
"qcy": 3
},
"qDimensionInfo": [ ... ],
"qMeasureInfo": [ ... ],
...
}
}
}
}The "qSize" property indicates both the number of columns ("qcx") and the number of rows ("qcy").Excluded combinationsOne might think that there should be a row generated also for combinations that do not have values. After all, one could without lying say that the sales for "Amanda Honda" in 2019 was 0. But this is not how the engine works! And that is a good thing! If the engine didn't do this type of filtering, then most cubes would result in such a large number of rows that they would be completely unusable. Consider for example that all sales transactions had both a timestamp and a unique ID associated with them. Then we create a hypercube that lists the timestamp, the ID, and the sales amount for each transaction represented in 1000 USD. We could define our cube like this:"qHyperCubeDef": {
"qDimensions":
[ { "qDef": { "qFieldDefs": [ "Timestamp" ] } },
{ "qDef": { "qFieldDefs": [ "ID" ] } }
],
"qMeasures":
[ { "qDef": { "qDef": "Sales/1000" } } ]
}The number of rows this cube would yield would be equal to the number of IDs, as there is exactly one timestamp associated with each ID. Also, in this case it clearly makes no sense to return rows that show the sales for nonexistent combinations of "Timestamp" and "ID". In fact, if we did, then we would soon run into trouble. Imagine that there are 10 IDs. The resulting table would in that case contain 10*10=100 rows (if we assume that the timestamps are unique as well). If there were 1000 IDs then there would be a million rows. And if there were a million IDs then you'd better dive for cover and hope you don't bring down the entire engine when you try to compute the layout! Because the resulting number of rows would be astronomical!It's good to be aware of this engine behavior as there are in fact cases where you risk creating such huge cubes. I have accidentally done so myself on several occasions. And I can tell you, it's not a good thing... The problem occurs when you add two fields to a cube that have no association to each other what so ever. A company could for instance have data for both sales and purchases, both of which have unique ID's, but are stored in different tables. Then I create a table visualization with timestamp and ID, but accidentally choose the ID from the "Purchases" table instead of from the "Sales" table like this:"qHyperCubeDef": {
"qDimensions":
[ { "qDef": { "qFieldDefs": [ "Sales.Timestamp" ] } },
{ "qDef": { "qFieldDefs": [ "Purchases.ID" ] } }
],
"qMeasures":
[ { "qDef": { "qDef": "Sales.Sales/1000" } } ]
}Since all possible combinations of "Sales.Timestamp" and "Purchases.ID" are legal, this is likely to give me a hypercube of a rather intimidating size.SummaryHypercubes with multiple dimensions is a very common thing in Qlik Sense. Apart from the obvious example of a table visualization, most other visualizations allow for some form of multiple dimension configuration as well. Two examples that I frequently use are the bar chart and line chart visualizations, both of which will have cubes with multiple dimensions under certain configurations.The handling of hypercubes is at the very core of how the engine deals with computations, but there are of course limits to what the engine can handle. If you run into surprisingly long computation times or very high engine memory usage, then be on the lookout for visualizations that accidentally span very large sets of field value combinations. Such cubes can sometimes give you nasty surprises.
...View More
If you want to add custom interactions that depend on what shapes the user is interacting with in picasso.js, you're going to want to use the shapesAt() function for two reasons.First, while you can try to use the shape that is the target of the event when rendering in SVG with picasso.js, that won't work if you're rendering with Canvas in picasso.js since the event target will just be the Canvas element. And second, shapes in some charts, such as a scatter plot, may overlap, so simply using the shape that is the target of the event is not enough if you want the interaction to include all of the shapes the user is interacting with. With the shapesAt() function, whether rendering in SVG or Canvas, you can get all of the shapes at the point of the user interaction.Let's take a look at one use case and how to use it. Imagine that we have a scatter plot and when the user clicks, we want to select all items at the point of the cursor. We want the selection to apply to Qlik Sense, but we want the chart to having a brushing effect, so we'll use set analysis so that all of the rows return regardless of the selection state in Qlik Sense, but we'll change the opacity based on the selection.The setup for the hypercube, which we'll create with nebula.js, looks like this.nebula.render({
element: document.querySelector('#container'),
type: 'scatter-plot',
fields: [
'Product Sub Group Desc',
'=Sum({<[Product Sub Group Desc]=>} [Sales Margin Amount])/Sum({<[Product Sub Group Desc]=>} [Sales Amount])',
'=Sum({<[Product Sub Group Desc]=, [Year]={"2020"}>} [Sales Amount])',
],
});Notice the set analysis in the measures that ignores the selections in the dimension of the chart, so that when we apply selections in Qlik Sense to the dimension of the chart, every row in the hypercube still returns.Next let's take a look at the entire picasso.js definition.scatterPlotSn = {
const picasso = picassojs();
picasso.use(picassoQ);
return {
qae: {
properties: {
qHyperCubeDef: {
qDimensions: [],
qMeasures: [],
qInitialDataFetch: [{ qWidth: 3, qHeight: 100 }],
qSuppressZero: true,
qSuppressMissing: true,
},
showTitles: true,
title: '',
subtitle: '',
footnote: '',
},
data: {
targets: [
{
path: '/qHyperCubeDef',
dimensions: {
min: 1,
max: 1,
},
measures: {
min: 2,
max: 2,
},
},
],
},
},
component() {
const element = stardust.useElement();
const layout = stardust.useStaleLayout();
const rect = stardust.useRect();
const model = stardust.useModel();
const [instance, setInstance] = stardust.useState();
stardust.useEffect(() => {
const p = picasso().chart({
element,
data: [],
settings: {},
});
setInstance(p);
return () => {
p.destroy();
};
}, []);
stardust.useEffect(() => {
if (!instance) {
return;
}
const selected = layout.qHyperCube.qDataPages[0].qMatrix.filter((row) => row[0].qState === 'S').map((row) => row[0].qElemNumber);
instance.update({
data: [
{
type: 'q',
key: 'qHyperCube',
data: layout.qHyperCube,
},
],
settings: {
scales: {
x: {
data: {
extract: {
field: 'qMeasureInfo/0',
},
},
},
y: {
data: {
extract: {
field: 'qMeasureInfo/1',
},
},
invert: true,
}
},
components: [
{
type: 'point',
data: {
extract: {
field: 'qDimensionInfo/0',
props: {
x: { field: 'qMeasureInfo/0' },
y: { field: 'qMeasureInfo/1' },
},
},
},
settings: {
x: { scale: 'x' },
y: { scale: 'y' },
shape: 'circle',
size: 1,
opacity: (d) => selected.length === 0 || selected.includes(d.datum.value) ? 1 : 0.25,
},
},
],
interactions: [
{
type: 'native',
events: {
click(e) {
const chartBounds = element.getBoundingClientRect();
const cx = e.clientX - chartBounds.left;
const cy = e.clientY - chartBounds.top;
const shapes = this.chart.shapesAt({ x: cx, y: cy });
const values = shapes.map((shape) => shape.data.value);
model.selectHyperCubeValues('/qHyperCubeDef', 0, values, true);
},
},
},
],
},
});
}, [layout, instance]);
stardust.useEffect(() => {
if (!instance) {
return;
}
instance.update();
}, [rect.width, rect.height, instance]);
},
};
}There's a lot of code there, but if you're familiar with building custom picasso.js charts, much of it will be very familiar. If you've never built a custom picasso.js chart, you may want to check outhttps://community.qlik.com/t5/Qlik-Design-Blog/Picasso-js-What-separates-it-from-other-visualization-libraries/ba-p/1829951.So let's take a closer look at the part that is the topic of the blog post, the shapesAt() function. The shapesAt() function in this example is used in the 'click' interaction.click(e) {
const chartBounds = element.getBoundingClientRect();
const cx = e.clientX - chartBounds.left;
const cy = e.clientY - chartBounds.top;
const shapes = this.chart.shapesAt({ x: cx, y: cy });
const values = shapes.map((shape) => shape.data.value);
model.selectHyperCubeValues('/qHyperCubeDef', 0, values, true);
}The shapesAt() function is called from the chart instance, and in this case, we're simply checking for the shapes at the exact x and y. There are more advanced ways to use it, including detecting only certain shapes, or using an area instead of a point to detect shapes, and you can check out the docs for more information athttps://qlik.dev/apis/javascript/picassojs#%23%2Fdefinitions%2FChart%2Fentries%2FshapesAt.The big thing I want to point out here so you can avoid the struggle I had first using the shapesAt() function is how the x and y coordinates are calculated. You must adjust for where the chart is located in the viewport. So the clientX and clientY values from the event give you the x and y coordinates of where the click happened in the viewport, then you can adjust for the area outside of the chart using the left and top from element.getBoundingClientRect().You can check out all of this code athttps://observablehq.com/@fkabinoff/picasso-js-shapesat. Let me know if this post inspires you, and what else you'd like to know about picasso.js!
...View More
What is the difference between the Date# function and the Date function? These two functions seem quite similar, but they have different purposes. It took me a while to understand when to use one over the other. Now that I have a handle on it, I thought I would share what I have learned. The Date# function is an interpretation function. According to Qlik Help, “Date# evaluates an expression as a date in the format specified in the second argument, if supplied.” I use the Date# function when I am loading a value that I want to be perceived as a date. For example, in the partial script below, I loaded “YR” from an Excel file and to ensure that the value was evaluated as a date with the year format ‘YYYY’, I used the Date# function.The syntax of the Date# function is as follows:Date#(text[, format])The format parameter of the Date# and Date functions is optional. If it is not included in the expression, it uses the date format set in the system variables in the script. Using Date#() is an important step if the “Year” field is used later in the script to join data or to compare to data that has a date value. When evaluating 2 values, you want to ensure that they are formatted the same and that you are comparing apples to apples.The Date function is a formatting function. According to Qlik Help, “Date() formats an expression as a date using the format set in the system variables in the data load script, or the operating system, or a format string, if supplied.” I use the Date function to format a date a specific way. For example, I may format a date as ‘YYYY’ if I only am interested in seeing the year. I could also format the date like this ‘M/D/YYYY’ to see the month, day and year.The syntax for the Date function is as follows:Date(text[, format])In the example below, I am formatting the “Yr” field as a 4-digit year (i.e. 2021).I can also format a date as seen below. This expression will return 09/17/2021.On occasion, I have used both the Date# and the Date functions in the same expression. For example, if I am loading text with the format YYYYMM and I want to format it as MMM-YYYY, I cannot simply use the expression Date(text, ‘MMM-YYYY’) because it does not pick up that the text is a date. So, I need to first interpret the text as a date and then I can format it. This expression works:Date(Date#(text, ‘YYYYMM’), ‘MMM-YYYY’)The Date# function first identifies the text as a date and indicates the format the date is in (‘YYYYMM’). Then the Date function formats the text like this ‘MMM-YYYY’.Both the Date# and Date functions can be used in script and chart functions. Date#() interprets the data as a date and Date() formats the date as specified. I hope this was helpful.Thanks,Jennell
...View More
.png "Former Employee"){kind=icon}
