2018

be9734868170ef50ac66fcd2b59da561.jpg.pngHey Guys - I have been meaning to do a video on this topic for many years and I finally got around to it! Over the years as I taught myself Qlik, I would dabble with the idea of incremental loading - but never really had a grasp on it as my data sets were always used for demos or they were quite small.....until I started building my Qlik Sense Retro-Gaming Dashboard.

1-30-2018 4-20-48 PM.png

(I use the incremental loading best practice to insert new video game pricing data daily - which allows me to view the pricing history of my collection for "Loose" and "Complete in Box" values). As I was building my app I had a ton of resources available to me but I never really found a decent video on this topic that simply explained what it was or how it is used. I figured I would create something short and sweet with a simplistic example from what I have learned, with hope it helps you understand incremental loading a bit better than I did! There are a few prerequisites you should be aware of, such as understanding what .QVD files are and what there purpose is. As well as basic data model design that include key and date fields which are used to identify new and changed records. I've included some samples files and resources at the end of this post to help.


Let me know what you think and please leave your questions or comments below. Enjoy!


Incremental Load


Regards,

Michael Tarallo (@mtarallo) | Twitter

Qlik

 

Resources:

 

 

Sample Insert Only - Incremental Load Script

 

//Simple Insert - New Records from Source Data

// Get the last OrderID loaded into the .QVD
MaxKeyLoad:
LOAD Max(OrderID) as MaxID
FROM [lib://IncLoadLocation (desktop-aura45o_mto)/sales_data.qvd] (qvd);

//Set a variable to be used in the SQL
Let MaxID = peek('MaxID',0,MaxKeyLoad);

//Get the latest data from the source where the OrderID in the Source is greater than the orderID in the .QVD
[orders]:
Load *;
SQL SELECT * FROM "Sales"."dbo"."Orders$"
WHERE OrderID > '$(MaxID)';

// Data is automatically concatenated if the data model has the sames fields
// CONCATENATE forces concatenation (in this example it is not necessary)
// Load all data from the .QVD
CONCATENATE
LOAD * from [lib://IncLoadLocation (desktop-aura45o_mto)/sales_data.qvd] (qvd);

//Store the new larger qvd file
// This will store the existing data and the new record and overwrite the existing .qvd
STORE orders into [lib://IncLoadLocation (desktop-aura45o_mto)/sales_data.qvd] (qvd);

// Exits script - used for debugging and troubleshooting - will not execute the next tab.
exit script;





 

 

Can't see the video? YouTube blocked in your region or office? - Download the .mp4 to watch on your computer or mobile device.

 

In case you want to perform this test with MS SQL - I added the SQL Queries and the MS SQL Database .bak file as well.

Qlik String functions are an extremely handy set of functions to clean, manipulate, and ultimately normalize data before (or after) loading it into an app. String functions ‒ Qlik Sense

 

I was working with a new data set recently, among other fields the data set contains addresses, something like this:

 

Abbey Road, London

 

Since the data set doesn’t have a dedicated city field I decided to apply Subfield() to extract the City as a separate field. Easy task, right? I use the comma as a separator to split the string.

 

…
Subfield (address, ',', 2) as City,
…








 

Shortly after we decided to expand the data universe of the app to include other countries in our analysis, we ended up having multiple sources with slightly different address formats:

 

Abbey Road, London

Paseo del Prado, s/n , Madrid

91, Rue de Rivoli, 75001, Paris

 

Not great news, right? Due to the variable number of commas in the address field I can no longer use subfield to manipulate the string, or maybe I still can use it.

 

Searching from the end of the string

 

Subfield function not only takes positive values for the third parameter, it also accepts negatives values. A negative value searches from the end of the string. This will help us to extract the City while keeping my script highly readable. The new function will be something like:

 

Subfield (address, ',', -1)








 

The function above will return the first portion of the string separated by the delimiter comma (',') searching from the end of the string, right to left. The newly created "City" field contains:

 

     London

     Madrid

     Paris

 

Right to left search in String functions


The use of negative values, or right to left search (remember the read will still be performed from the start of the string) also apply to other string functions:


  • Subfield

    Subfield('Abbey Road, London', 'o',-2) 

    Returns 'nd' (the second occurrence of 'o' starting from the end)

  • Index

    Index('Abbey Road, London', 'o',-2) 

    Returns 14 (the second occurrence of 'o' starting from the end)

  • FindOneOf

    FindOneOf('Abbey Road, London', 'oes',-3) 

    Returns ‘8’ because the search is for any of the characters: o, e or s, and "o" is the third occurrence starting from the right, and is in position 8 starting from the left.

 

I hope you find it as useful as I do.

AMZ @arturoqv

1-22-2018 9-48-55 AM.png

UPDATE: Attached are several demos, PDFs and a PPT that are useful to be shared externally for customers and partners alike.


By now it is common knowledge in the Business Intelligence and Data Visualization space, that the term Big Data doesn’t equate to one technology. That being said, it is also true that Big Data doesn’t relate to one scenario, use case or infrastructure. There can be many differences from one organization to the next. Since every situation is different, Qlik offers multiple techniques which can be used individually or in combination, to best meet the Big Data needs of a particular organization. One of these approaches is On Demand App Generation, commonly referred to as ODAG. ODAG was first introduced as an extension for previous releases, and is now built in to Qlik Sense starting with our June 2017 release. It benefits from continued enhancements with each release.


Our latest video from Adam  will show and tell you more.


"On Demand App Generation is a methodology that can be used in any situation where each user wants to explore their own slice of the data and it is now a built in function since the June2017 release of Qlik Sense. Big data analytics can now be approached through a shopping cart like process where dimensional data can be filtered at an aggregated high level and data slices automatically generated on-demand for further detailed analytics by users, all in a secure and governed manner." - Adam Mayer

 

 


On Demand App Generation


Helpful Resources




Who is Adam Mayer?

AM_round Green Circle_2.jpg

 

Adam joined Qlik in 2016 as a Senior Manager in Technical Product Marketing. He is responsible for delivering the company’s Internet of Things (IoT) go-to-market strategy. With a strong technical background in computing spanning over 20 years, underpinned by an incisive engineering perspective, Adam is an avid follower of new technology and holds a deep fascination of all things IoT, particularly on the data analytics side and finding new ways to make it as translatable, visual and understandable to as many people as possible.


"My first Qlik Sense app tracked my spend on car fuel where I geeked out on the rise and fall of pricing data!"


You can follow Adam on Twitter here: Adam_Mayer (@AdamMayerwrk) | Twitter


Can't see the video? Access to YouTube blocked?


Download the attached .mp4 to watch on your computer or mobile device

I've been doing most of my projects lately with React and enigma.js, and developing some general patterns along the way that I've been working into a starter project that I can use as a starting point. I'd like to share that with you today, and potentially start a discussion around the design decisions I've made thus far.

 

The starter project can be found at GitHub - fkabinoff/qlik-react-starter: Starter project for rapidly developing QAP-powered apps. It's still very much in development, but there's a decent framework and dev environment in place, as well as a few useful components.

 

Overview

The project uses enigma.js to interface with the Qlik engine, along with React, webpack, Babel, ESLint, Sass, and a few other goodies. Really, a pretty standard React app with enigma.js added. The qDoc, which is the name the enigma.js docs use for the connection to the Qlik app is created in a plain Javascript module and simply imported to any file that needs to use it.

 

I use react-hot-loader, which allows you to change react components without refreshing the page or losing state, and I really like it. It has it's problems, most noticeably to me that it doesn't play very nicely with higher order components, but the ability to change my React components and not have to reload the page or lose state helps me develop faster.

 

I'm using the latest Babel preset, which enables es2015, es2016, and es2017, and Babel plugins which enable decorators, spread syntax, and class properties. I can't believe we actually wrote Javascript before this stuff.

 

I use the very popular airbnb config for ESLint. It helps keep the code healthy and uniform.

 

I make use of reactstrap, a library of Bootstrap 4 components for React, and SASS for customizing and mixing in Bootstrap styles, as well as creating a few utility classes. I prefer using reactstrap over some of the more feature-rich React UI libraries, because I often find it creates more work trying to make the components of those UI libraries, such as Material UI, play nicely with how I want a component in a Qlik-powered app to function since in a Qlik-powered app the Qlik engine acts as a state machine, and many of the UI libraries with more features expect the state to be managed and changed by the client. Reactstrap gives me what I need, without being too opinionated about it.

 

I use the "render prop" pattern in place of higher order components. The render prop pattern is simply passing a component as a prop to another component for that component to render it. It basically works like a higher order component does, but to me it's a bit simpler, a little more flexible, and doesn't mess with react-hot-reload or PropTypes checking.

 

Notably, I don't use a state manager like Redux or MobX, and this may be my most contentious decision. To me, the use of a state manager in most instances is redundant when building a Qlik-powered app. Qlik already manages my state, and shares it between components. I hardly, if ever, have state that needs to be managed outside of Qlik in my Qlik-powered apps, and using a state manager forces me to do extra work to keep the client state and Qlik state in sync. I don't like it.

 

It would be very interesting to hear what you think about these design decisions, and the decisions that you've made with your own Qlik projects.

 

Components

Now, let's take a look at the actual components. There are currently 4 Qlik-specific components, but I expect to add more as I continue developing this. I tried to design the Qlik-specific components to expect props which are identical to values defined in Qlik docs, and pass down values directly returned by the Qlik engine, in order to standardize as much as possible. Though in some instances, I broke that rule. For instance, since the QlikObject wrapper, which I'll talk about in a moment, is really only designed to work with an object that fetches one page, it didnt make sense to pass down the qDataPages returned by the Qlik generic object getData method. Instead, I simply pass down the object at the first index of qDataPages.

 

QlikObject.jsx

The QlikObject component implements the render prop pattern, and contains the logic to create and manage a Qlik generic object and its state. It passes down the Qlik generic object information to the component defined in the Component prop, which is responsible for rendering something.


QlikVirtualScroll.jsx

The QlikVirtualScroll component implements the render prop pattern, and paginates data on vertical scroll so only data in view gets rendered. It passes down a subet of the qMatrix to the component defined in the Component prop, which is responsible for rendering something. It is used by the QlikFilter and QlikTable components.


QlikFilter.jsx

The QlikFilter component renders a dropdown that acts as a Qlik Sense filter. It's meant to be used as the `Component` prop of a `QlikObject` component a type of `qListObject`. This component doesn't need to be passed any props other than the props passed to it by `QlikObject`.


QlikTable.jsx

The QlikTable component renders a table. It's meant to be used as the `Component` prop of a `QlikObject` component with a type of `qHyperCube`. This component requires a prop `columnWidths` to be passed to it, defining the widths of the columns of the table. This prop should be passed using the `componentProps` prop of the `QlikObject`.



Finishing up

Go to GitHub - fkabinoff/qlik-react-starter: Starter project for rapidly developing QAP-powered apps, download the zip, run npm install, and npm run dev, and you can check out the QlikFilter and QlikTable components in action, and check out the code. I'll be adding more components and styles as I'm still pretty actively developing it, so watch the repo if you're interested, and let me know what you think.

Jennell McIntire

Qlik Sense Histogram

Posted by Jennell McIntire Jan 12, 2018

What is a histogram?  A histogram is a chart that allows you to visualize the distribution of data over a continuous interval or certain period.  It is made up of bars like a bar chart but instead of displaying actual counts against a dimension, a histogram displays the frequency at each interval or bin allowing you to see where the values are concentrated and where there are gaps or odd values.  A histogram is also different from a bar chart in that it does not need measures.  Only a single numeric dimension is needed to create a histogram.  Once the dimension is added to the Qlik Sense histogram chart, the frequency is automatically calculated.

 

Let’s look at an example, in the histogram below, the dimension BMI (body mass index) is used to view the frequency.  After BMI was added to the chart, Qlik Sense automatically created the BMI bins based on the frequency distribution.  This chart shows that BMI is concentrated in the range 21 <= x < 22.75 (21 to 22.75, but not including 22.75).

histogram.png

The Qlik Sense histogram chart has some properties that can be used to customize the histogram.  There is the ability to set the number of bars to a maximum number or set the width of the bars if you would like the bars to be even intervals.  Qlik Sense Help has an example of a histogram using even intervals to show temperature.

num of bars.png               bar width.png

 

In the histogram above, the number of bars is set to 10.  When the bars are set to Auto, Sturges’ formula is used to determine the bins and this may change based on selections.

 

A histogram chart is ideal for a large amount of data when you need a quick visualization of frequency.  It is easy to create since only one dimension is required.  The key to remember is that the dimension field must be numeric.  Try it out with your numeric dimensions in your next Qlik Sense app.  If you are using QlikView, you may be interested in Henric Cronström’s blog Recipe for a Histogram on how to create a histogram in QlikView.  Also, check out Arturo Munoz’s blogs on the Distribution Plot chart and the Box Plot chart.  You may also find these helpful in your next Qlik Sense app.

 

Thanks,

Jennell

ny2018.pngWell guys - where do I begin? – Ah I got it, Happy New Year! <sigh> It feels like only yesterday that I was writing my first blog for 2017. However, after a long break, I’m excited to get back to work and contribute towards another prosperous year at Qlik! I didn’t want this to be another one of those “Year-in-review-type” posts…..so…I cleverly disguised it to highlight some of our product achievements during 2017. Think about it as a sitcom clip-show. So much Qlik goodness has transpired over the past year not only is it difficult to cover everything that has happened with the product, but as a consumer, I know it's difficult to keep track of it all, so I ‘ll do my best to summarize and consolidate where I can. This post is a bit long - so for some of you, I expect you to bookmark it and use it for later reference if needed.


ENTER TO WIN $50 Amazon Gift Card - Rules Below


So, let’s begin.

 

For those of you who want the short, short version:

 

  • Qlik GeoAnalytics
  • Qlik Sense 3.2
  • QlikView Converter
  • Qlik Sense Chat Bot
  • Data Connectivity
  • Qlik Sense June 2017
  • Advanced Analytics Integration
  • Qlik Sense September 2017
  • Qlik Sense November 2017
  • Qlik Sense Mobile
  • Qlik's Associative Difference

 

Qlik GeoAnalytics

 

Qlik GeoAnalytics Datasheet1-3.jpgIn January of 2017 – we kicked off the then new year with an acquisition of our Sweden-based partner, Idevio, creator of Idevio Maps. We introduced the recently acquired product as Qlik GeoAnalytics. Currently available as an extension for both Qlik Sense and QlikView, Qlik GeoAnalytics goes beyond traditional “points on a map” and adds a broader range of capabilities to support advanced geoanalytic use cases. During 2018 (no specific time-frame yet) we plan on integrating Qlik GeoAnalytics directly into Qlik Sense, so stay tuned.

 

ICYMI: Introducing Qlik GeoAnalytics

 

Qlik Sense 3.2

qlik_sense_3.2_highlights_0.jpg

 

Soon after, in March 2017 - we announced our first release of Qlik Sense for the year, version 3.2. 3.2 added the ability to easily create time-based measures (Calendar Measures), included additional chart options to support custom colors and color assignments, introduced the QlikView Converter and added a new configuration to improve the performance and stability of multi-node deployments (Shared Persistence)


ICYMI: Introducing Qlik Sense 3.2

 

 

The QlikView Converter

 

ui-qlikview-converter-example-01.png With version 3.2 we included a utility found in the Dev Hub that simplifies converting a QlikView document (.qvw) over to a Qlik Sense app (.qvf). You simply upload your .qvw to the converter to be analyzed, and it will extract the data model, script, measures, dimensions, expressions, variables and even compatible visualizations. The user can then choose what they want to convert to create a starter Qlik Sense app. Next they can create new or modify existing visualizations using the assets that were already available in QlikView.

 

ICYMI: QlikView converter - Qlik Sense - YouTube

 

Qlik Sense Chat Bot

 

1.pngApril 2017 set a new milestone for Qlik by pushing the boundaries of analytics.  We demonstrated the art of the possible using the Qlik platform, its awesome APIs, and various 3rd party technologies - to create the first ever Qlik analytics chat bot. Simply stated, imagine getting your answers to your business questions by asking your digital assistant. Since the original bot debuted at the Gartner BI Bake-off and Qonnections there has been tremendous interest in this concept, which has evolved rapidly over the 2nd half of 2017. There is now a new community group: https://community.qlik.com/groups/qlik-chatbots with many supporting resources and our own Todd Margolis is leading the charge to help enable our community to take advantage of our two open-source Qlik Bots.  Whether you’re a business user and want quick and easy access to your KPIs on the go or you’re a developer and want to extend them with new capabilities or integrate Qlik into your existing bots, we’ve got you covered.

 

ICYMI:

 

Data Connectivity

cloud+blog+2.pngOver the course of 2017 Qlik Sense and Qlik Sense Cloud had many improvements, especially in the area of connectivity. In May we introduced the first integration of our Web Connectors package to Qlik Sense Cloud – starting with access to Twitter, Facebook and Google Analytics.  Then in July we introduced on premise connections from Qlik Sense Cloud starting with MS SQL as well as the Web file connector. During 2017 we delivered over 20+ connectors to Qlik Sense and Qlik Sense Cloud Business including Dropbox, REST, Amazon Redshift, YouTube and many more.

 

 

 

ICYMI:

 

In addition to the aforementioned connectivity we also debuted our new release schedule and product naming convention when delivering Qlik Sense June 2017. No more dot releases for us – new releases are now identified by the month they are delivered in. June 2017 brought us new visualizations (Histogram, Distribution and Box plots), enhanced visual data preparation features that support data profiling and data quality and a new Advanced Analytics integration framework – allowing you integrate with advanced analytics engines such as R and Python.

 

 

7-20-2017+5-40-37+PM.png


Qlik Advanced Analytics Integration (AAI) enables direct server to server data exchange between Qlik Sense and 3rd party calculation / analysis engines via a connector or what we also call a plugin. So for example, a specific forecasting function that is part of an R library can now be called from within Qlik script and chart expressions and calculated on the fly. Passing the results back to the Qlik analysis engine and subsequent visualizations. With this capability we now support the APIs that provide connectivity to such engines and provide starter connector projects for R and Python.

 

ICYMI: Introducing Qlik Advanced Analytics Integration

 

Qlik Sense September and November 2017

 

9-5-2017 1-20-11 PM.png

Continuing our commitment to deliver iterative improvements and features every 10 weeks, we released our September and November 2017 releases, which introduced our new Waterfall chart, continuous transformations using visual data preparation, our Qlik Sense Mobile app, visualization and navigation improvements and the introduction of our extension certification pilot.

 

 

 

ICYMI:

 

Qlik Sense Mobile (offline)

 

552x414bb.jpg
Originally released to select participants in June, the Qlik Sense Mobile app for iOS became available to customers and partners in the Apple App Store later in the year. This iOS app enables its users to connect to their Qlik Sense Enterprise Server and download a local copy of their Qlik Sense app directly to their iOS device where they can analyze data on the device just as if they were connected to the server, allowing them to take Qlik Sense with them on the go in a disconnected state. (at this time Qlik Sense Mobile is for iPad only – a version for the iPhone is expected to be release early this this year.)

 

 

ICYMI:

 

Qlik’s Associative Difference

 

assoc_thumb.pngFinally, analytics is more than just creating charts, it’s about finding answers. Therefore, I’d like to give an honorable mention to our team, who has created resources to help our prospects, customers and partners clearly understand the benefits of Qlik’s Associative Difference, powered by our patented associative technology. Sometimes referred to by many different names (Associative Experience, Associative Model, Green-White-Grey), Qlik’s Associative Difference separates us from those traditional query-based visualization tools. Qlik’s Associative Difference allows you to explore data freely in any direction, automatically associating every dimension in the data model which visually reveals how the data are related using the colors green, white and grey - AND -  without leaving any data behind.

 

ICYMI:

 

Well guys – that’s it – your Qlik 2017 Year In Review – oh wait. - I said I wasn't doing a "year-in-review" post, ah @#$%&!.

 

Personal Giveaway Rules:

 

Do you have some interesting Qlik highlights or discoveries during 2017 that you would like to share? We want to know. Post them in the comments below OR if you would like to participate in my personal giveaway with a chance to win a $50 Amazon Gift Card (claim code) post a tweet using twitter http://twitter.com/ with your 2017 Qlik highlight or discovery and tag me using @mtarallo along with the hashtag:

 

#Qlik2017Highlights

 

...you will then be entered into a reviewed, random drawing to receive $50 Amazon Gift Card(claim code). Winner will be selected using Qlik's Twitter Connector and Qlik Sense app which will draw a random tweet using the hashtag: #Qlik2017Highlights. Winner will be announced on Twitter by @mtarallo on February 6th 2018 [UPDATE: Feb 13th 2018] and in the Tuesday edition of the Qlik Design Blog. Entry must be an interesting or valid Qlik highlight or discovery - if not it will be discarded and the next entry will be selected at random.

 

Contest starts Tuesday Jan 9 2018 and ends on Tuesday Jan 30 2018. We want to hear from you!

 

Enjoy!

 

Regards,

Michael Tarallo (@mtarallo) | Twitter
Qlik

Qlik_BOT_Headers_Skype.jpg

 

Last month I talked about our new QlikBotNode, setting it up and getting it ready for Telegram Qlik Sense Bot with Node.js and Enigma.js.

 

Today, I will talk about the steps needed to make your bot available for Microsoft Skype as well!

 

Assuming that you have already forked the code from GitHub - qlik-bots/QlikBotNode: QlikBotNode is a server build in Node.js, Express.js, Enigma.js and MySql that connects … and you have already went through the previous tutorial and set up the Telegram bot.


Installation


  • First you need to "Register a Bot with Bot Service". Login to the Azure Portal http://portal.azure.com/
  • Click the New button found on the upper left-hand corner of the Azure portal, then select AI + Cognitive Services > Bot Channels Registration.
  • Click the Create button to start the creation process.

registration-create-bot-service-blade.png

 

  • Enter a Bot Name as you want to appear in your contact list
  • You can use the free Pricing Tier for testing
  • For the Messaging Point add you full route like https://{your-server}/api/sense-bot/microsoft/
  • You can skip the Application Insight for now
  • Once registered, go to settings and get the AppId and the Password. You will need to add them in your environmental variables

registration-settings-manage-link.png

 

  • Click on Manage to get the password

registration-generate-app-password.png

 

  • Put the AppId and Password in your Environmental variables.
  • For Windows, go to "Control Panel" -> "System" -> "Advanced System Settings" -> "Environment Variables"
  • - For Linux, from your directory (~), type "nano .bash_profile" or "vim .bash_profile" and enter the variables there like
  • export SKYPE_BOT_ID_SCV=''
    • export SKYPE_BOT_PASSWORD_SVC=''
  • While still in settings, go to "channel" and add skype.


Thats it!


For more Details on Register a bot with Bot Service (https://docs.microsoft.com/en-us/bot-framework/bot-service-quickstart-registration)


Usage


  • Type help for all of the available commands.

chat-help.png

 

  • Click on Salesforce to get the available commands for the Salesforce App and then click on the Dashboard to get a list of KPIs

chat-salesforce.png

 

  • Click on CIO Dashboard and then on Management

chat-cio.png

 

  • - If you want to change your language click on Language and select one of the available ones. So far we have only 3, English, Spanish and Greek but please feel free to add as many more as you want

chat-language.png

That's it!

 

Make sure you follow:

 

 

Best,

Yianni

Filter Blog

By date:
By tag: