The June, 2011 Gartner report, “Emerging Technology Analysis: Visualization-Based Data Discovery Tools” (available only to Gartner subscribers) is chock full of insights. Here are a few of my favorites:

  • Business users are driving adoption of data discovery platforms. Because data discovery tools have light infrastructure requirements, are fast to deploy, and have relatively short sales cycles, they are spreading to places traditional BI tools haven't been able to touch. The gap between traditional BI platforms and data discovery platforms is widening because business users find the benefits of data discovery tools so compelling that they make this choice despite the risk of creating fragmented silos of data, definitions, and tools.
  • The data discovery platform will surpass $1 billion USD in sales as early as 2013. Data discovery tools comprise one of the fastest-growing segments of the BI platforms market, mainly because they enable self-service BI. Gartner expects that by the end of 2013, data discovery tools will surpass $1 billion in annual software sales, and that until 2015 data discovery tools will outgrow the overall BI platforms market by a factor of three.
  • IT organizations should consider deploying data discovery platforms as analysis sandboxes. Gartner recommends that instead of trying to prevent the spread of data discovery tools, IT organizations should position these tools as analysis sandboxes. (See related QlikView blog posts, “Analysis Sandboxes: Indispensable Tools for Insight Discovery” and “Traditional Analysis Sandbox Approaches Fall Short” and “Analysis Sandboxes the QlikView Way.”) Gartner points out that business users who "play with" these sandboxes will make discoveries that can then be shared with the broader user community, and that new data models can serve as rapid prototypes for sanctioned, corporate data models. 

What Is a "Visualization-Based Data Discovery Platform," Anyway?

Below are Gartner's descriptions of the characteristics of visualization-based data discovery platforms, as well as some specifics about QlikView's capabilities.

 

Characteristics of visualization-based data discovery platformsQlikView's capabilities

A proprietary data structure to store and model data gathered from disparate sources, which minimizes reliance on predefined drill paths and dimensional hierarchies

With QlikView, developers or business users extract the data needed for analysis from multiple sources (e.g., spreadsheets, web services, databases, and enterprise applications). This extracted data is stored in a single QlikView file (.QVW). The QVW also contains the user interface and scripts—everything the user needs to perform analysis. QlikView holds all data needed for analysis in memory, where it is available for immediate exploration by users. Users can quickly and easily see relationships and find meaning in the data, for a quick path to insight. A user can continue to click on field values in the application, further filtering the data based on additional questions that come to mind.

A built-in performance layer that obviates the need for aggregates, summaries, and precalculations

With QlikView, users experience zero wait time as the QlikView engine performs the calculations needed to deliver the aggregations users request. QlikView stores common calculations and shares them among users, so they don’t have to be recalculated every time someone needs them. QlikView also compresses data down to 10% of its original size. As a result, QlikView can scale to handle very large data sets rather than duplicating hardware investment costs to simply move the entire data set into memory. And unlike technologies that simply “support” multi-processor hardware, QlikView is optimized to take full advantage of all the power of multi-processor hardware, thereby maximizing performance and the hardware investment.

An intuitive interface that enables users to explore data without much training

Business users can get up and running with QlikView in no time at all. Once they learn the core concepts of QlikView's associative experience, they can immediately begin exploring data. Users can ask a question in QlikView in many ways, such as lassoing data in charts and graphs and maps, clicking on items in list boxes, manipulating sliders, and selecting dates in calendars. Instantly, all the data in the entire application filters itself instantly around these selections.

With QlikView, users can literally see relationships in the data. They can see not just which data is associated with the user’s selections—they can just as easily see which data is not associated. The user’s selections are highlighted in green. Field values related to the user’s selection are highlighted in white. Unrelated data is highlighted in gray.

 

Gartner summed up its take on data discovery tools with: “Although still a small part of the market, visualization-based data discovery tools have far-reaching implications for how business information is consumed, and will take an increasingly large stake as the front end for analysis, querying, exploration and ‘dashboarding.’”

At QlikTech we are seeing that business users (not “end users”) are buying QlikView to explore and solve business problems (not data problems). Thus the emergence of the term Business Discovery. For more info, please see the QlikView White Paper, “Business Discovery: Powerful, User-Driven BI.”

At QlikTech, we’re in the throes of the Business Discovery Tour. We’re travelling the globe, talking about QlikView and Business Discovery to anyone who will listen . QlikView product advocate Donald Farmer has been ratcheting up the frequent flyer miles as he keynotes many of these events. We recorded Donald’s presentation at our event in Chicago in June. You can watch the video on YouTube.


In this presentation, Donald shared some of his perspectives about Business Discovery:

  • The term reflects who’s making buying decisions. The term “Business Discovery” is not just another marketing term. It reflects an important change in the BI software market. In the January, 2011 Gartner “Magic Quadrant for Business Intelligence Platforms,” the analysts wrote, “With ‘ease of use’ now surpassing ‘functionality’ for the first time as the dominant BI platform buying criterion in research conducted for this report, vocal, demanding and influential business users are increasingly driving BI purchasing decisions, most often choosing easier to use data discovery tools over traditional BI platforms — with or without IT's consent.” Donald remarked that this is a really important change not so much in the tools, or the way people use technology, but in the way they are buying. What we are seeing at QlikTech is that business users are buying QlikView to explore and solve business problems, not data problems. Thus the emergence of Business Discovery.
  • Business Discovery was born out of the consumerization of technology. An example Donald gave is the iPad. He remarked that 90% of iPads are purchased by individuals but 60% of them are used for business. What this means is that people are buying their own business technology. This is very different from the way it used to be. The consumerization trend is not just about devices, it’s also about software. Consumers have grown accustomed to getting answers to questions with a straightforward Internet search, collaboratively making decisions using social sites and tools (“People don’t trust data, they trust other people”), interacting with “focus, personal, and shared” apps, and taking their mobile devices everywhere they go.
  • Business Discovery marks the “end of the end user.” Donald expressed some serious dislike for the term “end user,” commenting that there are no end users any more. He said there used to be end users of BI tools because all people got were reports, analysis, charts, and printouts. But in modern BI, there are no end users. Today when people get BI reports they export the data to Excel so they can do something else—something more—with it. Traditional BI has become an expensive ETL (extract, transform, and load) technology for Microsoft Excel users. Business users want to go further, to do more. They are not at the end of a process.

A new class of consumer is bringing skills learned outside of work into the workplace. It’s not just young people; iPad usage is often driven by more senior members of the organization. The consumerization of technology has lead to a new behavior in business intelligence: Business Discovery. And QlikTech is, according to Gartner in the 2011 Magic Quadrant, the poster child for a new user-driven approach to BI. For more info, please download the QlikView White Paper, “Business Discovery: Powerful, User-Driven BI.”

Hans Rosling is a global health expert and data visionary who is a master at presenting complex data in a way that’s easy to understand. He is a storyteller with a Swedish soul. Here’s a good example, in a 9-minute TED video, “Hans Rosling and the magic washing machine.” This video is a great example of storytelling with data.

Rosling started out his presentation with a story, re-enacted on the stage: the day his mother got her first washing machine when he was four years old. On the first day it was to be used, even his grandmother was invited to come see the new machine in action. Rosling’s grandmother thought the washing machine was a miracle.

Rosling smoothly transitioned into a discussion about women all around the globe who wash laundry by hand, often with water from streams and rivers, heated with firewood. He then bucketed all of humanity into four categories based on our electricity generation and usage. A few points from his presentation—presented below in words far less effectively than he presented it in his simple, straightforward, and visual way:

  1. The “fire people”—29% of humans—consume <8% of the earth’s fossil fuel energy. Two billion of the 7 billion human beings on the planet live on less than $2 USD a day. They don’t have electricity. They heat water and cook food over open fire. They don’t always have enough food. The fire people use less than one unit of the world’s 12 units of fossil fuel energy (oil, coal, or gas).
  2. The “bulb people”—43% of us—consume ¼ of the energy. Three billion people live between the poverty line and the “wash line” (meaning people have washing machines). The bulb people have electricity but no washing machine. They use 3 units of the world’s 12 units of fossil fuel energy.
  3. The “wash people”—14% of the population—use 17% of the energy. One billion people live above the wash line. They have washing machines, but not a house full of other machines and devices. They live on about $40 a day. The wash people use about 2 units of the world’s energy.
  4. The “air people”—1/7 of the world’s population—consume ½ of its energy. The remaining 1 billion humans spend more than $80/day on their consumption. They live above the “air line.” They have a house full of machines and travel around the globe in airplanes. Air people consume 6 of the 12 units of the earth’s fossil fuel energy.

Rosling wrapped up his presentation with a brief analysis of the future. By 2050, he predicted, total energy usage will be 22 units (compared to 12 units in 2010). The richest people will still use most of the units. Two trends that will affect growth in the use of energy: population growth and economic growth.  The vast majority will come from economic growth. The solution?  We will need more energy efficient machines, changes in behavior, and more green energy.

Rosling said that when he was a kid, he liked the washing machine because his mother would load it up and then take him to the library. “Thank you steel mill, chemical processing industry – you gave us time to read books.” The beauty in wrapping data in a story is that then people can then internalize the meaning of the numbers. And that’s the whole point, right?

Arthur Lee

Metadata Made Easy

Posted by Arthur Lee Jul 19, 2011

In an earlier blog post, “Qlikview’s Pragmatic Approach to Metadata,” we explained how QlikView takes a fresh approach to metadata that doesn’t involve a lot of overhead to create or maintain since it is naturally exposed.  Perhaps the biggest difference is that most other solutions offer a separate module sometimes at an additional cost that is often difficult to set up and maintain.

 

Recently, we released a utility called the Meta Model that helps developers and administrators automate the entire process of understanding the metadata of a QlikView deployment (download here).  With this utility, it is now possible to gain deep insight into QlikView’s metadata.

 

The Meta Model is composed of two QlikView documents:

 

  • MetaScanner.qvw – this is the metadata “collector” application. Run this application each day after your daily reloads are completed and it will collect and accumulate your QlikView metadata into a “Meta Model.”  You can then leverage this Meta Model for many monitoring, cataloging, analysis, and support capabilities.

 

  • MetaMonitor.qvw – this is a fully functional demonstration application based on the Meta Model.  This application is not required as part of the Meta Model solution but can be used as a starting point to discover some of the great things in the Meta Model such as gaining a better understanding of the data lineage, server performance statistics, detailed history of data loads and much more.

Example of the Meta Monitor application.png

One advantage of QlikView’s approach to metadata is that developers can easily create their own applications on top of the Meta Model and/or extend the Meta Model for their specific needs.   You can see one such example at the QlikCommunity Meta Model User Group which is dedicated to this utility. This example uses Google’s charting API to enable QlikView developers and administrators to visually see the data connections within your deployment.

 

If you are using QlikView today, we encourage you to download the utility here and experience the power of the Meta Model.  If you are new to QlikView, you can be assured that the metadata of your future applications can be unlocked without the headaches of most other BI solutions.

In an earlier blog post, “Qlikview’s Pragmatic Approach to Metadata,” we explained how QlikView takes a fresh approach to metadata that doesn’t involve a lot of overhead to create or maintain since it is naturally exposed.  Perhaps the biggest difference is that most other solutions offer a separate module sometimes at an additional cost that is often difficult to set up and maintain[WU1] .

Recently, we released a utility called the Meta Model that helps [EDR2] developers and administrators automate the entire process of understanding the metadata of a QlikView deployment (download here).  With this utility, it is now possible[WU3] [EDR4] to gain deep insight into QlikView’s metadata.


[WU1]Erica – I changed the sentence structure around – hopefully it makes more sense.

[EDR2]Helps who – developers? Administrators?

[WU3]So since I addressed it in the previous sentence – I probably don’t need to repeat it.

[EDR4]Possible for who – developers? Administrators?

In an earlier blog post, “Qlikview’s Pragmatic Approach to Metadata,” we explained how QlikView takes a fresh approach to metadata that doesn’t involve a lot of overhead to create or maintain since it is naturally exposed.  Perhaps the biggest difference is that most other solutions offer a separate module sometimes at an additional cost that is often difficult to set up and maintain.

Recently, we released a utility called the Meta Model that helps developers and administrators automate the entire process of understanding the metadata of a QlikView deployment (download here).  With this utility, it is now possible to gain deep insight into QlikView’s metadata.

The Meta Model is composed of two QlikView documents:

·         MetaScanner.qvw – this is the metadata “collector” application. Run this application each day after your daily reloads are completed and it will collect and accumulate your QlikView metadata into a “Meta Model.”  You can then leverage this Meta Model for many monitoring, cataloging, analysis, and support capabilities.

·         MetaMonitor.qvw – this is a fully functional demonstration application based on the Meta Model.  This application is not required as part of the Meta Model solution but can be used as a starting point to discover some of the great things in the Meta Model such as gaining a better understanding of the data lineage, server performance statistics, detailed history of data loads and much more

In an earlier blog post, “Qlikview’s Pragmatic Approach to Metadata,” we explained how QlikView takes a fresh approach to metadata that doesn’t involve a lot of overhead to create or maintain since it is naturally exposed.  Perhaps the biggest difference is that most other solutions offer a separate module sometimes at an additional cost that is often difficult to set up and maintain.

Recently, we released a utility called the Meta Model that helps developers and administrators automate the entire process of understanding the metadata of a QlikView deployment (download here).  With this utility, it is now possible to gain deep insight into QlikView’s metadata.

The Meta Model is composed of two QlikView documents:

·         MetaScanner.qvw – this is the metadata “collector” application. Run this application each day after your daily reloads are completed and it will collect and accumulate your QlikView metadata into a “Meta Model.”  You can then leverage this Meta Model for many monitoring, cataloging, analysis, and support capabilities.

·         MetaMonitor.qvw – this is a fully functional demonstration application based on the Meta Model.  This application is not required as part of the Meta Model solution but can be used as a starting point to discover some of the great things in the Meta Model such as gaining a better understanding of the data lineage, server performance statistics, detailed history of data loads and much more

In an earlier blog post, “Qlikview’s Pragmatic Approach to Metadata,” we explained how QlikView takes a fresh approach to metadata that doesn’t involve a lot of overhead to create or maintain since it is naturally exposed.  Perhaps the biggest difference is that most other solutions offer a separate module sometimes at an additional cost that is often difficult to set up and maintain.

Recently, we released a utility called the Meta Model that helps developers and administrators automate the entire process of understanding the metadata of a QlikView deployment (download here).  With this utility, it is now possible to gain deep insight into QlikView’s metadata.

The Meta Model is composed of two QlikView documents:

·         MetaScanner.qvw – this is the metadata “collector” application. Run this application each day after your daily reloads are completed and it will collect and accumulate your QlikView metadata into a “Meta Model.”  You can then leverage this Meta Model for many monitoring, cataloging, analysis, and support capabilities.

·         MetaMonitor.qvw – this is a fully functional demonstration application based on the Meta Model.  This application is not required as part of the Meta Model solution but can be used as a starting point to discover some of the great things in the Meta Model such as gaining a better understanding of the data lineage, server performance statistics, detailed history of data loads and much more

Are we taking a simplistic approach when we visualize the data and forget about the information aesthetic? I think so.

 

Data Flow, written by R. Klanten, N. Bourquin, S. Ehmann and F. van Heerden, is a great book that takes a fresh approach to data visualization. To be honest, when I first started reading the book, I was a little bit disappointed as I was expecting more “how-tos” on displaying information graphically. Instead, the book approaches information visualization from a design perspective. But by the end, it really helped me see how differently we can think when we visualize data in new ways.

 

The authors talk about the importance of presenting data in a form dictated by the viewer’s needs, rather than the needs of the originator, the limitations of visualization tool, or even the data itself. Data Flow examines the various metaphors and uses the cultural, emotional, and existing world of the viewer to visualize information.

 

SearchClock.png

Source: Search clock by Chris Harrison, Data Flow pg 14

 

In business, we are quite used to using pie charts, line graphs, and diagrams as tools to visualize data. The authors advocate that we challenge and playfully re-engineer the overly simplistic approach of “business data = pie chart.” What about imagining data as a cloud, river, or body?

 

The example shown above is called a “search clock.” Inspired by an example provided in the book, I created this visualization in QlikView. It shows a great way of using multiple dimensions on a pie chart while still showing the distribution of search data by hour over time—but in a more entertaining and engaging way. This search clock displays the most popular search terms for each hour of the day from 1997 to 2000. It provides insight into how people used the web. The chart shows that the earliest searches were dominated by a desire for contact with people. Over time, the trend moved towards people looking for information. This insight would be hard to glean from a traditional pie chart, line chart, or bar chart.

 

Pie charts are one of the basic visualization tools for exact comparison, useful for highlighting the effect of large differences. This example shows how we can take the two-dimensional nature of pie charts a step further, blending layer upon layer to create stories behind the data.

 

In Data Flow, individuals creating data visualization are called “Information Designers.” These are individuals who shape an experience, or view, of the data with a particular goal in mind. Maybe we should all take a step back while creating visualizations for business users to think about their needs and design charts and other visualizations that tell stories behind the data in a more engaging way.

The other night I was at home listening to music and a thought occurred to me. The way I discover new music today is so different—and so much more satisfying!— from the way I discovered music 10-15 years ago. Similarly, the way people explore and interrogate data at work for insights and decision support is very different—and much more satisfying—from the way it was done in the past.

Today, music discovery is mobile, instant, social, and app-driven

How I discovered music in the pastHow I discover music today
Influence of friendsI would hear new music I liked while at a friend’s house or riding in their car. I’d write down the name of the band so I could look for the compact disc in a record store at a later time.My friends frequently post YouTube videos of music they like on Facebook. My friend David has particularly good taste; I always check out what he recommends. I can also follow bands on Facebook and connect with my friends via Ping, which is iTunes’ social network.
Discovering music in the carI listened to whatever was broadcast on the few channels I received on FM radio. If I liked a song, I hoped the DJ would say the name of the band so I could write it down. If the DJ didn’t announce it, I had to wait until I heard the song again.I listen to satellite radio, with hundreds of music channels to choose from. When I hear a song I want to buy, I hold my iPhone up in the air and click the big orange button on the SoundHound app. The software recognizes the song and stores metadata about it for me.
Discovering music in the storeI would head to Tower Records—now defunct, like so many other record stores. I’d pick up the CD I’d heard at my friend’s place, and wander through the pop and rock aisles, where CDs were arranged in alphabetical order. I would browse through racks of CDs to see if any of my favorite bands had new albums out. Pretty much the only way to explore music by bands I’d never heard of was to stand at a station where I could don a set of well-worn headphones and sample tunes from a half dozen CDs the record store was promoting.My music store is now online. I go into iTunes to buy a few songs I had stored in my SoundHound list. Searching is easy. If I want to find Coldplay’s new song, for example, I can find it by typing the words “Coldplay” and “waterfall” into a search box. The song “Every Teardrop Is a Waterfall” shows up in a list alongside other songs in that album, and other songs by Coldplay. I can click around, listening and exploring. By clicking on links, I can pursue a trail of tunes, discovering songs I like. Exploring a compilation or soundtrack, for example, takes me to artists I had never heard of before. I wander through the music, exploring and making discoveries. Also, on the home page of the iTunes store, I can see recommendations iTunes has specifically for me based on my purchase history. There’s also iTunes Genius, which creates playlists for me and introduces me to music I might like but haven’t already purchased.
Purchasing musicEven if I liked just one or two songs on a CD, I had to purchase the entire album.I buy individual songs based on my tastes and preferences, and based on personalized recommendations from my friends and iTunes. I can purchase songs from iTunes through SoundHound, right on my iPhone. Or I can wait until I’ve collected a handful of songs in my SoundHound history and then sit down with my laptop for an evening of musical exploration.
Playing musicI listened to one album at a time or—when I got a fancy 6-CD player, I could mix and match songs from 6 CDs and play them in random order. Or I could burn mix CDs—hard-coded collections of a dozen or songs.I can play songs by any artist in any order I like. I can easily organize my music any way I want. I can create playlists as large or small as I want, based on mood, memory, group of artists, or any other criteria.

 

Likewise, Business Discovery is mobile, instant, social, and app-driven

A similar change has taken place in the workplace, as people use Business Discovery platforms to derive insights and inform decisions. Traditional BI solutions give users access to lots of data—just as traditional, physical record stores gave shoppers access to lots of music. But how does the user explore all that data, to discover meaningful relationships and associations, and pursue their own path to insight? With traditional BI solutions, users can explore pre-determined drill paths (like the alphabetically-arranged CDs on shelves ordered by genre) and preconfigured queries (like the headset in the store, through which I could sample the music from a half-dozen new CDs).

In contrast, with Business Discovery platforms, users can search and explore all the relevant data at their disposal (like exploring all the music in the entire iTunes store). They can access their dashboards, analytics, and reports from anywhere, with their mobile device. They can click around in the application, see clearly what data sets are related and unrelated, and quickly ask and answer their business questions (like quickly finding new music I like, buying it, adding it to a playlist, and immediately beginning to enjoy it). They can draw others into the discussion, and share their insights and perspectives with a larger group.

I shared a draft of this blog post with Elif Tutuk, technical advisor, and she had a great perspective to share: “Business intelligence should not be only about the charts, visualization, numbers, reports etc. It should be a user experience where business users get information even before they search. Imagine a world in which BI tools give users multiple perspectives on information displayed on charts, as users interact with the chart. Or, as BI gets more social, imagine a world where BI tools display to users the top 10 charts viewed by others when they do a search.” The future of BI is exciting!

Lots of QlikTech’s customers are incorporating data from Salesforce.com into their QlikView apps. Sales reps use QlikView with Salesforce.com data to track to their targets and stay on top of their potential commissions. Sales management uses it to analyze opportunities, study win/loss ratios, and track team performance. Marketing managers use it to analyze campaign effectiveness, determine which marketing campaigns to run, and create product bundles for cross-selling and up-selling.

Fig 1 graphic for salesforce connector blog post.png

In fact, here at QlikTech we use QlikView with Salesforce, ourselves. We’ve built an application that our field marketing managers and marketing operations team use to populate marketing campaigns with campaign targets. Field marketing managers use QlikView to narrow down the targets for inclusion in a campaign. They can filter by sector, industry, sub industry, SIC code, revenue, number of employees, data source system, job level, job function, lead source, and many other criteria. Once a marketing manager has finalized the list of people to target in the campaign, with a click of the button the campaign in Salesforce is populated with the list.

Fig 2 graphic for salesforce connector blog post.png

Exploring Salesforce.com data in QlikView is easier than you might think

While QlikTech has supported Salesforce data for quite some time now, we recently released a brand new version of our Salesforce connector. QlikView for Salesforce.com is a custom connector that enables our customers to extract data from Salesforce and bring it into QlikView, just as they would from any SQL database, spreadsheet, or web site.

I talked recently about the new connector with Ian Crosland, global product manager for connectivity. The new connector:

  • Has been rewritten and is now managed in-house. Prior to the current version, we provided an open database connectivity (ODBC) connector to Salesforce.com that we had OEMed from a third party. To streamline the code and bring the connector up to date, we brought it in house and rewrote it as a dynamic link library (DLL).
  • Is simple to install. Once you’re sure you’ve got the prerequisite security settings and security token from Salesforce.com, it takes just a few minutes to install QlikView for Salesforce. Select the tables you want and start working with your Salesforce data instantly.
  • Offers new functionality. The new connector now enables customers to pull archived records from Salesforce into QlikView and extract data from Salesforce Chatter. It also supports the latest API from Salesforce (API 21).

For more information, see the data sheet “QlikView for Salesforce.com” or click here to download the connector. You can also interact with or download a demo app showing QlikView for Salesforce here.

Filter Blog

By date:
By tag: