Business Discovery Blog

11 Posts authored by: john Callan

For a Business Discovery platform to meet the expectations of today’s information worker (fast response times, high degrees of interactivity, self-service data exploration and discovery) and scale across an enterprise, it’s now widely accepted that the use of in-memory processing is required.  Here’s a quote from our partner Teradata, which comes from a disk-based heritage: “Naturally for the data which is being used heavily on a day to day basis then there will a more than convincing business case to store this data in-memory to deliver the performance which is required by the business” (http://blogs.teradata.com/anz/are-in-memory-databases-the-answer-or-part-of-the-answer/ )

 

This is no surprise to QlikTech as this is the approach we pioneered 20 years ago, and is now being taken up by pretty much all competing vendors.

 

However, we sometimes come across claims that visualization tools querying direct to disk-based databases are a viable alternative approach.  To suggest that a deployment that only utilizes a dynamic query to disk approach will meet performance expectations is simply not a reality. While some business discovery providers (including QlikView via the Direct Discovery capability) can directly query sources such as Teradata, it’s important to acknowledge that direct query alone is a) much slower and b) utilizes network traffic in an unbounded fashion. Whilst a direct query capability such as Direct Discovery is a very valuable ‘relief valve’ for access to very large data sets, ALL data discovery providers (including QlikView) recommend the use of a performance optimization layer.  In fact, this is one of the defining characteristics of data discovery software according to Gartner (data discovery is their term for Business Discovery):

 

”Data discovery tools are an increasingly prominent class of BI offering that provide three attributes:

1. A proprietary data structure to store and model data gathered from disparate sources, which minimizes the reliance on predefined drill paths and dimensional hierarchies.

2. A built-in performance layer that obviates the need for aggregates, summaries or pre-calculations.

3. An intuitive interface enabling users to explore data without much training.”*

 

The reality is that any BI system meant to satisfy business users has to replicate some or all of the data to deliver acceptable performance. Different vendors take different approaches; QlikView uses its associative in-memory engine (which offers up to 90% compression of source data), other vendors use less intelligent in-memory caches, but all the same, they still replicate data. For 20 years QlikTech has developed an in-memory approach that provides a unique high-performance, associative, intelligent data store. In addition we have developed tooling that very effectively manages the data to allow QlikView deployments to scale to many thousands of concurrent users. Any vendor claiming to deliver genuinely useable, fast discovery without recourse to some data replication in memory (or the use of an in-memory database further down the stack – still a rarity) is misguided.  

 

Related content: QlikView Scalability White Paper. QlikView Architecture and Systems Resource Usage Technical Brief

*Source: Gartner ‘The Rise of Data Discovery Tools’, 26 September 2008, ID:G00161601

With the “Enabling the New Enterprise” theme of “QlikView.next” (the code name for the next generation of the QlikView Business Discovery platform), IT pros will be able to optimize their QlikView environments and offer self-service Business Discovery to growing numbers of users while utilizing ever-growing volumes and variety of data.

Blog Post Oct 2012 pic.png

 

This theme is about making security, user management, availability, and scalability easily manageable by any customer, not just the largest ones. We want to make QlikView extremely easy to administer and give administrators the same kind of gorgeous and genius experience other users get, with several product scenarios:

  • Global Deployments. With this scenario we will deliver cross-geography clustering (ensuring higher availability) and software licensing to support the new way of working. We will also provide tools and documentation to make it easier for system administrators to recover QlikView servers when a machine fails and another one needs to start up.
  • Improved IT Insight. We are re-imagining the user experience for IT professionals, focusing on making it easier for IT pros to manage users and licenses regardless of whether the deployment covers ten or tens of thousands of users, is in a single or multiple geographies, or has one app or hundreds. One way we will accomplish this is with “QlikView on QlikView”– an interface to the management console for embedding a QlikView app that will enable IT professionals to monitor the health of their environments and identify problem areas and opportunities where they can take proactive action.
  • Safe and Secure. This scenario is about making QlikView security management simple for administrators to understand and configure according to their organization’s policies and compliance requirements. This scenario has two main facets: security standardization and support for emerging technologies.
  • Streamlined Application Deployment. This scenario will make it easier to build and deploy QlikView apps. This scenario is focused on making it easier for developers, testers, and approvers to move QlikView applications through the development, test, validation, and production supply chain in a simple and straightforward way. QlikView users will be able to more easily push QlikView applications into a workflow and through the deployment process.

To learn more about “QlikView.next,” the code name for the next generation of the QlikView Business Discovery platform, download the white paper, The “QlikView.next” Product Scenarios.

I recently gave two presentations on trends in BI at IT conferences in Sweden and South Africa and one of the trends we are seeing very clearly is the trend towards enabling discovery:

 

Discovery (dis-ˈkə-v(ə-)rē): Find (something or someone) unexpectedly or in the course of a search. (ref. Google)

 

To understand the reasons for this we need to understand why discovery is such an important aspect of the decision-making process that BI is supposed to support. As the definition states, discovery is about finding something unexpectedly during a search. While many discoveries are irrelevant and ultimately useless, others are so groundbreaking in their significance as to make all other useless discoveries worth the waste of time.

 

Discovery_Fleming.jpg

                                                                                                                                                                          (Source: www.creativeclipsonline.com)


I wonder how many useless discoveries Sir Alexander Fleming made before he went on vacation in 1928, inadvertently leaving a petri-dish filled with bacteria to grow a mold that would ultimately be the genesis of the anitbiotic that has had such a profound impact on humankind over the last 80 years. The discovery of penicillin was neither expected nor even the reason why he was conducting his research. But it was discovered nonetheless.

The scientific world is full of examples of game-changing discoveries like Sir Alexander’s. Some have a smaller impact than his, yet are still very notable. From the discoveries that led to Post-It Notes, Velcro, cellophane, brandy (cognac), the microwave, Teflon, even Viagra: the world is full of examples of individuals or groups discovering something unexpectedly in the course of a search (often for something else!).

Discovery isn’t – and shouldn’t – be limited to the scientific community. Discoveries happen every day in the business and consumer worlds. In business, discovering that your company is selling more of a product in one region than another, or that on a given day of the week, at a particular time, the quality of product on the production line drops, or that when Product A is placed alongside Product B on your store shelves both products sell more – all of these have an extremely valuable place in our professional lives.

Up until recently, our business software has failed in its attempt to facilitate the discovery process. This is not an indictment of the people charged with implementing and running the projects; rather it is a reflection of the fundamental architectures that the traditional BI vendors were forced to choose when decision support became prevalent. Constrained by disk-based and query-based architectures because of the high cost of memory at the time, the natural free-flowing exploration of data that might ultimately lead to discoveries was not permitted to occur. In order to try to alleviate this restriction, the notion of OLAP and cube-based technologies came to the fore, but without the desired results. The fundamental restriction was the underlying reliance on disk-based and query-based technologies that were a) too slow and b) technically difficult to master. Ultimately, what BI became was the provider of standardized reports with well-defined KPIs and other metrics and dashboards which offered the users a limited data exploration experience.

 

Discovery_EndUser.jpg

It was here that "end users" truly became END users; they were the end of a process of data collection, modeling, and cleansing, and report generation. At QlikTech we talk about the "end of the end user." With Business Discovery, the person actually using the software to make decisions through interesting discoveries is not thought of as the "end" of anything. Rather, they are the beginning of something. The beginning of the discovery process. The beginning of a process that allows them to ask and answer their own questions without restriction, without needing to follow pre-defined and restricted exploration paths.

The trend towards discovery in BI has been made possible with QlikView, which adopts a radically different approach to data access and analysis. The dramatic reduction of the cost of memory coupled with innovations such as associative data modeling and data exploration has given business users the ability to explore data to encourage those key discoveries. Gone are the restrictions around old, pre-defined data exploration paths, and are now replaced by an unrestricted approach to data discovery where business users can ask and answer their own questions, regardless of how the data has been structured further up the chain.

Imagine if Sir Alexander had been restricted by not being allowed to see the famous mold forming on his petri dish. In a traditional BI world, this path of insight would have been shut off to him, simply because it wasn’t something that had been pre-determined.

In this, the final part of the series examining the role of IT in Business Discovery, I highlight how IT professionals extract great value themselves in using QlikView for their own Business Discovery purposes.

 

Part 4: Business Discovery for IT: Making sense of all those log files

 

In addition to being the central element in supporting QlikView deployments, IT groups have adopted QlikView themselves to address their own business discovery requirements. QlikView is used by IT professionals in hundreds of organizations to help them analyze and make sense of system log files, helpdesk ticketing and asset management requirements (i.e. SLA monitoring, location and status of laptops, desktops, mobile devices, server infrastructure).

 

Below is a screenshot of a QlikView application that allows IT managers to monitor their helpdesk operations. Managers can see a high-level dashboard and can drill down into specific operators, departments and customers and can look at the volume of calls per shift, average wait time etc.

   Helpdesk.jpgBusiness Discovery for IT: Helpdesk monitoring

 

Another good example of IT professionals using QlikView is for the management of assets under IT control (e.g. laptops, smartphones). In many organizations these assets are in a constant state of flux – some fall out of warranty, some need repairing, some need replacing completely. The application below (and located on our demo site here) is a very good example of an solution that allows IT hardware managers to monitor their assets.

 

Asset Management.jpgBusiness Discovery for IT: IT Asset Management

 

This concludes the series on the role of IT in Business Discovery. In this series I’ve outlined how Business Discovery liberates IT and allows them to focus on areas that are core to their skill-set: data preparation, data and application governance and infrastructure provisioning and monitoring. Because Business Discovery enables a true self-service approach to BI, IT professionals can remove the backlog of requests for new reports and new queries from the business.

With QlikView, The business drives BI at the micro level while at the macro level the IT BI group facilitates enterprise initiatives like governance, data standardization, security and scalability.

In this, the 3rd part in the series of posts about the role of IT in Business Discovery (see Part 1 here and Part 2 here), I'll explain how IT groups play a central role in ensuring that there is proper control and governance over Business Discovery deployments while ensuring that they still can continue to empower the business functions to be creative and flexible.

Part 3: Business Discovery: Providing Flexibility at the Edges while ensuring Discipline at the Core

Many IT professionals get worried at the mention of ‘Self Service’ anything. And for good reason: with limited resources, IT groups must ensure that the mission-critical systems that the organization uses are operational at all times and are providing the service they were originally provisioned for. The only way to achieve these service levels is to ensure a degree of standardization and control. For example, imagine the support nightmare if everyone in the organization was using a different email system? Or if there were 100 different operating systems to support? Standardization has ensured that businesses can run profitably for a long time.

Disc at core Flex at edges2.jpg

 

 

Unfortunately, too much standardization and process can very often get in the way of the business’ legitimate needs to stay ahead of the competition and remain profitable and growing. This has led to the classic chasm between IT groups and business groups that is too prevalent today.

Business Discovery is user-enabled BI and provides a largely self-service approach for business users to interpret their data in the manner they wish so that they can remain effective in meeting the needs of the business. It allows them to ask and answer their own questions – instantly - explore and make discoveries in the data, without having to constantly return to the IT department for every new request. This is made possible because QlikView’s "app approach" uses a pre-built data model (i.e., it is not a direct query solution) meaning that business users have all the data they need for a given analysis. QlikView also provides associative data capabilities that allow users to navigate in the data in any manner possible, ensuring that they do not need to ask IT every time a new drill down path needs to be created. I encourage you to refer to the white paper “What Makes QlikView Unique” for a deeper understanding.

Successful Business Discovery implementations strike a healthy balance between the needs of the IT organization to ensure control and standardization and the business sides’ needs to remain flexible. QlikView provides a rich set of capabilities to meet the needs of IT when it comes to ensuring control, including (but not limited to):

  • Multi-tiered environments, including data, application and presentation tiers
  • Security integration with Active Directory and almost every single sign-n solution
  • Built-in row-level security, linked to LDAP roles
  • Straightforward, yet extremely powerful, automated metadata on everything from data usage/lineage, application usage, common KPIs, license usage.
  • Clustered/failover environments to ensure SLA adherence
  • External triggering and alerting of data reloads and application availability
  • Management Services API for integration with existing IT command-and-control infrastructure
  • Governance best practices for application promotion (Dev/Test/Prod), data usage, application usage
  • Direct integration with SAP, Salesforce.com, Trillium, Informatica and others
  • Integration with source control systems for deployment and change management

In the final part of the series, I'll show how IT professionals can use QlikView themselves, to support them in their own decisions about systems usage, SLA monitoring, IT asset management and more.

In my first post in this series about the role of IT in Business Discovery (located here), I talked about how Business Discovery is helping IT professionals to become champions to the business once again because it liberates them from the mundane and - frankly - inefficient tasks associated with Traditional BI (such as constant report writing, for example). This 'Win-Win' situation is helping fuel the enormous expansion of QlikView's impact across thousands of organizations across the world every day.

 

In this, the secord part of the series, I examine the various business and IT roles specifically and their impact on a Business Discovery deployment.

 

Part 2: Business and IT: Sharing the benefits together by understanding everyone's role

 

IT professionals have varied roles to contribute within a Business Discovery deployment. It’s quite typical for there to be an ‘overlap’ point between the business functions and the IT group. Business Discovery fosters a closer and more productive relationship between the business and IT professionals because this overlap point promotes a very collaborative approach to data provisioning and application design and –ultimately – application usage.

 

Roles in BD.jpg

 

The image above (click to expand) shows a simplified and generalized view of the various ‘actor’s' in a typical Business Discovery deployment, from both IT and the business. It’s worth remembering that the roles are often blended together, particularly in smaller organizations.

 

  • CIO/VP of IT: Their interaction with QlikView is typically by means of a dashboard view of IT-operations data such as asset management and procurement, systems SLA KPI’s (Service Level Agreement Key Performance Indicators) and headcount. For the CIO or VP of IT, a Business Discovery implementation means more than just dashboard views however: They are charged with ensuring the IT function supports the business as efficiently and effectively as possible. In the provisioning of Business Intelligence, QlikView enables them to grow their BI throughput without growing the costs associated with this growth because of its speed of deployment, more efficient use of IT resources and the secure self-service nature of QlikView.

 

  • Enterprise Architect: Once deployed, an Enterprise Architect will typically analyze usage, configuration and capabilities delivered by QlikView in the organization, ensuring that the correct infrastructure resources are being provisioned and that security policies are being adhered to correctly. The Enterprise Architect will also assess the integration capabilities between QlikView and other tools in the enterprise information chain. In the procurement process, the Enterprise Architect has a critical role in ensuring that QlikView will fit within the organization’s existing infrastructure and governance models.

 

  • Data Analyst: Their role in a centralized, enterprise-wide deployment of QlikView is one of provisioning data models that meet the changing needs of the business. In typical organizations today, new data requests are frequent and the role of the Data Analyst is to ensure correct ETL (Extract Transform and Load) models are in place and that the data being provisioned is of high quality. Data Analysts will use QlikView Developer to build the ETL scripts, create –and maintain - the data layer and ensure that data is both relevant and current.  They will also use the many free QlikView utilities to monitor and support the data layer within QlikView.

 

  • Business Analyst: They have a critical role in any Business Discovery deployment: they are typically the ‘crossover’ point between IT and the business functions. In many instances – particularly in larger organizations – BA’s will be attached to the business directly. The BA will typically use directly, or change, existing data models based on their own needs (usually provisioned by a Data Analyst) and will build the applications that are ultimately used by the business users themselves. With the rapidly changing requirements of businesses, BA’s will often create new applications on a very regular basis. This is one of the key tenets of QlikView: the rapid application development process that allows businesses to react a ever-changing environment.

 

  • Business Users: Rather than being ‘end’ users, business users are the start of the discovery process: with the applications that are built with QlikView, they can – themselves -- explore, interact and interpret the data using QlikView’s unique associative technology, without having to constantly return to the IT group to have a new report or query generated. QlikView even allows business users to build their own dashboards –from within the zero-install AJAX web client – using the high quality and secure data that has been provisioned to them from the IT group.

 

 

Existing IT Skills Transfer easily with QlikView


IT departments with existing traditional Business Intelligence deployments can re-use a large proportion of the existing skill-sets for a QlikView deployment. This ensures that QlikView deployments maintain a lower overall TCO (Total Cost of Ownership) and also ensures that QlikView can sit side-by-side with those existing deployments within the same department or BI Competency Center. This topic was covered in the blog post “QlikView and IT: Like Chocolate and Peanut Butter”.  Figure 4 highlights where there is significant skills overlap of general BI skills with those that are needed for large QlikView deployments, such as SQL scripting, data modeling, testing, integration and project governance models. The areas that require new skills training are relatively small and include a knowledge of the QlikView UI environment as well as – for system administrators – the QlikView Management Console. Both areas are relatively straightforward to get familiar with, especially for IT professionals.

 

The image below (click to expand) illustrates the various skill sets that are needed for QlikView deployment, and how many of them typically exist within an organization already:

 

Skills Transfer.jpg

In the next part of the series I'll look at what 'Self Service BI' means for IT professionals responsible for a Business Discovery deployment.

Business Discovery is a new – and different – approach to traditional, hierarchical BI, enabling business users to make discoveries in their data themselves and ask and answer their own questions, without needing to return to IT every time a new query, report or visualization is needed. The role of IT is a critical element for the success of any Business Discovery implementation. QlikView enables a true IT-led Business Discovery approach allowing IT to manage the deployment at the macro level and allowing the business to manage it at the micro level. IT is placed in the center of all successful deployments and Business Discovery enables IT to become champions to the business once again.


This post is part of a series that examines the role of IT in Business Discovery.

 

Part 1: Traditional BI falls short and has cost too much

The drawbacks to the traditional approaches to BI are well documented and understood (see the white paper “Business Discovery – Powerful, User-Driven BI”) A report-centric approach to BI requiring multiple elements of a vendor’s stack and a team of skilled data analysts with deep SQL skills is exacerbated by the real frustration within the business (i.e. non-IT) community of inadequate access to data to help them make decisions due to a backlog in requests for new reports and new queries. IT professionals are frustrated by the wasted investment in building large BI systems due to a lackluster adoption by the business functions that they are supposed to be enabling. “We built it and nobody came” is a disheartening reality for many IT professionals involved in Traditional BI projects. For many, the promise offered by Traditional BI has been seen as a failure. As a result, tensions between the business and IT communities increase, data-driven decision making suffers and – worryingly – business units take matters into their own hands by utilizing the most ubiquitous of BI tools out there – the spreadsheet - resulting in ‘Spreadsheet Anarchy’.

It wasn’t meant to be this way. Business Intelligence – once known as ‘Decision Support’ – was supposed to allow people and groups to make better and more informed decisions because of easy access to and consumption of centralized and approved data. Unfortunately, as often happens, technology got in the way and the promise went unfulfilled.


Business Discovery Liberates IT from Mundane Tasks: A ‘Win-Win’ situation

Business Discovery is a ‘bottoms up’ approach to the age-old problem of data access: rather than following a top-down hierarchical approach, Business Discovery gives the decision makers – the business units – the flexibility to explore their data, find insights and turn that data into information, without requiring a costly and inefficient intervention from IT at every step. The effect on the organization is two-fold: firstly, business units can more quickly make better data-driven decisions, without technology (and lack of deep skills to use it) getting in the way. Secondly, IT organizations can now be freed up to focus on their core competencies because the time previously needed for mundane data access and report writing tasks can now be spent on higher value projects for the organization. Not only does the business side’s productivity increase, IT’s efficiency increases, resulting in a classic ‘win-win situation’.

IT at the center.jpg

Figure 1:The Evolving BI Landscape and IT’s Changing Role

A new approach to ‘Self-Service BI’

The concept of self-service BI is not a new one: some Traditional BI vendors have attempted to enable business units to take some control over data access and report generation. Unfortunately what has happened is that these approaches have not satisfied the business’ needs because they have been narrow in scope, have required heavy maintenance from the very IT professionals whom they are trying to liberate, have poor performance and have been – for most business professionals –hard to use. The idea – while worthy – failed in its execution.


QlikView takes a different approach to self-service BI: based on an inherently easier-to-build and deploy model via its associative in-memory technology, IT provisions secure and ‘clean’ centralized business data, the infrastructure and security privileges, while the business units are given the flexibility to explore the data along the paths that they define, introduce additional local data to augment their discoveries and even build their own applications. All of this occurs under a governance framework within QlikView that IT can tailor to meet the specific requirements of the organization. In this way, IT controls the deployment at the macro level while the business units run it at the micro level.

 

Figure 1 highlights the central role that IT takes in any successful Business Discovery deployment: At the center, provisioning data and services while the business takes ownership of its own analytic needs.

 

Figure 2 provides a simple view of the impact of Business Discovery on an IT organization. With Business Discovery, requests to the IT department are within a range of requests for new data elements and data sources, new data models (e.g. ‘cleansed’ departmental data) and infrastructure support (e.g. license management). With Traditional BI, requests to the IT department are more frequent: new reports, new data queries (including new OLAP cubes), new views on the data, new data access needs and infrastructure support – to name a few. The net result is that IT departments are typically backlogged with requests from the business, which has a negative impact on the overall efficiency of the IT department. 

 

BD liberates the IT organization.jpgFigure 2: Business Discovery liberates the IT organization

Figure 2 shows ranges in the amount of requests made to IT: every organization is different in how they implement Business Discovery or Traditional BI. In some, the IT requests are smaller than others. In general, however, Business Discovery implementations require less intervention by IT and allows them to focus on other, more productive aspects of their critical role within an organization. The reasons for this are examined in the next posts in this series.
In the next part of the series, I'll examine the various business and IT roles and their impact on a Business Discovery deployment.

  “Cloud” and ”SaaS” (software as a service) are two terms often bundled together and used to describe the same thing. The way we at QlikTech use the term 'cloud' is to refer to the use of off-premise, often distributed, computing environments for the purpose of running applications and managing, storing, and analyzing data. We use the term 'SaaS' to refer to a specific distribution and implementation model for applications running in the cloud (software running on multi-tenant servers in the cloud and vendors charging customers a subscription fee to use the software).

 

QlikView and The Cloud:

QlikView can be deployed on premise or in the cloud. In fact, QlikTech’s own demo site (http://demo.qlik.com) runs on servers in the Amazon EC2 cloud. Customers that want to put QlikView on servers in the cloud purchase licenses from QlikTech just as they would if they were deploying the software on premise. This is an attractive solution for customers that:

 

  • Do not want the upfront capital costs of acquiring server hardware
  • Need a flexible environment that can easily scale as their deployment needs scale.

 

 

What’s important to note is that in cloud-based environments the data resides in the cloud. This somewhat obvious point is an important one, particularly with respect to business intelligence solutions: The “raw materials” for any successful BI deployment are the data that underlie the deployment. The location of this data will often determine whether a BI solution is brought in house or deployed in the cloud.  If data is on premises and your analytics are in the cloud, then potentially large data sets must be uploaded to the cloud for analysis. If your data is in the cloud, then it still needs to be transferred to your analytic environment. Of course, both of these scenarios are dependent on great bandwidth.

 

On the other hand, if your data already lives in the cloud, in the same environment as your analytic applications then it makes sense to use cloud analytics.

Through connectors to cloud-based offerings such as Salesforce.com, QlikView can be used to access and analyze data resident in the cloud while maintaining the strict security requirements needed. Therefore, there are no obstacles to QlikView working within a cloud environment: the degree to which it does is determined by how customers choose to deploy QlikView.

 

QlikView and SaaS:

Current trends in the BI platforms market indicate that BI SaaS offerings are not gaining the type of traction that other cloud-based solutions (such as CRM, for example) are gaining. According to Forrester Research, cloud BI will continue to chip away at on-premises BI, but it’s still a long road ahead. Heavy customization and integration of enterprise BI platforms, tools, and applications done by subject-matter experts and consultants will not go away. Another major reason for this is that most organizations continue to maintain their core operations data on premises. 

 

Currently, QlikTech has many OEM partners that are using QlikView to deliver SaaS-based BI offerings to their customers. For these partners, their customer base and business model lends itself to providing solutions via a SaaS model. For example, one partner provides a SaaS-based call center transaction solution to their customers. For an additional fee, their customers can avail of an ‘analytics module’ (built with QlikView) where they can analyze and discover insights in their call center data, to help them with resource allocation, RMA trends and so on.

 

At present, QlikTech does not offer QlikView via a SaaS model. (This would involve QlikView running on multi-tenant servers in the cloud and QlikTech charging customers a subscription fee to use the software). This is due to limited customer demand for such an offering. As the prevalence of cloud-based data increases, so may the demand for SaaS-based BI increase.

 

We continue to closely watch the growing trends in both Cloud and SaaS environments, and where appropriate to our customers' needs, will continue to provide innovative solutions to meet their Business Discovery requirements.


 


We are seeing lots of interest in and hype around the topic of “big data” because data volumes are on the rise and strategic thinkers across industries are looking for opportunities to maximize its value. According to McKinsey Global Institute and others, the term 'big data' refers to data sets whose size is beyond the ability of typical database software tools to capture, manage, and process within a tolerable elapsed time. Depending on the industry, this can mean data sets ranging from a few dozen terabytes to multiple petabytes. In addition, the term ‘big data’ is associated not only with the volume of data but also the variety (i.e. the types of data, structured or unstructured etc.) and the ‘velocity’ of data, i.e. the dynamic or changing nature of the data as new data flows into, and old data exits, a system.


During the last two decades, organizations have made significant investments in automating business processes using software applications that generate substantial amounts of data, which must then be manipulated before business professionals can usefully access, explore, and analyze it. This data is in myriad formats and its sheer volume is daunting. Business users are challenged to efficiently access, filter, and analyze the data — and gain insight from it — without using powerful data analytics solutions, which require specialized skills. They need better ways to navigate through the massive amounts of data to find what’s relevant to them, and to get answers to their specific business questions.

 

The growth in adoption of massively parallel processing (MPP) solutions for handling ever larger volumes of data — whether structured or unstructured —  is driving demand for analysis tools to enable business users to derive insights from the data.

 

QlikView takes a two-pronged approach to this challenge:

 

Firstly, QlikView’s approach has always been to understand what it is that business users require from their analysis, rather than to force-feed a solution that might not be appropriate. Providing the appropriate data for the appropriate use case is more valuable to users than providing all the data, all the time. For example, local bank branch managers may want to understand the sales, customer intelligence, and market dynamics in their branch catchment area, rather than for the entire nationwide branch network. With a simple consideration like this, the conversation moves from one of large data to one of relevance. In any organization, the number of people who need to analyze extremely large data volumes is typically relatively small. For example, a retail bank might have thousands of branches, however only about 100 business analysts in a centralized, corporate role. While branch managers only need slices of data that are relevant to their operations, the corporate analysts may need access to much large data volumes. QlikView is designed to accommodate both environments and enables users to focus on the data that is relevant to them and is of the highest value to them and their area of interest. By taking appropriate slices of the data – big or small – QlikView acts as an analytical environment downstream of the data source, to provide business analysts and casual business users alike the insight they need from the data that is most relevant.

 

Secondly QlikView has been addressing, and continues to address, the big data challenge by ensuring that targeted QlikView applications can address the amounts of data that are needed to ensure the relevancy of the application for business users:

 

  • Recent trends in large memory spaces available on standard Intel hardware allow QlikView to handle ever-larger volumes of data.
  • QlikView best practices promote an architecture-led deployment when handing very large data sizes, such as making proper use of distributed servers in a clustered environment; constructing appropriate applications for the intended audience; using sophisticated data reload engines; and using document chaining where necessary to allow aggregated views to be coupled with detail-level views while optimizing hardware resources.       
  • QlikView provides an open data protocol (QVX) via a series of API's for developers to allow them to interface with the API's of Hadoop-based data source providers. QlikView's QVX protocol can be used to connect to Hadoop based systems via two different methods
    • Disk based QVX file extracts from Hadoop  - PUSH
    • “Named pipe” QVX connector for Hadoop – PULL
  • A QVX SDK is available to all 3rd party developers who wish to build custom connectors for any system with an open API.  QlikTech has partnered with DataRoket which  has an ETL tool to connect with Hadoop, in addition they have produced a QVX named pipe connector for QlikView to link directly to their ETL tool

 

In conclusion, the QlikView Business Discovery platform is all about relevance. It’s about putting tools in the hands of business users to enable to them to ask and answer their own streams of questions, without having to go back to IT or business analysts for a new report or a new query every time they come up with a follow-on question.

 

(My colleague, Elif Tutuk, also wrote a blog post entitled 'An App Model Approach to Big Data' that is well worth a read to learn more about the QlikView approach to Big Data')

QV11 Enterprise Blog image.jpg

 

Not long ago the conventional wisdom held that, while QlikView was a tremendous data visualization tool, it didn't really have what it takes to be deployed across large organizations, providing business discovery capabilities to thousands of business users. The criticism held that managing large deployments was difficult and too costly from an IT perspective. People even questioned whether QlikView could handle the large data sizes used in mission critical business intelligence solutions.


In the very earliest days of QlikView, it's fair to say that some of this criticism was justified. QlikView 'grew up' as a tool to empower business professionals - and not just IT - to access and analyze their own data, enabling them to turn that data into actionable information without a reliance on others to do it for them. Fundamentally, QlikView still holds this ideal to be true: putting the data and the analyses into the hands of the business user.

What has changed over the years however is QlikView's relationship with IT: IT is now a valuable and trusted partner in every successful QlikView deployment around the world. Why? Because as organizations began to rely on QlikView, IT were increasingly asked to take a leadership role. QlikView responded to this demand by ensuring that a strong focus on enterprise capabilities was a critical part of every new release, to support the needs of IT professionals charged with managing QlikView deployments.


QlikView 11 is no different. Whether making reload tasks faster and easier to manage, providing more granular security options or providing an ability to finely tune a clustered deployment, QlikView 11 is the best release yet for IT professionals to manage their QlikView deployments.

Read more about the new Enterprise capabilities in QlikView 11 here.

Understanding how QlikView approaches the topic of security starts with an understanding of the QlikView Business Discovery Platform and how the components work together in a tiered environment. Having this knowledge can dramatically reduce the number of questions about how QlikView handles security.

 

 

While Security is usually a topic at the top of mind of an IT professional, it's worth reminding the business user reading this that while QlikView continues to pride itself on having a very fast and highly flexible environment to allow the business to ask and answer their own questions, without having 'discipline at the core', one cannot achieve 'flexibility at the edges'. This discipline starts with security.

It's important to understand the role of QlikView Server, QlikView Publisher and the QlikView Developer, and how they all fit together. In the first half of the accompanying video, I outline the roles and responsibilities of each within a tiered architecture. This video is one of an 8-part series on the topic of QlikView security that will be published on our website very soon. In addition, a new Technology White Paper, entitled "QlikView Security Overview" is now available in the Resource Library of our website.

Tiered architectures are something that IT professionals are well accustomed to, with firewall-protected regions providing inherent security against authorized access to data and applications. QlikView deployments are no different. By employing a tiered data and application architecture as standard, IT organizations can prevent unauthorized access to sensitive data and can be secure in the knowledge that their QlikView deployments are safe.

A fundamental concept IT pros need to understand is how QlikView authenticates end users and what mechanism it uses to provide authorization to data and applications. Authentication is almost always done outside of QlikView; QlikView relies on third-party authentication methods (such as Integrated Windows Authentication (IWA), single sign-on software, etc.) rather than handling authentication itself. Integrating with IWA or an SSO package is very straightforward from within QlikView. Authorization is achieved using a variety of methods; each is outlined in the second half of the video. The second half of the the video covers file-level authorization as well as data-level (e.g., row- and field- level authorization).

Amongst the biggest concerns that IT professionals have when considering a new software vendor is whether the vendor has a trusted security model and whether the software is flexible enough to adhere to the organization's security standards. Understanding the basic QlikView deployment architecture and the roles of each of the product components can go a long way toward answering many of the initial questions about security in QlikView.

Filter Blog

By date:
By tag: