Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
This space offers a variety of blogs, all written by Qlik employees. Product and non product related.
Product Innovation
By reading the Product Innovation blog, you will learn about what's new across all of the products in our growing Qlik product portfolio.
Support Updates
The Support Updates blog delivers important and useful Qlik Support information about end-of-product support, new service releases, and general support topics.
Qlik Academic Program
This blog was created for professors and students using Qlik within academia.
Community News
Hear it from your Community Managers! The Community News blog provides updates about the Qlik Community Platform and other news and important announcements.
Qlik Digest
The Qlik Digest is your essential monthly low-down of the need-to-know product updates, events, and resources from Qlik.
Qlik Education
The Qlik Education blog provides information about the latest updates of our courses and programs with the Qlik Education team.
Recent Blog Posts
-
A Historical Odyssey: QlikView 7, QVD files and the Aggr() function
For QlikView 7 we developed a number of features that would enable developers to make even more advanced applications. One such feature was the Buffer prefix – a prefix that you could put in front of a Load or a SELECT that would store the data on the local machine and automatically use the local data when appropriate. With it, you could e.g. load data from a slow ODBC connection just once a day and for other script runs use the buffer. The buf... Show MoreFor QlikView 7 we developed a number of features that would enable developers to make even more advanced applications. One such feature was the Buffer prefix – a prefix that you could put in front of a Load or a SELECT that would store the data on the local machine and automatically use the local data when appropriate. With it, you could e.g. load data from a slow ODBC connection just once a day and for other script runs use the buffer.
The buffered data needed to be compact and something that QlikView could load fast. So, the QVD file was invented to solve this need. Today, hardly anyone uses the Buffer prefix, but QVD files created with the Store command are often used.
Another new feature was the Aggr() function. Already for QlikView 4 we had a solution for having an aggregation as dimension. Well, solution is perhaps not the right word… There was a kludgy, hidden feature with which you could use a count of a field as dimension. Internally it was called the Doctors’ Special. We solved a customer’s problem at the time, but we were not satisfied: we instead wanted a general, good solution for nested aggregations.
Six years later – after much thinking – Håkan (the Inventor) came up with the Aggr() function. It is a general function that can be used both as dimension and as measure. It can be nested and it can internally use any aggregation. In other words: it is a general function for nested aggregations. A by-product was the calculated dimension, which was necessary in order to use the Aggr() as dimension.
The Intel Itanium processor was first released in 2001, but its sales had still not picked up when QlikView 7 was released four years later in 2005. Instead, a new kid appeared on the block: The AMD X64 architecture. The X64 turned out to be both cheaper and faster than the Itanium, and as a consequence it sold better.
We already had an Itanium edition of QlikView and we realized that we also needed an X64 edition. Porting the code was straightforward and QlikView 7 was now shipped in three editions: X86, IA64 and X64.
QlikView 7 also brought the calendar object, the block chart, the box plot, the expression overview, the variable overview, the alerts and the reports.As a curiosity, I can also mention that QlikView 7 was the first version with scroll bars for the sheet. None of the previous versions had had this. We had always had the opinion that scroll bars would steal screen space and not add any relevant functionality. Instead we had a zoom function so that you could fit the work area to the screen. But - it’s never too late to change one’s mind…
Further reading on the Qlik history:
A Historical Odyssey: QlikView 6 and Multi-threading
-
Extending QlikView Expressor with the New Extensions SDK
Last month QlikTech released QlikView Expressor 3.9 which introduced some key features that not only strengthen data governance practices within a QlikView deployment, but also enable developers to extend QlikView Expressor functionality. Enhancements include:Conformance Mapping - enables consistent naming of attributes representing a given data element. It reduces the proliferation of unnecessary variations that can cause inconsistent results in... Show MoreLast month QlikTech released QlikView Expressor 3.9 which introduced some key features that not only strengthen data governance practices within a QlikView deployment, but also enable developers to extend QlikView Expressor functionality.
Enhancements include:
- Conformance Mapping - enables consistent naming of attributes representing a given data element. It reduces the proliferation of unnecessary variations that can cause inconsistent results in data processing. Conformance Mapping makes it simple to unify the attributes mapped to fields in a Schema.
- Metadata Analysis Search and Usage - helps locate QlikView Expressor artifacts (dataflows, schemas, types, connections, attributes, fields, etc.) within workspaces, projects and libraries that are used to manage and prepare data for QlikView and other data targets.
- The Extension SDK - enables a set of toolkits and wizards that provide a framework with common functions for developing Extensions for QlikView Expressor.
In this article I will briefly cover the QlikView Expressor Extension SDK and provide a supporting reference with examples that demonstrate how it can be used. (You can learn more about these features in this Technical Brief.)The Extension SDK
Typically, software packages provide a method to extend their functionality using a development kit or API. This allows a developer to build in new capabilities that otherwise would not be available in the out-of-the-box software. The QlikView Expressor Extension SDK enables a set of toolkits and wizards that provide a framework with common functions for developing extensions for QlikView Expressor. With extensions, developers can provide clients with tools to integrate data from specialized sources and perform specialized transformations on data. An extension’s operators, connections and schema integrate into the QlikView Expressor Desktop interface and seamlessly work in conjunction with other standard, enterprise artifacts and operators.
The Extensions SDK provides a set of compilation, utility, and data type conversion functions using built-in Datascript Modules (DSM). DSMs are the standard QlikView Expressor scripting module for Datascript, based on the Lua open-source scripting language. Extensions built on these common functions add functionality to QlikView Expressor Desktop to support custom data sources and specialized transformations.
Figure 1 - The Extension Builder
What is a QlikView Expressor Extension?
A QlikView Expressor Extension is a plug-in (or add-on) that adds new Operators and Metadata artifacts to those already included in the core QlikView Expressor product. Extensions can provide read,write,transform and work-flow-like operations that are not available with the standard operators and artifact types. Once developed they can be easily packaged and distributed with the QlikView Expressor Extensions Manager.
Figure 2 - Custom Operators for reading files from a directory or FTP
The Extension SDK Tutorial
Want to give it a try? Check out this tutorial and sample project to learn how extensions are built using QlikView Expressor.
NOTE: Please make sure you have QlikView Expressor 3.9.1 installed.
Samples available in the project include:
- Reading Fixed Width Files
- Reading delimited files from a directory
- Reading a data file via FTP
- Encrypting and Decrypting Data with a custom transform
Regards,
Michael Tarallo
Senior Product Marketing Manager
QlikView and QlikView Expressor
@mtarallo
-
Why is the question "Why" important in a design process
When we start working on a new project, the customer requirements form the guiding rails for us to begin the process of designing. As we start brainstorming and throwing out ideas, we begin to fill the empty spaces with uncertain experiments. We then start making tiny decision which we stitch together part by part in trying to complete the big picture. Initially to get started with drawing an outline of the project, we naturally tend to ask ourse... Show MoreWhen we start working on a new project, the customer requirements form the guiding rails for us to begin the process of designing. As we start brainstorming and throwing out ideas, we begin to fill the empty spaces with uncertain experiments. We then start making tiny decision which we stitch together part by part in trying to complete the big picture.
Initially to get started with drawing an outline of the project, we naturally tend to ask ourselves the basic questions of why, what, where, how. As the project progresses, we come across unforeseen hurdles, feedback, opinions and try to work our way through it to achieve the big goal. However, in the process of doing so, there are very strong chances of digressing from the main purpose of the project. In trying to put the small pieces together, and focusing on how to make it happen, we sometimes tend to lose the purpose of why we are doing what we are doing. This is when the question “Why” always comes handy to validate our work.
A design can never be a solution if it doesn’t fulfill the underlying purpose of its existence. Asking the question “Why” each time we complete our little milestones, that we set for ourselves, can do wonders to the final solution.
In the book called “The Shape of Design”, Frank Chimero points out - “Our mistake was the same as that of the creative person who places too much focus on How to create her work, while ignoring Why she is creating it. Questions about How to do things improves craft and elevates form, but asking Why unearths a purpose and develops a point of view. We need to do more than hit the right note. The creative process, in essence, is an individual in dialogue with themselves and the work. Why is usually neglected, because How is more easily framed.”
The process of design is most successful when it is an iterative process rather than a linear one. Asking the question “Why” at every stage to validate your work can not only help in creating a strong argument for your work but also help in delivering a legitimate story. So, a reality check by asking the question “why” at every stage of the process can help in creating the solution that one set out to provide in the first place.
-
Storing and Reusing QlikView Specific Expressions with QlikView Expressor
"Good Data"Cultivating culture that emphasizes consistency and reusability is vital when introducing successful data governance practices. Common problems with many decision support systems are the amount of variation, redundancy and overlap that exists within the data models and business logic used across multiple analytical applications. These problems can delay critical decisions and disrupt IT operations while users struggle to verify the tru... Show More"Good Data"
Cultivating culture that emphasizes consistency and reusability is vital when introducing successful data governance practices. Common problems with many decision support systems are the amount of variation, redundancy and overlap that exists within the data models and business logic used across multiple analytical applications. These problems can delay critical decisions and disrupt IT operations while users struggle to verify the truth in data. Having data is one thing, having “good data” is another. With the volume of data increasing it is important to create a structured and consolidated data management layer that contains reusable and consistent definitions. This in turn gives developers and business users assurance that the data they are using, whether to develop applications or make decisions, is “good data”. It also expedites the process of creating new applications and eliminates much of the guesswork in maintaining applications as business requirements evolve over time.
Storing and Reusing QlikView Expressions
Using QlikView Expressor (QVE) to manage and prepare data for QlikView is a great step towards adding data governance and data management to your QlikView deployment. Not only can you visualize where data originates and its final destination, but you can also create reusable parameterized business rules that can be shared across multiple applications.
By design QVE uses a Transform Operator to store Expression and Function rules to manipulate and add data. When transforming data - a simple QVE expression is used. The results yield transformed and/or new data columns to be used in the final output of the QlikView table model.
But what if you want to store and reuse an actual QlikView specific scripted expression and not just the resulting column output? This would be an ideal method to reference a single version of that expression in a unified manner. In turn it could reduce maintenance significantly if changes are made since there is only one place to make modifications, QlikView Expressor. This approach would also increase productivity and data confidence as it creates a single common expression stored in a centralized reusable repository.
Want to learn more and see it in action? Download the complete document and sample below.
If you are viewing this from http://community.qlik.com/blogs - you must click this link to get the sample:
(http://community.qlik.com/blogs/qlikviewdesignblog/2013/04/24/defining-qlikview-expressions-with-qlikview-expressor)The document is also available here: http://community.qlik.com/docs/DOC-4214
Regards,
Michael Tarallo - Senior Product Marketing Manager
Bill Kehoe - Product Manager and founder Expressor
-
Show More, Show Less: Progressive Disclosure in Business Intelligence
Knowing when to show more information and when to show less is not only good design it's good usability. The more information you present the greater the cognitive load a user has to juggle to accomplish a task. Sometimes you want to have lots of information visible to make the best choice but other times you don't. There is a point of diminishing returns where you have given your users too much information too soon. Sometimes people need to ease... Show MoreKnowing when to show more information and when to show less is not only good design it's good usability. The more information you present the greater the cognitive load a user has to juggle to accomplish a task. Sometimes you want to have lots of information visible to make the best choice but other times you don't. There is a point of diminishing returns where you have given your users too much information too soon. Sometimes people need to ease into an application, get acquainted with it, and then proceed to learn more.
Progressive disclosure is when information is sequenced out across several pages or screens to help a user process information and to avoid overwhelming them with too much information. Additional information, or more advanced or rarely used features, are hidden away until needed. This is a technique that was used by IBM in the 1980's when developing user interfaces. They realized that progressively disclosing additional tasks and information through a series of menus was better than having everything present up front. It managed complexity by clearing up clutter. It helped new users get familiar with the UI and as they became more savvy users they used the menu systems to find more advanced functionality.
The attached Technical Paper discusses a bit more how progressive disclosure is useful in BI. Progressive disclosure helps the DAR methodology to give users the general summary on the Dashboard, more advanced functionality on the following pages, and then the real deep dive information for Reporting. Progressive disclosure helps you design for other people at a variety of skill levels to get the most out of your applications.
-
Use of Icons in Design
Have you ever noticed how information is constantly communicated to you via icons in the world of electronic media?Right from the first screen that appears after you start your computers, tablet or phone to checking your emails and then moving on to surfing the web, you are constantly being informed by these little visuals which help you navigate your way through information and effectively alert you of various situations.Icons are a great way of... Show MoreHave you ever noticed how information is constantly communicated to you via icons in the world of electronic media?
Right from the first screen that appears after you start your computers, tablet or phone to checking your emails and then moving on to surfing the web, you are constantly being informed by these little visuals which help you navigate your way through information and effectively alert you of various situations.Icons are a great way of communicating information quickly and precisely.
If we look at some of the top usability rated websites and applications, we will find that they rely very heavily on the use of icons to facilitate excellent user experience. For instance, Dropbox uses icons for almost every function which not only makes it quick to grasp but also makes it visually appealing.In our case as QlikView application designers, we need to consider the use of icons as a way to draw the users’ attention, to alert them of a situation or to alarm them of an impending. Also icons can be used very effectively to display priorities for KPIs which can be used in combination with the traffic light chart (Red, yellow, green). 
The other part to the story is that the use of icons has become very popular in the industry today, especially the web world, so people tend to associate certain actions to certain icons. In our case, while designing application, it becomes important to consider the fact that the users’ psychological paradigm works in alignment to the surfing the web when trying to work with an application. They try to find similarities between using a website and using an application. So, the use of icons becomes all the more relevant in designing applications for us.
So, what are some of the reasons that can justify why icons can be effective in designs? There are a few:-
•Effective visual communication
•Easy identification of information
•Draw users’ attention and alert them of a given situation – this is especially important in dashboards
•To provide a visual relief and enhance visual display of information
•To save screen real estate by replacing text with iconsAlthough icons can be a great way of conveying an idea, their use, in design can be slightly tricky. There are a few dos and don’ts that one needs to keep in mind when incorporating icons in design. That said, Icons can enhance the look and feel of a design and at the same time can enhance the readability of content and thus the usability of an application.
A detailed description with examples of best practices of using icons can be found in the technical brief here.
-
Do we support Windows 8 and IE10 yet?
Hi everyone,We are currently in the process of testing version 11.2 with Windows 8, 2012 Server and also Internet Explorer 10 with a view to introducing support (which may be limited) for the above in a future 11.2 Service Release. Retrospective support in version 10 and already released v11 Service Releases will not be added.We would advise users of Windows 7 not to upgrade their Internet Explorer browser to IE10 at this time. Be aware that... Show MoreHi everyone,
We are currently in the process of testing version 11.2 with Windows 8, 2012 Server and also Internet Explorer 10 with a view to introducing support (which may be limited) for the above in a future 11.2 Service Release. Retrospective support in version 10 and already released v11 Service Releases will not be added.
We would advise users of Windows 7 not to upgrade their Internet Explorer browser to IE10 at this time. Be aware that Windows Update may automatically install IE10. Some known issues with IE10 include (use as a guide only, not an exhaustive list):
- Color gradients in captions/backgrounds not working
- Unable to move columns in table boxes
- Scrolling issues in table boxes
- QMC task creation dialogs not working correctly
- WebParts do not work correctly
regards,
Global Support
-
QlikView 11.20 Service Release 1 is now available - UPDATED
Hi everyone,We are pleased to announce that QlikView 11.20 Service Release 1 is now available for download on http://www.qlik.com/download.11.20 SR1 contains many fixes, details of which can be found in the attached release notes.Thank you to all our customers and partners who participated in the beta program, we really appreciate your input!UPDATE: The initial 11.20 SR1 build , 11.20.11716 has been replaced by a new build, 11.20.11718. There a... Show MoreHi everyone,
We are pleased to announce that QlikView 11.20 Service Release 1 is now available for download on http://www.qlik.com/download.
11.20 SR1 contains many fixes, details of which can be found in the attached release notes.
Thank you to all our customers and partners who participated in the beta program, we really appreciate your input!
UPDATE: The initial 11.20 SR1 build , 11.20.11716 has been replaced by a new build, 11.20.11718. There are two reasons for this new build
- Build numbers in the pdf documentation did not match the actual build number (11.20.11715 instead of 11.20.11716)
- Reported bug 58950 with increased memory usage during reloads for specific script field concatenation has been fixed. If a document script had many field concatenations such as Field1 & Field2 & Field3 & Field4 & Field5 & Field6 as NewFieldName, you may have seen an increase in memory usage during document reloads.
If you have used 11.20.11716 without any evidence of memory increase during reloads then it is not necessary to install the new build. We apologize for any inconvenience.
regards,
Global Support
Note: In line with QlikView's maintenance policy and published release management policy, support for version 10 ends at the end of November 2013. A separate post will follow shortly.
-
Personas in UX Design
When UX professionals get together to redesign a service or product a common practice they will employ will be the use of personas. Personas are representations of different kinds of people / users of the thing you are redesigning. They are the archetypal kinds of people using the thing you are looking to improve. They are the novices, the savvy experts, the casual users, the die-hard fans, the skeptics, etc. You give them names, you describe wha... Show MoreWhen UX professionals get together to redesign a service or product a common practice they will employ will be the use of personas. Personas are representations of different kinds of people / users of the thing you are redesigning. They are the archetypal kinds of people using the thing you are looking to improve. They are the novices, the savvy experts, the casual users, the die-hard fans, the skeptics, etc. You give them names, you describe what their current needs are, how they use the product today, as well as their expectations for the future.
While you can't please all the people all the time, this exercise forces you to account for a variety of users and to remember that you are designing for a variety of needs and expectations and not just your own. It pushes you to step outside of yourself. Personas have the added benefit of putting a face to a series of requirements. It helps everyone involved keep organized by having a name to go with a list of features & requirements.
So where to begin?
You can start a few ways. If you have the luxury of time & access you can interview stakeholders in the project from a variety of levels. From the hands-off to the hands-on users, from the executives down, you will gain insights from people all along the way. If you don't have that sort of time, brainstorm the kinds of users you are designing for. What will the executive level user want or do with your application? What about the person who is doing the day-to-day maintenance of the app? Create lists of the kinds of users, how they are doing things today, what they want in a future design, what their roll will be, their level of technical sophistication, etc. From here you can begin to design for a variety of consumers.
A word of caution.
Personas aren't real people. It seems obvious but it is worth mentioning. Personas are a great way to get organized and remember to account for a variety of users but they aren't the same as actually testing your design with real people. You should still validate your ideas with a variety of people and see what's working and what isn't. A persona will guide you to a solution but you still need to confirm, with real people, if that solution works.
-
4 tips to start working with QlikView
A few weeks ago a former colleague, who has recently started to work with QlikView, contacted me and was wondering how to start with QlikView and how to help his team to better visualize data, here are some tips I shared with him that could help you to start too: 1 Find inspiration and reuseMake sure you visit QlikView Demo site prior to starting your project. One of the best things about the site is the fact that you are able to download one app... Show MoreA few weeks ago a former colleague, who has recently started to work with QlikView, contacted me and was wondering how to start with QlikView and how to help his team to better visualize data, here are some tips I shared with him that could help you to start too:
1 Find inspiration and reuse
Make sure you visit QlikView Demo site prior to starting your project. One of the best things about the site is the fact that you are able to download one app and reuse the content for your own. Don’t forget to check the brand new demo image gallery with visualization highlights and samples of QlikView’s unique approach to data viz.
If you need advice on when to use which chart, take a look at Data Visualization App, it has specific information and best practices on how to show your data in the most effective ways.
2 Identify the needs
You need to understand the business process that is behind the analysis first. The more you know about the business needs the more easily you will be able to design for the business by choosing the appropriate KPIs. So your first task is to identify the KPIs for your new dashboard and assign them a relative weight so later on you will be able to use hierarchy to emphasize the heaviest, most important ones - this is key.
3 Use pen and paper
To create a QlikView app start by picking a piece of paper and your favorite pencil and drawing your best approach to the dashboard, you don’t need to be very specific, just sketch it out and create the basic layout to emphasize the main KPIs and comparisons that are need in your project. Don’t forget to validate your design with other users and add their feedback to your design.
4 Choose a leitmotiv
Think of the fashion designers. They base their collections on a certain decade or style and then they are consistent in applying that style to the entire collection. When working with QlikView you can be a fashion designer in a way, choose a theme, your leitmotiv for the app, and keep it in mind when creating charts and when choosing the color palette. Despite using your corporate branding, it could be a mess if you mix several colors, just keep neutral colors for backgrounds and main elements and use contrast to emphasize the most important KPIs.
Some good examples of thematic design could be founded in QlikView Developer Toolkit App, an excellent starting point for your next QlikView App, you will also find many design elements as buttons, grids, shadows and rulers and lines.
These are just some basic tips to start working with QlikView but if you want to go further in the design process to developing QlikView applications, then you should review Michael’s technical papers you will find at the bottom of his post called How To Improve Your Design Process Working With Qlikview.
Enjoy Qliking!
Arturo Muñoz
-
The Importance of Data Visualization
In 1973 English statistician Francis Anscombe published the paper Graphs in Statistical Analysis to stress the importance of data visualization. He wanted to show that graphs are essential to good statistical analysis. Commonly referred to as Anscombe's Quartet, he created four data sets of x & y coordinates whose statistical properties of mean, variance, etc. are nearly identical. He then graphed these data sets as four scatter plot charts and d... Show MoreIn 1973 English statistician Francis Anscombe published the paper Graphs in Statistical Analysis to stress the importance of data visualization. He wanted to show that graphs are essential to good statistical analysis. Commonly referred to as Anscombe's Quartet, he created four data sets of x & y coordinates whose statistical properties of mean, variance, etc. are nearly identical. He then graphed these data sets as four scatter plot charts and demonstrated just how different the data sets really were.
These charts are an integral part of comprehending the data because we can instantaneously see the differences between the scatter plots, a task that would take much longer if we were to rely solely on the data table. The chart and the table are working together to give us the complete story. Anscombe's point was that the chart isn't subordinate to the data table. Both the data table and the chart help the reader understand the data but in different ways.
Good data visualization works because our brains are hardwired to rapidly process visual stimuli using very little conscious effort. Even before actively reading the charts your brain has pre-attentively processed the information to spot the patterns, the similarities and the differences.
So when planning your application consider tables AND charts rather than tables OR charts. They are two ways to learn the complete story in the data.
-
Metadata Management the Customer’s Way - Part 2: The Solution
RecapIn my last article Metadata Management the Customer’s Way – Part 1, I covered a customer’s specific metadata challenge and proposed a QlikView Expressor (QVE) solution to address it. To summarize, our customer needed to know which existing QlikView applications (.qvw) were in compliance with the newly established rules appointed by the organization’s Business Intelligence Competency Center. This article and companion video (below) will prov... Show MoreRecap
In my last article Metadata Management the Customer’s Way – Part 1, I covered a customer’s specific metadata challenge and proposed a QlikView Expressor (QVE) solution to address it. To summarize, our customer needed to know which existing QlikView applications (.qvw) were in compliance with the newly established rules appointed by the organization’s Business Intelligence Competency Center.
This article and companion video (below) will provide some details about the proposed solution along with QlikView and QlikView Expressor samples. (attached in this post)
We know from the previous article that the customer’s metadata, such as column labels and validation flags, are stored within database tables as part of their metadata management application. For simplicity I will use an Excel spreadsheet to simulate the customer’s metadata repository and focus on the two fields “ColumnName” and “Validated”.
Leveraging the QlikView Governance Dashboard
I've implemented this solution using data files produced by the QlikView Governance Dashboard (QVGD) scanning process and a QlikView Expressor Dataflow. The QVGD working data files (*.qvx) contain QlikView deployment metadata such as column labels, expressions and field names. Typically, this metadata creates the associative data model read by the QVGD’s information sheets. Leveraging these data files with QVE’s QlikView Read operator provides significant advantages when developing custom QlikView Metadata Management solutions such as this. Since we already have the data files from the QVGD scan, there’s no need to create a custom program to extract QlikView label metadata from the QlikView applications.
The QlikView Expressor Solution
Using the QlikView Expressor Design Studio a multi-step Dataflow can be created that will:
- Extract and load the custom metadata into a QVE Lookup table for later processing
- Read and join the appropriate QVGD data files on their appropriate keys
- “Lookup” what labels have been validated or not
- Provide appropriate business terms to all validated, non-validated and missing labels
- Capture the labels that are not in the custom metadata repository and write them back to the repository for later approval
- Write the results to a new QlikView data file to be used for analysis in QlikView
Fig. 1 - QlikView Expressor Dataflow that prepares the data for QlikView analysis and updates the metadata repository
Fig. 2 - Simple QlikView application used to analyze the compliance of the lables used in existing QlikView applications
Review the below video (full screen 480p or higher) to learn more and see the solution in action:
If the video does not display - please use this link: http://www.youtube.com/watch?v=IADq2IT1U6Y
Regards,
Michael Tarallo
Senior Product Marketing Manager
QlikView and QlikView Expressor
-
A Historical Odyssey: QlikView 6 and Multi-threading
Although QlikView 5 was a stable, good version with lots of functionality, it wasn’t good enough for the new hardware that came along. We had already seen the arrival of the 64-bit Itanium processor and we had responded by developing an Itanium edition. So far, so good. But when we experimented with large data amounts, we realized that although QlikView now could load and address very large amounts of data, QlikView couldn’t process the data fast... Show MoreAlthough QlikView 5 was a stable, good version with lots of functionality, it wasn’t good enough for the new hardware that came along. We had already seen the arrival of the 64-bit Itanium processor and we had responded by developing an Itanium edition. So far, so good. But when we experimented with large data amounts, we realized that although QlikView now could load and address very large amounts of data, QlikView couldn’t process the data fast enough. QlikView needed more processor power. The amount of memory was no longer the limiting factor – instead it was the processor power.
About this time, Intel had just launched its first dual core processors. There were already multi-processor motherboards, but the dual core processors were cheaper and could in addition share cache. The introduction of the dual core processors and QlikView’s need for more processor power forced us to look at how we better could utilize parallel processing.
Hence: Time for a re-write! Multi-threading – here we come.
Already in the QlikView 5 server, there was some basic multi-threading: Each session was evaluated in its own thread. But now, we needed to take multi-threading several steps further; we needed to make the evaluation of every click multi-threaded.
Multi-threading is not easy. The software process needs to have mechanisms that can determine that one thread should be split into several, and other mechanisms that merge several threads into one. The threads are asynchronous and sometimes a thread needs to stop and wait for the result of another thread. This means that you can get situations where two threads wait for each other. So you also need mechanisms to avoid such deadlocks.
Anyway, we re-wrote large parts of the code and when we released QlikView 6 in the spring of 2003, it was multi-threaded.
In the solution, the logical inference for a single click is multi threaded and the evaluation of several threads are merged together to form the final result. After the logical inference, the sheet objects need to be calculated. Each object is then a thread of its own that subscribe to all changes in the data model. “Subscribe” means that it is re-evaluated every time there is a change in the selection state. In addition, the chart engine is multi threaded, so that if an aggregation is made over a table with a large number of records, different parts of the table are aggregated in different threads.
QlikView 6 also brought the LEF files, which enabled Client Access Licenses (CALs) for the server. It also brought the qvp protocol, the plug-in client, the OCX, the table viewer, copy and paste objects, the layout themes, semi transparency (GDI+), the color functions, variables in the layout, the grid chart, the gauge chart, the radar chart, the slider object and the language dll:s.
We also introduced different license levels for the stand-alone version of QlikView 6: Enterprise, Professional and Analyzer, so that a customer could have different capabilities for developers, power users and standard users.
QlikView 6 should be remembered for bringing the multi-threading that we use still today. This is the core of the modern QlikView that can use the power of today's multi-core hardware.
Further reading on the Qlik history:
A Historical Odyssey: QlikView 5 and the Rainbow Border
A Historical Odyssey: QlikView 7, QVD files and the Aggr() function
-
Wildcard data loading (*.blah......)
Regularly in a QlikView application, you need to load from a directory or set of identical files which simply have different file names but contain the same type of data. For example, log files generated by systems contain the same type of data in a set structure, but the filenames often reflect the time and date or the system the log file was generated from.In this blog, I will show you how to (really simply) load in to QlikView, a set of files ... Show MoreRegularly in a QlikView application, you need to load from a directory or set of identical files which simply have different file names but contain the same type of data. For example, log files generated by systems contain the same type of data in a set structure, but the filenames often reflect the time and date or the system the log file was generated from.
In this blog, I will show you how to (really simply) load in to QlikView, a set of files that are identical in makeup but are not named the same.
So, imagine we have a directory full of log files on your server as below...
I need to load all of these log files in to my application, but I do not want a single load statement in my script for each of the 100+ files. So, firstly we need to perform a standard load from our table (.log) files. When you have done this using the qlikview script editor, you will have a simple load statement...
LOAD Sequence#,
Timestamp,
Level,
Hostname,
Logger,
Thread,
Id,
User,
Exe_Type
FROM
[vm-qvs12__Application_QVS_2012-12-04T13.25.38Z.log]
(txt, codepage is 1252, embedded labels, delimiter is '\t', msq);So this statement would load all of the columns listed from the specific file in the FROM statement. However, we want to utilise one single LOAD statement for all of our log files. By simply changing the filename to contain an asterisk, provided all of the columns that you are placing in your load statement exist in all of your files, using an asterisk will cycle through all of the .log files that start with the text "vm-qvs12__Application_QVS_" in the directory, and load the data in to your application.
example:
FROM
[files\Log\QlikViewServer\vm-qvs12__Application_QVS_*.log]
(txt, codepage is 1252, embedded labels, delimiter is '\t', msq);Now we have all of the data in our application, we may want to know from which log file the data in the application came from. So, we can use some functions to give us some extra data.
We can use the Filebasename() function to give us a column in the data that contains the filename of the loaded log file.
LOAD filebasename() as SourceFilename,
Sequence#,
Timestamp,
Level,
Hostname,
Logger,
Thread,
Id,
User,
Exe_Type
FROM
[files\Log\QlikViewServer\vm-qvs12__Application_QVS_*.log]
(txt, codepage is 1252, embedded labels, delimiter is '\t', msq);Now, that was a simple example of the wildcard load. What if I wanted to load data from multiple files and enhance the script with something like a preceding load? To do this I would have to utilise a simple function/variable and a small loop. In the example below, you can see that I am loading from multiple xlsx files, again using the filebasename function, but I am also creating a column called Total in my preceding load. The For Each and Next keywords tell QlikView to loop through the files in the location that conform to the wildcard in the Filelist function.
For each vFile in FileList('C:\Users\aby\Desktop\*wildcard.xlsx')
Load Col2+Col3 as Total,
*;
Load *,
Filebasename() as Source
from [$(vFile)]
(ooxml, embedded labels, table is Sheet1);Next vFile
Hopefully I have managed to simplify some of the scripts you use today!
ABY
-
QlikView 11.2 is available!
Yesterday we announced QlikView 11.2. The main new feature in this release is QlikView Direct Discovery. With this exciting feature, QlikView does not require loading all data into the QlikView in-memory engine anymore. QlikView Direct Discovery is a hybrid approach that leverages both in-memory data and data that is dynamically queried from an external source.Some of the main benefits of QlikView Direct Discovery are:Consolidate relevant data fr... Show MoreYesterday we announced QlikView 11.2. The main new feature in this release is QlikView Direct Discovery. With this exciting feature, QlikView does not require loading all data into the QlikView in-memory engine anymore. QlikView Direct Discovery is a hybrid approach that leverages both in-memory data and data that is dynamically queried from an external source.
Some of the main benefits of QlikView Direct Discovery are:
- Consolidate relevant data from multiple sources, including Big Data repositories
In addition to the current decision support environments like EDW, data marts and operational data stores, organizations are now also considering the big data storage environments, such as Hadoop, BigQuery or Teradata etc. The challenge is enabling business users to analyze data from all of these data sources in the same analytics application. QlikView Direct Discovery offers a hybrid approach solving this challenge. It allows users to seamlessly run queries on the big data stores while they do discoveries on the data extracted from their ERP systems, data marts, EDW or even from their excel files into memory.
- Maintain associations among all the data, regardless of where it is located
The level of details that is stored in the big data stores is usually very granular. Business users need an easier navigation means in this bulk of data. QlikView Direct Discovery leverages associations in the data, making extremely large datasets manageable.
For example, a policy analyst who uses a QlikView app to analyze regional loss and revenue information on a daily basis, would know the region names, but he would not have any clues on the specific policy numbers for these regions. With Direct Discovery, he can select the regional info, as he would do every day, and QlikView would automatically associate and query the billions of policy-level information on the big data store and display the query results with the in-memory metrics on the same QlikView app for the selected region.
- Access Big Data without complex data modeling or programming
It is very easy to use Direct Discovery feature on a QlikView application. The only thing that is changed is to use “DIRECT SELECT” on the load script instead of using “SQL SELECT”. This new keyword would indicate QlikView that the data source is a direct discovery source. In this case, QlikView would only load the field names and will bring the data to the user interface when a direct discovery field is used on a QlikView chart. More information on the technical aspect of this feature can be found in this technical addendum paper. A how-to video is also available here.
Please note that QlikView 11.2 is a new release of QlikView, with one new feature, QlikView Direct Discovery. Please read the QlikView 11.2 Release Note document that is available on the download site before upgrading to QlikView 11.2 to learn more about the release and some of the bug fixes with this release. With this release, the QlikView 11 branch will be stopped and new service releases will be made for QlikView 11.2 instead. For more information on this subject, please contact the QlikTech Support team.
Since we announced this new feature in October, we have seen great excitement in our customer and partner community as they see the potential of QlikView 11.2 enabling Business Discovery with Big Data, without any data size limitations. QlikView 11.2 is now available on our download site if you also would like to expand your business discovery capabilities on the data sets that were previously used separately, or not used, because of their bulk and the development effort required!
-
Holiday Gift! - Organize the files on your machine
Do you use the generic search engine on your OS to find the document you are looking for? Have you had any luck finding it within one minute? Well, I haven’t. So I decided to go back to the old fashion way. QlikView way. The associative way. Here is my background. I use Windows, so I search for something like “How QlikTech uses QlikView” in “Search programs and files” box above the “start” button. This is where the disaster begins. It fou... Show MoreDo you use the generic search engine on your OS to find the document you are looking for? Have you had any luck finding it within one minute? Well, I haven’t. So I decided to go back to the old fashion way. QlikView way. The associative way.
Here is my background. I use Windows, so I search for something like “How QlikTech uses QlikView” in “Search programs and files” box above the “start” button.
This is where the disaster begins. It found 1,270 items. I know I didn’t make that many documents with that name. What it’s finding is all the files that contain at least one of the key words in the file name, including as an attachment to emails. It also searches for the content of the document. It’s overwhelming to find what I am looking for from this long list, but Windows doesn’t give me an option to filter by a document type or by a folder these ppt may belong to.
I just needed a way to simply find my files in my folder by the name and not the content . So I created this application called Find My Files.qvw. If you’d like to try it, download the app and specify the root directory in the box and hit “Reload” button in QlikView. If you start with the highest hierarchy directory, it may take longer than expected depending on the number of files you have. For example, I have 106GB used in my C: drive, and I care about only what’s in my user folder. So I loaded all the files in that folder, which is 37GB in total. It took only 1 minute to read in all the files. If you don’t want to read all the files, you can specify the extensions in the loop such as qvw, xls, xlsx, doc, docx, ppt, pptx etc. The reload time will be faster, especially if you are reading the entire hard disk.
The script of this app is pretty simple, and it is one of the help sample codes. Many people may be already familiar with it.
Using this app, I know there are 18 files in my machine for “Making a good design great” ppt and not 1270.
Yes, there are various desktop search software available out there, but why is this app any better? This app also helps you organize your hard drive. For example, when you are running out of disk space and need to know which folder is consuming the most disk space, you may want to know which folder I can consider eliminating. Windows Explorer doesn’t give you the folder size, so you need to right click and see the properties to see how large the folder is. If the folder is 10GB, you sit and wait for the properties box to appear. Bummer. Use this application, and you will be more efficient with your work life.
Download this application and the technical brief from here.
Ho Ho Ho… Happy Holidays to you (belated)!
Shimanta (Shima-Santa)
-
Helping Hands
The holiday season is often a time when we consider our blessings, our shortcomings, our needs, and the needs of others. As a special Christmas blog post, I felt it may be good to share my experiences working with Tiny Hands International (THI), a non-profit organization that works specifically with orphans, street children, and sex-trafficking victims. It was an eye-opening experience for me that gave me insight into the needs of the victims as... Show More
The holiday season is often a time when we consider our blessings, our shortcomings, our needs, and the needs of others. As a special Christmas blog post, I felt it may be good to share my experiences working with Tiny Hands International (THI), a non-profit organization that works specifically with orphans, street children, and sex-trafficking victims. It was an eye-opening experience for me that gave me insight into the needs of the victims as well as the organization itself.I was having lunch several months ago with the founder of THI, John Molineux, who is a good friend of mine from college. At one point in our conversation, John expressed that they had been looking for a good way to analyze the data they collect when they intercept trafficking victims at border checkpoints in Nepal. Each year, 10-15,000 women are deceived and trafficked out of Nepal where they are then sold as sex slaves. More specifically, John mentioned to me that they had been hoping for a tool that would allow them to visualize the paths the victims take when they are trafficked. This could be useful in identifying the most commonly used routes, and it could perhaps give insight into how they could best focus their efforts. Of course, I knew QlikView could be a great help to them, and we got to work loading THI’s data. Once they identified the paths of the victims geographically in the data, it was easy to use a map extension to visualize the paths, drawing the more highly traveled paths with thicker lines:
The data is very raw, and THI is thinking of ways to optimize their collection and identification of paths, but even so, some trends and useful information can be seen with the map. For example, the dark red area on the east side of the map shows that a lot of activity is occurring at this checkpoint, including some of the thicker lines. Of course, using QlikView’s associative filtering, we can choose to display only the top 15 routes:
Or focus in on a specific border checkpoint like the eastern one I mentioned earlier:
We were also able to use an expression to color the most traveled routes in red, while coloring the less traveled routes in yellow and green:
Overall, we were very excited to see start seeing the story the data was telling us, and ideas were flowing on how they could further optimize their data collection and leverage QlikView’s analysis in their planning.
What I realized in working with Tiny Hands International is that charity organizations are businesses too. They also need the ability to analyze data and be as efficient as possible. Knowledge is power after all, and wasted time, money, and resources mean a less effective fight against injustice.
More than that, working with THI was a staunch reminder for me that there are people in the world right now doing selfless honorable work to improve the plight of their fellow humans. Oftentimes I sit at my warm desk feeling separated from this world, but I hope that as I'm humbled through experiences like this, I will consider the ways I might help to affect positive change in the world.
If you’d like to support the work of Tiny Hands International, please go to their website. A small gift from us could have a large impact on those in need.
-
Roman Emperors and the Month Names
Our current month names come from the Roman calendar. The original Roman calendar had ten months: Martius, Aprilis, Maius, Junius, Quintillis, Sextilis, September, October, November and December. The first four months were named after gods: Mars, Venus (Aphrodite in Greek and Apre in Etruscan), Maia and Juno. The origin of Aprilis is debated, but we know that the month was sacred to Venus because the Festum Veneris et Fortunae Virilis was held o... Show More
Our current month names come from the Roman calendar. The original Roman calendar had ten months: Martius, Aprilis, Maius, Junius, Quintillis, Sextilis, September, October, November and December. The first four months were named after gods: Mars, Venus (Aphrodite in Greek and Apre in Etruscan), Maia and Juno. The origin of Aprilis is debated, but we know that the month was sacred to Venus because the Festum Veneris et Fortunae Virilis was held on its first day. The last six months are based on the latin words for five, six, seven, etc.
Each year started in March and ended in December, 304 days later. It was then followed by a period of festival between the years.
But the calendar was soon changed by the king Numa Pompilius around 700 BC, who added Januarius (after the god Janus) and Februarius (after the purification festival Februa). He also moved the beginning of the year to Januarius.
However, the year was still too short - only 355 days - so the Pontifices occasionally had to add an extra month to keep the calendar in sync with the seasons. This decision was political and was sometimes done just to extend the term of a particular public official. Or it wasn’t done at all, if the official was an opponent.
By the 1st century BC, the calendar had become hopelessly confused, so in 46 BC, Julius Caesar initiated a reform that resulted in the establishment of a new calendar, the Julian, which was a vast improvement: Leap years were introduced and the year in the Julian calendar was on the average 365.25 days, so no extra intercalary month was needed. After Julius’ death, the month of Quintilis was renamed Julius in his honor, hence July.Julius Caesar was succeeded by Augustus, and after his death the senate renamed Sextilis after him. At the same time, the senate also suggested that September be renamed after the reigning Caesar Tiberius. But Tiberius refused with the words: “And what will you do if there be thirteen Caesars?”
Today most countries use the Gregorian calendar, which is based on the Julian and still has the Roman month names from 2000 years ago.
In QlikView and in Qlik Sense, the abbreviated month names are defined in the environment variable MonthNames.
Set MonthNames = 'Jan;Feb;Mar;Apr;May;Jun;Jul;Aug;Sep;Oct;Nov;Dec';
This variable is used by several functions, e.g. Month() and Date(). The format code ‘MMM’ is always mapped against the content of this variable.
You can also create an environment variable for the long month names, corresponding to the format code ‘MMMM’:
Set LongMonthNames =
'January;February;March;April;May;June;July;August;September;October;November;December';If you have these variables, you can use the long format codes not only for formatting of date fields, but also for interpretation of fields, e.g.
Date(MonthStart(Date),'MMMM YYYY') as YearMonth, // Creates and formats YearMonth
Date#(Date,'MMMM DD, YYYY') as Date, // Interprets the Date='October 9, 2012' as a date
Dual(MonthTxt, Month(Date#(MonthTxt,'MMM'))) as Month, // Interprets the MonthTxt='Oct' as a monthBottom line: You should define fields for months in your master calendar and add an environment variable for long month names.
Further reading related to this topic:
-
Version 11 Service Release 2 Update 2 is now available for download
Hello everyone,We are pleased to announce that Version 11 SR2 Update 2 - build 11.00.11440 is now available. This Update replaces Update 1 (11.00.11426). Unfortunately Update 1 contained two degrades which affected users of Windows XP and Windows Server 2008 (not Server 2008 R2).We regret the inconvenience caused. You can download the installation files for Update 2 from http://www.qlik.com/download.Please see attached release notes for fur... Show MoreHello everyone,
We are pleased to announce that Version 11 SR2 Update 2 - build 11.00.11440 is now available. This Update replaces Update 1 (11.00.11426). Unfortunately Update 1 contained two degrades which affected users of Windows XP and Windows Server 2008 (not Server 2008 R2).
We regret the inconvenience caused. You can download the installation files for Update 2 from http://www.qlik.com/download.
Please see attached release notes for further information.
regards,
Global Support Team
-
Tool vs. Toy
Back in 2001, with the release of the now cult-classic film Donnie Darko, there was a website that came out to accompany the film. The site was (and is) as cryptic and vague as the movie. There is this quasi linear path through the site but at the same time no clearly delineated objective - you experience the site and you do so just for the sake of experiencing it. It doesn't need to tell you what theaters the movie was being released in, or when... Show MoreBack in 2001, with the release of the now cult-classic film Donnie Darko, there was a website that came out to accompany the film. The site was (and is) as cryptic and vague as the movie. There is this quasi linear path through the site but at the same time no clearly delineated objective - you experience the site and you do so just for the sake of experiencing it. It doesn't need to tell you what theaters the movie was being released in, or when it was being released because you could get that information other places. This site was for enthusiasts, for die-hard fans. It was something to play with that added to the richness of the film by extending the experience of the film.
Now contrast this with a site that sells movie tickets. There is a clear objective. You use it to buy tickets online to save you from having to buy tickets in person. I go to these sites with a purpose, I complete my task, I leave. It's a tool to do a job.
The balance between creating a tool vs. creating a toy isn't always this clear. You want to create a pleasant experience, even something cool, but if the primary objective is to enable people to complete a task then you should focus on building a useful tool. Help your users do the thing they came here to do. In the case of QlikView this means building applications that focus on the user's needs, apps that clearly show the data, that help users explore their data and find new insights.
I've seen more than a few applications where people, with the best intentions, tried to create a fun environment but ended up adding too many pieces of unnecessary visual "flare." When considering a background, a chart, an animated logo, or a photograph you need to ask yourself: "is this going to help people use this application?" If the answer is no, then perhaps it is best left out. There are plenty of well designed applications that take a very minimalistic approach to design. The application should be well designed, but well designed this isn't synonymous with having a trendy aesthetic. You don't need to cram in all the design tricks you know.
Real design solves problems and the application you are building should be a tool to help people solve problems and complete tasks.
