I am new to QlikView BI solution ... I need some help in understanding the basic QlikView( associative,in-memory) importance than traditional BI
I have two doubts:
1. What time does a QlikView publisher takes to refresh the data( eg...a million records) from the sources to QVD files???
2. What is the optimum size of the source data can the QlikView solution adopt ( Unlike traditional BI supports much larger data warehouses)???
3. In which business areas ( like only SMBs) is this solution apt??
It would be a great help if anyone clear these above doubts......Confused about the actual implementation view of QlikView......Please let me know if any resources are available to detail in and out of QlikView and its practical implemenataions and success stories.....
Thanks & Regards,
Hi Vamsi Krishna,
I think these questions are really hard to answer:
1) Time to load depends on many things: Sources (time to query databases etc), script calculations (for example doing some aggregations in script, comparisons, grouping etc.) and of course also used hardware infrastructure (server hardware, network etc.)
So I think there is not one simple answer to you first questions.
2) The optimum size of the source data? I am again unable to give you an easy answer. It depends on how you further model these source data in your QlikView data model.
In terms of performance, also the type of source data is relevant, since QlikView stores compressed data in memory / qvd but compression performance is impacted by source data type (e.g. free text vs. numbers) and if these values are distinct or not.
3) I would say: all kind of business areas, but QlikView has of course strengthes and weaknesses, so I would not recommend it for all kinds of BI reporting issues.
I hope this helps, but I doubt it myself ;-)
1) Depends on data source, hardware, and how efficient your code is. A typical load time for us for a million rows from a database might be ten minutes. On the other hand, we typically use incremental loads and don't load a million rows at a time, even if we have millions or tens of millions of rows of actual data. For example, our largest QVD probably has twenty or thirty million rows, and is a two gigabyte QVD file. The actual database load took a minute of wall time last night (probably less than half of that spent in the DBMS itself), then there were six more minutes spent in QlikView processing and storing the final, updated QVD. But that's very much just an example, and is not likely to match anyone else's exact performance.
2) I think I remember some semi-official statement a few years ago to the effect that QlikView is best for up to tens of millions of rows, which matches my actual experience on our actual hardware. However, I'm not aware of any limitation other than hardware that would keep it from being used for billions of rows of data. Hardware to handle billions of rows of data IN RAM could probably get expensive, though. I assume there's some point beyond which it is simply not cost effective to run QlikView or any other in-memory BI tool. (That said, I expect in-memory and on-disk BI to converge in the future onto a keep-it-wherever-is-most-efficient-but-is-possible-on-your-hardware BI. That could be a while, of course, or maybe some BI tools have already nailed it. I don't keep up to date.)
3) I'm not sure I even understand the question.
You might search for the Gartner magic quadrant report, or whatever it's called, for BI tools. That could probably give you a better overview of your options and some pros and cons of each.
Thank you for your reply.........I would like to know if suppose there is a million record source how much time it takes in each phase like:
1. loading from database to memory as QVD format( typically 10 mins based on ur previous reply)
2. Perform transformations and made available to the end -user ( how much time it takes once loaded as qvd file?????)
please give a sample flow timings based on your usage experience of qlikview so that i can clearly understand from database to final end -user.......
Hi Vamsi Krishna,
could you give some examples of transformations you want to apply?
As John also indicated, you normally try to put as many time extensive operations into the load script, so that after the load, the response to the end user (selecting a data subset) should be really fast (within some few seconds as our general goal).
But the actual time really depends on many factors (data structure, intended operation, HW).
Actually I Dont knowmuch about transformations....but studying the performance of QlikView BI solution to present the practical experience of qlikview as a session..........Please share your experience of how many transformations and rows of data you have used and what is the performance statistics based on your experience of qlikview........ If possible full time and memory overview of its usage.....
Hi Vamsi Krishna,
I am not sure if this is helpful, I know of an (unofficial?) rule of thumb for memory consumed by an QlikView application:
If you build a QlikView application, the RAM footprint is approximately (but very roughly) 30 to 40% of the original data size (i.e. uncompressed, e.g. in a flat file).
Another 3-4 % of the original size should be added per concurrent user, so if you have something like 30 users, you need extra RAM the size of the original data size only for the users (i.e. storing the selections, user based calculations etc.).
As an example, if you have original data size of 1 GB raw data and 10 users, you should plan with 600 to 800 MB RAM consumed by QlikView.
From an answer time performance perspective, we try to get a response to each user action within 1 to 2 seconds. If some calculations take more time, it is often possible to transfer these calculations in the script, performing these during load time. But again, actual performance does depend on run time calculations as well as on HW and you must have enough RAM in your server (that's why I added above rule of thumb).
According to this document:
Compression from raw data should approach 10X, and you need another 5-10% for each concurrent user. If so, then for 1GB raw data and 10 concurrent users, you might expect to need around 200 MB RAM.
I checked my sources - I am referring to the QlikView Enterprise Framework where they give some information about sizing (sorry, I have no direct public link available).
They start with a approx. 8% footprint of a .qvw file compared to raw data (which is close to 10x compression I think), but then talk about 4x RAM footprint after loading. That's why I mentioned 30 to 40% for the RAM footprint.
I will try to find out why the RAM footprint should be considered so much larger than the qvw file size (which is next to the different per user RAM consumption the major difference between those two sources of information).
@Vamsi: I think the article John posted gives some good ideas about the upper limits of the QlikView technology, but besides our more theoretical talks, I forgot to ask you if you have already tried QlikView yourself. You probably know that there is a personal edition for free for download to evaluate the technology (with lots of demos also on the web page).
It is also quite easy to get a feeling not only about performance issues but the look and feel.
@Stefan: The RAM footprint is larger than the qvw file size because the qvw is typically compressed. See
for more info.