9 Replies Latest reply: Jul 12, 2011 8:17 PM by Rob Wunderlich RSS

    Need help in QlikView basic understanding..

      Hi All,

       

                   I am new to QlikView BI solution ... I need some help in understanding the basic QlikView( associative,in-memory)  importance than traditional BI

       

       

       

      I have two doubts:

       

       

      1. What  time does a QlikView publisher takes to refresh the data( eg...a million records)  from the sources to QVD files???

      2. What is the optimum size of the source data  can the QlikView solution adopt ( Unlike traditional BI supports much larger data warehouses)???

      3. In which business areas ( like only SMBs) is this solution apt??

       

       

      It would be a great help if anyone clear these above doubts......Confused about the actual implementation view of QlikView......Please let me know if any resources are available to detail in and out of QlikView  and its practical implemenataions and success stories.....

       

       

      Thanks & Regards,

      Vamsi Krishna.

        • Need help in QlikView basic understanding..
          Stefan Wühl

          Hi Vamsi Krishna,

           

          I think these questions are really hard to answer:

           

          1) Time to load depends on many things: Sources (time to query databases etc), script calculations (for example doing some aggregations in script, comparisons, grouping etc.) and of course also used hardware infrastructure (server hardware, network etc.)

          So I think there is not one simple answer to you first questions.

           

          2) The optimum size of the source data? I am again unable to give you an easy answer. It depends on how you further model these source data in your QlikView data model.

          In terms of performance, also the type of source data is relevant, since QlikView stores compressed data in memory / qvd but compression performance is impacted by source data type (e.g. free text vs. numbers) and if these values are distinct or not.

           

          3) I would say: all kind of business areas, but QlikView has of course strengthes and weaknesses, so I would not recommend it for all kinds of BI reporting issues.

           

          I hope this helps, but I doubt it myself ;-)

           

          Regards,

          Stefan

          • Need help in QlikView basic understanding..
            John Witherspoon

            1) Depends on data source, hardware, and how efficient your code is.  A typical load time for us for a million rows from a database might be ten minutes.  On the other hand, we typically use incremental loads and don't load a million rows at a time, even if we have millions or tens of millions of rows of actual data.  For example, our largest QVD probably has twenty or thirty million rows, and is a two gigabyte QVD file.  The actual database load took a minute of wall time last night (probably less than half of that spent in the DBMS itself), then there were six more minutes spent in QlikView processing and storing the final, updated QVD.  But that's very much just an example, and is not likely to match anyone else's exact performance.

             

            2) I think I remember some semi-official statement a few years ago to the effect that QlikView is best for up to tens of millions of rows, which matches my actual experience on our actual hardware.  However, I'm not aware of any limitation other than hardware that would keep it from being used for billions of rows of data.  Hardware to handle billions of rows of data IN RAM could probably get expensive, though.  I assume there's some point beyond which it is simply not cost effective to run QlikView or any other in-memory BI tool.  (That said, I expect in-memory and on-disk BI to converge in the future onto a keep-it-wherever-is-most-efficient-but-is-possible-on-your-hardware BI.  That could be a while, of course, or maybe some BI tools have already nailed it.  I don't keep up to date.)

             

            3) I'm not sure I even understand the question.

             

            You might search for the Gartner magic quadrant report, or whatever it's called, for BI tools.  That could probably give you a better overview of your options and some pros and cons of each.

              • Need help in QlikView basic understanding..

                Thank you for your reply.........I would like to know if suppose there is a million record source  how much time it takes in each phase like:

                 

                1. loading from database to memory as QVD format( typically 10 mins based on ur previous reply)

                2. Perform transformations and made available to the end -user ( how much time it takes once loaded as qvd file?????)

                 

                 

                please give a sample flow timings based on your usage experience of qlikview so that i can clearly understand  from database to final end -user.......

                 

                 

                Thanks,

                Vamsi Krishna.

                  • Need help in QlikView basic understanding..
                    Stefan Wühl

                    Hi Vamsi Krishna,

                     

                    could you give some examples of transformations you want to apply?

                     

                    As John also indicated, you normally try to put as many time extensive operations into the load script, so that after the load, the response to the end user (selecting a data subset) should be really fast (within some few seconds as our general goal).

                    But the actual time really depends on many factors (data structure, intended operation, HW).

                     

                    Regards,

                    Stefan

                      • Need help in QlikView basic understanding..

                        Actually  I  Dont knowmuch about transformations....but studying  the performance of QlikView BI solution to  present the practical experience of qlikview as a session..........Please share your experience of how many transformations and rows of data you have used and  what is the performance statistics based on your experience of qlikview........  If possible full time and memory overview of its usage.....

                         

                         

                        Thanks,

                        Vamsi Krishna.

                          • Need help in QlikView basic understanding..
                            Stefan Wühl

                            Hi Vamsi Krishna,

                             

                            I am not sure if this is helpful, I know of an (unofficial?) rule of thumb for memory consumed by an QlikView application:

                             

                            If you build a QlikView application, the RAM footprint is approximately (but very roughly) 30 to 40% of the original data size (i.e. uncompressed, e.g. in a flat file).

                            Another 3-4 % of the original size should be added per concurrent user, so if you have something like 30 users, you need extra RAM the size of the original data size only for the users (i.e. storing the selections, user based calculations etc.).

                             

                            As an example, if you have original data size of 1 GB raw data and 10 users, you should plan with 600 to 800 MB RAM consumed by QlikView.

                             

                            From an answer time performance perspective, we try to get a response to each user action within 1 to 2 seconds. If some calculations take more time, it is often possible to transfer these calculations in the script, performing these during load time. But again, actual performance does depend on run time calculations as well as on HW and you must have enough RAM in your server (that's why I added above rule of thumb).

                             

                            Regards,

                            Stefan