2 Replies Latest reply: Jul 1, 2010 1:48 PM by jayant130 RSS

    Load with QVD store Qs

    jayant130

       

      Below QlikView datamodel has tables tblPnl which contains around 200 mil rows, tblPosition around 1 mil rows and tblHrchy around 2500.



       

      tblPnltblPositiontblHrchy
      POS_IDPOS_IDBook
      CCYCUSIP[Business Line]
      SCEN_DATECUST_IDDesk
      TOTALBookPortfolio


       

      I want to create one QVD file for each portfolio, so that in case I have any updates to tblPosition/tblPnl only specific portfolio QVD file can be reloaded without entire reload.

       

      I would really appreciate any pointer/views/ideas on how to go about doing this.

       

       



        • Load with QVD store Qs

          I'm not sure that it makes sense to do that. The load script would not be pretty... or maintainable.

          QVD loading is pretty optimized. It takes about a minute to load 200 million records, in my experience.

          One of my client databases has in excess of 250 million rows in a single table; i just reload it during off-peak hours. The data is slightly out of date as a result, but it suits the purpose for this client.

          If that isn't enough, you may need to look into doing a real-time data integration. I've seen a demonstration of the capabilities, but not really looked under the hood.

          I don't think that kludging a solution using complex load scripting is the wisest approach. I think you might need to take a step back from the solution you've come up with, and try another approach to the problem.

          (what is the problem, by the way? time to reload?)

            • Load with QVD store Qs
              jayant130

              Thank you! Ryan.

              You are right, the real issue is - time to reload. I am fairly new to QlikView, I was thinking about some other options :

              1. When ever there is update to certain slice of data, I still load entire data into another QVW and then rename the it to production QVW file, don't know if it is possible while users are connected to the QVW ?

              2. Split the files into smaller files and load them up in parallel.

              I will also look into 'real-time data integration'.

              Thanks again.