3 Replies Latest reply: Oct 13, 2017 9:52 AM by omar bensalem RSS

    Can you guys please help me incremental load in qlik sense.

    Aashish Gupta

      Can you guys please help me incremental load in qlik sense. as we have around 60-70 lakhs of rows with 200-400 columns in each. So its too much time taking process to load the complete data in qlik sense.

        • Re: Can you guys please help me incremental load in qlik sense.
          omar bensalem

          I'll guide with theoritically;

           

          What you have to do :

           

          1) Load all the data the first time:

          The first day, you'd want to have all the available data till the loading date:

           

          table:

          load *

          from source;

           

          // then store all of it into a qvd;

           

          store table into table.qvd(qvd);

           

          once you do this, you need to alter your script.

          No need to load everything from now on, but only the data of the current day (for example):

           

          so comment the old script and replace with :

          2) load new data + concatenate to all existing data + store into SAME qvd:

           

          // load new data

          table:

          load * from source WHERE DATE=Today();



          //concatenate new data with existing data

          concatenate

          load * from table.qvd(qvd);

           

          //storing the new concatendated data into the same QVD

          store table into table.qvd(qvd);



          With that, every day you load your app, you just import the data of the day, concatenate it with all the existing data and store the whole thing into the SAME qvd and repeat.

           

          Hope this is helpfull.

          Omar BEN SALEM.

          • Re: Can you guys please help me incremental load in qlik sense.
            Vishnu Chakravaram

            Omar's suggestion is somewhat correct but you'll use a where not exists (unique_identifier) after the from statement,

            otherwise the solution won't work.

              • Re: Can you guys please help me incremental load in qlik sense.
                omar bensalem

                It will work; what the where not exists(unique_identifier) does is making sure to update the content.

                 

                table:

                load * from source WHERE DATE=Today();



                //concatenate new data with existing data

                concatenate

                load * from table.qvd(qvd) where not exists (unique_identifier); // this will import all the new data, and will import from the already stored data only the lines where the unique_identifier wasn't imported in the new data; with that, we do the update.

                if we don not use this, we'll have duplicated lines (if the same line exists in the loaded sqvd and in the new data)

                 

                //storing the new concatendated data into the same QVD

                store table into table.qvd(qvd);