10 Replies Latest reply: Sep 4, 2015 4:57 AM by leila leila RSS

    Incremental Load (Append Into QVD File)

      hello dears.

      i implemented Incremental Load (Insert ,Update ,Delete)

      i have a problem .

      for the first time , i created qvd File. (i don't problem)

               

      let vQVDExistsFile=isnull(QvdCreateTime('$(vSourceQvdFile)SaleQVD.qvd'));

      if vQVDExistsFile=-1 then

        Sale:

        LOAD OrderId,

            ProductId,

            CustomerId,

            Amount,

            Date_Updated,

            Num(Date_Updated) as NumDate

        FROM

        $(vSourceData)Sale.xlsx

        (ooxml, embedded labels, table is Sheet1);

       

        Store Sale into $(vSourceQvdFile)SaleQVD.qvd(qvd);

      END IF

       

      I found that change records for the second time

      But i Can not Append into SaleQVD.qvd

      All New Records (Update,Delete) Override into SaleQVD,Not Append

        • Re: Incremental Load (Append Into QVD File)
          Petter Skjolden

          There is no append option when you write to a QVD file. It will always have to be rewritten in full. That is why you have to do the appending in-memory and concatenate the old data with you new data. So you have the old data stored in a QVD and new data has to be read from another source and concatenated with the table that was read from the QVD.

           

           

          You have to have some statements between the LOAD .... (qvd) and the Store. Which should look a bit like this:

           

          CONCATENATE LOAD (Sale)

          *;

          SQL

          .....

          ;

           

          or

           

          CONCATENATE LOAD (Sale)

          *

          FROM filesource.txt (txt);

          • Re: Incremental Load (Append Into QVD File)
            neetha P

            hi leila,

             

            Insert and Update

            The next case is applicable when data in previously loaded records may have changed between script executions. The following conditions apply:

            • The data source can be any database.
            • QlikView loads records inserted into the database or updated in the database after the last script execution
            • A field ModificationDate (or similar) is required for QlikView to recognize which records are new.
            • A primary key field is required for QlikView to sort out updated records from the QVD file.
            • This solution will force the reading of the QVD file to standard mode (rather than optimized), which is still considerably faster than loading the entire database.

            Script Example:

            QV_Table:

            SQL SELECT PrimaryKey, X, Y FROM DB_TABLE

            WHERE ModificationTime >= #$(LastExecTime)#;

             

            Concatenate LOAD PrimaryKey, X, Y FROM File.QVD

            WHERE NOT Exists(PrimaryKey);

             

            STORE QV_Table INTO File.QVD;

             

            Regards

            Neetha

              • Re: Incremental Load (Append Into QVD File)

                hello neetha

                Can you see Attachment File.?

                I've done exactly that

                  • Re: Incremental Load (Append Into QVD File)
                    neetha P

                    hi leila

                     

                    please paste the full code here,mine is personnel edition

                      • Re: Incremental Load (Append Into QVD File)

                        Thanks

                        Incremental.zip

                        Attached Full Code

                        • Re: Incremental Load (Append Into QVD File)

                          set vSourceQvdFile='QVDFile\';

                          set vSourceData='Source\';

                           

                          let vQVDExistsFile=isnull(QvdCreateTime('$(vSourceQvdFile)SaleQVD.qvd'));

                           

                           

                          if vQVDExistsFile=-1 then

                           

                           

                            Sale:

                            LOAD OrderId,

                                ProductId,

                                CustomerId,

                                Amount,

                                Date_Updated,

                                Num(Date_Updated) as NumDate

                            FROM

                            $(vSourceData)Sale.xlsx

                            (ooxml, embedded labels, table is Sheet1);

                           

                            Store Sale into $(vSourceQvdFile)SaleQVD.qvd(qvd);

                            EXIT Script;

                           

                          END IF

                           

                          NoConcatenate

                          Sale:

                          LOAD OrderId,

                               ProductId,

                               CustomerId,

                               Amount,

                               Date_Updated,

                               NumDate

                          FROM

                          $(vSourceQvdFile)SaleQVD.qvd(qvd);

                           

                           

                          NoConcatenate

                          SaleOrder:

                            Load Max(NumDate) as MaxDate

                            Resident Sale;

                           

                            let vMaxUpdateDate=peek('MaxDate',0,'SaleOrder');

                           

                           

                           

                            Drop Table SaleOrder;

                           

                          NoConcatenate

                          IncrementalLoad:

                          LOAD * where NumDate>$(vMaxUpdateDate);

                          LOAD *,Num(Date_Updated) as NumDate;

                          LOAD OrderId,

                                ProductId,

                                CustomerId,

                                Amount,

                                Date_Updated

                               

                          FROM

                          $(vSourceData)Sale.xlsx

                          (ooxml, embedded labels, table is Sheet1)

                          ;

                           

                           

                           

                           

                          Concatenate

                          LOAD OrderId,

                               ProductId,

                               CustomerId,

                               Amount,

                               Date_Updated,

                               NumDate

                          FROM

                          $(vSourceQvdFile)SaleQVD.qvd(qvd)

                          where not Exists(OrderId);

                           

                           

                           

                           

                           

                           

                           

                           

                           

                           

                          CountOfRecords:

                          Load count(OrderId) as CountOrderId

                          Resident IncrementalLoad;

                          let vCountofRecords=Peek('CountOrderId',0,'CountOfRecords');

                           

                           

                          if vCountofRecords<>0 then

                            store IncrementalLoad  into $(vSourceQvdFile)SaleQVD.qvd(qvd);

                            //store IncrementalLoad  into $(vSourceQvdFile)Sale1.qvd(qvd);

                          end if

                           

                           

                          //Drop table IncrementalLoad;

                    • Re: Incremental Load (Append Into QVD File)
                      Peter Cammaert

                      Is it really useful to do an incremental load from an Excel file? How many rows does the original Excel contain?