Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Incremental Load (Append Into QVD File)

hello dears.

i implemented Incremental Load (Insert ,Update ,Delete)

i have a problem .

for the first time , i created qvd File. (i don't problem)

         

let vQVDExistsFile=isnull(QvdCreateTime('$(vSourceQvdFile)SaleQVD.qvd'));

if vQVDExistsFile=-1 then

  Sale:

  LOAD OrderId,

      ProductId,

      CustomerId,

      Amount,

      Date_Updated,

      Num(Date_Updated) as NumDate

  FROM

  $(vSourceData)Sale.xlsx

  (ooxml, embedded labels, table is Sheet1);

  Store Sale into $(vSourceQvdFile)SaleQVD.qvd(qvd);

END IF

I found that change records for the second time

But i Can not Append into SaleQVD.qvd

All New Records (Update,Delete) Override into SaleQVD,Not Append

10 Replies
petter
Partner - Champion III
Partner - Champion III

There is no append option when you write to a QVD file. It will always have to be rewritten in full. That is why you have to do the appending in-memory and concatenate the old data with you new data. So you have the old data stored in a QVD and new data has to be read from another source and concatenated with the table that was read from the QVD.

You have to have some statements between the LOAD .... (qvd) and the Store. Which should look a bit like this:

CONCATENATE LOAD (Sale)

*;

SQL

.....

;

or

CONCATENATE LOAD (Sale)

*

FROM filesource.txt (txt);

Anonymous
Not applicable
Author

hi leila,

Insert and Update

The next case is applicable when data in previously loaded records may have changed between script executions. The following conditions apply:

  • The data source can be any database.
  • QlikView loads records inserted into the database or updated in the database after the last script execution
  • A field ModificationDate (or similar) is required for QlikView to recognize which records are new.
  • A primary key field is required for QlikView to sort out updated records from the QVD file.
  • This solution will force the reading of the QVD file to standard mode (rather than optimized), which is still considerably faster than loading the entire database.

Script Example:

QV_Table:

SQL SELECT PrimaryKey, X, Y FROM DB_TABLE

WHERE ModificationTime >= #$(LastExecTime)#;

Concatenate LOAD PrimaryKey, X, Y FROM File.QVD

WHERE NOT Exists(PrimaryKey);

STORE QV_Table INTO File.QVD;

Regards

Neetha

Not applicable
Author

hello peter

Can you see Attachment File.?

I've done exactly that

Not applicable
Author

hello neetha

Can you see Attachment File.?

I've done exactly that

Anonymous
Not applicable
Author

hi leila

please paste the full code here,mine is personnel edition

Not applicable
Author

Thanks

Incremental.zip

Attached Full Code

Not applicable
Author

set vSourceQvdFile='QVDFile\';

set vSourceData='Source\';

let vQVDExistsFile=isnull(QvdCreateTime('$(vSourceQvdFile)SaleQVD.qvd'));

if vQVDExistsFile=-1 then

  Sale:

  LOAD OrderId,

      ProductId,

      CustomerId,

      Amount,

      Date_Updated,

      Num(Date_Updated) as NumDate

  FROM

  $(vSourceData)Sale.xlsx

  (ooxml, embedded labels, table is Sheet1);

  Store Sale into $(vSourceQvdFile)SaleQVD.qvd(qvd);

  EXIT Script;

END IF

NoConcatenate

Sale:

LOAD OrderId,

     ProductId,

     CustomerId,

     Amount,

     Date_Updated,

     NumDate

FROM

$(vSourceQvdFile)SaleQVD.qvd(qvd);

NoConcatenate

SaleOrder:

  Load Max(NumDate) as MaxDate

  Resident Sale;

  let vMaxUpdateDate=peek('MaxDate',0,'SaleOrder');

  Drop Table SaleOrder;

NoConcatenate

IncrementalLoad:

LOAD * where NumDate>$(vMaxUpdateDate);

LOAD *,Num(Date_Updated) as NumDate;

LOAD OrderId,

      ProductId,

      CustomerId,

      Amount,

      Date_Updated

     

FROM

$(vSourceData)Sale.xlsx

(ooxml, embedded labels, table is Sheet1)

;

Concatenate

LOAD OrderId,

     ProductId,

     CustomerId,

     Amount,

     Date_Updated,

     NumDate

FROM

$(vSourceQvdFile)SaleQVD.qvd(qvd)

where not Exists(OrderId);

CountOfRecords:

Load count(OrderId) as CountOrderId

Resident IncrementalLoad;

let vCountofRecords=Peek('CountOrderId',0,'CountOfRecords');

if vCountofRecords<>0 then

  store IncrementalLoad  into $(vSourceQvdFile)SaleQVD.qvd(qvd);

  //store IncrementalLoad  into $(vSourceQvdFile)Sale1.qvd(qvd);

end if

//Drop table IncrementalLoad;

Peter_Cammaert
Partner - Champion III
Partner - Champion III

Is it really useful to do an incremental load from an Excel file? How many rows does the original Excel contain?

Peter_Cammaert
Partner - Champion III
Partner - Champion III

Usually an incremental load is performed in order to save both time and resources. But your scripts reads the full Excel file anyway, sometimes twice (the very first time)...