Qlik Community

QlikView Scripting

Discussion Board for collaboration on QlikView Scripting.

Not applicable

Incremental Load (Append Into QVD File)

hello dears.

i implemented Incremental Load (Insert ,Update ,Delete)

i have a problem .

for the first time , i created qvd File. (i don't problem)

         

let vQVDExistsFile=isnull(QvdCreateTime('$(vSourceQvdFile)SaleQVD.qvd'));

if vQVDExistsFile=-1 then

  Sale:

  LOAD OrderId,

      ProductId,

      CustomerId,

      Amount,

      Date_Updated,

      Num(Date_Updated) as NumDate

  FROM

  $(vSourceData)Sale.xlsx

  (ooxml, embedded labels, table is Sheet1);

  Store Sale into $(vSourceQvdFile)SaleQVD.qvd(qvd);

END IF

I found that change records for the second time

But i Can not Append into SaleQVD.qvd

All New Records (Update,Delete) Override into SaleQVD,Not Append

10 Replies
MVP
MVP

Re: Incremental Load (Append Into QVD File)

There is no append option when you write to a QVD file. It will always have to be rewritten in full. That is why you have to do the appending in-memory and concatenate the old data with you new data. So you have the old data stored in a QVD and new data has to be read from another source and concatenated with the table that was read from the QVD.

You have to have some statements between the LOAD .... (qvd) and the Store. Which should look a bit like this:

CONCATENATE LOAD (Sale)

*;

SQL

.....

;

or

CONCATENATE LOAD (Sale)

*

FROM filesource.txt (txt);

neetha_p
Honored Contributor

Re: Incremental Load (Append Into QVD File)

hi leila,

Insert and Update

The next case is applicable when data in previously loaded records may have changed between script executions. The following conditions apply:

  • The data source can be any database.
  • QlikView loads records inserted into the database or updated in the database after the last script execution
  • A field ModificationDate (or similar) is required for QlikView to recognize which records are new.
  • A primary key field is required for QlikView to sort out updated records from the QVD file.
  • This solution will force the reading of the QVD file to standard mode (rather than optimized), which is still considerably faster than loading the entire database.

Script Example:

QV_Table:

SQL SELECT PrimaryKey, X, Y FROM DB_TABLE

WHERE ModificationTime >= #$(LastExecTime)#;

Concatenate LOAD PrimaryKey, X, Y FROM File.QVD

WHERE NOT Exists(PrimaryKey);

STORE QV_Table INTO File.QVD;

Regards

Neetha

Not applicable

Re: Incremental Load (Append Into QVD File)

hello peter

Can you see Attachment File.?

I've done exactly that

Not applicable

Re: Incremental Load (Append Into QVD File)

hello neetha

Can you see Attachment File.?

I've done exactly that

neetha_p
Honored Contributor

Re: Incremental Load (Append Into QVD File)

hi leila

please paste the full code here,mine is personnel edition

Not applicable

Re: Incremental Load (Append Into QVD File)

Thanks

Incremental.zip

Attached Full Code

Not applicable

Re: Incremental Load (Append Into QVD File)

set vSourceQvdFile='QVDFile\';

set vSourceData='Source\';

let vQVDExistsFile=isnull(QvdCreateTime('$(vSourceQvdFile)SaleQVD.qvd'));

if vQVDExistsFile=-1 then

  Sale:

  LOAD OrderId,

      ProductId,

      CustomerId,

      Amount,

      Date_Updated,

      Num(Date_Updated) as NumDate

  FROM

  $(vSourceData)Sale.xlsx

  (ooxml, embedded labels, table is Sheet1);

  Store Sale into $(vSourceQvdFile)SaleQVD.qvd(qvd);

  EXIT Script;

END IF

NoConcatenate

Sale:

LOAD OrderId,

     ProductId,

     CustomerId,

     Amount,

     Date_Updated,

     NumDate

FROM

$(vSourceQvdFile)SaleQVD.qvd(qvd);

NoConcatenate

SaleOrder:

  Load Max(NumDate) as MaxDate

  Resident Sale;

  let vMaxUpdateDate=peek('MaxDate',0,'SaleOrder');

  Drop Table SaleOrder;

NoConcatenate

IncrementalLoad:

LOAD * where NumDate>$(vMaxUpdateDate);

LOAD *,Num(Date_Updated) as NumDate;

LOAD OrderId,

      ProductId,

      CustomerId,

      Amount,

      Date_Updated

     

FROM

$(vSourceData)Sale.xlsx

(ooxml, embedded labels, table is Sheet1)

;

Concatenate

LOAD OrderId,

     ProductId,

     CustomerId,

     Amount,

     Date_Updated,

     NumDate

FROM

$(vSourceQvdFile)SaleQVD.qvd(qvd)

where not Exists(OrderId);

CountOfRecords:

Load count(OrderId) as CountOrderId

Resident IncrementalLoad;

let vCountofRecords=Peek('CountOrderId',0,'CountOfRecords');

if vCountofRecords<>0 then

  store IncrementalLoad  into $(vSourceQvdFile)SaleQVD.qvd(qvd);

  //store IncrementalLoad  into $(vSourceQvdFile)Sale1.qvd(qvd);

end if

//Drop table IncrementalLoad;

Re: Incremental Load (Append Into QVD File)

Is it really useful to do an incremental load from an Excel file? How many rows does the original Excel contain?

Re: Incremental Load (Append Into QVD File)

Usually an incremental load is performed in order to save both time and resources. But your scripts reads the full Excel file anyway, sometimes twice (the very first time)...

Community Browser