topic Re: Incremental Load (Append Into QVD File) in QlikView

Incremental Load (Append Into QVD File)

Fri, 04 Sep 2015 08:22:04 GMT

hello dears.

i implemented Incremental Load (Insert ,Update ,Delete)

i have a problem .

for the first time , i created qvd File. (i don't problem)

let vQVDExistsFile=isnull(QvdCreateTime('$(vSourceQvdFile)SaleQVD.qvd'));

if vQVDExistsFile=-1 then

Sale:

LOAD OrderId,

ProductId,

CustomerId,

Amount,

Date_Updated,

Num(Date_Updated) as NumDate

FROM

$(vSourceData)Sale.xlsx

(ooxml, embedded labels, table is Sheet1);

Store Sale into $(vSourceQvdFile)SaleQVD.qvd(qvd);

END IF

I found that change records for the second time

But i Can not Append into SaleQVD.qvd

All New Records (Update,Delete) Override into SaleQVD,Not Append

Re: Incremental Load (Append Into QVD File)

petter — Fri, 04 Sep 2015 08:35:49 GMT

There is no append option when you write to a QVD file. It will always have to be rewritten in full. That is why you have to do the appending in-memory and concatenate the old data with you new data. So you have the old data stored in a QVD and new data has to be read from another source and concatenated with the table that was read from the QVD.

You have to have some statements between the LOAD .... (qvd) and the Store. Which should look a bit like this:

CONCATENATE LOAD (Sale)

SQL

.....

;

CONCATENATE LOAD (Sale)

FROM filesource.txt (txt);

Re: Incremental Load (Append Into QVD File)

Anonymous — Fri, 04 Sep 2015 08:37:29 GMT

hi leila,

Insert and Update

The next case is applicable when data in previously loaded records may have changed between script executions. The following conditions apply:

The data source can be any database.
QlikView loads records inserted into the database or updated in the database after the last script execution
A field ModificationDate (or similar) is required for QlikView to recognize which records are new.
A primary key field is required for QlikView to sort out updated records from the QVD file.
This solution will force the reading of the QVD file to standard mode (rather than optimized), which is still considerably faster than loading the entire database.

Script Example:

QV_Table:

SQL SELECT PrimaryKey, X, Y FROM DB_TABLE

WHERE ModificationTime >= #$(LastExecTime)#;

Concatenate LOAD PrimaryKey, X, Y FROM File.QVD

WHERE NOT Exists(PrimaryKey);

STORE QV_Table INTO File.QVD;

Regards

Neetha

Re: Incremental Load (Append Into QVD File)

Fri, 04 Sep 2015 08:45:41 GMT

hello peter

Can you see Attachment File.?

I've done exactly that

Re: Incremental Load (Append Into QVD File)

Fri, 04 Sep 2015 08:50:27 GMT

hello neetha

Can you see Attachment File.?

I've done exactly that

Re: Incremental Load (Append Into QVD File)

Anonymous — Fri, 04 Sep 2015 08:53:20 GMT

hi leila

please paste the full code here,mine is personnel edition

Re: Incremental Load (Append Into QVD File)

Fri, 04 Sep 2015 08:57:40 GMT

Thanks

Incremental.zip

Attached Full Code

Re: Incremental Load (Append Into QVD File)

Fri, 04 Sep 2015 08:59:44 GMT

set vSourceQvdFile='QVDFile\';

set vSourceData='Source\';

let vQVDExistsFile=isnull(QvdCreateTime('$(vSourceQvdFile)SaleQVD.qvd'));

if vQVDExistsFile=-1 then

Sale:

LOAD OrderId,

ProductId,

CustomerId,

Amount,

Date_Updated,

Num(Date_Updated) as NumDate

FROM

$(vSourceData)Sale.xlsx

(ooxml, embedded labels, table is Sheet1);

Store Sale into $(vSourceQvdFile)SaleQVD.qvd(qvd);

EXIT Script;

END IF

NoConcatenate

Sale:

LOAD OrderId,

ProductId,

CustomerId,

Amount,

Date_Updated,

NumDate

FROM

$(vSourceQvdFile)SaleQVD.qvd(qvd);

NoConcatenate

SaleOrder:

Load Max(NumDate) as MaxDate

Resident Sale;

let vMaxUpdateDate=peek('MaxDate',0,'SaleOrder');

Drop Table SaleOrder;

NoConcatenate

IncrementalLoad:

LOAD * where NumDate>$(vMaxUpdateDate);

LOAD *,Num(Date_Updated) as NumDate;

LOAD OrderId,

ProductId,

CustomerId,

Amount,

Date_Updated

FROM

$(vSourceData)Sale.xlsx

(ooxml, embedded labels, table is Sheet1)

;

Concatenate

LOAD OrderId,

ProductId,

CustomerId,

Amount,

Date_Updated,

NumDate

FROM

$(vSourceQvdFile)SaleQVD.qvd(qvd)

where not Exists(OrderId);

CountOfRecords:

Load count(OrderId) as CountOrderId

Resident IncrementalLoad;

let vCountofRecords=Peek('CountOrderId',0,'CountOfRecords');

if vCountofRecords<>0 then

store IncrementalLoad into $(vSourceQvdFile)SaleQVD.qvd(qvd);

//store IncrementalLoad into $(vSourceQvdFile)Sale1.qvd(qvd);

end if

//Drop table IncrementalLoad;

Re: Incremental Load (Append Into QVD File)

Peter_Cammaert — Fri, 04 Sep 2015 09:07:32 GMT

Is it really useful to do an incremental load from an Excel file? How many rows does the original Excel contain?

Re: Incremental Load (Append Into QVD File)

Peter_Cammaert — Fri, 04 Sep 2015 09:09:42 GMT

Usually an incremental load is performed in order to save both time and resources. But your scripts reads the full Excel file anyway, sometimes twice (the very first time)...

Re: Incremental Load (Append Into QVD File)

Fri, 04 Sep 2015 09:15:58 GMT

no , i want Implemented Incremental Load on fact Tables With millions Record.

i want Test And Applied in Fact Table