topic How to handle incremental load in qlik view in QlikView

How to handle incremental load in qlik view

sakshikaul — Wed, 25 Nov 2020 16:16:04 GMT

Hi all ,

I have a query regarding handling incremental load in which I have the following scenario:-

I have a business which deals with two seasons winter and summer as 18A and 18P respectively.Here the data for both seasons is available ( full load) on FTP (Data source)

All the incremental files are also available on ftp .On daily basis the incremental data is also created on same FTP. This data do not contain any duplicate records. Please help how to handle the same ?

FULL LOAD

Material_Master:

LOAD distinct MATERIAL,

COLOR,

SIZE,

[Size Group],

Description,

COMPOSITION,

BRANDCD,

BRAND,

SEASON,

SEASONDESC,

THEME,

DELYCODE,

GENDER,

ProductHierarchy,

MaterialGroup,

[Core/Fashion],

FIT,

PATTERN,

[Key/NonKey],

RANGE,

ORIGIN,

[Top/Bottom],

SLEEVE,

NECK,

WEATHER,

OCCASSION,

CustomDesc,

MaterialGrp4,

MGRP4DESC,

EAN,

ALTEAN,

PurchaseGroup,

PurchaseGroupDesc,

CATEGORY,

SUBCAT,

BASECOMP,

MaterialType,

ValuationClass,

COST,

MRP,

ChangeDt

FROM

[ftp://ftpIPQ:ftpben16@10.47.77.207/IFR/IAP/QLK/MAT_MAS/MAT_MAS_FULL.txt]

(txt, codepage is 1252, embedded labels, delimiter is '|', msq);

store Material_Master into C:\Users\dmsadmin\Desktop\New folder\QVD\extract_qvd\Material_Master_Full.qvd;

drop table Material_Master;

NOTE

MAT_MAS_20120780_00003.txt and etc is the incremental load files whereas MAT_MAS_FULL.txt is full load

Re: How to handle incremental load in qlik view

BalaBhaskar_Qlik — Thu, 02 Aug 2018 14:41:03 GMT

May be like this:

Let ThisExecTime = Now( );

QV_Table:

SQL SELECT PrimaryKey, X, Y FROM DB_TABLE

WHERE ModificationTime >= #$(LastExecTime)#

AND ModificationTime < #$(ThisExecTime)#;

Concatenate LOAD PrimaryKey, X, Y FROM File.QVD

WHERE NOT EXISTS(PrimaryKey);

Inner Join SQL SELECT PrimaryKey FROM DB_TABLE;

If ScriptErrorCount = 0 then

STORE QV_Table INTO File.QVD;

Let LastExecTime = ThisExecTime;

End If

Check these:

https://www.resultdata.com/three-types-of-qlikview-incremental-loads/.

https://www.analyticsvidhya.com/blog/2014/09/qlikview-incremental-load/

Re: How to handle incremental load in qlik view

sakshikaul — Thu, 13 Sep 2018 04:12:01 GMT

when I am implementing what is given in the above link I am getting type D inconsistency error in my script

Re: How to handle incremental load in qlik view

Anonymous — Thu, 13 Sep 2018 04:40:40 GMT

Kindly Refer to this it might help you.

QlikView Incremental Load

Internal inconsistency Types explained

Thanks

Re: How to handle incremental load in qlik view

BalaBhaskar_Qlik — Thu, 13 Sep 2018 08:40:30 GMT

Type D, Its related to your memory of server . if you application require more memory compare to available memory, normally this error comes. If it is unable to consume memory during application reloading or debugging then this issue occurs.

This is when a scripting or data error (instead of inner or left join) results in a Cartesian join of two sizeable tables, i.e. QV tries (and fails) to creates millions of rows. A symptom of this particular scenario is that QV gobbles up all memory before crashing.