How to handle incremental load ?

[Anonymous]

Hi All,

I have to implement incremental load and I have following material master on  FTP now how to handle incremental load? Here itemnumber is a primary key

Material_Master:

   LOAD

  MATERIAL,

     COLOR,

     SIZE,

     MATERIAL&COLOR&SIZE as itemnumber,

     [Size Group],

     Description,

     COMPOSITION,

     BRANDCD,

     BRAND,

     SEASON,

     SEASONDESC,

     THEME,

     DELYCODE,

     GENDER,

     IF(GENDER ='LADIES', 'WOMEN', IF(GENDER='MENS','MENS',  IF(GENDER='BOYS' or GENDER='GIRLS' or GENDER='BABY','KIDS', GENDER))) as Gender_Revised,

     ProductHierarchy,

     MaterialGroup,

     [Core/Fashion],

     FIT,

     PATTERN,

     [Key/NonKey],

     RANGE,

     ORIGIN,

     [Top/Bottom],

     SLEEVE,

     NECK,

     WEATHER,

     OCCASSION,

     CustomDesc,

     MaterialGrp4,

     MGRP4DESC,

     EAN,

     ALTEAN,

     PurchaseGroup,

     PurchaseGroupDesc,

     CATEGORY,

     SUBCAT,

     BASECOMP,

     MaterialType as Promotional_1,

     ValuationClass,

     COST,

     MRP as MRP_INR,

     if (MRP <1000, '<1000', if (MRP <2000, '1000-2000',if (MRP <3000, '2000-3000',if (MRP <4000, '3000-4000',if (MRP <5000, '4000-5000',if (MRP <6000, '5000-6000', '>6000')))))) as [MRP Bucket]

FROM

(qvd)

where Exists itemnumber;

[crusader_]

ResultingTable:

LOAD * FROM new.qvd (qvd);

LOAD * FROM X.qvd (qvd)

WHERE NOT EXISTS(UniqueID);

*UniqueID in your case is itemnumber

//Andrei

[Anonymous]

So each time I have to load all the new icr files and concatenate in old files? rite?

[crusader_]

Yes, that's right.... sequentially adding new data to full and overwrite full.

inc1 + old => old,

inc2 + old => old,

etc.

Andrei

[Anonymous]

I didnt get the last point. How to implement the last step ? ie  overwrite full?

[crusader_]

yes, otherwise how are you going to keep updated full file?

You will not keep all inc files forever, right? so you need a have common place to store changes, so full is your common place.

That's why I said you need to split your initial full and work on copy.

//Andrei

[Anonymous]

I have implemented the same but still not getting the correct figures. i have a field named gender, which i have pulled after implementing following above scenario., so please tell me is this incorrect data  because the values in gender feilds present in files on FTP are not correct right ?

[crusader_]

If you don't have any modifications on GENDER field, than most likely one of files contains this.

It couldn't appear just from nowhere or in the result of incremental load

You can load qvd file separately to check what is there or use EasyQlik QViewer tool.

//Andrei

[Anonymous]

orrected?

If I am selecting D under Gender field the chart is not showing any values. This means there is some deliminiter issue in master files? rite?

[crusader_]

Could you just copy and open file?

It might be delimiter issue as well as bad data... hard to guess without seeing data...

//Andrei

[Anonymous]

So, i can assume due to that I am Not getting correct data?