Incremental Load

[Anonymous]

Hi All,

I want to Implement Incremental Load for Below Scenario,

• On daily Basis i will receive the new CSV file,i want to pick up the recent CSV file (Coding has been Done for this part)
• If CSV file as new data or updated data Qlikview want to display new and updated record ,otherwise it will want to display the old data fetched from the previous CSV file

Date Column used is: Date_Update

Thanks in Advance,

[Anil_Babu_Samineni]

How is the CSV file names looks like in your folder?

[chris_andersson]

What type of incremental load would you say you are implementing? Should QV display old data from the previous CSV-file even if there's new data from the latest CSV?

Reference:
Using QVD files for incremental load ‒ QlikView

[Anonymous]

For Getting Recent file from the Folder:

FOR EACH file1 in FileList('$(vFilePath2)*.csv');  // Loops each excel file in the given Folder 

LET vYears1 = mid(file1,26,28);

LET vDates1 = mid(vYears1,15,14);

LET vCSVData = Rangemax(vDates1,vCSVData);  // Gets the max year for every iteration 

Next

CSVdata File Format=CSV_Data_20170316032409

Vfilepath2=Holding the path location where CSV data will get downloaded on daily basis from SAP system.

[Anonymous]

need to display the old data from previous CSV file and New and updated record from the New CSV file.

[Anil_Babu_Samineni]

If this is from SAP basis, Then good to know.

Anyway, To incr. You could try this

FOR EACH file1 in FileList('$(vFilePath2)*.csv');  // Loops each excel file in the given Folder 

LET vYears1 = mid(file1,26,28);

LET vDates1 = mid(vYears1,15,14);

LET vCSVData = Rangemax(vDates1,vCSVData);  // Gets the max year for every iteration 

Next

Yours:

LOAD Date_Update, Another;

Store Yours into [FilePath...];

Drop Table Yours;

Sample:

Load * From Yours;

Last_Update_Reload:

Load Max(Date_Update) as Max_Date Resident Sample;

Let VarForMax = Peek('Max_Date', 0, 'Last_Update_Reload');

Drop Table Sample;

First:

Load * From Sample_External_Source Where Date_Update > $(VarForMax);

Concatenate

Load * From <Sample Qvd Path>;

Store First into ...;

Drop Table First;

Load * From <First>;

[chris_andersson]

Will you be able to use QVD files to store the Incremental load?

From Qlik Help:

[Anonymous]

i will try and ping u if i need any clarification

[Peter_Cammaert]

It all starts with "how do you identify your rows in your data source"? The technical question would be: what is the primary key of your rows.

If you have a primary key, it would simply be a matter of reading the latest CSV file in its entirety, and then adding rows from all older CSV files that you haven't loaded already. The new ones will be loaded, the updated ones will be loaded and not overwritten by older ones, and the old ones will be loaded only if there are no newer copies available in the internal table.

As suggested by Christoffer Andersson, it would be very interesting if you could save the current internal table as a QVD containing the historical data at the end of your script. Next time you reload, you only load from a new CSV (newer than your last reload) and add historical data from the QVD. Performance will be much better.

Best,

Peter

[Anonymous]

Hi I am Facing Error: Garbage in your Statement after Where Date_Update > $(VarForMax)