Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable

Incremental Load

Hi All,

I want to Implement Incremental Load for Below Scenario,

  • On daily Basis i will receive the new CSV file,i want to pick up the recent CSV file (Coding has been Done for this part)
  • If CSV file as new data or updated data Qlikview want to display new and updated record ,otherwise it will want to display the old data fetched from the previous CSV file

Date Column used is: Date_Update


Thanks in Advance,


10 Replies
Anil_Babu_Samineni

How is the CSV file names looks like in your folder?

Please add me Anil_Babu_Samineni to interact faster when reply back. Speak low think High.

Before develop something, think If placed (The Right information | To the right people | At the Right time | In the Right place | With the Right context)
chris_andersson
Partner - Contributor III
Partner - Contributor III

What type of incremental load would you say you are implementing? Should QV display old data from the previous CSV-file even if there's new data from the latest CSV?

Reference:
Using QVD files for incremental load ‒ QlikView

Anonymous
Not applicable
Author

For Getting Recent file from the Folder:


FOR EACH file1 in FileList('$(vFilePath2)*.csv');  // Loops each excel file in the given Folder 

LET vYears1 = mid(file1,26,28);

LET vDates1 = mid(vYears1,15,14);

LET vCSVData = Rangemax(vDates1,vCSVData);  // Gets the max year for every iteration 

Next

CSVdata File Format=CSV_Data_20170316032409

Vfilepath2=Holding the path location where CSV data will get downloaded on daily basis from SAP system.

Anonymous
Not applicable
Author

need to display the old data from previous CSV file and New and updated record from the New CSV file.

Anil_Babu_Samineni

If this is from SAP basis, Then good to know.

Anyway, To incr. You could try this

FOR EACH file1 in FileList('$(vFilePath2)*.csv');  // Loops each excel file in the given Folder 

LET vYears1 = mid(file1,26,28);

LET vDates1 = mid(vYears1,15,14);

LET vCSVData = Rangemax(vDates1,vCSVData);  // Gets the max year for every iteration 

Next


Yours:

LOAD Date_Update, Another;

Store Yours into [FilePath...];

Drop Table Yours;

Sample:

Load * From Yours;

Last_Update_Reload:

Load Max(Date_Update) as Max_Date Resident Sample;

Let VarForMax = Peek('Max_Date', 0, 'Last_Update_Reload');

Drop Table Sample;

First:

Load * From Sample_External_Source Where Date_Update > $(VarForMax);

Concatenate

Load * From <Sample Qvd Path>;

Store First into ...;

Drop Table First;

Load * From <First>;

Please add me Anil_Babu_Samineni to interact faster when reply back. Speak low think High.

Before develop something, think If placed (The Right information | To the right people | At the Right time | In the Right place | With the Right context)
chris_andersson
Partner - Contributor III
Partner - Contributor III

Will you be able to use QVD files to store the Incremental load?

From Qlik Help:

dr_gen_QvdPic01_290x332.png

Anonymous
Not applicable
Author

i will try and ping u if i need any clarification

Peter_Cammaert
Partner - Champion III
Partner - Champion III

It all starts with "how do you identify your rows in your data source"? The technical question would be: what is the primary key of your rows.

If you have a primary key, it would simply be a matter of reading the latest CSV file in its entirety, and then adding rows from all older CSV files that you haven't loaded already. The new ones will be loaded, the updated ones will be loaded and not overwritten by older ones, and the old ones will be loaded only if there are no newer copies available in the internal table.

As suggested by Christoffer Andersson, it would be very interesting if you could save the current internal table as a QVD containing the historical data at the end of your script. Next time you reload, you only load from a new CSV (newer than your last reload) and add historical data from the QVD. Performance will be much better.

Best,

Peter

Anonymous
Not applicable
Author

Hi I am Facing Error: Garbage in your Statement after Where Date_Update > $(VarForMax)