Load data from Excel file that changes overnight

[badicsis]

I have an excel file called 'Daily data'  dropped into a folder every morning (let's say including 01 Jan 2020 data). I need to get this data automatically uploaded into QlikSense and Store it. Then next day there will be the file called the same name 'Daily data' but now it includes 02 Jan 2020 data (it will not include 01 Jan 2020 data anymore) and I need to get this uploaded into Qlik sense too and store it and so on, every day. I know how to schedule a daily recurring task, the question is how do i achieve that Qlik stores the data I've already uploaded so when new data uploads next day it will add to the previous days data.

Thanks!

[jwjackso · Accepted solution]

After you load the excel file, add a test to see if a QVD (that you are going to create) exists.  if it exists, concatenate the data in the QVD to the data you just loaded.

 Data:

Load Excel file;

If IsNull(QvdCreateTime(Some File Name) ) <> Null() Then

   Concatenate;

   Load QVD

End If

// Now create the QVD with both the new data and the old data

Store  Data into Data.QVD (qvd);

[fosuzuki]

I can think of two approaches:

1. Create an Extractor app which will generate one QVD for each load ("DailyData_YYYYMMDD.qvd"), containing only the data for the respective day. Then, you can have a second app loading the data from all QVDs using something like "LOAD * FROM DailyData_*.qvd".

2. Create an Extractor with Incremental Load approach. Check this help article.The output will be a single QVD, accumulating all the data.

[badicsis]

Thanks Jwjackso, i like the idea, but the new excel data simply overwrote the previous data in the app. I did the below after loading the excel file. I wonder how to fix it please?

If IsNull(QvdCreateTime(T_RadCalls.qvd)) <> Null() then
Concatenate

Load [T_RadCalls.qvd];

End if

Store RadCalls into [lib://OtherSources\T_RadCalls.qvd]
(qvd);
Drop Table RadCalls

[NitinK7]

 first you create a QVD maker to store all data in QVD, see below code for QVD maker

Set vQVD_File = '$(v_QVDPath)/filename.qvd';
Let vQVD_File_exist=If(FileSize('$(vQVD_File)')>0,-1,0);

If $(vQVD_File_exist) Then

tablename:

load *

from your Excel file;

concatenate

load *

from QVD file;

Else

tablename:

load *

from your Excel file;

end if

store  tablename into [your file path/filename.qvd] (QVD);

after loading this QVD maker

you will try to use this qvd file in your dashboard/application.

Thanks,

Nitin.

[badicsis]

Thanks guys for all the help, I kind of came up with my own simple solution based on your helpful notes, basically Loaded my first original excel sheet and created a qvd,
then i replaced my excel sheet with the new data and replaced my script to Load this new excel sheet fields first then load the fields from the qvd file (they are the same fields)
so Qlik Sense automatically concatenates them.

Original script for first load:

Tablename:
LOAD * FROM Excel
Store Tablename into [Path\filename.qvd]
(qvd);

The above created a qvd file so I changed my script as per below:

Tablename:
LOAD * FROM Excel;

LOAD * FROM [Path\filename.qvd]
(qvd);

STORE Tablename into [Path\filename.qvd]
(qvd);
Drop Table Tablename