Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi [Qlik Community],
I have one table called sql.Table where from I am loading the data everyday.
//Full Reload
table:
Load * FROM sql.table;
//Partial Reload
table:
Add Load * FROM sql.table where date = today()-3;
The problem with the partial reload is, it is always adding the data into the same table while it is the right purpose of the Partial Reload.
What I need, I want to load only the latest data that is available in the same table.
Later, I scheduled this app via Partial Reload in QMC.
Thanks
What you want to do here is persist the data you are loading to QVD, using the STORE statement.
If you have a unique ID in sql.table you can then use WHERE EXISTS to only load rows from the persisted QVD which are not in the latest load. Something like this:
// Get recent rows
table:
LOAD * FROM sql.table WHERE date >= today()-3;
// If QVD present add from here
if alt(FileSize('lib://DataFiles/MyQVD.qvd'), 0) > 0 then
CONCATENATE(table)
LOAD * FROM [lib://DataFiles/MyQVD.qvd] (qvd)
WHERE NOT EXISTS (ID);
end if
// Write all data back to QVD
STORE table INTO [lib://DataFiles/MyQVD.qvd] (qvd);
The store and load to a QVD is very quick and doing it this way means the data is persisted and can be shared between different applications. The partial load functionality in Sense is not used.
If there is no unique ID on the table then you need to think a bit more about how the incremental load will work. Trying to get to the point where a WHERE EXISTS can be used is important, as this allows an optmised QVD load.
If you have a transactional table you don't even need to go back three days, you could get the time of the last load and simply load since that time (perhaps with a little added to account for daylight saving once a year and a bit of a buffer).
You can find more reading on incremental loads here:
https://www.quickintelligence.co.uk/qlikview-incremental-load/
Hope that helps,
Steve