Skip to main content
Announcements
Introducing Qlik Answers: A plug-and-play, Generative AI powered RAG solution. READ ALL ABOUT IT!
cancel
Showing results for 
Search instead for 
Did you mean: 
MassicotPSCU
Contributor III
Contributor III

Incremental Load to compile news records daily and update existing ones.

I'm trying to do an incremental load that updates existing records but also compiles new records daily into a main qvd as well. I've been following the documentation here to do so (Insert and Update); however it's not doing what I expect. Loading new and updated records with incremental load | Qlik Sense on Windows Help

Instead of compiling the new and updated records into my main qvd daily, it just keeps the last set of inserted and updated records from 3 days ago and drops everything that was previously in the main qvd.

With my direct pull from the DB, I want to only load the last 3 days of both new records and updated records. I want to then concatenate that derived table daily to my main qvd that is supposed to have all records going back about 5 months ago (50,000 rows). The problem is my current script only keeps the last 3 days of records from the direct pull and drops everything that was previously in main qvd.  Here is an example of my current script. Please let me know if anything is happening that would drop all the data in my main qvd that was not from 3 days ago.  I think it has something do with the where not exists but I'm not sure as I'm following the documentation. 

LET vMinDate = date(today()-3);

DB:
LOAD
"Product ID",
"Date",
QTY,
"Product ID" &'-'& "Date" as "ID"
FROM [lib://AttachedFiles/Incremental Reload Test.xlsx]
(ooxml, embedded labels, table is Sheet1)
WHERE Date >= $(vMinDate);

 


CONCATENATE (DB)


/*QVD*/
LOAD
"Product ID",
"Date",
QTY,
"Product ID" &'-'& "Date" as "ID"
FROM [lib://AttachedFiles/Incremental Reload Test QVD.xlsx]
(ooxml, embedded labels, table is Sheet1)
WHERE not exists("Product ID" &'-'& "Date");

STORE DB INTO [lib://AttachedFiles/Incremental Reload Test QVD.xlsx]

 

Labels (4)
1 Solution

Accepted Solutions
rubenmarin

Hi, the concatenate should add data from the qvd, that's a file with qvd extension, and the store could also be a file  with qvd extension, using a qvd file the loads will be faster.

Also, as you suspect, the syntax for Exists() expects the first parameter as the field to seearch, and the second the value to search, so it should be: WHERE not exists(ID, "Product ID" &'-'& "Date");

View solution in original post

2 Replies
rubenmarin

Hi, the concatenate should add data from the qvd, that's a file with qvd extension, and the store could also be a file  with qvd extension, using a qvd file the loads will be faster.

Also, as you suspect, the syntax for Exists() expects the first parameter as the field to seearch, and the second the value to search, so it should be: WHERE not exists(ID, "Product ID" &'-'& "Date");

MassicotPSCU
Contributor III
Contributor III
Author

Thank you @rubenmarin ! The 1st argument in the exists was exactly what I was missing..