Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

Announcements
Join us to spark ideas for how to put the latest capabilities into action. Register here!
cancel
Showing results for 
Search instead for 
Did you mean: 
user8
Contributor III
Contributor III

Incremental Load - Lines fetched

Hi,

I created an incremental load script as below:

// Loading data from QVD
SRO:
LOAD 
	[Form Ref No],
	[Form Name],
    CompletionDate
FROM [lib://SRO.qvd] (qvd);

// Find last modified date
Last_Updated_Date:
load max (CompletionDate) as Maxdate
resident SRO;

// Store last modified date to a variable
Let Last_Updated_Date=Date(peek('Maxdate',0,'Last_Updated_Date'));
TRACE >>> $(Last_Updated_Date);

// Delete Table SRO
Drop table SRO;

LIB CONNECT TO 'SQL_ODBC';

// Load incremental data
Incremental:
LOAD 
	[Form Ref No],
	[Form Name],
    CompletionDate;

SELECT
	"Form Ref No",
	"Form Name", 
     CompletionDate 
	 FROM vwSROHeader where CompletionDate > $(Last_Updated_Date);
     
// Concatenate with QVD
Concatenate
LOAD 
	[Form Ref No],
	[Form Name],
    CompletionDate
FROM [lib://SRO.qvd] (qvd) where not Exists ("Form Ref No");

// Replace old QVD file
Store Incremental into [lib://SRO.qvd] (qvd);

// Drop Incremental Table
Drop table Incremental;     

Started loading data
SRO << SRO
(QVD (row-based) optimized)
Lines fetched: 1,181                           ====> There were 1,181 lines in the existing QVD file
Last_Updated_Date << SRO
Lines fetched: 1
>>> 12/9/2019                                       
Incremental << vwSROHeader
Lines fetched: 1,189                           ====> There were 1,189 lines in DB, which means 8 new lines will be added to the QVD file
Incremental << SRO
(QVD (row-based) optimized)
Lines fetched: 1,189                           ====> Here it shows 1,189 lines fetched. Does this mean it fetched all the 1,189 lines again or just the 8 new lines to the QVD File?
Creating search index
Search index creation completed successfully
App saved
Finished successfully

Please refer to my comments in red as above. How do I verify the script is working as expected i.e. fetching only the new lines (8) and not all the lines (1,189) into the QVD file?

Thank you.

 

Labels (2)
1 Reply
andoryuu
Creator III
Creator III

That number is the total rows fetched into the current table so yes, it's pulling the correct rows. RowNo() will show you the current row count of the transaction - so if it is concatenating it'll up that row count. If it were adding more than just the 8 rows that number would be higher.
When in doubt, I like to store out all my intermediary tables into QVDs and take a look at them with EasyQlik QViewer.