Data Lineage in Qlik Sense needs to improve. Current feature has some problem
We have some problem with data lineage functional. Below I try to explain you what I mean.
Intro: we have Qlik Sense Enterprise Server and November 2020 SR3 version of software. We work with different api services for getting data from there. And often we use cycles for data extract processes. Sometimes we see that size of our qvd files only increases but its files can have very small counts of data rows.
I researched this problem and came to the following conclusion: Qlik writes all the previous information from the moment of launch to the headers of the QVD files.
Let's take a look at the following example:
LOAD * Inline [
For iRowIterationTBL = 0 to NoOfRows('IterationTBL') - 1
LET vIterationNum = Peek('iteration',$(iRowIterationTBL),'IterationTBL');
Trace vIterationNum = $(vIterationNum);
'$(vIterationNum)' as iteration_no
STORE [LOG $(vIterationNum)] into 'C:\Data_Temp\LOG $(vIterationNum).qvd' (qvd);
DROP Table [LOG $(vIterationNum)];
As a result, we got 20 QVD files with different sizes.
And if we open header Lineage section we find that in LOG 1.qvd has following info: