Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
We have some problem with data lineage functional. Below I try to explain you what I mean.
Intro: we have Qlik Sense Enterprise Server and November 2020 SR3 version of software. We work with different api services for getting data from there. And often we use cycles for data extract processes. Sometimes we see that size of our qvd files only increases but its files can have very small counts of data rows.
I researched this problem and came to the following conclusion: Qlik writes all the previous information from the moment of launch to the headers of the QVD files.
Let's take a look at the following example:
IterationTBL:
LOAD * Inline [
iteration
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
];
For iRowIterationTBL = 0 to NoOfRows('IterationTBL') - 1
LET vIterationNum = Peek('iteration',$(iRowIterationTBL),'IterationTBL');
Trace vIterationNum = $(vIterationNum);
[LOG $(vIterationNum)]:
LOAD
'$(vIterationNum)' as iteration_no
AutoGenerate 1;
STORE [LOG $(vIterationNum)] into 'C:\Data_Temp\LOG $(vIterationNum).qvd' (qvd);
DROP Table [LOG $(vIterationNum)];
NEXT iRowIterationTBL
As a result, we got 20 QVD files with different sizes.
And if we open header Lineage section we find that in LOG 1.qvd has following info:
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 1.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
If you look at the information in the log 20.qvd, you can find all the preceding lines.
<LineageInfo>
<Discriminator>AUTOGENERATE;</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>INLINE;</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 1.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 10.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 11.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 12.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 13.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 14.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 15.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 16.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 17.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 18.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 19.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 2.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 20.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 3.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 4.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 5.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 6.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 7.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 8.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
<LineageInfo>
<Discriminator>{STORE - 'C:\Data_Temp\LOG 9.qvd' (qvd)};</Discriminator>
<Statement></Statement>
</LineageInfo>
Due to the fact that a qlik accumulates metadata in itself and cannot reset lineage information at subsequent stages, a situation may arise when qvd with 1 data line has 31 MB size.
Is there some way to reset the lineage metadata during script execution?
Qlik Sense Business @Sonja_Bauernfeind @Yaaser_Ahmed_Khan @Daniel_Seo @Damien_V
I've created https://community.qlik.com/t5/Ideation-Discussions/Create-a-mechanism-for-resetting-the-cache-for-th... Idea. Please like it! 😉