Skip to main content
Announcements
Global Transformation Awards! Applications are now open. Submit Entry
cancel
Showing results for 
Search instead for 
Did you mean: 
i_shamaev
Creator
Creator

Data Lineage in Qlik Sense needs to improve. Current feature has some problem

We have some problem with data lineage functional. Below I try to explain you what I mean.

Intro: we have Qlik Sense Enterprise Server and November 2020 SR3 version of software. We work with different api services for getting data from there. And often we use cycles for data extract processes. Sometimes we see that size of our qvd files only increases but its files can have very small counts of data rows. 

I researched this problem and came to the following conclusion: Qlik writes all the previous information from the moment of launch to the headers of the QVD files.

Let's take a look at the following example:

 

 

 

 

IterationTBL:
LOAD * Inline [
    iteration
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
];


For iRowIterationTBL = 0 to NoOfRows('IterationTBL') - 1

    LET vIterationNum = Peek('iteration',$(iRowIterationTBL),'IterationTBL');

    Trace vIterationNum = $(vIterationNum);

    [LOG $(vIterationNum)]:
    LOAD 
        '$(vIterationNum)' as iteration_no 
    AutoGenerate 1;

    STORE [LOG $(vIterationNum)] into 'C:\Data_Temp\LOG $(vIterationNum).qvd' (qvd);
    DROP Table [LOG $(vIterationNum)];

NEXT iRowIterationTBL

 

 

 

 

As a result, we got 20 QVD files with different sizes.

i_shamaev_0-1612470649883.png

And if we open header Lineage section we find that in LOG 1.qvd has following info:

 

 

 

<LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 1.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>

 

 

 

If you look at the information in the log 20.qvd, you can find all the preceding lines.

 

 

 

    <LineageInfo>
       <Discriminator>AUTOGENERATE;</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>INLINE;</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 1.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 10.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 11.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 12.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 13.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 14.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 15.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 16.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 17.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 18.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 19.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 2.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 20.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 3.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 4.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 5.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 6.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 7.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 8.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>
     <LineageInfo>
       <Discriminator>{STORE - 'C:\Data_Temp\LOG 9.qvd' (qvd)};</Discriminator>
       <Statement></Statement>
     </LineageInfo>

 

 

 

Due to the fact that a qlik accumulates metadata in itself and cannot reset lineage information at subsequent stages, a situation may arise when qvd with 1 data line has 31 MB size.

i_shamaev_0-1612471950400.png

Is there some way to reset the lineage metadata during script execution?
Qlik Sense Business @Sonja_Bauernfeind @Yaaser_Ahmed_Khan @Daniel_Seo @Damien_V 

Labels (1)
10 Replies
i_shamaev
Creator
Creator
Author