Qlik Community

Qlik Sense App Development

Discussion board where members can learn more about Qlik Sense App Development and Usage.

vlad_komarov
Valued Contributor III

Empty QVD files are growing in size if generated in the loop

I am running a data extraction from large database and I have to load the data one day at a time. I am saving these daily files into QVD and concatenating them afterwards.

Some of the extractions are returning 0 records and some of these QVD files are empty.

But I’ve noticed that these empty files are growing in size after every cycle…

Screen Shot 2018-02-24 at 2.05.56 PM.png

This growth depends on the number of fields in the table, of course, but I could not find any reason for that.

I’ve tried this on both QS 3.2 and QS Sept 2017, but results are the same.

I’ve saved these empty daily tables to CSV files, and they are identical in size:

Screen Shot 2018-02-24 at 2.08.19 PM.png

I’ve created a sample app that generates empty files and here are results (attached).

Any idea why it’s happening?

1 Solution

Accepted Solutions

Re: Empty QVD files are growing in size if generated in the loop

This is a known problem where lineage data is accumulated in the QVD.  See this thread:

Expansion of QVD file size during iterated process

-Rob

7 Replies
Digvijay_Singh
Honored Contributor III

Re: Empty QVD files are growing in size if generated in the loop

Can you share your script to check the code writing  daily records into QVDs.

vlad_komarov
Valued Contributor III

Re: Empty QVD files are growing in size if generated in the loop

Sure (see attached)

Digvijay_Singh
Honored Contributor III

Re: Empty QVD files are growing in size if generated in the loop

Not sure at what step you get the problem. I ran your script and I see just 3kb qvd file, I executed script again and again and don't see any size change etc. Let me know what I am missing in attached qvf.

One way size could have grown if it doesn't have statement "Drop Table Cycle_QVD_TEST;", But its there in your sample so it won't concatenate with the previous file.

Re: Empty QVD files are growing in size if generated in the loop

This is a known problem where lineage data is accumulated in the QVD.  See this thread:

Expansion of QVD file size during iterated process

-Rob

vlad_komarov
Valued Contributor III

Re: Empty QVD files are growing in size if generated in the loop

Thank you, Rob!

Did not realize that this problem was carried on from QV time..

Interesting to see if Lineage information storage could be disabled in QS as well.


Regards,

Vlad

vlad_komarov
Valued Contributor III

Re: Empty QVD files are growing in size if generated in the loop

Interesting point made by olaf.gschweng in the post you've mentioned.

"After all tables have been dropped the lineage info seems to be cleared"...


It does not look like my case...

I am dropping tables every time the Store command is completed....


But it still produces larger QVD files in each cycle...


VK

Re: Empty QVD files are growing in size if generated in the loop

The lineage is cleared form the QVW, not the QVD.

-Rob

Community Browser