Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
vlad_komarov
Partner - Specialist III
Partner - Specialist III

Empty QVD files are growing in size if generated in the loop

I am running a data extraction from large database and I have to load the data one day at a time. I am saving these daily files into QVD and concatenating them afterwards.

Some of the extractions are returning 0 records and some of these QVD files are empty.

But I’ve noticed that these empty files are growing in size after every cycle…

Screen Shot 2018-02-24 at 2.05.56 PM.png

This growth depends on the number of fields in the table, of course, but I could not find any reason for that.

I’ve tried this on both QS 3.2 and QS Sept 2017, but results are the same.

I’ve saved these empty daily tables to CSV files, and they are identical in size:

Screen Shot 2018-02-24 at 2.08.19 PM.png

I’ve created a sample app that generates empty files and here are results (attached).

Any idea why it’s happening?

1 Solution

Accepted Solutions
rwunderlich
Partner Ambassador/MVP
Partner Ambassador/MVP

This is a known problem where lineage data is accumulated in the QVD.  See this thread:

Expansion of QVD file size during iterated process

-Rob

View solution in original post

7 Replies
Digvijay_Singh

Can you share your script to check the code writing  daily records into QVDs.

vlad_komarov
Partner - Specialist III
Partner - Specialist III
Author

Sure (see attached)

Digvijay_Singh

Not sure at what step you get the problem. I ran your script and I see just 3kb qvd file, I executed script again and again and don't see any size change etc. Let me know what I am missing in attached qvf.

One way size could have grown if it doesn't have statement "Drop Table Cycle_QVD_TEST;", But its there in your sample so it won't concatenate with the previous file.

rwunderlich
Partner Ambassador/MVP
Partner Ambassador/MVP

This is a known problem where lineage data is accumulated in the QVD.  See this thread:

Expansion of QVD file size during iterated process

-Rob

vlad_komarov
Partner - Specialist III
Partner - Specialist III
Author

Thank you, Rob!

Did not realize that this problem was carried on from QV time..

Interesting to see if Lineage information storage could be disabled in QS as well.


Regards,

Vlad

vlad_komarov
Partner - Specialist III
Partner - Specialist III
Author

Interesting point made by olaf.gschweng in the post you've mentioned.

"After all tables have been dropped the lineage info seems to be cleared"...


It does not look like my case...

I am dropping tables every time the Store command is completed....


But it still produces larger QVD files in each cycle...


VK

rwunderlich
Partner Ambassador/MVP
Partner Ambassador/MVP

The lineage is cleared form the QVW, not the QVD.

-Rob