Empty QVD files are growing in size if generated in the loop

[vlad_komarov]

I am running a data extraction from large database and I have to load the data one day at a time. I am saving these daily files into QVD and concatenating them afterwards.

Some of the extractions are returning 0 records and some of these QVD files are empty.

But I’ve noticed that these empty files are growing in size after every cycle…

This growth depends on the number of fields in the table, of course, but I could not find any reason for that.

I’ve tried this on both QS 3.2 and QS Sept 2017, but results are the same.

I’ve saved these empty daily tables to CSV files, and they are identical in size:

I’ve created a sample app that generates empty files and here are results (attached).

Any idea why it’s happening?

[Digvijay_Singh]

Can you share your script to check the code writing  daily records into QVDs.

[vlad_komarov]

Sure (see attached)

[Digvijay_Singh]

Not sure at what step you get the problem. I ran your script and I see just 3kb qvd file, I executed script again and again and don't see any size change etc. Let me know what I am missing in attached qvf.

One way size could have grown if it doesn't have statement "Drop Table Cycle_QVD_TEST;", But its there in your sample so it won't concatenate with the previous file.

[rwunderlich · Accepted solution]

This is a known problem where lineage data is accumulated in the QVD.  See this thread:

Expansion of QVD file size during iterated process

-Rob

[vlad_komarov]

Thank you, Rob!

Did not realize that this problem was carried on from QV time..

Interesting to see if Lineage information storage could be disabled in QS as well.

Regards,

Vlad

[vlad_komarov]

Interesting point made by olaf.gschweng in the post you've mentioned.

"After all tables have been dropped the lineage info seems to be cleared"...

It does not look like my case...

I am dropping tables every time the Store command is completed....

But it still produces larger QVD files in each cycle...

VK

[rwunderlich]

The lineage is cleared form the QVW, not the QVD.

-Rob