topic Re: Empty QVD files are growing in size if generated in the loop in App Development

Empty QVD files are growing in size if generated in the loop

vlad_komarov — Sat, 24 Feb 2018 20:10:51 GMT

I am running a data extraction from large database and I have to load the data one day at a time. I am saving these daily files into QVD and concatenating them afterwards.

Some of the extractions are returning 0 records and some of these QVD files are empty.

But I’ve noticed that these empty files are growing in size after every cycle…

This growth depends on the number of fields in the table, of course, but I could not find any reason for that.

I’ve tried this on both QS 3.2 and QS Sept 2017, but results are the same.

I’ve saved these empty daily tables to CSV files, and they are identical in size:

I’ve created a sample app that generates empty files and here are results (attached).

Any idea why it’s happening?

Re: Empty QVD files are growing in size if generated in the loop

Digvijay_Singh — Sat, 24 Feb 2018 21:19:31 GMT

Can you share your script to check the code writing daily records into QVDs.

Re: Empty QVD files are growing in size if generated in the loop

vlad_komarov — Sat, 24 Feb 2018 21:45:24 GMT

Sure (see attached)

Re: Empty QVD files are growing in size if generated in the loop

Digvijay_Singh — Sun, 25 Feb 2018 03:17:36 GMT

Not sure at what step you get the problem. I ran your script and I see just 3kb qvd file, I executed script again and again and don't see any size change etc. Let me know what I am missing in attached qvf.

One way size could have grown if it doesn't have statement "Drop Table Cycle_QVD_TEST;", But its there in your sample so it won't concatenate with the previous file.

Re: Empty QVD files are growing in size if generated in the loop

rwunderlich — Sun, 25 Feb 2018 16:19:27 GMT

This is a known problem where lineage data is accumulated in the QVD. See this thread:

Expansion of QVD file size during iterated process

-Rob

Re: Empty QVD files are growing in size if generated in the loop

vlad_komarov — Sun, 25 Feb 2018 21:21:19 GMT

Thank you, Rob!

Did not realize that this problem was carried on from QV time..

Interesting to see if Lineage information storage could be disabled in QS as well.

Regards,

Vlad

Re: Empty QVD files are growing in size if generated in the loop

vlad_komarov — Mon, 26 Feb 2018 14:10:24 GMT

Interesting point made by olaf.gschweng in the post you've mentioned.

"After all tables have been dropped the lineage info seems to be cleared"...

It does not look like my case...

I am dropping tables every time the Store command is completed....

But it still produces larger QVD files in each cycle...

Re: Empty QVD files are growing in size if generated in the loop

rwunderlich — Mon, 26 Feb 2018 17:59:44 GMT

The lineage is cleared form the QVW, not the QVD.

-Rob