Qlik Community

QlikView App Dev

Discussion Board for collaboration related to QlikView App Development.

cancel
Showing results for 
Search instead for 
Did you mean: 
ericarru
Creator
Creator

Data Vanish on Oversized Document

Hello,

I am facing a very strange behaviour on QlikView which i think that could be due to a bug. I am currently creating a script of an incremental load which is downloading only the new data of a DB/QVD/CSV and storing it to QVD/CSV contatenated to the old data. The script is a llitle bit complex but i made sure to drop all the tables before finish the reload because de document just needs to process the data not keep it.

My surprise is that when I do this procedure for several tables (I use 9) with a lot of data, when the reload finish the document end up weighing 50MB on the QV server.  I copied to my desktop pc and then opened with QV and saved it with another name just fort testing. Just by doing that the document now weights 30MB. Thinking that it might be the data i reloaded the document with en exit script just before the use of that it has. Notthing happened.

Playing a little bit with the document i figured out that if i declare some specific variables the document size passes from 30MB to 200KB. There are several of those and one example of this variables is the following:

let vMinStartDataDate= 42430;

This variable has nothing special on it it is just to set a minimum start date but the same happen also with other variables such as:

Let vLoadSource='sql';

let vUserTag='MV';

let TablesFileTag='Test';

let vDocumentStep='Extract';

Any of these is fine, just using one at the begginging of the document is enoght.

Also I found that there is some point where if you declare the variable after that, the document stays with its 30MB. If instead you declare it before the document occupies 200KB. Also I tested that his still happen even if you comment the script in between. Just for completion i let here the script with the two points.

let vMinStartDataDate= 42430; // Point Before.

// if vDocumentStep = 'Extract' THEN

// Let LoadTalbeSheet = 'DB';

// Let SaveTalbeSheet = 'DB' ;

// Let vLoadSource='sql'; // Type 'SQL','QVD' or 'CSV'

// ELSEIF vDocumentStep='Transform' THEN

// Let LoadTalbeSheet = 'DB' ;

// Let SaveTalbeSheet = 'CT' ;

// Let vLoadSource='qvd'; // Type 'SQL','QVD' or 'CSV'

// ELSE

// Let LoadTalbeSheet = 'CT' ;

// Let SaveTalbeSheet = 'CT' ;

// Let vLoadSource='qvd'; // Type 'SQL','QVD' or 'CSV'

// ENDIF

let vMinStartDataDate= 42430; // Point After

Does anyone have an ide of how to solv this?

8 Replies
vamsee
Specialist
Specialist

That's a huge difference in size just because of the variables.

Anyway, as a workaround Maybe try running a macro at the end to delete all variables.

sub cleanVariables
Set VariableCollection = ActiveDocument.GetVariableDescriptions
for i = 0 to VariableCollection.Count - 1
set tempVar = VariableCollection.Item(i)
tempVarName = tempVar.Name

ActiveDocument.RemoveVariable tempVarName
next
msgbox "Deleted all variables "
end sub

Thanks.

ericarru
Creator
Creator
Author

Hi vamsee99999‌ ,

Thank you for your response.I tested the macro. It deleted all the variables but seems that the document still the same size . It seems like there is some memory that should have been previously deleted but it's not until you redefine one of those variables.

marcus_sommer
MVP & Luminary
MVP & Luminary

The filesize could change by a different compressing-setting (document properties in tab general) whereby it sounds are unlikely in your case by 50 MB vs 30 MB because usually their impact is bigger.

Another point is section access which "could" reduce the number of records by opeing the application if there not matches for all records. Just take a look within the script if there is any section access and go in the document properties in tab open and disable there section access and check it again.

Beside this you should find any differences in the number of records in the tableviewer (usually is a look on the fact-tables enough).

- Marcus

ericarru
Creator
Creator
Author

Hi marcus_sommer‌,

Thank you for your response.I tested with the compression and the QV stills with the same 30MB size (since it is the size that ends with once you save as a copy).


For my QV there is neither section access used nor activated option for it. If I understood well what you said is possible to have an oversized document due to the option it's activated. Since I had it deactivated I tried to activate it just to be sure. Surprisingly now the document sizes 200KB. What I will also try is if reloading the full document with this option activated never get to the initial 50MB which is the real problem.

As for the tables, at the end of the process I just have 2 tables with les than 500 rows each one. Also is possible to reload the document eliminating all the tables keeping the 30MB size.

So: Using the vamsee99999 I have a document of 30 MB size without any variable and any table on it.

marcus_sommer
MVP & Luminary
MVP & Luminary

Just to make it clear - copying the app from the server to your local machine and opening and saving it without any further change and/or reload altered the filesize from 50 MB to 30 MB? Beside section access (are you really sure that there is no one active ?) there are probably not many possibilities - one might be different releases of QlikView.

If there is instead a reload and maybe even some changes on variables or the script the results could of course be very different. Beside this seems 30 or 50 MB a bit too much for just 1000 records.

- Marcus

ericarru
Creator
Creator
Author

I did some more testing on the document. There is also an other ways to decrease the size of the document to 200KB.

The way is reload it without the 2 tables (again, they are less tan 1000 rows and 20 different fields). And then delete some sheet object. Any random sheet object is ok.

Also happen if you reload the document and do Save As. It seems that the document may be corrupted.

ericarru
Creator
Creator
Author

Yes, you are right. Now I'll try to replicate it so I can confirm 100% if I have to open the document and save/save as or only coping is enough.

ericarru
Creator
Creator
Author

Hello I got to a workaround solution just by throwing the tables at the end of the execution of the script (which is strange because if you have the document saved an then just throw the tables by reloading it you don't het the same result).

This workaround is working because i don't need the tables but if i needed them...