Qlik app loading or processing consumes C Drive Space ?

[rohitk1609]

Dear Techies,

I am facing a problem where C drive free space gradually decrease when I run Qlik app.  My Qlik apps mainly pull Qvds , process it and dump it to again in Qvd form. 

If I explain our app processing it takes 25 GB Qvd , 150,000,000 records, then it apply aggregates and joins then dump it to new qvds.

Our server is windows 2016 R2 and has 300 GB space in C Drive. When I execute above explained Qlik app process it consumes 140 GB space. when I my Qlik task is over still memory doesn't release instantly. 

When i reboot the server it releases 140 GB space from C Drive. 

If I run Qlik app again same space gets consumed.

My concern is, this is how Qlik works or something is not configured at windows level ?

I hope a positive response.

Thanks,

Rohit

[dplr-rn]

Most likely qlik is creating temp files while overwriting the documents but these should get released post reload. Did you check with qlik support?

[Levi_Turner]

Is this about memory or about disk space?

For disk space, I'd use a utility like Wiztree to see what file(s) are taking up the space. Based on the fact that the disk space becomes free after reboot, I'd guess it's the page file for the Windows OS. But you'd want to confirm that. If confirmed this means that the OS is out of RAM resources and thus writes to disk in order to continue operating.

[rohitk1609]

Thanks Levi for your important note.

Can I know more about wiztree software, is it paid or free and any side affect or bad consequences on production server ?

Secondly, on first look I thought the same it is because of pagination. Yes we have large size of Qvds 25 GB, when we load it we do aggregation and grouping on millions of records. More than 100 GB space get occupied by this pagination gradually. So how should we manage it, increase size in  C drive ? reboot every week ? or anything you would like us to suggest ?

Thanks,

Rohit

[rohitk1609]

One solution I know is cleanse or delete pagefile.sys file time to time. If you agree with me please confirm it and if you can suggest any article to follow which would guide us how to manager pagefile.sys then it will be great ?

Thanks,

Rohit

[marcus_sommer]

I doubt that's a great idea to intervene in any way into the OS. I think that increasing the amount of RAM and/or an optimizing to the Qlik processes would be more sensible.

Optimizing means to check if the amount of data could be reduced in regard to columns, records, formatting of the values and also the cardinality of the fields. Further if (more) incremental approaches could be applied and at least some kind of (more detailed) slicing from the data to avoid an overflow of RAM.

- Marcus

[rohitk1609]

cardinality of the fields means ?

[marcus_sommer]

The-Importance-Of-Being-Distinct

[rohitk1609]

Dear Marcus,

I have increased the size of C drive by 200 GB but there is a weird thing, newly added 200 GB is also getting consumed fully with same Qlik tasks.

Bizarre thing is that Qlik tasks are over but space is not released automatically. Can i know how much time Windows OS takes to free the space occupied by pagination ?

Thanks,

Rohit 

[rohitk1609]

I have checked Virtual Memory setting on my server, thee is unchecked on Automatically manage paging on all drives and it is set to 1024 MB. So I am not sure why virtual memory is expanding to 400 GB ?

I have checked the size of pagefile.sys file and it is 1  GB, then  how could 400 GB space is consumed gradually by big tasks ?

Thanks,

Rohit