Qlik app loading or processing consumes C Drive Space ?

[rohitk1609]

Dear Techies,

I am facing a problem where C drive free space gradually decrease when I run Qlik app.  My Qlik apps mainly pull Qvds , process it and dump it to again in Qvd form. 

If I explain our app processing it takes 25 GB Qvd , 150,000,000 records, then it apply aggregates and joins then dump it to new qvds.

Our server is windows 2016 R2 and has 300 GB space in C Drive. When I execute above explained Qlik app process it consumes 140 GB space. when I my Qlik task is over still memory doesn't release instantly. 

When i reboot the server it releases 140 GB space from C Drive. 

If I run Qlik app again same space gets consumed.

My concern is, this is how Qlik works or something is not configured at windows level ?

I hope a positive response.

Thanks,

Rohit

[Levi_Turner]

I am not aware of any memory leak with any version of Qlik Sense Enterprise (or QlikView for that matter) which is remotely modern (e.g. I can't speak to Qlik Sense Enteprise 1.x).

Qlik will attempt to use memory as requested and will attempt to stay under the low working set defined in QMC > Engines > Advanced > Min memory usage.

Qlik will not drop memory before the min memory usage threshold is set, period. 

I'd follow up with the Microsoft SQL Server line of inquiry because both Qlik and MS SQL use Working sets, and both are memory intensive applications. If both are in use and have high demands, it seems quite plausible that a Windows server will go to page.

[rohitk1609]

Thanks Levi for your input.

Yes I found SQL server is installed and I am gonna disable its service.

Now, You are talking about Min memory usage and Max memory usage in Engine part. It is set to default, min 70 and max 90. If I am right this 70 and 90% are set for RAM not physical memory of C drive or primary partition ?

My problem is C drive space gradually decrease every minute when I run Qlik reloads. we have 500 GB total and last time it was 500 mb left , i rebooted and it came to 400 GB space free. I reboot the server, right after it, 471 GB free but after a minute it was drop to 363 Gb free

Earlier I thought it is happening due to pagination but I have checked pagination file size it is 1 GB. So I am not sure where exactly 400 GB is going after some time or few days.

Do you have any idea what exactly is happening ? 

Thanks,

Rohit

[marcus_sommer]

Because of the fact that the consume of drive space is so heavy and goes so fast it should be rather easy to detect by looking into the resource-monitor (provides a more detailled view as the taskmanager and could be opened from there) which process accessed the drives.

- Marcus

[rohitk1609]

Currently, 200 GB consumed from C Drive out of 500 GB.
I have checked all folder combined size which is 100 GB.When I looked in resources monitor I can't find out which process or file is consuming 100 GB space.

I attaching images of resources monitor where you can notice, 300 GB is free but where 100 GB is going not sure.

Please have a look?

[marcus_sommer]

I didn't meant to look to the used/available disc-space after your tasks were running else during they run to see in the first/second window which processes cause a very high read/write amount. I'm not an expert to this admin-stuff but if any "normal" tool is the reason it should be visible here. By system-processes which are responsible for shadow copies, recovery points and similar I'm not sure if they could be detected here - this could be done with the process monitor (but this should be rather the worst case).

Did you enable to see all folders/files on your system - per default many are hidden and might be also not considered within the view of folder/file properties?

That those "invisible" data are released by a re-start is a strong indicator that some kind of caching or exhaustedly logging is the cause.

- Marcus

[rohitk1609]

Yes I did enabled all hidden files and here is the result which will help you to investigate more. I have talked to few people, often people face this problem none has solution, so if we solve this, this will help not only me but many. 

[marcus_sommer]

You couldn't compare the consumed space on this way because the counting over the folder-properties won't consider many of the NTFS specialties like ADS and probably also by the different kinds of hidden- and/or system files and maybe also in regard to the various ways to set (hard/soft) links and similar stuff.

AFAIK there is no direct way in the UI to see all of this but it's possible with powershell. Take a look to commands like:

dir -force

- Marcus

[tomovangel]

Hello Rohit, 

I think this might be related to the "swap file" or "virtual memory file"

Basically when your RAM is full, windows is generating this file to compensate for the missing Ram memory. 

You can change the location of this file to disk D:, but you better find some of your IT guys and they will do it. 

[rohitk1609]

Virtual memory or pagination creates pagefile.sys file which has 1 GB space. So I don't think it is the one who is consumed 100 GB space.

Thanks for your input.

Rohit

[rohitk1609]

Hey Marcus,

I ran the command you asked in powershell and here is the result: