Qlik app loading or processing consumes C Drive Space ?

[rohitk1609]

Dear Techies,

I am facing a problem where C drive free space gradually decrease when I run Qlik app.  My Qlik apps mainly pull Qvds , process it and dump it to again in Qvd form. 

If I explain our app processing it takes 25 GB Qvd , 150,000,000 records, then it apply aggregates and joins then dump it to new qvds.

Our server is windows 2016 R2 and has 300 GB space in C Drive. When I execute above explained Qlik app process it consumes 140 GB space. when I my Qlik task is over still memory doesn't release instantly. 

When i reboot the server it releases 140 GB space from C Drive. 

If I run Qlik app again same space gets consumed.

My concern is, this is how Qlik works or something is not configured at windows level ?

I hope a positive response.

Thanks,

Rohit

[marcus_sommer]

How do you check that Qlik consumed all storage space on C (you should consider to install Qlik the next time on another drive and also to keep all program data and your source-data there)?

There are a lot of still not mentioned informationen about your environment - like running everything with a desktop client and/or a server with/without a publisher?

Because of the huge amount of consumed storage I would assume that the cause of it should be rather easy to detect by monitoring the processing within the taskmanager and the file-explorer but if not you could use Qlik to scan all (or at least the Qlik related) folders. It's quite easy done with a looping per dirlist/filelist and writing path/file/filetime/filesize and so on into a table - ones after a server re-start and then again when the Qlik tasks has been running - and both results could be directly compared. A good example to the mentioned scanning could you find in the help by "for each".

Some good candidates for the cause is a Windows / Qlik logging (there are multiple loggings for each possible) with a high verbosity. Also thinkable are BUFFER statements or auto-save settings in Qlik, OS services like the windows shadow copies, security tools which cache and/or log things, ....

- Marcus

[rohitk1609]

I am answering your question step wise step:

We are using Qlik Sense Feb 2019 SR4. Windows Server 2016. 500 GB space of C drive or primary partition. 416 GB RAM and 64 Processors 64 Bit. We have multi node site, 2 Scheduler , 2 Consumer nodes and 1 central.

Problem of pagination occurs at Central server.

1.  I installed Qlik(Program files ) folder is in D Drive. When I run Qlik app reloads which picks 25 GB Qvd, apply aggregation and joins means process it and dump it to another set of qvds.  

2.  I went to task manager=> Details=> largest memory consuming task is sqlserver.exe which is consuming 2392556 K.  

sqlsevr.exe is for Qlik postgressql database ? We haven't installed any sql server on central node. I am not sure how this service comes to our central node.

3.  I will try the loop next time to find the space of each directory in C. but again I checked the space of all folder combined in C drive it was 90 GB and rest who knows who is consuming. So I am not sure loop wil catch who is consuming 400 GB space

4. page file was 1 GB. 

5 When it was 18 GB left in C drive out of 500 GB, I reboot the server, right after it, 471 GB free but after a minute it was drop to 363 Gb free. might be becuase Qlik tasks have been started. I am just assuming.

6. It could be the right case of memory leak ?

Please add your input Marcus, your every input is valuable for us.

Thanks,

Rohit

[Levi_Turner]

> sqlsevr.exe is for Qlik postgressql database ? We haven't installed any sql server on central node. I am not sure how this service comes to our central node.

No. PostgreSQL spawns executables named postgres.exe, sqlsevr.exe is Microsoft SQL Server which is not included with Qlik Sense Enterprise. 

[rohitk1609]

Thanks Levi for adding your valuable input.

We haven't installed Microsoft SQL server on central node, how could this service has come ? Can you help us to know how it could come.

I have noticed it on my test site central node too, it has also same named service. Is it a default services ?

Thanks,

Rohit

[Levi_Turner]

No, it is not a default Windows Service. Someone installed Microsoft SQL Server.

Task Manager > Details:

Task Manager > Details > Goto Service:

[Levi_Turner]

Task Manager > Details:

[marcus_sommer]

Do you pull any data from a database and using a MS sql driver? Such driver may buffer some data ...

- Marcus

[rohitk1609]

Dear Marcus and Levi,

We do have excel driver installed becuase we have configured User Directory Connector in Qlik Sense to pull meta data for users. but we run it manually once in a blue moon.

I am not sure why this service is running all time.

Secondly, I couldn't find any Microsoft SQL studio on central node.My IT infrastructure team doesn't know anything about it. 

When I looked into install programs I found few programs with name sql server, you can find it in below image: 

Are the packages shows in image default one comes with windows server 2016 ?

Image is attached. please download to view it

Thanks,

Rohit

[marcus_sommer]

AFAIK is the sql server never been a part of the OS and someone must have installed it. Maybe the sql server is just a part of an OS image - I assume your IT doesn't installed each time an OS from the scratch else the use of images is quite common and they might have one created which doesn't contain a rather naked OS else they enriched with certain tools respectively enabled various server-roles and other configurations. IMO your IT infrastructure team should know something about it.

I think I would just try to disable the autostart of it and see what happens ...

- Marcus

[rohitk1609]

I am gonna talk to IT team again for it.
Next, If you read my earlier input, the way C drive space gradually decrease, do you think it is a case of memory leak and problem in Feb 2019 version ?

I can check the folder size, I did , folders are not consuming the space, Pagination file is of 1 GB, 400 GB space is going where I have no idea,
Is there any other way to track what could consume this much space ?

Thanks,
Rohit