shared memory load qvd

[valmanar]

Good morning, I have an application (A) that its only function is the loading of data and its subsequent recording in .qvd files, it acts as a central database to serve data to all applications, it has no graphical functionality. I have another application (B) that is responsible for consuming the data from application A, and is the one that has all the graphical functionality, application B loads the data as follows, for example:
d_client]:
LOAD * FROM [lib://qvd_sales_qlik (hiperusera_upcinf060)/d_client.qvd](qvd);
[d_buyer]:
LOAD * FROM [lib://qvd_sales_qlik (hyperuser_upcinf060)/d_buyer.qvd](qvd);

The application B is used by 10 people with different types of access on the data loaded from the .qvd, my doubt refers to the management in memory of the data loaded from .qvd by the application B:
Does each of these 10 people who use application B load the .qvd of application A into memory every time they open it?
User 1 opens application B, the two .qvd are loaded with the sentence Load from above
User 2 opens application B, being loaded again the two .qvd therefore you have duplicated in memory the data for each user.
Or, on the contrary, when user 1 opens application B for the first time, the .qvd are loaded from application A and remain in memory and this data is shared among all users who open application B without having to load the .qvd again.
Thank you and greetings.

[ArnadoSandoval]

Hi @valmanar 

Very interesting question, I found these two articles explaining how Qlik manage user data, perhaps they will help you:

• QIX Engine Under the hood (this is a presentation about how the Qlik Engine works, it is very illustrative)
• The QlikView Cache  
• Qlik Sense cache 
• Cache Initializer  - Talking about cache, this is an interesting tool !!!

Hope this helps,

[valmanar]

Thank you very much Arnado, I have reviewed the links you indicated and I have learned a lot about the internal management of the cache memory in Qlik.
In spite of them there is a point that I have not been able to find or understand, the question is if an application that in its load editor obtains a .qvd from the disk like that:
d_client]:
LOAD * FROM [lib://qvd_sales_qlik (hiperusera_upcinf060)/d_client.qvd](qvd);

This .qvd has the total of the records, the application has an access section so that each user only see their own records.

If this application is opened by two different users:
1 .- Depending on the user read into memory only their own records from the .qvd, or load the entire .qvd?
2.- If the whole .qvd has been loaded, when does the second user open the application, checks if its registers are already in memory and does not read them from the .qvd nor duplicate them in memory?
3.- If the whole .qvd has been loaded, when does the second user open the application, does he load the whole .qvd into memory and duplicate it with the one loaded by the first user who opened the application?

Thanks and greetings.

[ArnadoSandoval · Accepted solution]

Hi @valmanar 

I haven't read a Qlik document directly documenting how Qlik load QVDs into memory and how it shares memory resources, perhaps, Levi_Turner's reply give us some insights, but the document never clarifies how it is done in a multi-users environment  when two or more users load the same application. This is the link to the reference document The Qlik® Associative Engine memory management and CPU usage ; still it does not dwell into your very specific question.

What should you do then to find an answer?

You may do your own research, which is not hard to know, you need:

1. Your own Qlik Application
2. Two or more workstations
3. Two or more login accounts for different users
4. Performance Monitor running on another terminal.
5. Select a date and time with minimum QlikSense workload.

 You first start the Performance Monitor tool on a terminal, record the initial memory usage, make sure the memory usage is stable (low workload times are best for this test); go to your first workstation, login with the first user account, load/open the Qlik Application under study, check the "Performance Monitor" and record the memory impact introduced by the first application, you can also record the time taken by the first application to load/open; now go to the second workstation, login with the second user account, load/open the Qlik Application under study, record the time it takes to load/open, go to the "Performance Monitor" and record the memory impact introduced by the second application instance (here you may get the memory went up by a factor of two, or it increase very little); go to the third workstation and repeat the steps.

The data collected memory (from the Performance Monitor) and times (from the time taken by each load/open process) should give you an idea on how Qlik is managing memory.

Another option is to open a ticket with Qlik to request some one of their engineers do this test for you, or speak with your Qlik vendor for assistance.

Now, observing the definition of your issue:

LOAD * FROM [lib://qvd_sales_qlik (hiperusera_upcinf060)/d_client.qvd](qvd);

That load statement will always load the WHOLE qvd as not filters are defined.

the application has an access section so that each user only see their own records.

This is the help document for Managing security with section access which make me think that Section Access is memory enforced, so Section Access does not filter data at the time of loading the data.

If the whole .qvd has been loaded, when does the second user open the application, does he load the whole .qvd into memory and duplicate it with the one loaded by the first user who opened the application?

The test I proposed above should help to answer this question.

hth.

[valmanar]

Thanks Arnaldo for the detail of the explanation, I'm going to perform the test you propose and see what happens.
Thank you and greetings.

[ArnadoSandoval]

@valmanar 

Please write back to this topic the outcome of the tests, I am looking forward for the results.

hth