topic Re: Memory Management in QlikView

Memory Management

Tue, 26 Aug 2014 11:18:33 GMT

How to reduce the reload time of the application as it take 127 GB Ram out of 128 GB ram allocated to the system

I am not able to open other application

Re: Memory Management

ashfaq_haseeb — Tue, 26 Aug 2014 11:22:44 GMT

Hi,

Its totally depend on your model and how you design it.

How many records do you load?

you can try by splitting your data into smaller qvd's.

Make sure you don't have multiple sessions opened by end users.

Do not load unnecessary fields.

Regards

ASHFAQ

Re: Memory Management

salto — Tue, 26 Aug 2014 11:25:07 GMT

Hello Shweta,

have you tried to reload the document with a subset of data, and then check your data model?

Re: Memory Management

vinay_hg — Tue, 26 Aug 2014 11:26:58 GMT

Hi,

1. do not consider the fields which are not required in ETL and in presentation qvw.

2. try to use proper data schemas in data modelling.

3. try not to use nested if conditions in expressions, instead use set analysis.

4.dont use too much variables for calucaltions in presentation QVW.

5. dont right big expressions in the presentaions, try to bring calculated values from model level.(script level). i mean split your application into layers like extract(qvw) where you will load data nd creat raw qvd, then load required data only from qvds to qvw file model layer. i nscript level try to do all calculation. end result save it in again qvd or load this qvw into presentation qvw.

so by all this load on the presentation qvw file will be lesser. and takes less RAM while loading and run/open faster.

Regards

vinay

Re: Memory Management

Tue, 26 Aug 2014 11:30:31 GMT

they have records in crores

We have already divided the coding application into 2 parts ,But still its 1st part takes 21 hours and 2nd parts take 6 hours

Re: Memory Management

ashfaq_haseeb — Tue, 26 Aug 2014 11:33:05 GMT

In that case make sure you have enough ram.

Look at below post for memory management and ram sonsumption.

http://community.qlik.com/message/455415#455415

http://community.qlik.com/message/455396#455396

Regards

ASHFAQ

Re: Memory Management

vinay_hg — Tue, 26 Aug 2014 11:36:27 GMT

are you using proper data modelling or schema in 2 parts? is all crore data columns are required? is it able comment some column?

and according to me it should not take 21hours and 6hours to load. how many tables data you are loading in each part? what kind of relation b/t them you had created?

Re: Memory Management

Tue, 26 Aug 2014 11:39:57 GMT

we follow 3 tier architecture -

1.first qvd generator takes 3 hours

2.data model app divided into 2 parts -1st take 21 hours and other take 6 hours

3.main qvw( having design)- take 15 min

All the calculation has done at 2nd level ,i used variable & set analysis

Re: Memory Management

vinay_hg — Tue, 26 Aug 2014 11:45:23 GMT

ok tell me in data modelling, 2 parts how many tables or qvds you r loading to dod calculation?

and what is the data schema you had prepared b/t tables(star or sowflex or all data concatinated to single table) in each parts?

Re: Memory Management

Tue, 26 Aug 2014 11:50:33 GMT

Re: Memory Management

vinay_hg — Tue, 26 Aug 2014 11:59:57 GMT

what i see here a simple model.. it should not take much time.. may thecalculations you have used taking time..

suggestions in model

1.y u r using fiscal in model? load this in presentation qvw file directly, remove from model use directly in presentaion qvd.

2.t-transformationentry and mapping selection tables contains 3 columns?if u required only one column from each table means use applymap function, take that filed directly to main table. then no need of that 2 tables

3.are you using any resident load inside calculation?

Re: Memory Management

Tue, 26 Aug 2014 12:06:56 GMT

yes they have used lot of resident load in the calculation

T_TransSalesEntryMain is t he combination of two table header and sales,we have applied the increamental load

to fetch the data from this table .Instead of this table take max time to reload

Re: Memory Management

vinay_hg — Tue, 26 Aug 2014 12:18:13 GMT

you need to reduce the usage of more resident of same table if data size is high. and remove fiscal mapping from main table . use the same in presentation .qvw file

y you are doing incremental load in model. it should be in the genarator right? in case it is incremental load then core data reload is not required right.. old data u can process and save in different qvd may be 2012 .qvd only new set of data only u can process.

Re: Memory Management

Tue, 26 Aug 2014 12:20:06 GMT

make sure you are creating QVD along with , which are as comparative faster than all other type of loads.

one thing i have observe while practicing that we should load table from qvd always , it is not good to load table from sql query and hitting the DB server directly.