Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
andriesb
Creator II
Creator II

mapping load : from resident or qvd : best practice

To add different information extracted from an already saved qvd, I have to create 5 "mapping load"  's.

What is better and faster:

do a single load and then load the 5 different mappings information from resident memory

or

only load the mapping info 5 times

in code:

tempList:

Load *

from source.qvd (qvd)

mapping load

...

resident templist

(...5 times)

or

5 times:

mapping Load

field1,

field2

from source.qvd (qvd)

I could imagine loading only 5 times the mapping would result better in memory usage but would be slower to execute.

Any ideas/ experience to share on this?

7 Replies
jagan
Luminary Alumni
Luminary Alumni

Hi,

First load the QVD and then using Resident load the mapping tables.  This has better performance than loading the QVD 5 times.  Resident Load is faster than loading data from QVD.

tempList:

Load *

from source.qvd (qvd)

mapping load

...

resident templist

(...5 times)

Regards,

Jagan.

andriesb
Creator II
Creator II
Author

and according to memory usage ? Loading all 5 milj. records versus only the information I actually needs for the mapping?

Loading from memory will always be faster, but the extra memory requered could be a drawback.

Speed is not always the first priority, as a 'reload' could be sceduled during night time.

amien
Specialist
Specialist

"Resident Load is faster than loading data from QVD."

is this really always the case? i just tried loading a QVD and than a resident load on the same table. QVD is faster.

rwunderlich
Partner Ambassador/MVP
Partner Ambassador/MVP

No, it's not always the case. In many cases loading from a QVD is faster than resident.

-Rob

Not applicable

Rob is it fair to say that an optimized QVD load will generally always be faster than a Resident load, whereas a QVD load that is unoptimized is not?

rwunderlich
Partner Ambassador/MVP
Partner Ambassador/MVP

I'm not sure that is correct. More to the point, I think it's rarely relevant because most of the time, we are doing operations in a Resident load that would not qualify for an optimized load.

Like:

LOAD max(Date) from ....

which suprisingly, is usually faster from a QVD even thought it's not optimized.

-Rob

phaneendra_kunc
Partner - Specialist III
Partner - Specialist III

I agree with Rob..QVD load is faster when compared to Resident....

One scenario i have is 30 mill records table needs to be reload twice with manipulations. so with my test i came to know that resident is taking longer than normal QVD load.

So we came up with a approach where First Load statement data will be stored into a temporary QVD and again the next LOAD statement will pick up this temp QVD instead of a resident..and does its job quicker..