Performance question - split one LOAD into two heaps?
Hi,
I know I have asked this kind of question before, but now I have an even more specific example, plus the other thread is closed:
I have one table (a transaction table) from which I used to do one LOAD to have it in RAM and just do RESIDENT LOADs from that point on to speed things up.
Now I thought, the table has about 19mio records (beginning in April) and I already load it filtered with a start_date.
=> Splitting it into two would speed things up even more because one part would be really small - less than 1mio records (because per item ("track"), there is only one receiving_booking and then many transfer_bookings.
=> So now I have two LOADs from the same table, one for the small part and one for the big. The filter is on a field TRAN_CODE, for the value 'SPRECEIVE' (small part) and all others (big part).
PLUS: There is a SWITCH_CASE statement because that time_filter (for the start_date) can be used in only one of three ways that report can be used 😉
I remember that in another tool I worked with, when I wanted to split a table into, say, the values "red" and "green", I could, instead of having two LOADs, parse it just once and tell the tool like
>Look at this record and if it is red, put it on the right, otherwise put it on the left; Then move on<
Can I do something like this in QlikView?
Thanks a lot!
P.S.: Of course I could LOAD the table as a whole and then split it using two RESIDENT LOADs, that might also make sense, but it is not the same ...
I think you will always need two loadings because of the way how qlik handled the data internally within symbol-tables and data-tables - and they will be faster as each kind of alternatives.
I think you will always need two loadings because of the way how qlik handled the data internally within symbol-tables and data-tables - and they will be faster as each kind of alternatives.
I will look for some other way to further improve the performance because this report is now already being used by three persons and every hour we have a risk of conflicts with our primary data_loading script - the one reads, the other one writes, you know that, I've written about that before. By making it ever faster, I hope that this risk can be minimized, though not completely banned.
