topic Re: Performance question - split one LOAD into two heaps? in QlikView

Performance question - split one LOAD into two heaps?

datanibbler — Tue, 22 Nov 2016 07:31:43 GMT

Hi,

I know I have asked this kind of question before, but now I have an even more specific example, plus the other thread is closed:

I have one table (a transaction table) from which I used to do one LOAD to have it in RAM and just do RESIDENT LOADs from that point on to speed things up.

Now I thought, the table has about 19mio records (beginning in April) and I already load it filtered with a start_date.

=> Splitting it into two would speed things up even more because one part would be really small - less than 1mio records (because per item ("track"), there is only one receiving_booking and then many transfer_bookings.

=> So now I have two LOADs from the same table, one for the small part and one for the big. The filter is on a field TRAN_CODE, for the value 'SPRECEIVE' (small part) and all others (big part).

PLUS: There is a SWITCH_CASE statement because that time_filter (for the start_date) can be used in only one of three ways that report can be used 😉

I remember that in another tool I worked with, when I wanted to split a table into, say, the values "red" and "green", I could, instead of having two LOADs, parse it just once and tell the tool like

>Look at this record and if it is red, put it on the right, otherwise put it on the left; Then move on<

Can I do something like this in QlikView?

Thanks a lot!

P.S.: Of course I could LOAD the table as a whole and then split it using two RESIDENT LOADs, that might also make sense, but it is not the same ...

Re: Performance question - split one LOAD into two heaps?

marcus_sommer — Tue, 22 Nov 2016 07:41:32 GMT

Hi DataNibbler,

I think you will always need two loadings because of the way how qlik handled the data internally within symbol-tables and data-tables - and they will be faster as each kind of alternatives.

- Marcus

Re: Performance question - split one LOAD into two heaps?

datanibbler — Tue, 22 Nov 2016 08:02:26 GMT

Okay.

Thanks Marcus!

I will look for some other way to further improve the performance because this report is now already being used by three persons and every hour we have a risk of conflicts with our primary data_loading script - the one reads, the other one writes, you know that, I've written about that before. By making it ever faster, I hope that this risk can be minimized, though not completely banned.

Well, up to now there have been no mayor issues.

Best regards,

DataNibbler