Section access code in beginning of end versus end of script

[spcpeters]

Hey all,

We found through testing that the average reload time of a dashboard seems to be significantly lower when the section access code is run as the final step of the code versus when it is in the top of the code. In one case it is roughly 8 minutes versus 6 minutes (as run on the QMC). Would anybody have an idea what the technical reason for this would be? I couldn't find anything interesting only that would explain it.

Thanks

[Miguel_Angel_Baeyens]

Were you able to test it with other apps? Also, which version are you running?

I cannot run any script right now, but it _should_ not make any difference at all. The execution of the script is sequential, and the tables are associated at the end of the script, after the last instruction has been successfully executed.

In my experience with several SRs of QlikView 11.20 and QlikView 12.10 and 12.40 I have not experienced that, not at least in a noticeable way.

It comes to my mind, though, and this is an wild guess so take it like that, that since section access also works during reloads, i.e.: you can only reload what you (actually, the account performing the reload) are allowed to see, that security check happens for every table (assuming the tables are limited during load time, which is in itself a wild assumption).

It would work like the SQL part brings the entire table as it is defined in the script, whereas the LOAD part already reduces the data to be stored into Qlik.

I'm sure @hic can shed some light here.

[marcus_sommer]

I think the document- and task-log should return some insights which parts of the script differ to eachother. All or certain load-statements or the last steps of building the datamodel and storing the application.

I don't know exactly how section access is technically implemented - but I'm not sure that's really related to your observation or if it's just related to the order of your load-statements because it could have a significantly impact on the work which Qlik needs to do to store the data.

The reason for it is that Qlik stores the data in symbol-tables (each field get an own table with only distinct values) and the data-tables which just contain a bit-stuffed pointer to the symbol-tables. If now at first a field is loaded with many distinct values and then a second (much smaller) load is also loading this field - each field-value needs to be checked if it already exists in the symbol-table and this happens then against a large table. If the order is reversed the check starts against a much smaller table which then gradually grows.

Quite recently I had a bit similar case of concatenating multiple fact-tables. At first I loaded the both biggest ones and then the other smaller ones. Then I reversed the order and the reload-times reduced to about the half.

Therefore the order of your loadings could matter in regard of the load-times (of course always depending on your data).

- Marcus

[Miguel_Angel_Baeyens]

That's a good observation @marcus_sommer. I'll see if I can reproduce that behavior.

I have been through that exercise of moving tables around in a very big app (several GB in disk) yet I saw no difference with 12.10 SR4 and 12.40. I cannot say I'm missing some settings somewhere since both environments used to be QilkView 11.20 and then upgraded.

[Brett_Bleess]

Stephan, I think Miguel may have been on to something, can you share what the Section Access record for the account doing the reload looks like, its values in the Section Access table and what fields you are using in Section Access?  If you are loading dynamic reduction fields, and that account does not have 'blank' in the reduction fields, that might potentially explain things, as things have to figure out which values all the other users in the Section Access table do have access to and then restrict the account doing the load to those values etc.  We will see if Henric eventually can have a look at this too, but I think i may be on the right track thanks to Miguel.  Marcus could be on to something too, but my gut right now is on what Miguel said initially.

Regards,
Brett