Simple LOAD - taking too long ...

[datanibbler]

Hi,

I have an issue:

- In my current app, I have multiple ways of reloading the script (or parts of it) based on user_input on the GUI:

     - I have two buttons, one for a full (regular) and one for a partial reload;

     - These two are again split in two using conditional execution (a >> IF ... THEN << bracket)

=> Because I cannot simulate this (not without mayor "retooling" to be repeated every time, anyway), I use TRACE statements to tell which part of the code has been executed.

There is not much data in there yet, just a few records.

My issue is such: The LOAD itself is seemingly finished in a second, just like it should - at least the TRACE statement highlighting the end of it is displayed pretty quickly.

<=> But then it takes a while (from a few seconds to a few minutes) until the button "OK" becomes available.

Can you tell me what might be the reason for this? This app is eventually supposed to be like a database where multiple users will enter data in the qvd file.

Thanks a lot!

Best regards,

DataNibbler

[marcus_sommer]

Hi DataNibbler,

this will be most often happens if the load generate some synthetic keys or circular references which could need a lot of time to calculate and/or there are some store-statements which could be take some time by large amounts of data.

- Marcus

[datanibbler]

Hi Marcus,

thanks for the answer!

However, there is just one table, so there cannot be a synthetic key or circular reference.

There is a STORE statement in two of the blocks of code to execute - but the table has 46 records, so that should not take long.Something needs some time to calculate, that's for sure, and I guess it is in the script - the GUI is only calculated after the execution_window has been closed, no?

P.S.: I will experiment a bit with the start and end of that >> IF NOT ISPARTIALRELOAD() << bracket, there might be a code that takes so long after the LOAD has finished.

[marcus_sommer]

Hi DataNibbler,

yes, this should be very fast. Are there further statements involved maybe some loops? What said the log? What happens if this runs in the debugger?

- Marcus

[datanibbler]

Hi,

I am just checking the code. The entire script is subdivided into several parts:

- The first part (standard elements) is executed in all cases

I cannot use

- Then there  is the > IF NOT ISPARTIALRELOAD() < bracket

   => This is subdivided by use of the > IF [variable] = X THEN ... ELSE < function into two

- After the ENDIF statement, there are again some statements to be executed in all cases

- Then there is the > IF ISPARTIALRELOAD() < bracket

   => This is again subdivided by the same function (another variable)

The log says nothing about this: It looks completely normal and the entire execution (of one branch) takes just about a second. The log finishes with "Script execution finished" after a second or so. Nothing visible there.

I cannot easily use the debugger because I cannot distinguish between full (regular) and partial reload in the debugger. So I would have to change around my code to fit the debugger and there is usually a chance that somewhere along that line, the actual error goes missing, to turn up again when you stop using the debugger and return to normal execution 😉

[marcus_sommer]

I could imagine that the reason is the partial load then these data must be included in the datamodel by rebuilding the symbol-tables. I think if you looked to the taskmanager you will see some activities to this time.

- Marcus

[datanibbler]

yes, I guess it must be something connected with that.

Too bad - having multiple ways in which to reload my script (or parts of it) is crucial to this app.

Maybe I will deactivate the partial_reload_branch for testing and remove the ISPARTIALRELOAD() function and try with just the conditional execution. If that helps, I would just abandon the approach with the partial reload and use more conditional_execution_branches instead.

P.S.: I noticed something else. I had overlooked those >> IF ... THEN << clauses were not properly closed off - at least that's how it seems. I had one opening IF ... THEN for the full or partial reload and inside that another >> IF THEN << opening (for the variable for the conditional execution) but only one >> ENDIF <<.

[marcus_sommer]

Yes, I think partial loads will have only an advantage if you loads huge amounts of data quite often (maybe hourly). If not normal incremental loads will be fast enough and easier to handle.

- Marcus

[datanibbler]

Hi,

I'm with you there. The issue is, in this case the partial load has nothing to do with the amount of data to load. There are completely different blocks of code to execute.

That does not seem to have been the problem, either. I deactivated everything to do with the partial reload, leaving only the conditional execution, but I still had the same issue - the script finishes in about 1sec, then comes the final TRACE statement, the button OK becomes available - and I see the cursor indicating that something is still calculating and I have to wait another 20sec approximately until I can click the button.

This seems to be a very difficult one - but then, QlikView is not your typical tool to build a database. I didn't expect it to be easy ...

[marcus_sommer]

I'm not sure if to use qlikview as database is very practical. It needs many logic to develop and test things which others have already developed and tested. So I think it could be easier to use qlikview as a frontend and control-tool but the backends are real databases, maybe these postings are helpful:

https://community.qlik.com/search.jspa?q=write+database&type=document

- Marcus