Complex reload process - Need a safety valve

[Anonymous]

Hi all,

At a current client there is a need for loading a large amount of data as quickly as possible, which we're currently accomplishing using some hefty hardware and a series of 40 QVWs! This has allowed us to improve performance tenfold over the previous iteration which used a single load script.

For background, basically we have a starting process which loads a couple dozen small tables from the database server into QVD files and takes ~20-30 seconds. The completion of that process triggers the processes that can run with only those first QVDs as well as the next, somewhat larger database load, which triggers processes that can run with only the tables from the 1st and 2nd database load as well as a third database load. Once all of the processes beforehand are complete, a multiple task trigger sets off the reload and distribute of the UI.

To meet the need for "as quickly as possible" we want the first load task to be triggered upon completion of the last (UI load & distribute) task. This also works OK, but it introduces issues in case of failures: If *any* of the 40 QVW loads were to fail, the load chain would be stopped until someone manually triggered the first process again.

We're looking for a "safety valve" to ensure that the loading process is self-recovering. We did get one method to work - A QVW that resets a few things, is triggered by the failure of any of the other 40 QVWs, and triggers the first load of the process to get it going again - But it made the QMC almost unusable: Since it is triggered by any process failing, the Status page shows it as a subprocess of every QVW in the chain; since it triggers the first process in the chain, every process shows up as a child of every other process in the QMC. If any of them are loading, they all show as loading, if any of them fail they all show as failed and it makes finding the source of issues a royal pain.

Is there a better way to do this? We were hoping that dependencies would work and we could set up our "safety valve" process on a time trigger with dependencies for the other processes not running. However, dependencies seem to not care whether things are currently running as long as they haven't failed.

Any help is much appreciated!

[Peter_Cammaert]

The most obvious alternative method is to eliminate reasons of failure in the reloads (can they be categorised? Just a few of those QVWs fail, and due to a limited set of causes?) or make the reload scripts themselves (almost) failure-proof by embedding a retry logic at every possible point of failure.

[Anonymous]

The most common reasons for failure are mostly filesystem-related, like the OS not being done writing a file when the next QVW wants to read it and that sort of thing. We also see "Warning" fairly frequently because the log file can't be read/written. The scripts are solid and run fine. We just need a way to get the process going again if there's a failure.

I've also noticed that multiple-event triggers are a bit flaky. Let's say you have A.qvw, B.qvw, C.qvw, and D.qvw set up so that B and C are triggered by A, and D is triggered by B and C running. B takes longer to run than C. C fails at some point. You either wait for things to finish or kill the remaining running processes, fix C if necessary, and start A. A finishes and starts B and C. C finishes but B is still running, and D, which should be triggered when both B and C have finished, starts running even though B hasn't finished because B finished once before. I'd really love to have a way to reset those triggers when starting A.

Finally, I noticed that when you use a multiple-event trigger and you use more than about 8 events, it just seems to ignore the remaining ones. I'm guessing Qlik doesn't test these sorts of scenarios as much as they should.