At a current client there is a need for loading a large amount of data as quickly as possible, which we're currently accomplishing using some hefty hardware and a series of 40 QVWs! This has allowed us to improve performance tenfold over the previous iteration which used a single load script.
For background, basically we have a starting process which loads a couple dozen small tables from the database server into QVD files and takes ~20-30 seconds. The completion of that process triggers the processes that can run with only those first QVDs as well as the next, somewhat larger database load, which triggers processes that can run with only the tables from the 1st and 2nd database load as well as a third database load. Once all of the processes beforehand are complete, a multiple task trigger sets off the reload and distribute of the UI.
To meet the need for "as quickly as possible" we want the first load task to be triggered upon completion of the last (UI load & distribute) task. This also works OK, but it introduces issues in case of failures: If *any* of the 40 QVW loads were to fail, the load chain would be stopped until someone manually triggered the first process again.
We're looking for a "safety valve" to ensure that the loading process is self-recovering. We did get one method to work - A QVW that resets a few things, is triggered by the failure of any of the other 40 QVWs, and triggers the first load of the process to get it going again - But it made the QMC almost unusable: Since it is triggered by any process failing, the Status page shows it as a subprocess of every QVW in the chain; since it triggers the first process in the chain, every process shows up as a child of every other process in the QMC. If any of them are loading, they all show as loading, if any of them fail they all show as failed and it makes finding the source of issues a royal pain.
Is there a better way to do this? We were hoping that dependencies would work and we could set up our "safety valve" process on a time trigger with dependencies for the other processes not running. However, dependencies seem to not care whether things are currently running as long as they haven't failed.
Any help is much appreciated!