Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
We've recently switched our reload processes from using the OCX to using QlikView reload tasks. This was done because the OCX has severe reliability problems and appears to have little or no support with those sorts of issues.
But one of the things we are trying to do is bulk reload multiple documents at the end of a warehouse load process. So consequently, what we want to do is fire off all the document tasks at once, at the point the warehouse load completes. We're using EDX triggering to do this. Is this a reliable way to handle it? Is there any downside or limit to triggering a batch of tasks essentially simultaneously? Is the Qlikview Task Reload process robust enough to handle the multiple requests and properly parallelize the reload processes? We're talking around 20 at the moment, but that has been growing over time. And some of these are fairly large documents that can take anywhere from a few minutes to more than an hour or even more to reload based on the amount of data involved. Since this is deployed to hundreds of customers, we have not attempted to fine-tune any configuration parameters such as CPU affinity, etc., and leave all of those at the defaults, at least initially. We hope we are taking advantage of the QlikView internal threading by submitting all the tasks at once and letting QlikView sort out the threading process. Is this a reasonable way to make use of the reload tasks? And if not, what should we do different?
--
Keith Doyle
I would not necessarily recommend firing all tasks at once, because the Publisher will try to allocate each task to an available processor, and when it runs out of processors, some tasks might start failing. I'd recommend layering some of the tasks in time and not allowing more tasks to be released than the number of available CPU cores.
I'd either spread the tasks over time, or define dependencies between tasks that could ensure that one task doesn't begin before the other task is successfully finished.
cheers,
Oleg Troyansky
Check out my new book QlikView Your Business
Similar suggestion to Oleg and similar cautions.
To gain a bit more throughput you could create multiple sets of dependent tasks. For example, you could define the task setup below (or something similar). Triggering the "Top Level Task" via EDX will result in each set of task dependencies below it executing concurrently.
Top Level Task
group 1: task A1
group 1: task A2
group 1: task A3
group 2: task B1
group 2: task B2
group 2: task B3
group 3: task C1
group 3: task C2
group 3: task C3