Skip to main content
Announcements
Qlik Connect 2024! Seize endless possibilities! LEARN MORE
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Are there any limitations of task-based reload scheduling?

We've recently switched our reload processes from using the OCX to using QlikView reload tasks.   This was done because the OCX has severe reliability problems and appears to have little or no support with those sorts of issues.

But one of the things we are trying to do is bulk reload multiple documents at the end of a warehouse load process.   So consequently, what we want to do is fire off all the document tasks at once, at the point the warehouse load completes.  We're using EDX triggering to do this.   Is this a reliable way to handle it?   Is there any downside or limit to triggering a batch of tasks essentially simultaneously?   Is the Qlikview Task Reload process robust enough to handle the multiple requests and properly parallelize the reload processes?    We're talking around 20 at the moment, but that has been growing over time.   And some of these are fairly large documents that can take anywhere from a few minutes to more than an hour or even more to reload based on the amount of data involved.   Since this is deployed to hundreds of customers, we have not attempted to fine-tune any configuration parameters such as CPU affinity, etc., and leave all of those at the defaults, at least initially.   We hope we are taking advantage of the QlikView internal threading by submitting all the tasks at once and letting QlikView sort out the threading process.    Is this a reasonable way to make use of the reload tasks?   And if not, what should we do different?

--

Keith Doyle

2 Replies
Oleg_Troyansky
Partner Ambassador/MVP
Partner Ambassador/MVP

I would not necessarily recommend firing all tasks at once, because the Publisher will try to allocate each task to an available processor, and when it runs out of processors, some tasks might start failing. I'd recommend layering some of the tasks in time and not allowing more tasks to be released than the number of available CPU cores.

I'd either spread the tasks over time, or define dependencies between tasks that could ensure that one task doesn't begin before the other task is successfully finished.

cheers,

Oleg Troyansky

Check out my new book QlikView Your Business

jlongoria
Creator
Creator

Similar suggestion to Oleg and similar cautions.

To gain a bit more throughput you could create multiple sets of dependent tasks. For example, you could define the task setup below (or something similar). Triggering the "Top Level Task" via EDX will result in each set of task dependencies below it executing concurrently.

Top Level Task

   group 1: task A1

       group 1: task A2

          group 1: task A3

   group 2: task B1

       group 2: task B2

          group 2: task B3

   group 3: task C1

       group 3: task C2

          group 3: task C3