2 Replies Latest reply: Mar 24, 2016 6:36 PM by Joe L RSS

    Are there any limitations of task-based reload scheduling?

    Keith Doyle

      We've recently switched our reload processes from using the OCX to using QlikView reload tasks.   This was done because the OCX has severe reliability problems and appears to have little or no support with those sorts of issues.

       

      But one of the things we are trying to do is bulk reload multiple documents at the end of a warehouse load process.   So consequently, what we want to do is fire off all the document tasks at once, at the point the warehouse load completes.  We're using EDX triggering to do this.   Is this a reliable way to handle it?   Is there any downside or limit to triggering a batch of tasks essentially simultaneously?   Is the Qlikview Task Reload process robust enough to handle the multiple requests and properly parallelize the reload processes?    We're talking around 20 at the moment, but that has been growing over time.   And some of these are fairly large documents that can take anywhere from a few minutes to more than an hour or even more to reload based on the amount of data involved.   Since this is deployed to hundreds of customers, we have not attempted to fine-tune any configuration parameters such as CPU affinity, etc., and leave all of those at the defaults, at least initially.   We hope we are taking advantage of the QlikView internal threading by submitting all the tasks at once and letting QlikView sort out the threading process.    Is this a reasonable way to make use of the reload tasks?   And if not, what should we do different?

       

       

      --

       

      Keith Doyle

        • Re: Are there any limitations of task-based reload scheduling?
          Oleg Troyansky

          I would not necessarily recommend firing all tasks at once, because the Publisher will try to allocate each task to an available processor, and when it runs out of processors, some tasks might start failing. I'd recommend layering some of the tasks in time and not allowing more tasks to be released than the number of available CPU cores.

           

          I'd either spread the tasks over time, or define dependencies between tasks that could ensure that one task doesn't begin before the other task is successfully finished.

           

          cheers,

          Oleg Troyansky

          Check out my new book QlikView Your Business

            • Re: Are there any limitations of task-based reload scheduling?
              Joe L

              Similar suggestion to Oleg and similar cautions.

               

              To gain a bit more throughput you could create multiple sets of dependent tasks. For example, you could define the task setup below (or something similar). Triggering the "Top Level Task" via EDX will result in each set of task dependencies below it executing concurrently.

               

              Top Level Task

                 group 1: task A1

                     group 1: task A2

                        group 1: task A3

                 group 2: task B1

                     group 2: task B2

                        group 2: task B3

                 group 3: task C1

                     group 3: task C2

                        group 3: task C3