1 Reply Latest reply: Jul 15, 2011 3:45 PM by Daniel Rozental RSS

    QV 10 Multithreading: Pitfalls and Easter Eggs

    Chris Cammers

      Hello everyone,

       

      I'm working with a client who is preparing to upgrade to V10 from V9 and I am looking for strategies I can suggest with them to help speed up load scripts taking advantage of Mutilthreading enhancement.

       

      Here is my understanding of the Multitheading(MT) process

       

      When executing the load script Qlikview will look forward in the script and execute any commands that do not have dependencies on another task. Once all the dependent tasks are complete it will execute the next task and continue scanning forward looking for dependent tasks where it would have to wait again and so-on.

       

      So Does that sound like a good understanding of the process?

       

      Based on that my optimization suggestion is to perform all database loads and then join together based on the resident data.

       

      So you would be tempted to load a table then immediately perform a join then another join and another. But if you load all the tables from the database first then you will take advantage of the MT process and then the joins will be performed with a single thread.(? maybe)

       

      What about loads that are conditional? can Qlikview scan ahead to maintain multiple threading if I am passing parameters to a procedure that is executed multiple times, except differently based on the arguments?

       

       

      So that is what I have so far, I'm hoping people will contribute ideas and we can try/prove them out.

       

      Thanks

       

      Chris

        • Re: QV 10 Multithreading: Pitfalls and Easter Eggs
          Daniel Rozental

          Chris,

           

          I don't believe that's the way QV10 multithreads scripts. What it does is use many cores to resolve expresions or formulas but it won't change the sequence things are being done or something like that.

           

          i.e. if you're doing something like

           

          LOAD

               right(FIELD1,2) AS FIELD1,

               left(FIELD2,2) AS FIELD2

          FROM ...

           

          QV will now use separate threads to resolve each expression faster but will not start to process anything else until it's done with that load.

           

          Only thinkg that will make sense changing is avoiding functions that will not multithread, although I wouldn't really know what those are.