8 Replies Latest reply: Aug 14, 2015 5:44 AM by Omer Demir RSS

    Single-threaded calculations

      Hi,

       

      I was reading through some of the documents section and found the following:

       

      http://community.qlik.com/docs/DOC-2063

      http://community.qlik.com/docs/DOC-2073

       

      The first mentions to watch the task manager to see if your QV document is calculating using only 1 core and mentions that there are certain calculations that are single-threaded (only use 1 core). The second link says that the count() function is one such calculation that forces single-threading. Macros have been mentioned before to cause single-threading.

       

      Does anyone know of any other functions that cause single-threading? The first document says that there will be future to documents talking about single-threaded calculations but I can only see the count() one on there.

        • Single-threaded calculations

          Anyone got any ideas on this? Sorry for bumping.

          • Single-threaded calculations
            Henric Cronström

            When you make a selection, QlikVew first evaluates the logical inference of this selection in the data model. I.e. finds out which field values are possible. This process is multi threaded.

             

            Then QlikView starts to calculate the different objects. This is also a multi threaded process: Each object gets a thread of its own.

             

            Most objects can be calculated quickly, but charts can be quite CPU intensive. In a chart calculation, the first step is to expand the cube, i.e. generate all combinations of field values where calculations should be made. In a second step the calculations actually take place. The first step is single threaded, but normally it is very fast, so it is not a problem. However, if the expression contains fields from different tables, this step could take some time.

            The second step is normally multi threaded. However, the distinct clause will cause a single threaded calculation.

             

            So, to make a long story short: The count() function is not single threaded in it self - but the distinct clause can make any aggregation function single threaded. And - No, there are no other functions that are single threaded. But, the "generate all combinations" step is single threaded and this is noticeable if you have a lot of data and expressions with fields from several tables.

             

            /HIC

              • Single-threaded calculations

                Thanks for the great information Henric.

                • Re: Single-threaded calculations
                  Jason Michaelides

                  Henric, I hope you don't mind a follow-up question to this.  In my script I am loading a fairly large table.  From this table I then create a second table which is an aggreagated version of the first table. (Normally I don't aggregate in the script but for various reasons it is necessary here).  I noticed while watching the task monitor that during the grouping only one core is used - can you confim this should be the case, or is it maybe my CrossTable() function?

                   

                  Here's the section of the script:

                   

                   

                  GEQ_temp_MTD:   //MTD aggregations
                  CrossTable([SEC Type],[P&L amount (MTD GEQ)],3)
                  LOAD
                  %KEY_Date  AS   %KEY_Date_GEQ        //Won't be used whenever GEQ_P&L is used but needed for EXISTS() function below (needs to be different name as main %KEY_Date)
                                  ,%KEY_GEQ       //Portfolio/Desk and Source System
                                  ,%KEY_GEQ_Date  //Portfolio/Desk and Source System and Date
                                 
                  ,
                  Sum(
                  If([Dividend/Interest]='Dividends'
                  ,
                  RangeSum(%m_Adj_MTD_Trading_USD,%m_Adj_MTD_Fees_USD,%m_Adj_MTD_Dividends_USD)
                  ,
                  RangeSum(%m_Adj_MTD_Trading_USD,%m_Adj_MTD_Fees_USD)
                  )
                  )                   
                  AS   [Principal Transactions]

                  ,
                  Sum(%m_Adj_MTD_Commissions_USD)                                                     AS     [Commissions]
                  ,
                  Sum(If([Dividend/Interest]='Interest',%m_Adj_MTD_Dividends_USD,0))                  AS   [Net Interest]
                  ,
                  Sum(%m_Adj_MTD_IB_USD)                                                                   AS   [Investment Banking]
                  ,
                  Sum(%m_Adj_MTD_AsstMgmtFee_USD)                                                     AS     [ASMT]
                  ,
                  Sum(%m_Adj_MTD_Other_USD)                                                           AS     [Other]
                  ,
                  Sum(%m_Adj_MTD_CapCharge_USD)                                                       AS     [Capital Charges]
                  ,
                  Sum(%m_Adj_MTD_Funding_USD)                                                         AS     [Financing]

                  RESIDENT [P&L]
                  GROUP BY %KEY_Date,%KEY_GEQ,%KEY_GEQ_Date

                   

                   

                  Many thanks,

                   

                  Jason

                    • Re: Single-threaded calculations
                      Henric Cronström

                      My previous answer in this thread concerns how QlikView works when you analyze data - when you are in "analysis mode". Your question concerns how QlikView works during script execution, and that is a different issue.

                       

                      I am not surprised that your script uses one core only. In fact, it is what I would expect. The script evaluation is inherently different from the analysis evaluation of data. In the script, the order of the records matter and this poses limitiations on the algorithm. We have made the script execution multi-threaded, but in principle the multi threading is on a record per record base: A thread needs to wait for the other threads to finish before it can process the next record.

                       

                      Further, both the crosstable transformation and the group by transformation involve multiple records, which also makes multi threading difficult (or maybe impossible).

                       

                      So, yes, this is what I would expect.

                       

                      HIC

                    • Re: Single-threaded calculations
                      Omer Demir

                      I think, It is necessary to address visitors to a more up to date post by HIC who explains Count(Distinct...) is no more single threaded, so that readers are not confused.

                       

                      Here is the post:

                      A Myth About Count(distinct …)