Single-threaded calculations

[#1]

Hi,

I was reading through some of the documents section and found the following:

http://community.qlik.com/docs/DOC-2063

http://community.qlik.com/docs/DOC-2073

The first mentions to watch the task manager to see if your QV document is calculating using only 1 core and mentions that there are certain calculations that are single-threaded (only use 1 core). The second link says that the count() function is one such calculation that forces single-threading. Macros have been mentioned before to cause single-threading.

Does anyone know of any other functions that cause single-threading? The first document says that there will be future to documents talking about single-threaded calculations but I can only see the count() one on there.

[#2]

Anyone got any ideas on this? Sorry for bumping.

[hic · Accepted solution]

When you make a selection, QlikVew first evaluates the logical inference of this selection in the data model. I.e. finds out which field values are possible. This process is multi threaded.

Then QlikView starts to calculate the different objects. This is also a multi threaded process: Each object gets a thread of its own.

Most objects can be calculated quickly, but charts can be quite CPU intensive. In a chart calculation, the first step is to expand the cube, i.e. generate all combinations of field values where calculations should be made. In a second step the calculations actually take place. The first step is single threaded, but normally it is very fast, so it is not a problem. However, if the expression contains fields from different tables, this step could take some time.

The second step is normally multi threaded. However, the distinct clause will cause a single threaded calculation.

So, to make a long story short: The count() function is not single threaded in it self - but the distinct clause can make any aggregation function single threaded. And - No, there are no other functions that are single threaded. But, the "generate all combinations" step is single threaded and this is noticeable if you have a lot of data and expressions with fields from several tables.

/HIC

[#4]

Thanks for the great information Henric.

[Jason_Michaelides]

Henric, I hope you don't mind a follow-up question to this.  In my script I am loading a fairly large table.  From this table I then create a second table which is an aggreagated version of the first table. (Normally I don't aggregate in the script but for various reasons it is necessary here).  I noticed while watching the task monitor that during the grouping only one core is used - can you confim this should be the case, or is it maybe my CrossTable() function?

Here's the section of the script:

GEQ_temp_MTD:   //MTD aggregations
CrossTable([SEC Type],[P&L amount (MTD GEQ)],3)
LOAD
%KEY_Date  AS   %KEY_Date_GEQ        //Won't be used whenever GEQ_P&L is used but needed for EXISTS() function below (needs to be different name as main %KEY_Date)
                ,%KEY_GEQ       //Portfolio/Desk and Source System
                ,%KEY_GEQ_Date  //Portfolio/Desk and Source System and Date
               
,Sum(
If([Dividend/Interest]='Dividends'
,RangeSum(%m_Adj_MTD_Trading_USD,%m_Adj_MTD_Fees_USD,%m_Adj_MTD_Dividends_USD)
,RangeSum(%m_Adj_MTD_Trading_USD,%m_Adj_MTD_Fees_USD)
)
)                   
AS   [Principal Transactions]

,Sum(%m_Adj_MTD_Commissions_USD)                                                     AS     [Commissions]
,Sum(If([Dividend/Interest]='Interest',%m_Adj_MTD_Dividends_USD,0))                  AS   [Net Interest]
,Sum(%m_Adj_MTD_IB_USD)                                                                   AS   [Investment Banking]
,Sum(%m_Adj_MTD_AsstMgmtFee_USD)                                                     AS     [ASMT]
,Sum(%m_Adj_MTD_Other_USD)                                                           AS     [Other]
,Sum(%m_Adj_MTD_CapCharge_USD)                                                       AS     [Capital Charges]
,Sum(%m_Adj_MTD_Funding_USD)                                                         AS     [Financing]

RESIDENT [P&L]
GROUP BY %KEY_Date,%KEY_GEQ,%KEY_GEQ_Date

Many thanks,

Jason

[hic]

My previous answer in this thread concerns how QlikView works when you analyze data - when you are in "analysis mode". Your question concerns how QlikView works during script execution, and that is a different issue.

I am not surprised that your script uses one core only. In fact, it is what I would expect. The script evaluation is inherently different from the analysis evaluation of data. In the script, the order of the records matter and this poses limitiations on the algorithm. We have made the script execution multi-threaded, but in principle the multi threading is on a record per record base: A thread needs to wait for the other threads to finish before it can process the next record.

Further, both the crosstable transformation and the group by transformation involve multiple records, which also makes multi threading difficult (or maybe impossible).

So, yes, this is what I would expect.

HIC

[Jason_Michaelides]

Apologies for posting in the wrong thread, but thank you for your answer, Henric - much appreciated.

[juraj_misina]

Hi Henric,

yet another follow up question, if you don't mind. From what you wrote (a year ago) I understand why peek() is single-threaded. But it seems that preceding loads are single-threaded as well. Is it because of order of records or some is there some other reason? Thank you.

[omerfaruk]

I think, It is necessary to address visitors to a more up to date post by HIC who explains Count(Distinct...) is no more single threaded, so that readers are not confused.

Here is the post:

A Myth About Count(distinct …)