How do we do parallel processing in Qlikview?

[sujit_nath]

I have a QVS script where I am scoping the data and that is being done sequentially. Is there a way wherein we can divide them into blocks and each block should execute in parallel when I reload the application.

[swuehl]

As far as I know, only if you split your script into multiple scripts. You can execute the scripts in parallel and create a trigger that waits for successfull execution of all the tasks before continuing with another reload task.

[petter]

Hi Sujit,

I agree completely with Stefan.

You don't state whether you are asking about QlikView desktop or QlikView server.

Parts of the load script can be broken down and executed in parallel but the final load into a Qlik data model is not going to be in parallel. That's why creating QVDs that load extremely fast and only loading them as the last step is optimal.

So essentially by using QVDs as storage and interchange store for the various parallel load scripts and various steps you can achieve parallelism. It takes some effort to achieve it. For organizations having larger data volumes that needs to be prepared for QlikView/Qlik Sense it can improve load time and optimize the data flow for the data preparation significantly. The cost will be increased development time, increased complexity and more maintenance.

[sujit_nath]

Thanks Stefan and Petter. I'm about to use QV server in order to achieve this. Will look on the approaches you mentioned.  Best regards!!

[romansavchuk]

Hi Sujit,

Once time I had experience with paralleling data processing logic in QlikView using Powershell script (it allow to run several reload process of QlikView Application simultaneously).

This was manually written process, but we can reduce processing time several timed (it depends on quantity of simultaneously executing applications).
I write QlikView application (call it 'Parent') and 'Child' application, that store processing logic. Parent application separate data into parts and execute powershell script, that clone 'Child' application and parallel reload clones (See command line  reload of QlikView application here: https://community.qlik.com/thread/53660). Each clone of "Child" application takes his part of data and process it. Powerhshell script waits for execution of clones and after execution of powershell script finish, Parent prepare next batch of data to pass it to next Clones (see operator "Execute" in QlikView - it waits for execution of program - in our case it`s powershell script).

This allow to control parallelism process (I can set number of simultaneously executing child applications), but this process available only while reloading application using command line (Qv.exe /r "<path to application>"), not from QlikView Server. You can only run Parent application, but Childs executed from command line and they use computer resources, instead of QlikView server resources for reload.

One important thing in this scenario is monitor RAM on PC. Your number of simultaneously executed application (call it X) may be calculated by following formula:
X = Floor( F / M ) - 1  ( -1 - left memory for Parent application)
where:
F = "Free PC RAM in MB"
M = "Max RAM consumed of one clone application" in MB (in case, when all clone applications process similar data volume)