QlikView Expressor: Using Partitions in an Output Operator

QlikView Expressor supports the feature of partitions, which are multiple threads of execution within an operator.  Each thread carries out the same processing on a subset of the records; each record is procesed by only one thread.

 

As discussed in another article, one of the valuable uses of partitioning is to improve the speed at which records enter and are processed by a dataflow.  And with the Write Table operator, partitioning can also improve the speed at which records are emitted from a dataflow.  However, the output operators that write to files or to Salesforce can only emit records on a single partition.

 

When your dataflow uses a Write Table operator, you have a range of options when using partitioning.  You may choose to retain the partitioning utilized by the upstream operators, repartition based on a different key field, or repartition in a way independent of prior partitioning directives.

 

Open the Write Table operator's Partition Settings window.  By default, this operator is configured to use the same partitioning scheme as its immediate upstream operator employing the same number of partitions and the 'Keep Upstream Partitioning' method.

 

partition9.png

 

If you want to change the number of partitions, select 'Specify' and a partitioning method.

 

partition11.png

 

If you change the partitioning method, it is probably a good idea to select 'Buffer input data' to ensure that the repartitioning process executed by the Write Table operator does not restrict the emission of records from the immediate upstream operator.

 

When using operators that write to files or to Salesforce, your only option is to departition the input as output from these operators must be confined to a single partition.

 

By default, the partition count will be set to one and the collection method to 'Unordered.'  Records will be written as they are received from the upstream operator. 

 

partition12.png

 

You may, however, choose to change the collection method.

 

partition13.png

 

Again, it is a wise choice to select 'Buffer input data' when specifying a method other than 'Unordered.'

 

The following video shows how to configure output operator partitioning.