Data Flow Processors

[Jennell_Yorkman]

In this blog post, I will review some data flow processors that can be used to prepare your data in a data flow. Let’s start by quickly reviewing what a data flow is. In Qlik Cloud Analytics, a data flow is a no-code experience that visually allows you to prepare your data with drag and drop capabilities. It is intuitive and easy to use and does not require the user to have scripting experience. Data flow processors, along with sources and targets, are used to build a data flow. Each processor handles a specific data transformation task. Here you will find a full list of the data flow processors available.

This blog will touch base on a few processors to familiarize you with how they work and how easy they are to use. To begin, a data flow must first be created. There is more than one way to do this. From the Qlik Cloud Analytics catalog, click on the + Create new button and select Data flow or navigate to Prepare data from the menu and click on the add Data flow button at the top of the page.

+ Create new menu

+ Create new

Prepare data

MenuData flow

Once you name the new data flow, navigate to the Editor.

On the left, there are sources, processors and targets. The source is the data input, the processors are the data transformation types, and the targets are data outputs. Before we can look at the processors, we need to select our input data from the data catalog or a connection. Once that is in place, we can begin to explore the processor options. There are several data flow processors – too many to review in this blog but I will review three of the them - the Filter processor, the Join processor and the Unpivot processor.

Filter Processor

The filter processor filters data based on a condition. A processor can be added to the data flow canvas by dragging and dropping the processor onto the canvas or by clicking on the menu in the data source and selecting Add processor.

If you drag and drop the processor onto the canvas, you will need to connect the dots between the input and processor. If you add it from your data source menu, the dots will automatically be connected for you.

Each processor has a properties panel where the processor can be configured.  In this example, let’s use the filter to select employees who live in the United States. To do this, first select the field to process – Country. There is an option to apply a function but one is not needed in this example.  The operator will be equal, and the Value will be United States. Once the properties are entered, click the Apply button to save.

At the bottom of the page, I can preview the script (matching and not matching records) for the filter processor I just applied and see a preview of the data.

From the filter processor menu, there are a few options for my next step as seen below.

Add matching target will add a target to the data flow for the records that match the Country = United States filter. Add non-matching target will add a target to the data flow for the records that do not match the Country = United States filter. Matching and non-matching processors can also be added. For this example, I will add a matching target and in the properties panel, I will select the space, the extension (.qvd, .parquet, .txt or .csv) and the name of the target file. Like the sources, the target can be a data file or connection. Once I click Apply in the properties panel, I will see a message at the top right indicating that my flow is valid and ready to be run. Running the data flow will grab my Employees dataset, filter the data by country and store the results in a QVD named US Employess.

I now have a data file that has been transformed and prepared for use.

Join Processor

Now, let’s look at how we can join two data inputs into one data output. To do this, two data inputs are required. In the example below, ARSummary and ARSummary-1 are the two data inputs.

In the properties panel of the join processor, the join type is selected and the fields that should be used to link the two tables are selected. You can learn more about joins here. Once the target is added, the data flow can be run, and the result will be a single table with the records from the ARSummary table and the associated records from the ARSummary-1 table.

Unpivot Processor

If you are familiar with scripting, the unpivot processor is like a crosstable load. It allows you to rearrange a table so that column data becomes row data. It can transform a table like this:

To this:

Here is an example data flow with the unpivot processor:

In the properties panel of the unpivot processor, there are only a few settings to update. The first is the unpivot fields. Here is where the fields that we want to unpivot are selected. In this example, we want the year to be stored as row level data so we select them all.

The Attribute field name is the name we want to give to the unpivoted fields – in this case Year. The Value field name is the name of the data that is associated with the fields we are unpivoting – in this examples Sales.

After applying the changes and running the data flow, we will have a table transformed based on our specifications without any code.

In this blog post, we touched upon three of the many processors that can be used in a data flow. Note that a data flow can have many sources, processors and targets – it all depends on your needs. The visual interface of a data flow makes it easy to prepare your data without any code in an appealing design that is easy to follow. Try it out!

Thanks,

Jennell