How to extract changes from the Change Store (Write Table) and store them in a database using Qlik Automate

[Emile_Koslowski]

This article explains how to extract changes from a Change Store by using the Qlik Cloud Services connector in Qlik Automate and how to sync them to a database.

The example will use a MySQL database, but can easily be modified to use other database connectors supported in Qlik Automate, such as MSSQL, Postgres, AWS DynamoDB, AWS Redshift, Google BigQuery, Snowflake.

The article also includes:

• Two automation examples you can download and import (see Qlik Automate: How to import and export automations):
  • Automation Example to Extract Change Store History to MySQL Incremental.json
  • Automation Example to Bulk Extract Change Store History to MySQL Incremental.json
• Configuration instructions for the examples

Content

• Prerequisites
• Creating the automation
• Insert changes in MySQL one by one
• Making this incremental
• Bulk updates
• Attachment configuration instructions
• Bonus!
• Replace field names
• User email instead of user id
• Triggering the automation from a sheet

Prerequisites

1. A working Write Table with a set of editable columns and some example values already stored in it. More information about the Write Table chart can be found in Write Table | help.qlik.com.
2. A MySQL (or similar) database table with columns that match your editable columns.
3. Setting up the database, extend your database with metadata fields:
   • userId: to store the user ID of the user who made a change
   • updatedAt: to store the datetime when a change was saved

Here is an example of an empty database table for a change store with:

• a single PK “productId”
• editable columns are “AmountToOrder”, “Priority”, and “Note”.
• additional columns “userId” and “updatedAt”, which will be used to log user activity and merge separate changes into the same record

Creating the automation

1. Create a new automation. See Qlik Automate for details.
2. Add the List Change Store History block from the Qlik Cloud Services connector.
3. Configure this block with the change store ID. You can copy this from the write table chart.
4. Perform a manual run of the automation to make sure some records are returned. The List Change Store History block will return a list of objects for each cell that contains one or more changes. Every object includes the combination of primary key(s), the editable column for that cell, and a list of all values belonging to that cell.
5. You have two options on how to perform this sync:
   • Insert changes one by one
   • Insert changes in bulk (this option is more complex but also more performant)

Insert changes in MySQL one by one

1. Add a Loop block (A)inside the List Change Store History block and configure it to loop over the list of changes inside each object returned by the List Change Store History block (B) :
   
   
   

   
   

2. Search for the MySQL connector (A) in the automation block library and drag the Upsert Record block inside the Loop block (B) :
   
   
   
   

   

3. Create a new connection (C) to your MySQL database in the Connection tab of the Upsert Record block and activate it for the block by clicking it once created.
4. Configure the Upsert Record block as follows
   1. Table: the database table you have created for the Write Table
   2. Where: the key:value mappings for the granularity on which you want to save the changes. In this example, it is a combination of the primary key (productId) with userId and updatedAt.
      
      
      1. productId (primary key): this comes from the cellKey in the List Change Store History block.
         
         
         
      2. userId and createdAt: these are defined for each change, so they should be retrieved from the item in the Loop block.
         
         The userId maps to the createdBy parameter.
         
         
         
         
         
   3. Record: this is the key:value mapping for the individual change that should get updated. The Key maps to the columnName that is returned by the List Change Store History block, the Value maps to the cellValue parameter that is returned in the Loop block:
      
      
      
      
5. This is what your automation will look like now:
   
   
   
   

Run the automation manually by clicking the Run button in the automation editor and review that you have records showing in the MySQL table:

Making this incremental

Currently, there is no incremental version yet for the Get Change Store History block. While this is on our roadmap, the automation from this article can be extended to do incremental loads, by first retrieving the highest updatedAt value from the MySQL table. The below steps explain how the automation can be extended:

1. Add a Do Query block from the MySQL connector to the automation and configure the query as follows:
   

   SELECT MAX(updatedAT) FROM <your database table>

   
   
   
   
2. Run a test run of the automation without the other blocks attached to verify the result in the Do Query block’s History tab:
   
   
   
   
3. Add a Condition block (A) to the automation and configure it to evaluate the MAX(updatedAt) (B) field from the Do Query block.
   
   Because the Do Query block returns the value as part of a list, the automation editor prompts you to specify which item of the list you want to use. Use the default version option Select first item from list (C).
   
   
   
   

   

4. Configure the operator in the Condition block to is not empty. If an updatedAt timestamp is found, the Yes part of the Condition block will be executed. If no timestamp is found, the No part will be executed.
   
   
   
   
5. Add a Variable block (A)to the Yes part of the Condition block and create a new variable of type String named Filter. The setting is accessed from the Manage variables (B) button in the Variables block.
   
   
   

   
   

6. Add an operation to the Variable block to Set value of Filter and type updatedAt gt “ in the input field.
   
   
   
   Click in the input field to add a reference to the timestamp in the Do Query block, mirroring the Condition block configuration.
   
   
   
   After the mapping is added, append it with an additional double quote character:
   
   
   
   
7. Right-click the Variable block and duplicate it, then add the duplicated block to the No part of the Condition (A).
   
   Remove the Set value of Filter step and replace it with the Empty Variable (B) operation.
   
   Right-click each Variable block and add a comment explaining the respective function. 
   
   
   
8. Reattach the original automation after the Condition block. 
   
   Verify that it is attached after the block and not inside the Yes or No sections. 
   
   
   
9. In the List Change Store History block (A), map the Filter variable to the Filter parameter (B) :
   
   
   
10. Run the automation and confirm it only picks up new changes on new runs.

Bulk updates

The solution documented in the previous section will execute the Upsert Record block once for each cell with changes in the change store. This may create too much traffic for some use cases. To address this, the automation can be extended to support bulk operations and insert multiple records in a single database operation.

The approach is to transform the output of the List Change Store History block from a nested list of changes into a list of records that contains the changes grouped by primary key, userId, and updatedAt timestamp.

See the attached automation example:  Automation Example to Bulk Extract Change Store History to MySQL Incremental.json. 

1. Drag the Loop and Upsert Record block outside the List Change Store History loop, but do not delete them:
   
   
   
   
2. Add a Variable block inside the List Change Store History block loop (A) and create a new variable partialChangeRecord of type Object (B).
   
   This variable will be used to map each cell value to the primary key(s), userId and updatedAt timestamp.
   
   
   
   
3. Add the first operation to the Variable to Empty it (A).
   
   This is important to ensure that for every item in the loop, we start with an empty partialChangeRecord variable.
   
   Next, add a Set key/values operation to set the primary key(s) (B).
   
   In our example, we have a single primary key, productId, but if you have multiple fields, you should add them one by one.
   
   
   
   Set it to the cellKey.rowKey parameter returned in the output from List Change Store History
   
   
   
4. Drag the Loop block back into the automation and attach it to the Variable block.
   
   Disconnect the Upsert Record block.
   
   
   
   
5. Add a new Variable block inside the Loop block and configure the Variable parameter to the partialChangeRecord object.
   
   Now set additional Key/values in the variable for the primary userId, updatedAt, and the cellValue:
   
   
   1. userId: createdBy parameter returned in the loop
   2. updatedAt: updatedAt parameter returned in the loop
   3. For the Cell value, set the key to the columnName returned by the List Change Store History block and set the Value to the cellValue returned from the Loop block:
      
      
      
   4. uniqueKey: add a fourth keyValue pair uniqueKey.
      
      This will be used later to merge various cell changes into a single record.
      
      This combines the primary key(s), userId, and updatedAt timestamp, each separated by a pipe (|) symbol.
      
      Click the input field, map the first parameter, then type a | -symbol, and click the input field again to map the next parameter:
      
      
      
      
6. Add another Variable block inside the Loop block (A) and create a new list type variable listOfPartialChanges.
   
   Add an Add item operation to add the partialChangeRecord variable (B) to this list.
   
   
   
7. Add a Merge Lists block after the List Change Store History block. 
   
   The Merge Lists block will be used to merge the partial change records into full records for each change.
   
   Configure both List parameters in the Merge Lists block to use the listOfPartialChanges variable.
   
   
   
8. Go to the Settings tab of the Merge Lists block and apply this configuration:
   
   
   1. Item merge strategy: Merge list 1 item and list 2 item in one new item (default)
   2. List 1 unique key: uniqueKey
   3. List 2 unique key: uniqueKey
   4. On duplicate unique key: Merge item from list 2 into item from list 1
   5. When a property exists on both lists: Keep value from list 1 (default) 
      
      Due to an ongoing defect, this parameter is only available after refreshing the automation editor. As the parameters use the default values, this should not impact you.
      
      
      
      
9. Perform a manual run of the automation to verify that the Merge Lists block output is merging.
   
   You will notice that this list contains many duplicates.
   
   
10. Add a Deduplicate List block (A) and configure the List parameter to use the output from the Merge Lists block, then set the key to uniqueKey (B).
    
    
    
11. Perform another manual run and confirm that the Deduplicate List block now only contains unique entries.
12. The output still contains the uniqueKey parameter that is not compatible with the database.
    
    There are two options: either extend the database or remove uniqueKey.
    
    To remove it, add a Transform List block and set the Input List parameter to the Deduplicate list block:
    
    
    
13. Configure the Fields in output list parameter from the Transform List block.
    
    To make this easier, it is best to have example data from a manual run. If you haven’t performed a manual run yet, do one now.
    
    Click the Add Field button and configure new fields:
    
    
    1. For the Field parameter, click the input field and start by mapping your primary keys
       
       

       

    2. For the Value parameter, do the same and select the corresponding parameter in the Item in input list option
       
       
       
       
    3. Repeat this for all other items in the list except for the uniqueKey
       
       

       

14. Add a Loop batch block (A) to the automation and configure it to loop over the Transform List block output in batches of 50 (B).
    
    Adjust this batch size depending on your database and the number of editable columns.
    
    

    

15. Add an Insert Bulk block from the MySQL connector within the Loop Batch block loop.
    
    Configure it to use your database table variable (or hardcode your table name) and set the Values parameter to the Batch in Loop Batch parameter.
    
    
    
    
16. Run the automation and confirm that the database gets updated.
    
    
    
    
17. Optionally, collapse all empty loop blocks to clean up the automation and provide comments to explain what the blocks and functions do.
    
    This will help you understand this automation when you revisit it in the future.
    
    To add a comment, right-click on the block and click Edit comment.
    
    
    

Attachment configuration instructions

The provided automations will require additional configuration after being imported, such as changing the store, database, and primary key setup. 

Automation Example to Extract Change Store History to MySQL Incremental.json

1. Variable - databaseTable -> configure with the name of your database table
2. Variable - changeStoreId -> configure with your change store ID
3. Upsert Record - MySQL -> replace the productId with your primary key, add additional primary keys if necessary

Automation Example to Bulk Extract Change Store History to MySQL Incremental.json

1. Variable - databaseTable -> configure with the name of your database table
2. Variable - changeStoreId -> configure with your change store id
3. Variable - partialChangeRecord -> replace the productId with your primary key, add additional primary keys if necessary
4. Variable - partialChangeRecord in Loop block -> Update the uniqueKey field by replacing the productId with your primary key, add additional primary keys if necessary
5. Transform List -> replace the productId with your primary key, add additional primary keys if necessary

Bonus!

Replace field names

If field names in the change store don't match the database (or another destination), the Replace Field Names In List block can be used to translate the field names from one system to another.

1. Search the Replace Field Names List block and add it to your automation.
   
   
   
   
2. Provide the translations for the field names that need to be changed to match the destination system.
   
   
   
   

User email instead of user id

To add a more readable parameter to track the user who made changes, the Get User block from the Qlik Cloud Services connector can be used to map User IDs into email addresses or names.

1. Search the Get user block (A) in the Qlik Cloud services connector and add it to your automation.
   
   Configure it to use the createdBy parameter (B).
   
   
   
   
2. Update the Upsert Record block to use the output from Get User.

   A user's name might not be sufficient as a unique identifier. Instead, combine it with a user ID or user email.

   
   

Triggering the automation from a sheet

Add a button chart object to the sheet that contains the Write Table, allowing users to start the automation from within the Qlik app. See How to run an automation with custom parameters through the Qlik Sense button for more information.

Environment

• Qlik Cloud Analytics
• Qlik Automate