This article provides details about the new permissions available for Private automations, which will replace the previous Automation Creator role.

You can assign these permissions to a custom role or the User Default. More information about creating & managing custom roles is available here.

The existing Automation Creator role will be deprecated later in 2026.

Private automations

These permissions allow users to run and create automations in their personal space.

Shared automations

Earlier in 2025, we introduced the shared automation permission that allows users to run, create, and manage automations in shared spaces in which they have the correct permissions. More information about which space roles are required for the various automation actions is available here:

• Full & Professional users
• Analyzer users

Understanding and managing your Qlik Cloud subscription consumption is essential for maintaining predictable costs, ensuring uninterrupted service, and optimizing resource allocation across your organization. This guide provides you with the tools, strategies, and best practices to gain complete visibility into how your subscription is being consumed and implement proactive controls to stay within your capacity limits.

While Qlik Cloud measures consumption at the tenant level, you can achieve effective governance through strategic monitoring, automated alerting, and space-based management practices. This guide will walk you through the monitoring tools available, how to automate their deployment and refresh, and practical approaches to tracking consumption patterns and implementing controls that align with your organizational needs.

Content:

• Understanding the Administration activity center capacity consumption view
• The Home page capacity consumption dashboard
• How does this view fit into your monitoring strategy?
• Official consumption data: The Data Capacity Reporting App
• Automating the deployment of the consumption app
• Getting estimated usage with Qlik Cloud Monitoring Apps
• Automating Qlik Cloud Monitoring Apps deployment
• Proactive monitoring strategies: What to track and at what levels
• Monitoring automation runs
• Monitoring Data for Analysis
• Monitoring scheduled reloads
• Monitoring reports generated
• Practical approaches to controlling consumption
• 1. User education and communication
• 2. Alert-driven intervention workflow
• 3. Development vs. production separation
• Leveraging Qlik's alerting and distribution capabilities for consumption monitoring
• Available monitoring and distribution tools
• Setting up each monitoring approach
• Examples: Setting up consumption monitoring across key metrics
• Your action plan: From reactive to proactive
• Additional resources

Understanding the Administration activity center capacity consumption view

The Administration activity center Home page provides your first line of visibility into capacity consumption. Understanding what this view offers and how it complements the detailed monitoring apps will help you build an effective monitoring strategy.

The Home page capacity consumption dashboard

Navigate to the Administration activity center → Home to see a real-time dashboard summarizing capacity consumption. This view displays visual bar charts for consumption metrics relevant to your subscription:

Common metrics displayed:

• Data for Analysis: Purchased vs. used with color-coded breakdown (App Import, App Reload, Datafile); not available for user-based subscriptions
• Reports generated (monthly): Purchased count vs. used
• Standard automation runs (monthly): Purchased vs. used
• Third-party automation runs (monthly): Purchased vs. used
• Assistant questions asked (monthly): Purchased vs. used (if Qlik Answers is enabled)

Additional metrics may include Data Moved, Large App consumption, Qlik Predict deployed models, and others, depending on your subscription.

Metrics appear dynamically as features are adopted. If no one has asked an assistant question yet, that metric won't display until first use, keeping the dashboard focused on what you're consuming.

The Data for Analysis chart shows a current snapshot with the last update timestamp. Most metrics update multiple times per hour, providing near real-time visibility into your consumption position.

How does this view fit into your monitoring strategy?

The Administration activity center provides high-level consumption visibility designed for rapid assessment. For detailed analysis, investigation, and proactive monitoring, you'll complement this view with a set of monitoring apps.

Daily quick check (2 minutes):

1. Scan consumption bars to identify any metrics approaching capacity
2. Review indicators showing current usage levels
3. Verify Data for Analysis timestamp shows recent data

When you need more detail, the Home page tells you what is being consumed. The monitoring apps tell you who, where, when, and why. Capacity subscriptions should use the Data Capacity Reporting App as the source of truth, while the Qlik Cloud Monitoring apps can be treated as estimated consumption reports, for example:

• For automation consumption details: Use the Automation Analyzer
• For data consumption details: Use the Data Capacity Reporting App and Reload Analyzer
• For report consumption details: Use the Report Analyzer

Use the Home page for daily checks and status awareness. When consumption requires attention or you need to understand trends, drill into the appropriate monitoring app for detailed analysis.

For more information, see Monitoring resource consumption.

Official consumption data: The Data Capacity Reporting App

The Data Capacity Reporting App is your official, billable record of consumption for capacity-based subscriptions. This Qlik-supported application is generated once per day (morning Central European Time) and provides the definitive view of your consumption against your entitlement.

The app tracks eight key value meters across the current and previous two months:

• Data for Analysis: Total data loaded into or created in Qlik Cloud (measured by monthly peak usage)
• Data Moved: Volume of data moved to cloud destinations via Qlik Talend Data Integration
• Third Party Transformations: Datasets used in transformations but not delivered by Qlik Talend Data Integration
• Automation Runs: All automation executions using Qlik Automate
• Reports: Reports distributed using Qlik Reporting Service
• Job Executions: Number of Talend job task runs (Qlik Talend Cloud subscriptions)
• Job Duration: Total duration of Talend job executions (Qlik Talend Cloud subscriptions)

This app represents your billable consumption record. The data in this app is what Qlik uses for official capacity reporting and billing purposes. When there's any discrepancy between this app and other monitoring sources, the Data Capacity Reporting App is the authoritative source. This app refreshes only once daily, meaning you see yesterday's official position, not real-time consumption. For more frequent monitoring and estimated usage, you'll complement this with the Qlik Cloud Monitoring Apps.

For detailed information, see Monitoring detailed consumption for capacity-based subscriptions.

Automating the deployment of the consumption app

Rather than manually distributing the consumption app from the Administration activity center each day, automate this process using the Capacity consumption app deployer template in Qlik Automate.

Setup steps:

1. Create a new automation in Qlik Automate
2. Navigate to the App Installer category
3. Select "Capacity consumption app deployer" template
4. Configure variables (space names, versions to keep)
5. Test manually once
6. Schedule to run daily around midday CET

This automation creates or uses designated spaces, imports the latest version, publishes it to a managed space, and maintains version history according to your configuration. You now have a single source of truth that updates automatically each day. Create automations or alerts on the published app for automated insights.

For complete details, see the Qlik Community article: Automate deployment of the Capacity consumption app with Qlik Automate.

Getting estimated usage with Qlik Cloud Monitoring Apps

While the official consumption report updates once daily, the Qlik Cloud Monitoring Apps (community-supported) can be reloaded multiple times per day up to your contractual reload limits, giving you more timely estimated usage insights.

The Qlik Cloud Monitoring Apps provide estimated consumption data that may differ slightly from the official Data Capacity Reporting App. Use these apps for trend monitoring, troubleshooting, and proactive management, but always refer to the Data Capacity Reporting App for official billable consumption figures.

Particularly valuable monitoring apps include:

App Analyzer: Provides comprehensive application usage and operational analytics, including:

• App access patterns and session frequency by user
• Sheet and object usage tracking
• App metadata and lineage information
• Historical snapshots to track changes over time

Automation Analyzer: Provides detailed analysis of automation runs, including:

• Run frequency by automation, space, and user
• Success/failure rates and error patterns
• Execution duration trends
• Third-party block usage tracking

Reload Analyzer: Tracks data refresh activity, including:

• Reload frequency by app and space
• Duration and resource consumption
• Success rates and failure patterns
• Reload scheduling patterns

Access Evaluator: Analyzes user roles, access, and permissions across your tenant

Report Analyzer: Tracks report generation, including:

• Report task execution frequency
• Report distribution patterns
• Report generation consumption

Entitlement Analyzer: For user-based subscriptions, provides insights into:

• Professional and Analyzer seat allocation and utilization
• Analyzer capacity consumption by user (for Analyzer capacity licenses)
• User session patterns and license assignment

For a complete list of available monitoring apps, see the Qlik Community article: The Qlik Sense Monitoring Applications for Cloud and On-Premise.

Automating Qlik Cloud Monitoring Apps deployment

The Qlik Cloud Monitoring Apps deployer template simplifies installation and maintenance of these community apps.

What it handles:

• Installs/upgrades selected monitoring apps from Qlik's OSS GitHub
• Creates required API keys and data connections
• Manages API key rotation to ensure connections remain operational
• Creates or uses existing shared and managed spaces
• Schedules regular reloads (configurable frequency)
• Verifies prerequisite settings and notifies if configuration changes are needed

Reload frequency considerations: You can reload these monitoring apps multiple times per day to get more current estimated usage data. However, each reload counts against your tenant's reload capacity limits. Consider your contractual limits when scheduling. For most organizations, reloading 2-4 times per day provides a good balance between timely insights and consumption.

For complete implementation details, see the Qlik Community guide: Qlik Cloud Monitoring Apps Workflow Guide.

The monitoring apps are also available on GitHub: qlik-oss/qlik-cloud-monitoring-apps.

Proactive monitoring strategies: What to track and at what levels

Effective governance comes from monitoring consumption at multiple levels and implementing proactive interventions. Here's how to approach monitoring for key consumption metrics.

Monitoring automation runs

Automation runs are counted across all automations in your tenant, regardless of owner or run mode (manual, scheduled, triggered, webhook, API). Test runs within the automation editor also count toward your limit.

What to monitor:

Tenant level:

• Total automation runs per month against your purchased capacity
• Rate of consumption (tracking whether you're on pace to exceed limits)
• Third-party block usage (may incur additional costs depending on subscription)

Space level:

• Which spaces are generating the most automation runs
• Identify spaces with unusually high consumption

Automation level:

• Individual automation run frequency
• Failed runs that retry excessively
• Scheduled vs. triggered vs. webhook run distribution

User level:

• Which users own the most frequently running automations
• Users with failed automations requiring attention

Example alert scenario: Using the Automation Analyzer, create alerts when:

• Total third-party (billable) runs reach 80% of purchased capacity
• Any single automation runs more than 100 times per day
• Failed run rate exceeds 20% for any automation
• A specific space's automations account for more than 40% of total runs

Monitoring Data for Analysis

Data for Analysis is measured by monthly peak usage. A single day's spike can impact your entire month's consumption.

This data is only available via the Data Consumption report; it is a lagging metric and currently lacks customer data such as app names, user names, and space names. As such, use of an automation template to provide notifications may be preferable to standard alerts, and some app size metrics may be better analyzed in the reload analyzer.

What to monitor:

Tenant level:

• Daily peak compared to purchased capacity
• Trend analysis: Are daily peaks increasing?
• Current month peak vs. previous months

App level:

• Which apps are contributing most to data volume
• New apps that significantly increase consumption
• Apps with unexpectedly large data reloads

Space level:

• Data consumption by space

Example alert scenario: Using the Data Capacity Reporting App and Reload Analyzer:

• Alert when the daily peak reaches 80%, 90%, 95% of purchased capacity in the Data Capacity Reporting App
• Alert when any single app reload increases data volume by more than 20GB in the Reload Analyzer
• Alert when a space's total data exceeds a defined threshold in the Data Capacity Reporting App

Monitoring scheduled reloads

Each subscription tier has limits on maximum concurrent reloads, and capacity subscriptions have daily reload counts. Exceeding concurrent limits causes queuing; exceeding daily limits can block operations.

What to monitor:

Tenant level:

• Scheduled reloads that exceed contractual limits
• Concurrent reload patterns during peak hours
• Failed reload rate

Space level:

• Reload frequency by space (wasted work)
• Reload schedule overlap that causes queuing

App level:

• Apps with multiple reload tasks
• Apps with long-running reloads that block concurrency

Example alert scenario: Using the Reload Analyzer:

• Alert when an app has more than one reload task active
• Alert when failed reloads exceed 10% in any 24-hour period
• Alert when reload queuing occurs during business hours

Monitoring reports generated

Report generation counts vary by subscription tier, with add-on packs available for purchase. Across all reporting capabilities, tenants have a maximum of 30,000 reporting-related requests per day.

What to monitor:

Tenant level:

• Monthly report generation against purchased capacity
• Rate of consumption throughout the month

Report task level:

• Which report tasks generate the most reports
• Report tasks using cycling (can generate up to 500 unique reports per task)

Example alert scenario: Using consumption reporting and monitoring apps:

• Alert when monthly reports reach 75% of capacity
• Alert when any single report task generates more than 100 reports in a day
• Alert when daily reporting requests approach 25,000

For detailed information on report limits, see Qlik Reporting Service specifications and limitations.

Practical approaches to controlling consumption

While Qlik Cloud measures consumption at the tenant level, you can implement effective governance practices that provide meaningful control over resource usage.

1. User education and communication

Make users aware of the impacts of their consumption and empower them to monitor their own usage.

Implementation:

• Publish monitoring apps to managed spaces accessible by content creators
• Regular communications showing tenant-wide consumption status to creators

2. Alert-driven intervention workflow

Create early warning systems that trigger well before official capacity notifications.

Implementation:

Alert tier 1 (60-70% of capacity):

• Notification to admins and space owners & managers
• Review consumption trends
• Identify optimization opportunities

Alert tier 2 (75-85% of capacity):

• Formal review meeting
• Suspend non-critical scheduled executions (automation runs, assistants, reload tasks, report tasks, etc)
• Contact the account manager about potential overage

Alert tier 3 (90%+ of capacity):

• Disable non-essential executions (automation runs, assistants, reload tasks, report tasks, etc)
• Defer development/testing activities to next month
• Discussion about capacity expansion or overage management

3. Development vs. production separation

Use strict space controls to prevent development consumption from impacting production limits, or procure a development subscription from Qlik to fully isolate capacity.

Implementation:

• Create clear development space boundaries
• Limit development automation runs through governance
• Schedule development reload activities during off-peak hours
• Regular cleanup of abandoned development content

For information on subscription types and capacity planning, see Qlik Cloud capacity-based subscriptions.

Examples: Setting up consumption monitoring across key metrics

The following examples demonstrate how to set up comprehensive monitoring for different consumption metrics. These examples assume you have deployed the Capacity consumption app deployer (running daily around midday UTC) and the Qlik Cloud Monitoring Apps deployer (running overnight) with default settings.

Explore the apps to discover a wide range of operational metrics you can monitor, alert, automate, and subscribe to.

Example 1: Third-party automation runs monitoring and alerts

Scenario: Your organization uses third-party automation blocks (such as Slack, ServiceNow, or Salesforce connectors), which incur additional costs based on consumption. You need to monitor third-party automation runs to prevent unexpected charges and identify which automations are driving costs.

Navigate to the Automation Analyzer and create the following alerts:

Alert 1: Third-party runs approaching limit

• Location: Create on the "Run Quotas" chart in the "Dashboard" sheet by right-clicking on the Billable bar. This will create an alert with three useful measures present:
  
  
  
  
• Condition: "Runs Billable Remaining (Bar)" Less than 2000 (for example):
  
  
  
  
• Schedule: When data changes
• Share with others: Add other key administrators/ managers

Alert 2: Individual user excessive third-party runs

• Location: On the "Dashboard" sheet, open the "Runs by executor" tab, and create an alert from this table.
• Condition: Where "Runs Billable CMTD" is Greater than 1000 (or a reasonable limit for an individual user in the tenant):
  
  
  
  
• Schedule: When data changes
• Share with others: Add other key administrators/ managers

Automation - Automation usage notifier: Automation or user email notifications

This approach allows you to send email notifications or take action directly on executing users or owners, while sending a fully customised template to notify them that they are approaching limits.

See Automation Usage Notifier | GitHub for details.

• Location: Create a new automation by importing the JSON file provided at automationNotifier. There is a readme present there, in summary:
  • Monitors automation run usage against tenant quota
  • Works directly with the Automation Analyzer monitoring app
  • Supports threshold limits by (automatic lowest-limit enforcement):
    • Absolute run count
    • Percentage of licensed quota
  • Assessment modes:
    • Per automation (notify owners)
    • Per user (notify heavy users)
  • Automatic fallback notification to admin group for orphaned automations or deleted users
  • Rich HTML email alerts with direct automation links
  • Safe to schedule daily or trigger after analyzer reload
  • Fully configurable via simple variables
  • Designed for soft governance and early quota protection

Example 2: Data for Analysis capacity monitoring

Scenario: Your Data for Analysis consumption is measured by monthly peak usage. You need early warning when daily peaks are trending upward and visibility into which apps are driving consumption.

Step 1: Create peak usage alerts in the Data Capacity Reporting App

Alert 1: Warning capacity threshold

• Location: Create on the "Data for Analysis" current month peak usage chart
• Condition: Set "Capacity remaining" to Less than 0.3 (for 30%; adjust to business need)
• Schedule: When data changes
• Share with others: Add other key administrators/ managers

Alert 2: Critical capacity threshold

• Location: Create on the "Data for Analysis" current month peak usage chart
• Condition: Set "Capacity remaining" to Less than 0.15 (for 15%; adjust to business need)
• Schedule: When data changes
• Share with others: Add other key administrators/ managers

Step 2: Create a weekly trend subscription

In the Data Capacity Reporting App:

1. Create a sheet showing: Weekly peak trend (last 8 weeks), Top 10 apps by data volume, Space breakdown
2. Create a subscription
3. Schedule: Monday at 9:00 AM
4. Recipients: Administrators, managers
5. Format: PDF
6. Name: "Weekly Data for Analysis consumption report."

Example 3: Comprehensive monthly consumption review package

Scenario: You want to create a comprehensive monthly review package that combines official billable data with estimated usage trends to facilitate informed capacity planning discussions.

Create a Qlik Automate automation that runs on the first business day of each month:

1. Trigger: Scheduled monthly on the first weekday at 9:00 AM
2. Block 1: Generate a snapshot from the Data Capacity Reporting App using the Qlik Reporting connector:
   • All value meters with official consumption vs. purchased capacity
   • Month-over-month trends
   • Year-to-date comparison
3. Block 2: Generate a snapshot from Automation Analyzer using the Qlik Reporting connector:
   • Top 10 consuming automations
   • Executor breakdown
   • Space breakdown
4. Block 3: Generate snapshot from Reload Analyzer using the Qlik Reporting connector:
   • Top 10 apps by reload duration
   • Reload frequency analysis
   • Failed reload summary
5. Block 4: Compile snapshots into a single PDF or PowerPoint
6. Block 5: Distribute to stakeholder group via email

Your action plan: From reactive to proactive

The key to managing Qlik Cloud consumption effectively is shifting from reactive (waiting for 80%/90%/100% notifications) to proactive (continuous monitoring with early intervention).

This week:

• Deploy the Capacity consumption app deployer automation (if on a capacity-based subscription)
• Deploy the Qlik Cloud Monitoring Apps deployer automation with your selected monitoring apps
• Review the deployed monitoring apps and familiarize yourself with available metrics
• Set up your first consumption alert targeting your highest-risk metric

This month:

• Configure a comprehensive tiered alert system across all relevant consumption metrics
• Establish consumption targets for high-usage spaces or functions
• Create documentation for users on consumption best practices
• Conduct the first monthly consumption review meeting

Ongoing:

• Daily: Quick review of the Administration activity center Home page for the current status
• Weekly: Review of alert notifications and trend analysis in monitoring apps
• Monthly: Formal consumption review meeting with stakeholders
• Quarterly: Comprehensive audit of automations, apps, and reload schedules to identify optimization opportunities
• Continuous: Regular communication with users about consumption practices and organizational targets

By combining automated monitoring through the official Data Capacity Reporting App and community monitoring apps, tiered alerts, clear governance policies, and proactive intervention workflows, you can effectively manage your subscription costs and maintain predictable, controlled consumption across your organization.

Additional resources

Qlik Help documentation:

• Monitoring resource consumption
• Monitoring detailed consumption for capacity-based subscriptions
• Qlik Cloud capacity-based subscriptions
• Understanding the subscription value meters
• Managing automations
• Qlik Automate limitations
• Qlik Reporting Service specifications and limitations

Qlik Community Official Support Articles:

• The Qlik Sense Monitoring Applications for Cloud and On-Premise
• Qlik Cloud Monitoring Apps Workflow Guide
• Automate deployment of the Capacity consumption app with Qlik Automate

Developer resources:

• Monitor your deployment (Qlik Developer Portal)
• Qlik Cloud Monitoring Apps on GitHub

The Qlik Cloud Monitoring Apps are community-supported and provided as-is. They are not officially supported by Qlik, though they are maintained through Qlik's Open-Source Software GitHub. The Capacity consumption app deployer and Qlik Cloud Monitoring Apps deployer are supported automation templates found in the template picker catalog.

When setting up a Microsoft SQL Server Always On Availability Group (AG) along with a Windows Failover Cluster, are there any additional SQL Server–side configurations or Talend-specific database settings required to run Talend Job against a MSSQL Always On database?

Answer

Talend Job need be adapted at the JDBC connection level to ensure proper failover handling and connection resiliency, by setting relevant parameters in the Additional JDBC Parameters field.

Talend JDBC Configuration Requirement

Talend should connect to SQL Server using either the Availability Group Listener (AG Listener) DNS name or the Failover Cluster Instance (FCI) virtual network name, and include specific JDBC connection parameters.

Sample JDBC Connection URL:

jdbc:sqlserver://<AG_Listener_DNS_Name>:1433;
databaseName=<Database_Name>;
multiSubnetFailover=true;
loginTimeout=60

Replace and with your actual values. Unless otherwise configured, Port 1433 is the default SQL Server port.

Key Parameter Explanations

multiSubnetFailover=true
Enables fast reconnection after AG failover and is mandatory for multi-subnet or DR-enabled AG environments.

applicationIntent=ReadWrite (optional, usage-dependent)
Ensures write operations are always routed to the primary replica.
Valid values:
ReadWrite
ReadOnly

loginTimeout=60
Prevents premature Talend Job failures during transient failover or brief network interruptions.

Best Practice Recommendation

Before promoting any changes to the Production environment, it is essential to perform failover and reconnection stress tests in the DEV/QA environment. This will help to validate the behavior of Talend Job during:

• AG role switchovers
• Network interruptions
  
• Planned and unplanned failover scenarios

Related Content 

Talend JDBC connection parameters | Qlik Talend Help Center 

Microsoft JDBC driver support for Always On / HA-DR | learn.microsoft.com 

SQL Server JDBC connection properties | learn.microsoft.com 

Environment

• Talend Data Integration 

This article explains how to extract changes from a Change Store by using the Qlik Cloud Services connector in Qlik Automate and how to sync them to an Excel file.

While the example uses a Microsoft Excel file, it can easily be modified to create a CSV as well.

The article also includes:

• An automation example you can download and import (see Qlik Automate: How to import and export automations): Automation Example To Extract Change Store Data and Store in Microsoft Excel.json
• A Qlik app example with an inline load script with example data: Write Table Purchase Order Demo.qvf
• Example purchase order template Excel file: Purchase order template.xlsx
• Configuration instructions for the example

Content

• Prerequisites
• Installing the example app
• Create the automation and get the SharePoint metadata
• Configure the automation
• Running the automation from the sheet
• Bonus: overwriting an existing Excel file
• Bonus: Sending the Excel file as an email attachment

Prerequisites

You will need the following:

1. A working Write Table with a set of editable columns and some example values already stored in it.
   
   For more information on the Write Table chart, see Write Table | help.qlik.com.
   
   Here is an example of the Write Table that will be used in this article. It has the following configuration:
   
   
   
   

    

   • Week start is included in the primary key because the purchasing process (making the changes) happens on a weekly basis.

   • Product Name is included in the primary key to make sure it is always returned when retrieving changes through the Get Current Changes From Change Store block in Qlik Automate.
     
     Below is an example of the table in an app:
     
     

2. An Excel file that will be the template for purchase orders.
   
   
   • This must be stored in SharePoint or OneDrive.
     
     
   • This file must contain a sheet with an empty table on the sheet. Take note of the sheet name/id and the table name and id.
     
     An example template file is attached to this article.
     
     
     
     
3. A destination folder in Microsoft SharePoint or OneDrive where the new purchase order files should be created.
   
   Example of the SharePoint structure:
   
   

Installing the example app

Optionally, you can use the app that is included in this article. Follow these steps to install the app and configure the Write Table:

1. Import the app into your Qlik Cloud tenant.
2. Go to the Data Load Editor and do a manual reload.
3. Open the Inventory management sheet and go to edit mode.
4. Drag a Write Table Chart (A) on top of the Straight Table.
   
   
   
   
5. Select Convert to: Write Table (B).
   
   

   

6. Go to the Change store section in the Write Table's configuration panel and define a new primary key.
   
   
   

    

7. Select Product ID, Product Name, and Order Date as columns for the primary key, then click Save.
   

   
   

8. Add the following editable columns to the Write Table:
   
   
   1. To purchase: Manual user input
      
      
      
      
   2. Priority: Single selection: High, Low
      
      Tip! Add an empty option by clicking the + button without providing a value.
      
      
      
      
   3. Note: Manual user input
      
      
      
      
9. Leave the edit sheet mode and take note of the change store id for the change store that is linked to the Write Table.
   
   
   
   
10. First, make a selection in the app, then provide some example changes in the Write Table to use as example data to configure the automation:
    
    
    

     

Create the automation and get the SharePoint metadata

1. Create a new automation. See Qlik Automate for details.
2. Before the automation can be configured, SharePoint metadata is required that cannot be retrieved dynamically in the automation.
3. Get the SharePoint Drive Id. You need access to the Drive in the SharePoint Site on which the Excel template file is stored. To do this, add the List Drives From Site block from the Microsoft SharePoint connector. Connect your SharePoint account to this block.
4. Click the input field for the Site Id in the Inputs tab (A) on the List Drives From Site block and use the do lookup functionality (B) to search for the site (C).
   
   
   

    

   
   

    

5. Right-click the List Drives From Site block (A) to perform a Test Run (B) of the automation.
   
   Once completed, review the automation’s run history to retrieve the correct Drive Id from the List Drives From Site block to find the Drive Id of the drive you want to use.
   
   Store this id for later use.
   
   
   
   
   
   

   

6. Add a List Items On Drive block from the Microsoft SharePoint connector to the automation.
   
   This block will be used to retrieve the folder id for the destination folder in which the purchase orders should be created. Configure the block with the Drive Id from step 5.
   
   Specify root as the Item Id.
   
   
   
   
7. Run the automation manually to review the automation run’s history and retrieve the correct Item Id for your destination folder from the output of the List Items On Drive block.
   
   
   
   Tip! If your destination folder is nested in other folders, you will need to repeat this step until you have the item Id of the destination folder. Start with root as the Item Id and then replace it with each folder’s item id as you go towards the destination folder.
   
   
8. Right-click both SharePoint blocks and collapse (A) their loops.
   
   Then right-click them again to disable (B) them as they are no longer needed for the regular automation runs.
   
   But you might want to keep them inside the automation if you want to take this automation to a different SharePoint team or folder structure in the future. If you are certain you will not need another drive id or folder id for this automation, you can delete the blocks.
   
   

   
   
   

    

Configure the automation

1. Add an Inputs block and configure it with one required parameter weekStart. 
   
   This will be used to capture the weekStart date from the app when a user triggers the automation from the app.

   
   

2. Add six Variable blocks to the automation and add the following variables of type String:
   
   
   • driveId: store the Drive Id from step 5 in the previous section
   • folderId: store the folder’s item id of step 7 in the previous section
   • templateFileName: name of the Excel file template
   • destinationFileName: name of the purchase order file to create
   • sheetName: name of the sheet that contains the table
   • tableName: name of the table
     
     
3. Set values for the variables that correspond with your Excel file template. If you are using the example template from this article, you can supply the following values:
   
   
   • driveId: store the Drive Id from step 5 in the previous section
   • folderId: store the folder’s item id of step 7 in the previous section
   • templateFileName: Purchase order template.xlsx
   • destinationFileName: Purchase order_{$.inputs.weekStart}.xlsx
     
     Tip! if your date format uses slashes, it will not work for the Excel file name, as Excel will create directories. Instead, use a different date format, such as MM-DD-YYYY or MM_DD_YYYY.
     
     
     
     
   • sheetName: Purchase order
   • tableName: Products
     
     

4. Add an Open File block from the Cloud Storage connector and configure it as follows:
   
   
   • Connector: Microsoft SharePoint
   • Path: templateFileName variable
   • Drive Id: driveId variable.
     
     
     

      

5. Add a Copy File block from the Cloud Storage connector.
   
   This block will copy the template Excel File and create a new empty Excel file to create the purchase order.
   
   Configure the block as follows:
   
   
   • Source File: select the “Open File on Microsoft SharePoint …” block
   • Destination connector: Microsoft SharePoint
   • Destination Path: folder path + / + destinationFileName variable
   • Drive Id: driveId variable
     
     
     

      

6. Add a List Items On Drive block from the Microsoft SharePoint connector.
   
   This block will be used to get the Item Id for the file created by the Copy File block.
   
   Configure it as follows:
   
   
   • Drive Id: driveId variable
   • Item Id: folderId variable
     
     Tip! Right-click on the List Items On Drive block and choose Collapse loop. This saves space in the automation and makes it more readable.
     
     
     
     
7. Perform a manual run of the automation to verify that the destination Excel file is created and is returned by the List Items On Drive block.
   
   
8. Add a Lookup Item In List block from the Lists blocks section.
   
   This block will be used to retrieve the item id of the file created by the Copy File block (because this id is not returned by the Copy File block).
   
   Configure the block as follows:
   
   
   • List: full output from List Items On Drive 2 (A)
   • Conditions:
     
     
     1. Click the first input box (Property from) in the Condition parameter (B) and select the name parameter (C).
        
        
        
        
     2. Set the next input field (operator) to equals (D).
        
        
        

         

     3. Set the third one (Value) to the destinationFileName (E) variable.
        
        

9. Add a Create Workbook Session block from the Microsoft Excel connector and configure it with the output from the Lookup Item In List block.
   
   
   
   
10. Add a List Current Changes From Change Store block from the Qlik Cloud Services connector.
    
    This block will return all saved changes from the change store.
    
    Configure the Store Id parameter with the write table’s change store id.
    
    

    

11. Add a Filter List block. This block will be used to filter orders for the correct week.
    
    Configure it as follows:
    
    
    • List: output of the List Current Changes From Change Store block
    • Conditions:
      
      
      • Property from: the Order Date (A) returned by the List Current Changes From Change Store block.
        
        Tip! Make sure this field is part of your primary key so it is included in the change data from the change store.
        
        
        

         

      • Operator: equals

      • Value: the weekStart input from the Inputs block
        
        

        

12. Add an Update Rows In Worksheet block from the Microsoft Excel connector.
    
    This block will only be used to update a single cell in the Excel template with the date of the Week Start.
    
    If your Order form does not have such a value, you can ignore this step.
    
    If it has multiple of these values, then you can repeat it for each of them.
    
    Configure the block as follows:
    
    
    • Drive Id: Drive Id from the “Parent Reference” from the Lookup Item In List block
    • Item Id: Id from the Lookup Item In List block
    • Worksheet Name: Sheet Name variable
    • Start Cell: Coordinate of the Excel cell that needs to be updated. In the example template, this is G2.
    • End Cell: Same coordinate as the Start Cell since we are only updating a single cell.
    • Values: The weekStart input from the Inputs block. Since it will only update a single cell, there is no need to create an array as described in the help text.
      
      

      

13. Add a Loop Batch block.
    
    This block will divide the output from the Filter List block across multiple batches that can be added to Excel batch by batch.
    
    Configure the block as follows:
    
    
    • Loop over items of list: Filter List block
    • Amount of items per batch: 100
      
      

14. Add a Variable block inside the Batch Loop block.
    
    Create a new variable RowsString of type string.
    
    This variable will be used to build a string containing the changes in a format that is accepted by the Microsoft Excel connector.
    
    Add the following operations in the variable block:
    
    
    • Empty RowsString: this makes sure that on every new batch, the variable is emptied.
    • Append to RowsString: this will add the first text value to the string. Configure this to a single square bracket [.
      
      

      

15. Add a Loop block.
    
    This block will iterate over all the changes in the current batch.
    
    Configure it as follows:
    
    Loop over items of list: Loop Batch > Batch
    
    

    

    Tip! Perform a manual run of the automation to make sure there are example values in the Loop block.
    
    
16. Add a Get User block from the Qlik Cloud Services connector inside the loop of the Loop block.
    
    This block will be used to retrieve the user information for the updatedBy parameter in each change.
    
    Configure it to use the updatedBy parameter as input for the User Id parameter.
    
    
    
    
17. Add a Variable block to the Get User block and configure it to use the RowsString variable.
    
    Add an Append to RowsString operation and configure the Value to be of this format [“value1”,“value2”,…] where every item in this list corresponds with a value from the change that should be stored in the Excel file.
    
    
    1. This string should be built value by value.
       
       Start by typing [“ and then click to add the first value Product ID (A)
       
       

       

       
       

        

    2. Add another double quote, a comma, and a new double quote (",") for the next value:
       
       

       

    3. Repeat steps a and b to add all the values.
       
       Finally, add the user’s name from the Get User block as the final value.
       
       Optionally, this can also be another indicator, such as an email address. Finish the operation with another square bracket and a trailing comma ("],).
       
       
       

        

18. Add an Add Rows To Table (Batch) block from the Microsoft Excel connector after the loop from the Loop block.
    
    This block will be used to update the Excel file with the generated string that represents the current batch of changes.
    
    Configure the block as follows:
    
    
    • Drive Id: Drive Id from the “Parent Reference” from the Lookup Item In List block
    • Item Id: Id from the Lookup Item In List block
    • Worksheet: Sheet Name variable
    • Table Id: Table Name variable
    • Rows: This will be the Rows String variable, but it is not fully ready yet and needs some modifications:
      
      
      1. Add the Rows String variable
         
         

         

      2. Click the field mapping to the variable and choose Add formula. This will open the formula picker.
         
         

         

      3. Search for the Right trim formula.

         
         

          

      4. Configure the Character to trim parameter to a single comma.

         

      5. Type a single square bracket after the field mapping in the Rows input field:

         
         

          

         
         

          

19. Add a Close Workbook Session block from the Microsoft Excel connector at the end of the automation after the loop from the Loop Batch block.
    
    Configure the block as follows:
    
    
    • Drive Id: Drive Id from the Parent Reference from the Lookup Item In List block
    • Item Id: Id from the Lookup Item In List block
    • Session Id: Id from the output of the Create Workbook Session block
      
      
      

       

20. Run the automation manually and review the generated Excel file to make sure the table in the order form is correctly populated.
    
    
    
    
    

     

Running the automation from the sheet

The automation is now configured and can be run manually. But ideally, a user can run it from within the Qlik Sense app whenever they are finished with creating orders through the Write Table.

This article will only cover the button’s configuration in a sheet. A step-by-step guide on configuring the button object to run automations is available in How to run an automation with custom parameters through the Qlik Sense button.

1. Add a Button object to the sheet that contains the Write Table, set the button's action to 'Execute automation', and configure the automation.
   
   Tip! The automation selector in the button object only returns the first 100 automations. If your automation is not shown, you might need to manually copy and paste the automation id from the automation URL.
   
   
2. Add a parameter for the weekStart input from the automation's Inputs block.
   
   Then configure it to use the GetFieldSelections formula for the [Order Date] field (this is used as Week start in the Write Table).
   
   
   

    

3. Select a date in the app and click the button to run the automation and ensure that the correct date is received by the Inputs block:
   
   
   

    

   
   

    

Bonus: overwriting an existing Excel file

The Copy File block will fail when there already exists an Excel file with the same name. Depending on the use case, that might be the right behavior or you might want to overwrite the file.
The overwrite process explained below will delete the existing file and then create a new file.

1. Go to the automation and disconnect the Open File on Microsoft SharePoint block from the  Variable - tableName block.
   
   
2. Search for the Check If File Exists block from the Cloud Storage connector and connect it to the Variable - tableName block.
   
   Configure the block as follows:
   
   
   • Connector: Microsoft SharePoint
   • Path: folder path + / + destinationFileName variable (this should be the same path as the one configured in the Copy File block)
   • Drive Id: driveId variable
     
     
     
     

3. Add a Condition block to the automation and configure it to evaluate the output from the Check If File Exists block.
   
   This block will return a Boolean (true or false) result. If it is true, the file exists.
   
   Configure the Condition block to evaluate that output using the Boolean 'is true' operator:
   
   

4. Add a Delete File block from the Cloud Storage connector to the 'Yes' part of the Condition block. This will then only be executed when a file already exists, and will then delete the file. Configure the block as follows:
   
   
   • Connector: Microsoft SharePoint
   • Path: folder path + / + destinationFileName variable (this should be the same path as the one configured in the Copy File block)
   • Drive Id: driveId variable
     
     
     
     
5. Collapse the 'No' part of the Condition block as this will not be used (when there is no file, the automation can continue and copy the template to create a new file). Right-click on the Condition block and select the 'Hide NO Condition' option.
   
   
   
   

   
   

6. Reconnect the Open File block (and the other attached blocks) to the Condition block.
   
   

   

7. Perform two runs of the automation for the same date (weekStart) and ensure the original file is deleted and then recreated.
   
   
   

    

   
   
   

Bonus: Sending the Excel file as an email attachment

Qlik Automate can also be used to share the purchase order with your purchasing team. This can be built in the same automation or in a separate automation. Below are the steps to add this to the same automation.

1. Add an Open File block from the Cloud Storage connector at the end of the automation.
   
   Configure it as follows:
   
   
   • Connector: Microsoft SharePoint
   • Path: folder path + / + destinationFileName variable
   • Drive Id: driveId variable
     
     
     

      

2. Add a Send Mail With Attachments block from the Microsoft Outlook 365 connector.
   
   This block will be used to send the Excel file as an email attachment to one or more recipients.
   
   Configure the block as follows:
   
   
   • To: one or more email addresses for the recipients of the purchase order
   • Subject: Purchase order { $.inputs.weekStart }
   • Type: text
   • Content Body: <your email body>
   • Attachments:
     
     
     1. Click the 'Add attachment' button
     2. Specify the output from the Open File 2 block
        
        
        
        

Tip! Update the button label to make it clear to users of your app that clicking it will also send the purchase order.

As an alternative, it is also possible to add these blocks to a new automation that is triggered from a second button.
 

Is it possible to use different IdPs in a Qlik Cloud multi-tenant deployment?

Qlik Cloud is designed to support a single interactive Identity Provider (IdP) per tenant. For details, see Why Qlik doesn't support multiple interactive identity providers on a Qlik Cloud tenant. Identity Fedration can be used to link user identities across IdPs. See Using Multiple concurrent Identity Providers with Qlik Cloud.

A multi-tenant Qlik Cloud deployment allows for additional flexibility. Specific IdPs can be assigned to different tenants. For example, two tenants could make use of two separate IdPs.

The following limitations and risks apply:

• Users are managed at the tenant level.
  
  While it is possible to have tenants using different IdPs in a multi-tenant environment, these tenants and their users are isolated, even if they belong to the same company or license.
  
  
• Duplication of license assignments.
  
  If you have multiple tenants associated with the same license, use the same IdP; otherwise, users risk duplicated license assignments. See Changing corporate identity providers for details. 
  
  This occurs because the IdPs do not treat the user's e-mail address as a unique parameter to identify the user. The user's subject is used, which is provided by the IdP. Different IdPs can give different parameters, meaning the same user can be seen differently across tenants. 

Environment

• Qlik Cloud Analytics

This article explains how to extract changes from a Change Store and store them in a QVD by using a load script in Qlik Analytics. 

The article also includes

• An app example with an incremental load script that will store new changes in a QVD
• Configuration instructions for the examples

Scenario

This example will create an analytics app for Vendor Reviews. The idea is that you, as a company, are working with multiple vendors. Once a quarter, you want to review these vendors.

The example is simplified, but it can be extended with additional data for real-world examples or for other “review” use cases like employee reviews, budget reviews, and so on.

The data model

The app’s data model is a single table “Vendors” that contains a Vendor ID, Vendor Name, and City:

Vendors:
Load * inline [
"Vendor ID","Vendor Name","City"
1,Dunder Mifflin,Ghent
2,Nuka Cola,Leuven
3,Octan, Brussels
4,Kitchen Table International,Antwerp
];

The Write Table

The Write Table contains two data model fields: Vendor ID and Vendor Name. They are both configured as primary keys to demonstrate how this can work for composite keys.

The Write Table is then extended with three editable columns:

• Quarter (Single select)
• Action required? (Single select)
• Comment (Manual user input)

Prerequisites

• A shared space
• A managed space (optional but advised for the tutorial)
• A connection to the Change-stores API to the Analytics REST connector in the shared space. A step-by-step guide on creating this connection is available in Extracting write table changes with the REST connector in Qlik Cloud.

Steps

1. Upload the attached .QVF file to a shared space
2. Open the private sheet Vendor Reviews
3. Click the Reload App (A) button and make sure data appears (B) in the top table
   

   
   

4. Go to Edit sheet (A) mode
5. Drag a Write Table Chart (B) on the top table, and choose the option Convert to: Write Table (C).
   
   This transforms the table into a Write Table with two data model columns Vendor ID and Vendor Name.
   
   
   

    

6. Go to the Data section in the Write Table’s Properties menu and add an editable column
   
   
   

    

7. This prompts you to define a primary key inside the table. Click Define (A) in the table and use both Vendor ID and Vendor Name as primary keys (B).
   
   You can also just use Vendor ID, but we want to show that this also supports composite primary keys.
   
   
   
   
   
   
8. Configure the editable column:
   
   
   • Title: Quarter
   • Show content: Single selection
   • Add options for Q1Y26 through Q4Y26.
     
     Tip! Also add an empty option by clicking the Add button without specifying a value.
     
     
     

      

9. Add another Editable column with the below configuration
   
   
   • Title: Action required
   • Type: Single select
   • Options:  Yes and No
     
     

     

10. Add another Editable column with the below configuration
    
    
    • Title: Review
    • Type: Single select
    • Options:  Yes and No
      
      
      
      
11. The Write Table is now set up.
    
    Go to the Write Table’s properties and locate the Change store (A) section. Copy the Change store ID (B).
    
    
    

     

12. Leave the Edit sheet mode. Then add changes for at least two records. Save those changes.
    
    
    

     

13. Go to the app’s load script editor and uncomment the second script section by first selecting all lines in the script section (CTRL+A or CMD+A) (A) and then clicking the comment button (B) in the toolbar.
    
    
    

     

14. Configure the settings in the CONFIGURATION part at the end of the load script
    
    
    

     

15. Update the load script with the IDs of the editable columns.
    
    The easiest solution to get these IDs is to test your connection. Make sure the connection URL is configured to use the /changes/tabular-views endpoint and uses the correct change store ID.
    
    
    

     

16. Copy and paste the example load script (for the editable columns only) and paste it in the app’s load script SQL Select statement that starts on line 159:
    
    
    

     

17. Replace the corresponding * symbols in the LOAD statement that starts on line 176:
    
    
    

     

18. Choose which records you want to track in your table by configuring the Exists Key on line 216.
    
    This key will be used to filter the “granularity” on which we want to store changes in the QVD and data model, as the load script will only load unique existing keys (line 235).
    
    
    • $(vExistsKeyFormula) is a pipe-separated list of the primary keys.
    • In this example, Quarter is added as an additional part of the exists key to keep track of changes by Quarter.
    • Optionally, this can be extended with createdBy and updatedAt to extend the granularity to every change made:
      
      
      

       

19. Reload the app and verify that the correct change store table is created in your data model. The second table in the sheet should also successfully show vendors and their reviews.

Environment

• Qlik Cloud Analytics

This article explains how to extract changes from a Change Store by using the Qlik Cloud Services connector in Qlik Automate and how to sync them to a database.

The example will use a MySQL database, but can easily be modified to use other database connectors supported in Qlik Automate, such as MSSQL, Postgres, AWS DynamoDB, AWS Redshift, Google BigQuery, Snowflake.

The article also includes:

• Two automation examples you can download and import (see Qlik Automate: How to import and export automations):
  • Automation Example to Extract Change Store History to MySQL Incremental.json
  • Automation Example to Bulk Extract Change Store History to MySQL Incremental.json
• Configuration instructions for the examples

Content

• Prerequisites
• Creating the automation
• Insert changes in MySQL one by one
• Making this incremental
• Bulk updates
• Attachment configuration instructions
• Bonus!
• Replace field names
• User email instead of user id
• Triggering the automation from a sheet

Prerequisites

1. A working Write Table with a set of editable columns and some example values already stored in it. More information about the Write Table chart can be found in Write Table | help.qlik.com.
2. A MySQL (or similar) database table with columns that match your editable columns.
3. Setting up the database, extend your database with metadata fields:
   • userId: to store the user ID of the user who made a change
   • updatedAt: to store the datetime when a change was saved

Here is an example of an empty database table for a change store with:

• a single PK “productId”
• editable columns are “AmountToOrder”, “Priority”, and “Note”.
• additional columns “userId” and “updatedAt”, which will be used to log user activity and merge separate changes into the same record

Creating the automation

1. Create a new automation. See Qlik Automate for details.
2. Add the List Change Store History block from the Qlik Cloud Services connector.
3. Configure this block with the change store ID. You can copy this from the write table chart.
4. Perform a manual run of the automation to make sure some records are returned. The List Change Store History block will return a list of objects for each cell that contains one or more changes. Every object includes the combination of primary key(s), the editable column for that cell, and a list of all values belonging to that cell.
5. You have two options on how to perform this sync:
   • Insert changes one by one
   • Insert changes in bulk (this option is more complex but also more performant)

Insert changes in MySQL one by one

1. Add a Loop block (A)inside the List Change Store History block and configure it to loop over the list of changes inside each object returned by the List Change Store History block (B) :
   
   
   

   
   

2. Search for the MySQL connector (A) in the automation block library and drag the Upsert Record block inside the Loop block (B) :
   
   
   
   

   

3. Create a new connection (C) to your MySQL database in the Connection tab of the Upsert Record block and activate it for the block by clicking it once created.
4. Configure the Upsert Record block as follows
   • Table: the database table you have created for the Write Table
   • Where: the key:value mappings for the granularity on which you want to save the changes. In this example, it is a combination of the primary key (productId) with userId and updatedAt.
     
     
     1. productId (primary key): this comes from the cellKey in the List Change Store History block.
        
        
        

         

     2. userId and createdAt: these are defined for each change, so they should be retrieved from the item in the Loop block.
        
        The userId maps to the createdBy parameter.
        
        
        

         

        
        
        
   • Record: this is the key:value mapping for the individual change that should get updated. The Key maps to the columnName that is returned by the List Change Store History block, the Value maps to the cellValue parameter that is returned in the Loop block:
     
     
     
     
5. This is what your automation will look like now:
   
   
   
   

Run the automation manually by clicking the Run button in the automation editor and review that you have records showing in the MySQL table:

Making this incremental

Currently, there is no incremental version yet for the Get Change Store History block. While this is on our roadmap, the automation from this article can be extended to do incremental loads, by first retrieving the highest updatedAt value from the MySQL table. The below steps explain how the automation can be extended:

1. Add a Do Query block from the MySQL connector to the automation and configure the query as follows:
   

   SELECT MAX(updatedAT) FROM <your database table>

   
   
   
   
2. Run a test run of the automation without the other blocks attached to verify the result in the Do Query block’s History tab:
   
   
   
   
3. Add a Condition block (A) to the automation and configure it to evaluate the MAX(updatedAt) (B) field from the Do Query block.
   
   Because the Do Query block returns the value as part of a list, the automation editor prompts you to specify which item of the list you want to use. Use the default version option Select first item from list (C).
   
   
   
   

   

4. Configure the operator in the Condition block to is not empty. If an updatedAt timestamp is found, the Yes part of the Condition block will be executed. If no timestamp is found, the No part will be executed.
   
   
   
   
5. Add a Variable block (A)to the Yes part of the Condition block and create a new variable of type String named Filter. The setting is accessed from the Manage variables (B) button in the Variables block.
   
   
   

   
   

6. Add an operation to the Variable block to Set value of Filter and type updatedAt gt “ in the input field.
   
   
   
   Click in the input field to add a reference to the timestamp in the Do Query block, mirroring the Condition block configuration.
   
   
   
   After the mapping is added, append it with an additional double quote character:
   
   
   
   
7. Right-click the Variable block and duplicate it, then add the duplicated block to the No part of the Condition (A).
   
   Remove the Set value of Filter step and replace it with the Empty Variable (B) operation.
   
   Right-click each Variable block and add a comment explaining the respective function. 
   
   
   

    

8. Reattach the original automation after the Condition block. 
   
   Verify that it is attached after the block and not inside the Yes or No sections. 
   
   
   

    

9. In the List Change Store History block (A), map the Filter variable to the Filter parameter (B) :
   
   
   

    

10. Run the automation and confirm it only picks up new changes on new runs.

Bulk updates

The solution documented in the previous section will execute the Upsert Record block once for each cell with changes in the change store. This may create too much traffic for some use cases. To address this, the automation can be extended to support bulk operations and insert multiple records in a single database operation.

The approach is to transform the output of the List Change Store History block from a nested list of changes into a list of records that contains the changes grouped by primary key, userId, and updatedAt timestamp.

See the attached automation example:  Automation Example to Bulk Extract Change Store History to MySQL Incremental.json. 

1. Drag the Loop and Upsert Record block outside the List Change Store History loop, but do not delete them:
   
   
   
   
2. Add a Variable block inside the List Change Store History block loop (A) and create a new variable partialChangeRecord of type Object (B).
   
   This variable will be used to map each cell value to the primary key(s), userId and updatedAt timestamp.
   
   
   
   
3. Add the first operation to the Variable to Empty it (A).
   
   This is important to ensure that for every item in the loop, we start with an empty partialChangeRecord variable.
   
   Next, add a Set key/values operation to set the primary key(s) (B).
   
   In our example, we have a single primary key, productId, but if you have multiple fields, you should add them one by one.
   
   
   
   Set it to the cellKey.rowKey parameter returned in the output from List Change Store History
   
   
   

    

4. Drag the Loop block back into the automation and attach it to the Variable block.
   
   Disconnect the Upsert Record block.
   
   
   
   
5. Add a new Variable block inside the Loop block and configure the Variable parameter to the partialChangeRecord object.
   
   Now set additional Key/values in the variable for the primary userId, updatedAt, and the cellValue:
   
   
   • userId: createdBy parameter returned in the loop
   • updatedAt: updatedAt parameter returned in the loop
   • For the Cell value, set the key to the columnName returned by the List Change Store History block and set the Value to the cellValue returned from the Loop block:
     
     
     

      

   • uniqueKey: add a fourth keyValue pair uniqueKey.
     
     This will be used later to merge various cell changes into a single record.
     
     This combines the primary key(s), userId, and updatedAt timestamp, each separated by a pipe (|) symbol.
     
     Click the input field, map the first parameter, then type a | -symbol, and click the input field again to map the next parameter:
     
     
     

      

     
     

      

6. Add another Variable block inside the Loop block (A) and create a new list type variable listOfPartialChanges.
   
   Add an Add item operation to add the partialChangeRecord variable (B) to this list.
   
   
   

    

7. Add a Merge Lists block after the List Change Store History block. 
   
   The Merge Lists block will be used to merge the partial change records into full records for each change.
   
   Configure both List parameters in the Merge Lists block to use the listOfPartialChanges variable.
   
   
   

    

8. Go to the Settings tab of the Merge Lists block and apply this configuration:
   
   
   • Item merge strategy: Merge list 1 item and list 2 item in one new item (default)
   • List 1 unique key: uniqueKey
   • List 2 unique key: uniqueKey
   • On duplicate unique key: Merge item from list 2 into item from list 1
   • When a property exists on both lists: Keep value from list 1 (default) 
     
     Due to an ongoing defect, this parameter is only available after refreshing the automation editor. As the parameters use the default values, this should not impact you.
     
     
     
     
9. Perform a manual run of the automation to verify that the Merge Lists block output is merging.
   
   You will notice that this list contains many duplicates.
   
   
10. Add a Deduplicate List block (A) and configure the List parameter to use the output from the Merge Lists block, then set the key to uniqueKey (B).
    
    
    

     

11. Perform another manual run and confirm that the Deduplicate List block now only contains unique entries.
12. The output still contains the uniqueKey parameter that is not compatible with the database.
    
    There are two options: either extend the database or remove uniqueKey.
    
    To remove it, add a Transform List block and set the Input List parameter to the Deduplicate list block:
    
    
    

     

13. Configure the Fields in output list parameter from the Transform List block.
    
    To make this easier, it is best to have example data from a manual run. If you haven’t performed a manual run yet, do one now.
    
    Click the Add Field button and configure new fields:
    
    
    1. For the Field parameter, click the input field and start by mapping your primary keys
       
       

       

    2. For the Value parameter, do the same and select the corresponding parameter in the Item in input list option
       
       
       
       
    3. Repeat this for all other items in the list except for the uniqueKey
       
       

       

14. Add a Loop batch block (A) to the automation and configure it to loop over the Transform List block output in batches of 50 (B).
    
    Adjust this batch size depending on your database and the number of editable columns.
    
    

    

15. Add an Insert Bulk block from the MySQL connector within the Loop Batch block loop.
    
    Configure it to use your database table variable (or hardcode your table name) and set the Values parameter to the Batch in Loop Batch parameter.
    
    
    
    
16. Run the automation and confirm that the database gets updated.
    
    
    
    
17. Optionally, collapse all empty loop blocks to clean up the automation and provide comments to explain what the blocks and functions do.
    
    This will help you understand this automation when you revisit it in the future.
    
    To add a comment, right-click on the block and click Edit comment.
    
    
    

     

Attachment configuration instructions

The provided automations will require additional configuration after being imported, such as changing the store, database, and primary key setup. 

Automation Example to Extract Change Store History to MySQL Incremental.json

1. Variable - databaseTable -> configure with the name of your database table
2. Variable - changeStoreId -> configure with your change store ID
3. Upsert Record - MySQL -> replace the productId with your primary key, add additional primary keys if necessary

Automation Example to Bulk Extract Change Store History to MySQL Incremental.json

1. Variable - databaseTable -> configure with the name of your database table
2. Variable - changeStoreId -> configure with your change store id
3. Variable - partialChangeRecord -> replace the productId with your primary key, add additional primary keys if necessary
4. Variable - partialChangeRecord in Loop block -> Update the uniqueKey field by replacing the productId with your primary key, add additional primary keys if necessary
5. Transform List -> replace the productId with your primary key, add additional primary keys if necessary

Bonus!

Replace field names

If field names in the change store don't match the database (or another destination), the Replace Field Names In List block can be used to translate the field names from one system to another.

1. Search the Replace Field Names List block and add it to your automation.
   
   
   
   
2. Provide the translations for the field names that need to be changed to match the destination system.
   
   
   
   

User email instead of user id

To add a more readable parameter to track the user who made changes, the Get User block from the Qlik Cloud Services connector can be used to map User IDs into email addresses or names.

1. Search the Get user block (A) in the Qlik Cloud services connector and add it to your automation.
   
   Configure it to use the createdBy parameter (B).
   
   
   
   
2. Update the Upsert Record block to use the output from Get User.

   A user's name might not be sufficient as a unique identifier. Instead, combine it with a user ID or user email.

   
   

    

Triggering the automation from a sheet

Add a button chart object to the sheet that contains the Write Table, allowing users to start the automation from within the Qlik app. See How to run an automation with custom parameters through the Qlik Sense button for more information.

Environment

• Qlik Cloud Analytics
• Qlik Automate

A Job design is shown below, using a tSetKeystore component to set the keystore file in the preJob, followed by using a tMysqlConnection to establish a MYSQL connection. However, MYSQL fails to connect. 

Nevertheless, by changing the order of the components as demonstrated below, the MYSQL connection is successful.

Resolution

To address this issue, you can choose from the following solutions without altering the order of the tSetKeyStore and tMysqlConnection components.

Disable useSSL parameter

Isolate the MYSQL connection in a separate child Job

To isolate the MYSQL connection, create a separate subJob. Use a tRunJob to execute the subJob and ensure the 'Use an independent process to run subJob' option is checked. The Job design is outlined below. 

Cause

tSetKeyStore sets values for javax.net.ssl properties, thereby affecting the subsequent components. Most recent MySQL versions use SSL connections by default. Since the Java SSL environment has been modified, the MySQL JDBC driver inherits these changes from tSetKeyStore, which can potentially impact the connection.

Environment

• Talend Data Integration 

A Job design is presented below:

tSetKeystore: set the Kafka truststore file.

tKafkaConnection, tKafkaInput: connect with Kafka Cluster as a Consumer and transmits messages.

However, while running the Job, an error exception occurred under the tKafkaInput component.

org.apache.kafka.common.KafkaException: Failed to construct kafka consumer 

Resolution

Make sure to execute the tSetKeyStore component prior to the Kafka components to enable the Job to locate the certificates required for the Kafka connection. To achieve this, connect the tSetKeystore component to tkafkaConnection using an OnSubjobOK link, as demonstrated below:

For more detailed information on trigger connectors, specifically OnSubjobOK and OnComponentOK, please refer to this KB article: What is the difference between OnSubjobOK and OnComponentOK?.

Environment

• Talend Data Integration 7.3.1, 8.0.1

This article explains whether changing integration credentials or the host address for a database integration requires an integration reset in Stitch. It will also address key differences between key-based incremental replication and log-based incremental replication.

Changing Integration Credentials

Updating credentials (e.g., username or password) does not require an integration reset. Stitch will continue replicating data from the last saved bookmark values for your tables according to the configured replication method.

This article aims to help you facilitate the most effective way to collaborate with Qlik Support. It defines your responsibilities as a partner, including reproducing issues, performing basic troubleshooting, and consulting the knowledge base or official documentation.

Before contacting Qlik Talend Technical Support, partners must complete the steps outlined in Qlik Responsible Partner Duties and should review the OEM/MSP Support Policy to understand the scope of support and the expectations on Partners.

• Partners are required to complete the steps outlined in this article before submitting a support case.
• It is the partner’s responsibility to answer the customer's questions to the best of their ability by referring to documentation, the Qlik Community, previous cases, and online resources before submitting a case or commenting on an existing one.
• For high-priority or urgent issues, it is acceptable to raise a ticket early. However, the responsibilities outlined in the article still apply at every stage of the case.

Content

• General Expectations Before Submitting a Case
• Provide Product and Version Information
• Reproduce the Issue in Your Environment
• Always Provide Complete (Not Partial) Logs
• Describe What You Have Already Investigated
• Submit All Related Files You Investigated Or Used For Replication
• Submit the Case Using Your Partner Account
• Partner Case Submission
• Partner Case Submission Template
• Summary of the Problem
• Customer Information
• Files Attached

General Expectations Before Submitting a Case

Isolate What Is Affected

Identify which Qlik product, source endpoint, target endpoint, environment, or system layer is experiencing the issue.

For example, if a task fails in Qlik Replicate on Windows, check whether the issue occurs in a single task or across all tasks, whether it happens during full load or CDC, and whether it is related to a specific table or the data itself.

Similarly, if the issue occurs in only one environment (e.g., Production), ask the customer to confirm whether it can be reproduced in a test environment, or test in your own environment to determine if the issue is environment-related.

Provide Product and Version Information

Always include the exact product name, version, source endpoint, target endpoint the customer is using. 

Many issues are version-related, endpoint-related and Support cannot accurately investigate the issue without this information.

If the product the customer is using has reached End of life (EOL), please plan the upgrade. If the issue can be reproducible on the latest version, please reach out to us so that we can investigate and determine whether it's a defect or working as designed. 

For End of Life or End of Support information, see Product Lifecycle

Reproduce the Issue in Your Environment

Partners are expected to recreate the customer’s environment (matching versions, configurations, and other relevant details) and attempt to reproduce the issue.

If you do not already have a test environment, please ensure one is set up. Having your own environment is essential for reproducing issues and confirming whether the same behavior occurs outside of the customer’s setup.

In some cases, it may also be helpful to test in a clean environment to rule out local configuration issues.

If the issue does not occur in the newer version or a clean setup, it may have already been resolved, and you can propose an upgrade as a solution.

See the Release Note for the resolved issues.

Regardless of whether the issue could be reproduced, please include: 

• The exact Qlik Product version and environment you used for testing
• The exact steps you followed
• Screenshots showing your process
• Any sample data, DDL, or setting used during testing
• A summary of your results (such as “Reproduced same error,” or “Could not reproduce – no error occurred”)

Always Provide Complete (Not Partial) Logs

While pasting a portion of the log into the case comment can help highlight the main error, it is still required to attach the Diagnostic Package with the entire original log file (using, for example, FileCloud).

Support requires the full verbose logs and task settings to understand the overall context and verify that the partial information provided is accurate and complete.

It is difficult to verify the root cause or provide reliable guidance without full verbose logs.

Additionally:

• Specify which logs you reviewed(If there are a large number of log files, please specify the exact log file name).
• Confirm that the customer has provided complete, untrimmed logs and decrypted logs if necessary for certain data errors.
• Include logs from both the customer's environment and your own reproduction tests
• Change the log level to VERBOSE temporarily, reproduce the issue, and share the Diagnostic Package with us.
  
  Providing both sets of logs allows Support to understand what tests have been performed and to verify whether the same error occurred during reproduction. 

Describe What You Have Already Investigated

Please do not simply forward or copy and paste the customer’s inquiry.

As a responsible partner, you are expected to perform an initial investigation. In your case submission, clearly describe:

• What have you reviewed (such as logs and configurations)?
• What steps did you take to isolate or reproduce the issue?
• What have you analyzed, tested, or attempted?
• What you concluded or suspected based on your investigation (such as why you think it is a bug, why you think the problem is occurring)

Sharing this thought process:

• Helps Support understand your assumptions
• Avoids duplicated effort
• Demonstrates that the case is ready for advanced-level investigation

Even if the issue is still unresolved, outlining what you have already tried helps Support address it more quickly and effectively.

Submit All Related Files You Investigated Or Used For Replication

Attach all relevant files you have received from the customer and personally reviewed during your investigation, as well as all relevant files you have used when reproducing the steps.

Providing both the customer’s files and your reproduction files enables Support to verify whether the same issue occurs under the same conditions and to determine if the problem is reproducible, environment-specific, or specific to the customer’s configuration.

This includes (but is not limited to):

• Log files (such as server logs, Windows Event logs, job logs)
• Screenshots: If an error appears in the UI, please include screenshots showing the error message or any screenshots that help illustrate the issue
• Sample data that causes the issue
• Affected Table DDL(both source and target)
• Output of change table (ct table)
• Any other relevant information that may help explain the issue
• Including specific details for the issue, such as:
  • The exact time of occurrence
  • Frequency
  • Whether it was a one-time issue or the issue occurs continuously
  • Specific affected table name, column name
  • Specify any triggering conditions or patterns
  • Has anything changed in the environment (including maintenance) or the design of the task
  • If this is related to an upgrade
  • Current status of the issue
  • Background of the issue(such as requirements)

Submit the Case Using Your Partner Account

All support cases must be submitted using your official partner account, not the customer's account.

If you do not yet have a partner's account, contact Qlik Customer Support to request access and to receive the appropriate onboarding.

Partner Case Submission

Review the support policy and set the case severity properly. See Qlik Support Policy and SLAs.

Partner Case Submission Template

This template provides guidance on what to include and how to structure your case.

Summary of the Problem

What happened? When did it happen? Where did it occur?

Clearly describe the issue, including:

• A summary of the impact (example: "Unable to resume the task")
• The exact time of occurrence (example: "Since July 10, from 13:55 JST")
• Product Name:
• Product Version:
• Source Endpoint:
• Source Version/ODBC driver version:
• Target Endpoint:
• Target Version/ODBC driver version:
• Operating System: 
  (Such as Windows Server 2019, RHEL 😎
• LogStream / FileChannel Involved

Specify if applicable:

• Task Type: Existing task / New task
• Environment: Production / Development
• Tenant information(only Qlik Cloud)

Find your Qlik Cloud Subscription ID and Tenant Hostname and ID

Customer Information

Only include what is needed based on the case type.

Files Attached

List the files you’ve included in the case and provide a brief description of each.

Summary of your Investigation

Explain what steps you took to investigate the issue before contacting Support.

Examples:

• Logs/files you have reviewed
• Patterns/errors observed
• Assumptions you made (such as suspected cause)
• Whether the issue is reproducible (If it's reproducible, describe the reproduction steps too)
• Troubleshooting steps attempted
• Expected Result
• Observed Result
• Is there any known workaround?
  (Yes / No – please describe)
• Related Community Discussions
  (Please provide a link to the related community, if available.)
• Document Reference Available:
  (Yes / No – provide link if possible)

Thank you! We appreciate your cooperation in following these guidelines.

This ensures that your cases can be handled efficiently and escalated quickly when necessary.

Environment

• Qlik Data Integration Products

This error occurs with the Google Cloud SQL PostgreSQL database integration and it displays as below in the extraction logs:

Fatal Error Occured - ERROR: temporary file size exceeds temp_file_limit 

Resolution

To resolve this issue, you need to increase the temp_file_limit parameter in your PostgreSQL configuration

Here are the steps to fix it:

1. Access your Google Cloud SQL instance settings.

2. Locate the database flags or parameters section.

3. Find the temp_file_limit flag and increase its value.

   • The value is specified in kilobytes (kB).

   • The default in PostgreSQL is -1, which means no limit. However, Cloud SQL may enforce a smaller custom value depending on your instance configuration.

4. If you’re unsure about the appropriate value, start by doubling the current limit and adjust as needed based on your workload. Increasing this limit allows larger queries to complete but may also increase storage usage, so monitor performance and disk space after making the change.

5. Save the changes.
   Updating database flags in Cloud SQL typically requires a restart of the instance for the new settings to take effect.

After modifying the temp_file_limit, restart your PostgreSQL instance (if required) and run an extraction in Stitch.

Cause 

The error message indicates that the temporary file size has exceeded the temp_file_limit in your Google Cloud SQL PostgreSQL database. This limit is set to control the maximum size of temporary files used during query execution.

Related Content

See Google’s documentation on configuring database flags here:
Configure database flags | cloud.google.com

Environment

Stitch 

To investigate Task failure, It is necessary to collect the Diagnostics Package from Qlik Cloud Data Integration.

Option One: Project-level

1. Click on the 3 dots menu of the task in question
2. Click View task logs
   
   
   
   

Option Two: Monitor view within the task

1. Click on the 3 dots menu in the right-hand corner
2. Click View task logs
   
   
   
   
   
3. Once the logs are open, click Download
4. Select Diagnostics package
5. Click Download
   
   
   
   
   
6. A small notification will pop up at the top of the page when the download starts:
   
   
   
   And then when it completes:
   
   
   
   
   

Often, Support will request that specific logging components be increased to Verbose or Trace in order to effectively troubleshoot. To modify, click on the "Logging options" located in the right-hand corner of the logs view. The options presented in the UI do not use the same terminology as what you see in the logs themselves. For better understanding, please refer to this mapping:

Please note that if the View task logs option is not present in the dropdown menu, it indicates that the type of task you are working with does not have available task logs. In the current design, only Replication and Landing tasks have task logs.

Environment

• Qlik Data Gateway  
• Qlik Cloud 

Qlik Automate is a no-code automation and integration platform that lets you visually create automated workflows. It allows you to connect Qlik capabilities with other systems without writing code. Powered by Qlik Talend Cloud APIs, Qlik Automate enables users to create powerful automation workflows for their data pipelines.

Learn more about Qlik Automate.

In this article, you will learn how to set up Qlik Automate to deploy a Qlik Talend Cloud pipeline project across spaces or tenants.

To ease your implementation, there is a template on Qlik Automate that you can customize to fit your needs.
You will find it in the template picker: navigate to Add new → New automation → Search templates and search for ‘Deploying a Data Integration pipeline project from development to production' in the search bar, and click Use template.

ℹ️ This template wil be generally available on October 1, 2025.

Use case and pre-requisites

In this deployment use case, the development team made changes to an existing Qlik Talend Cloud (QTC) pipeline.

As the deployment owner, you will redeploy the updated pipeline project from a development space to a production space where an existing pipeline is already running.

To reproduce this workflow, you'll first need to create:

• Two data spaces:
• A DEV space
• A PROD space
• Two source databases
• A DEV source database
• A PROD source database
• Two target databases
• A DEV target database
• A PROD target database

Using separate spaces and databases ensures a clear separation of concerns and responsibilities in an organization, reduces the risk to production pipelines while the development team is working on feature changes.

Workflow steps:

1. Export the updated pipeline project from the DEV space
2. Get the project variables from DEV
3. Update the project variables for PROD before import
4. Stop the pipeline currently running in production
5. Import your project to the PROD space
6. Prepare your project and check the status
7. Restart the project in PROD

ℹ️ Note: This is a re-deployment workflow. For initial deployments, create a new project prior to proceeding with the import.

Step-by-step workflow

1. Export the updated pipeline project from the DEV space

Use the 'Export Project' block to call the corresponding API, using the ProjectID.

This will download your DEV project as a ZIP file. In Qlik Automate, you can use various cloud storage options, e.g. OneDrive. Configure the OneDrive 'Copy File on Microsoft OneDrive' block to store it at the desired location.

To avoid duplicate file names (which may casue the automation to fail) and to easily differentiate your project exports, use the 'Variable' block to define a unique prefix (such as dateTime). 

2. Get the project variables from DEV

From the 'Qlik Talend Data Integration' connector, use the 'Get Project Binding' block to call the API endpoint.

The 'bindings' are project variables that are tied to the project and can be customized for reuse in another project. Once you test-run it, store the text response for later use from the 'History' tab in the block configuration pane on the right side of the automation canvas:

3. Update the project variables for PROD before import

We will now use the 'bindings' from the previous step as a template to adjust the values for your PROD pipeline project, before proceeding with the import.

From the automation, use the 'Update Project Bindings' block. Copy the response from the 'Get Project Binding' block into the text editor and update the DEV values with the appropriate PROD variables (such as the source and target databases). Then, paste the updated text into the Variables input parameter of the 'Update Project Binding' block.

ℹ️ Note: these project variables are not applied dynamically when you run the 'Update Bindings' using the Qlik Automate block. They are appended and only take effect when you import the project.

4. Stop the pipeline currently running in production

For a Change Data Capture (CDC) project, you must stop the project before proceeding with the import.

Use the 'Stop Data Task' block from the 'Qlik Talend Data Integration' connector. You will find the connectors in the Block Library pane on the left side of the automation canvas.

Fill in the ProjectID and TaskID:

ℹ️ We recommend using a logic with variables to handle task stopping in the automation. Please refer to the template configuration and customize it to your needs.

5. Import your project to the PROD space

You’re now ready to import the DEV project contents into the existing PROD project.

⚠️ Warning: Importing the new project will overwrite any existing content in the PROD project.

Using the OneDrive block and the 'Import Project' blocks, we will import the previously saved ZIP file.

ℹ️ In this template, the project ID is handled dynamically using the variable block. Review and customize this built-in logic to match your environment and requirements.
 

After this step is completed, your project is now deployed to production.

6. Prepare your project and check the status

It is necessary to prepare your project before restarting it in production. Preparing ensures it’s ready to be run by creating or recreating the required artifacts (such as tables, etc).

The 'Prepare Project' block uses the ProjectID to prepare the project tasks by using the built-in project logic. You can also specify one or more specific tasks to prepare using the 'Data Task ID' field. In our example, we are reusing the previously set variable to prepare the same PROD project we just imported.

If your pipeline is damaged, and you need to recreate artifacts from scratch, enable the 'Allow recreate' option. Caution: this may result in data loss.

Triggering a 'Prepare' results in a new 'actionID'. This ID is used to query the action status via the 'Get Action Status' API block in Qlik Automate. We use an API polling strategy to check the status at a preset frequency. 

Depending on the number of tables, the preparation can take up to several minutes.

Once we get the confirmation that the preparation action was 'COMPLETED', we can move on with restarting the project tasks.

If the preparation fails, you can define an adequate course of action, such as creating a ServiceNow ticket or sending a message on a Teams channel.

ℹ️ Tip: Review the template's conditional blocks configuration to handle different preparation statuses and customize the logic to fit your needs.

7. Restart the project in PROD

Now that your project is successfully prepared, you can restart it in production.

In this workflow, we use the 'List Data Tasks' to filter on 'landing' and 'storage' for the production project, and restart these tasks automatically.

All done: your production pipeline has been updated, prepared, and restarted automatically!

Now it’s your turn: fetch the Qlik Automate template from the template library and start automating your pipeline deployments.

Related Content

About Qlik Automate

Start a Qlik Talend Cloud® trial

How to get started with the Qlik Talend Data Integration blocks in Qlik Automate

How to use the Qlik Automate templates