Executing tasks or modifying tasks (changing owner, renaming an app) in the Qlik Sense Management Console and refreshing the page does not update the correct task status. Issue affects Content Admin and Deployment Admin roles.

The behaviour began after an upgrade of Qlik Sense Enterprise on Windows. 

Fix version:

This issue can be mitigated beginning with August 2021 by enabling the QMCCachingSupport Security Rule.

Solution for August 2023 and above:

Enable QmcTaskTableCacheDisabled.

To do so:

1. Navigate to C:\Program Files\Qlik\Sense\CapabilityService\
2. Locate and open the capabilities.json
3. Modify or add the QmcTaskTableCacheDisabled flag for these values to true
   
   {"contentHash":"CONTENTHASHHERE","originalClassName":"FeatureToggle","flag":"QmcTaskTableCacheDisabled","enabled":true}
   
   Where CONTENTHASHHERE matches the number in all other features listed in the capabilities.json.
   
   Example:
   
   
   
   This will disable the caching on the tasks table only, leaving the overall QMC Cache intact to gain performance. If you had previously set QmcCacheEnabled, QmcDirtyChecking, QmcExtendedCaching to false, please set it to true again.
   
   
4. Restart the Qlik Sense services

Workaround for earlier versions:

Upgrade to the latest Service Release and disable the caching functionality: 

To do so:

1. Navigate to C:\Program Files\Qlik\Sense\CapabilityService\
2. Locate and open the capabilities.json
3. Modify the flag for these values to false
   • QmcCacheEnabled
   • QmcDirtyChecking
   • QmcExtendedCaching
4. Restart the Qlik Sense services
   
   

NOTE: Make sure to use lower case when setting values to true or false as capabilities.json file is case sensitive.

Should the issue persist after applying the workaround/fix, contact Qlik Support.

Internal Investigation ID(s): 

QB-2096
QB-5168
QB-7655
QB-17238

Environment

Qlik Sense Enterprise on Windows 

Table of content: 

• Writeback use case
• Two ways to execute an automation from a Qlik Sense Sheet
• 1. Native 'execute automation' action in the Qlik button
• 2. Link a triggered automation from your Qlik Sense Sheet.
• Example 1: Open website or URL (button)
• Example 2: Add an action link in your straight table
• Example 2: Extension with input forms
• Additional Notes
• Demo application with automations

 

Writeback use case

Qlik Application Automation was released in Qlik Cloud on September 28th, 2021, and has helped many customers take action on their data and automated numerous operational day-to-day workflows.  

One of the key use cases we’ve seen in the first year is the writeback use case where users can, from a single click on a Qlik Sense dashboard, add, correct, or update data into their operational system (e.g. Salesforce, Jira, HubSpot, Service Now) or into the data platform (e.g. Snowflake, MySQL) used for the data load. 

On April 20th 2023 we released a new Qlik Sense Button that has improved functionality to trigger an automation - see https://community.qlik.com/t5/Official-Support-Articles/How-to-build-a-write-back-solution-with-native-Qlik-Sense/ta-p/2059047

Example Qlik Sense sheet using automations for writeback use cases (source: Alberto Vaghi - Solution Architect Qlik)

 

In this article, we want to share an overview of the current building blocks for writeback use cases that are natively available in Qlik Cloud by using the power of Qlik Application Automation.

Two ways to execute an automation from a Qlik Sense Sheet

In short, we have two mechanisms to execute an automation from a Qlik Sense Sheet. 

 

1. Native 'execute automation' action in the Qlik button

In the 'Actions' menu item from the native button, the option 'Execute automations' will link a button natively to an automation. More information on triggering an automation from the native Qlik Sense button can be found here: https://community.qlik.com/t5/Official-Support-Articles/How-to-build-a-write-back-solution-with-native-Qlik-Sense/ta-p/2059047 

 

Why choose this option?	Current Limitations
Native button No limitation on selection list (because we are using the bookmark)  Easy setup Access control (only private)	No parallel run execution Run queuing limit see limitation page

 

2. Link a triggered automation from your Qlik Sense Sheet. 

By making your automation triggered, you allow any system to connect to your automation. Inputs and data can be shared through the URL as query string params (get) or the body (post).

 

Example Automation that is triggered from a Qlik Sense Sheet with 2 inputs: (opportunity)Id and (opportunity)stage

 

In a Qlik Sense Sheet, you can link a triggered automation in a button by using the 'open website or email' option in the navigation section, or by adding the link in a straight table.  Details of these two options are shared below. 

 

Why choose this option?	Current Limitations
The button can be used by everyone that has access to the sheet Multiple records can be updated at once  The user can get feedback in the opened window or get redirected to another page	Opens new window No parallel run execution Run queuing limit see limitation page No access control on who can trigger

 

 

Example 1: Open website or URL (button)

 In this example, we have a list of opportunities that can be selected and assigned a new opportunity stage.

Example Qlik Sense Sheet with a button triggering an automation using the 'open website' option

 

The website URL configuration consists of :

1. The automation trigger URL
2. The input parameters added as query string params with the link to the selected items
3. Optional additional metadata parameters to be used for tracing purposes such as user or timestamp 

 Example:

https://<tenantname>/api/v1/automations/<automation_id>/actions/execute?X-Execution-Token=<token>&stage=' & GetFieldSelections([Stage Enum]) & '&opportunity_id=' & GetFieldSelections(Id) & '&user=' & SubField(OSUser(), '=', -1)

 

Example 2: Add an action link in your straight table

The same approach can be used as in Example 1, where we now configure the trigger URL as a column for each action option. The user will only be able to update one record at a time but does not need to make any selection upfront.  

Example of a Qlik Sense Straight table that triggers in the last 3 columns an automation to compel action

 

The configuration steps for the additional columns are:

1. Add per action option (in this case 'opportunity stage') a new column in the section 'data'.
2. Configure the triggered automation URL in that specific column using the data from another column (in this case 'opportunity id'), hardcoded action option (in this case 'opportunity stage'), and optionally some metadata for tracing purposes.
3. Set representation option to 'link' and set options 'link setting' = add label, 'link label' =open.

Example:

https://<tenantname>/api/v1/automations/<automation_id>/actions/execute?X-Execution-Token=<token>&stage=Open'& '&opportunity_id=' & [Id] & '&user=' & SubField(OSUser(), '=', -1)

 

Column configuration

 

 

Example 2: Extension with input forms

There are extensions available leveraging the same underlying principles and providing the user with an easy-to-use input form. An example is a great extension built by the Qlik principle solution architect Riley MacDonald: https://github.com/rileymd88/automation-trigger

Example of input form using the extension

 

 

Additional Notes

• In many cases, the app will need to be reloaded when the source data is updated by the writeback. It is advisable not to reload the application from within a triggered automation, such as is done in the attached demo examples. A triggered automation that includes an application reload may fail when there are multiple users interacting with the application since parallel reloads of the same app are not possible.

  We suggest using a separate automation to execute the reloads. This automation should be configured to run on a schedule. By doing this, the reloads can be scheduled at intervals and will pool multiple changes from writing back in the same reload. Database connectors and some other connectors have incremental blocks which allow you to only process new or updated records. Use these blocks in the automation to only perform a reload if any new or updated records were found.

  

   

• When using the triggered automation option, the system will open a new tab or window when clicking the button. You can configure the automation to:
  • Provide feedback to the users by using the 'output' block
  • Redirect the user to a specific page such as the opportunity record in the source system by using the 'redirect' block
  • Autoclose the tab by using the redirect block to a website that autocloses

Example using the output block

Example using the redirect block

 

• Using variables to provide feedback to the user can also be used. However, there might be challenges keeping the two sessions in sync - see this article.
• It is advised to keep track of the changes in a separate database or store them with metadata such as the changelog, user, and timestamp. 
• Many Qlik Partners provide more advanced solutions for writeback. This article focuses on the out-of-the-box features supporting basic use cases. 

 

Demo application with automations

Attached you will find a Zip file containing a demo application that shows different examples of how writeback can be implemented. Besides the application and the automations you will also find the required MySQL table data in CSV format for import. 

Configuration of MySQL

1. Import table writeback_source.
2. Import table writeback_changes.

Configuration of load script sections in this application

1. "Main" - Add MySQL connection.
2. "Original Data" - Configure to read from MySQL table writeback_source.
3. "Changelog" - Configure to read from MySQL table writeback_changes.
4. "Merged Data" - Configure writeback_source & writeback_changes.

Configuration of the automation

Triggered automation - Open a website or email.

Required block configurations:

1. "Do Query" - Add MySQL connection.
2. "Update Variable" - Use "do lookup" to configure App Id (Writeback demos) & Variable Id (last_update).
3. "Do Reload" - Use "do lookup" to configure App Id (Writeback demos).
4. "Save App" - Writeback demos.

Triggered automation - Extension

Required block configurations:

1. "Do Query" - Add MySQL connection.
2. "Update Variable" - Use "do lookup" to configure App Id (Writeback demos) & Variable Id (last_update).
3. "Do Reload" - Use "do lookup" to configure App Id (Writeback demos).
4. "Save App" - Writeback demos.

Execute automation - Native button

Required block configurations:

1. "Get Straight Table Data" - Use "do lookup" to configure App Id (Writeback demos), Sheet Id (Execute automation - Native button) & Object Id (the Id of the straight table: "Multiple selections of Ids allowed").
2. "Get Straight Table Data 2" - Use "do lookup" to configure App Id (Writeback demos), Sheet Id (Execute automation - Native button) & Object Id (the Id of the straight table: "Update sales stage").
3. "Do Query" - Add MySQL connection.
4. "Update Variable" - Use "do lookup" to configure App Id (Writeback demos) & Variable Id (last_update).
5. "Do Reload" - Use "do lookup" to configure App Id (Writeback demos).
6. "Save App" - Writeback demos.

This article provides a step-by-step guide on building a write back solution with only native Qlik components and automations. 

Content:

• Environment
• Write back use cases
• 1. Ticket creation
• 2. Data annotation
• 3. Update records
• Tutorial: ticket creation
• 1. Configuring the button in the app
• 2. Configuring the automation
• Configuring the toast notification
• Reloads
• Limitations

Disclaimer for reporting use cases: this solution could produce inconsistent results in reports produced with automations; when using the button to pass through selections, the intended report composition and associated data reduction for the report may not be achieved. This is due to the fact that the session state of Qlik Application Automation cannot be transferred to the report composition definition that is passed to the Qlik Reporting Service.

Environment

• Qlik Sense app
• Qlik Application Automation
• A business tool that's used as the data source (we'll use HubSpot in this example)

Write back use cases

When analyzing results in a Qlik Sense app, it could happen you spot a mistake in your data or something that seems odd. To address this, you may want someone from your team to investigate this or you may want to update data in your source systems directly without leaving Qlik. Or maybe your data is just fine but you want to add a new record from within Qlik without having to open your business application. These scenarios fit in the following use cases:

1. Ticket creation: create a ticket to ensure a data review or update task will get followed up by your data team
2. Data annotation: add a comment or tag to one or more records in your source data 
3. Data change: add new records or update existing ones
   
   

1. Ticket creation

This is the least intrusive form of writing back that delegates the change to someone in your data team. The idea is that you create a ticket in a task management system like Jira or ServiceNow. Someone from your team will then pick up the ticket, investigate your comment, and review the data. The difference with sending an alert or email is that the ticketing system guarantees that the request is tracked.

2. Data annotation

Another option to communicate changes is to write a comment or a tag for one or more records directly to the source system. This could be a comment on a deal record in your CRM or it could be stored in a separate database table if you're loading data from a database.

3. Update records

The final use case allows for updating records directly from within the sheet. Make sure you know who has access to the button before setting this up since this will allow users to change records directly.

Tutorial: ticket creation

All the above use cases can be realized in the same way: by configuring a native Qlik Sense button in your sheet to run an automation. Before you start this tutorial, make sure you already have an app and a new, empty automation. The tutorial has 2 parts:

1. Configuring the button in the app
2. Configuring the automation

1. Configuring the button in the app

To configure the app, we'll use the following native Qlik Sense components:

• Button: Used to trigger the automation.
• Variable input: Used to allow users to specify parameters to use in the automation (for example, a variable to store a comment).
• Container: Used to group the variable inputs and the button for a better user experience.
• Table: Used to easily select records for the data annotation or ticket creation use case.

Steps:

1. Open your app and add the Container component to a sheet that already contains a Table with records. Then add (a) Button to the Container component. To configure the button, click the (b) Container and go to (c) Edit properties on the button. Feel free to give the button a proper label, in this tutorial, we'll use "Create Jira ticket". 
   

   
   

2. Configure the button under the 'Actions and navigation' tab by adding a new action and setting it to 'Execute automation' and configure the automation parameter to use the empty automation you created earlier. You can only configure an automation that you own. It is not possible to configure someone else's automation. Tip: you can use the 'View this automation' link to open the automation editor in a new tab.
   

   
   

3. Check the "Include selections" option. This way, on each click, the button will create a temporary bookmark and send its ID to the automation. The temporary bookmark contains the selections that were active when the button was clicked and also the variable state which contains any values the user has supplied for variables through the Variable Input components.
   
   

   

4. Check the "Run mode: triggered" option as well. This will allow everyone who has access to the sheet to run the automation by clicking the button. If you don't select this checkbox, the automation will be executed over an API call, and only users with permission to run the automation will be able to run it through the button. Currently, only the automation owner has this permission.
   
   
   
   

5. Enable the "Show notifications" toggle, this will send a toast notification back to the user in the sheet after the automation completes. Feel free to increase the duration.

6. Add new variables to your app. Do not use load script variables for this; instead, create the variable through the Edit Sheet page. Create the following variables, only give them a name, and you can leave the other fields blank.
   
   1. title_for_jira
   2. comment_for_jira
   3. assignee_for_jira
      
      
7. Add a Variable Input component for each variable to the Container component that already has the Button from earlier. Re-arrange the components in the Container so the Button is last. Set the 'Show as' parameter for the Variable input for the ticket title and comment to the 'Input box'. For the ticket assignee, set it to 'Drop down'.
   
   Note: you'll find the Variable Input under the 'Custom objects' category.
   
   

   

8. To configure the dropdown, you'll need the user IDs of the possible assignees for the Jira ticket. One way of getting those is to use the List Users block in the Jira connector in an automation. Once you have the IDs, configure the dropdown 'Values' parameter by adding alternative values. The Value should be the user's ID and the Label should be the user's name.
   
   

    

9. Your sheet should now look like this:
   
   

   Tip: using a Container component will allow your variable inputs & button to scale better for smaller screens. 
   
   
   
   

2. Configuring the automation

1. Click the 'Copy input block' button in the Button's configuration section. This will copy a preconfigured Inputs block and Apply Bookmark block to your clipboard to get you started with the automation. Next, click the 'View this automation' link to open the automation editor in a new tab.
   

   
   

2. Right-click in the automation canvas and select 'Paste Block(s)'. Connect the new blocks to the Start block. The Apply Bookmark block will ensure that any selections applied by the user, together with the variables state (whatever the user configured for the variables before clicking the button), are available in the automation session.
   
   

   

3. Add a List Current Selections block, and configure it to use the App parameter provided through the Inputs block. This block will return all records the user selected (because they're sent over through the temporary bookmark).
   

   
   

4. Add the Get Expression block for each variable. Configure it to use the App parameter provided through the Inputs block. Use the following format for the expression, =variablename. For example: =title_for_jira. Do the same for the comment and assignee variables.
   
   

   

5. If you haven't already, this is a good point to make a selection, configure the Variable Inputs and click the button from within the app and make sure that each block is returning the correct selections/variable values. Doing this will make it easier to configure the other blocks.
   
   
6. Add a Create Issue block from the Jira connector and configure the following parameters by using the do lookup functionality. Project, Issue Type & Priority. These will be hard-coded for each new issue.
   
   Tip: use a staging or test project in Jira while building the automation.
   
   
7. Map the expression value for the Summary parameter to the output of the Get Expression Value block that's retrieving the ticket title. Do the same for the assignee and set the Comment variable as the ticket's description.
   
   

   
   
   

8. The deal IDs should be added as a comma-separated list to the issue's Description:
   
   1. Click the description input field and select the output from the List Current Selections block. Click the (a) values list and chose the 'Select first item from list' option on the next screen:
      

      
      

      
      
      

   2. Upon automation run, this will resolve to the first text value selected for the field hs_object_id (which corresponds to the deal ID from HubSpot). To update this to a comma-separated list of IDs, the mapping must first be changed to output a list of all values for hs_object_id. To do this, toggle the formula parsing:

      

   3. Change the final 0 in the formula to an asterisk '*', and turn the formula parsing back on.
      
      
      
      
   4. Click the mapping and choose 'Add formula' from the context menu. Then, search for the Implode formula. This formula will turn a list into a comma-separated list. Configure it to use a comma as a separator:
      
      
      
      
      
      
   5. Do a new test run by selecting multiple deals in your sheet and then clicking the button, this should create a new ticket in Jira with the deal IDs as a comma-separated list.
      

      
      
      
      

9. Add an Update Run Title block to the end of the automation. This will show a toast notification to the user who clicked the button after the automation is completed. Learn more about configuring this and who can see the notification in the next part of this article: "Configuring the toast notification".
   

   
   

   The resulting toast notification in the Qlik Sense app:
   
   

   

Bonus: add a link to the toast notification

Instead of showing a plain message in the toast notification, it's also possible to include a link to point the user to a certain resource. This can be done by configuring the Update Run Title block with the following snippet:

{"message":"Ticket created", "url": "https://<link to jira ticket>"}

Configuring the toast notification

Depending on the button's configuration and the automation run mode, use either the Update Run Title block or the Output block to show the toast notification.

See the below table for each option:

The run mode in the button can be configured by toggling the 'Include Selections' option in the button's settings: 

The run mode in the automation can be configured here in the Start block:

Reloads

After writing back to your source systems, you'll want to do a reload to see your changes reflected in the app. Be mindful of the impact of doing these reloads. If multiple people are using this button at the same time, you don't want to do a reload for each update. 

Problems: 

1. Reload concurrency: the same app can not be reloaded in parallel. If multiple people are using the button at the same time, it won't be possible to do reloads for every click on the button.

Improvements:

1. Partial reloads: use partial reloads to only load new data into the app. This way the reload takes less time.
2. Separate reloads: do not use the automation triggered by the button for reloads. Instead, use a separate process for this. For example:
   1. An additional button on the app that triggers reloads.
   2. A scheduled reload.
   3. Another automation with a "List Incremental" block that runs on a schedule and reads changes from the source. It will trigger a reload only if new records are available in the source. Note: not every connector in automations has an incremental block.

Limitations

1. Public access in triggered mode: when the button is set to run the automation in triggered mode, everyone who can access the sheet and click the button will be able to execute the automation.
2. No impersonation: when the automation runs, it will use the automation owner's Qlik account for the Qlik Cloud & Qlik Reporting connectors. It's not possible to impersonate the user who clicked the button. This means that any form of section access is not applied to the user who clicked the button. 
3. No parallel runs: the automation can not run in parallel. New runs get queued with a maximum queue size of 200 runs.
4. Execution token is visible: when the button is set to run the automation in triggered mode, clicking the button exposes the automation's Execution Token from the network traffic in the developer console. This token can only be used to trigger that automation and has no use for other automations or other APIs. Generating a new token can be done by copying the automation.
5. Toast notification timeout: the button will wait on the automation for a maximum of 10 minutes to return the toast notification. If the automation takes longer, it will still run to completion, but no toast notification will be shown.
6. General limits: more information on limitations can be found here: 
   
   1. Button limits
   2. Automation limits

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

The BigQuery component encountered an error, which reads:

"BigQueryException: Connection has closed: javax.net.ssl.SSLException: Connection reset."

Cause

The error "BigQueryException: Connection has closed: javax.net.ssl.SSLException: Connection reset" typically signifies that the network connection to BigQuery was unexpectedly terminated. This could be due to various network issues, such as transient disruptions or restrictive network configurations like firewalls or proxies that terminate idle connections.

Resolution

Here are some strategies to manage and potentially resolve this issue:

Implement a Retry Mechanism:

Use a retry mechanism (tBigQueryXxx-->onSubjobError-->) with exponential backoff to handle connection resets gracefully. When a connection reset occurs, catch the exception, wait for a progressively increasing interval, and then retry the operation. This is especially useful for transient network issues.

Review Firewall and Proxy Configurations:

Ensure that all firewalls and proxy servers within your network path are properly configured to permit long-lived connections necessary for your BigQuery operations. These systems may be closing connections that remain idle for an extended period.

Batch Processing with Pagination:

Instead of attempting to load all results at once, consider breaking your query into smaller chunks. You can modify your query to retrieve subsets of the data and process each subset separately. This approach limits the impact of any single query failure.

Environment

• Talend Data Integration 7.3.1, 8.0.1

You might encounter the error "Failed to connect to server The SSL connection could not be established, see inner exception." when trying to connect to a REST API endpoint using  the Qlik REST connector in the Qlik Sense cloud services (but if you test the same connection on an on premise Qlik Sense version the connection might still work):

Please note that this kind of unexpected connection errors in the Qlik cloud might be caused due to incompatibility reason with the REST endpoint ciphers and the .NET 5.0 version in the Qlik Sense cloud. 

Accordingly we recommend to check what kind of ciphers are required for your endpoint and compare with default ciphers for .NET 5.0. In case of an incompatibility we suggest to upgrade the ciphers on the endpoint or to use the Qlik DataTransfer.

Environment

• Qlik Sense SAAS

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

The Qlik Sense Desktop or Server installation fails with:

INSTALLATION FAILED
AN ERROR HAS OCCURRED
For detailed information see the log file.
 

The installation logs (How to read the installation logs for Qlik products) will read:

Error 0x80070643: Failed to install MSI package.
Error 0x80070643: Failed to configure per-user MSI package.
Detected failing msi: DemoApps

Error 0x80070643: Failed to install MSI package.
Error 0x80070643: Failed to configure per-user MSI package.
Detected failing msi: SenseDesktop
Applied execute package: SenseDesktop, result: 0x80070643, restart: None
Error 0x80070643: Failed to execute MSI package.
ProgressTypeInstallation
Starting rollback execution of SenseDesktop

CAQuietExec:  Entering CAQuietExec in C:\WINDOWS\Installer\MSIE865.tmp, version 3.10.2103.0
CAQuietExec:  "powershell" -NoLogo -NonInteractive -InputFormat None 
CAQuietExec:  Error 0x80070002: Command failed to execute.
CAQuietExec:  Error 0x80070002: QuietExec Failed
CAQuietExec:  Error 0x80070002: Failed in ExecCommon method
CustomAction CA_ConvertToUTF8 returned actual error code 1603 (note this may not be 100% accurate if translation happened inside sandbox)
MSI (s) (74:54) [10:44:37:941]: Note: 1: 2265 2:  3: -2147287035 
MSI (s) (74:54) [10:44:37:942]: User policy value 'DisableRollback' is 0
MSI (s) (74:54) [10:44:37:942]: Machine policy value 'DisableRollback' is 0
Action ended 10:44:37: InstallFinalize. Return value 3.

The Failed to Install MSI Package can have a number of different root causes.

Install C++ redistributable

Dependencies may be missing. Install the latest C++ redistributable. 

.NET Framework update installation error: "0x80070643" or "0x643".

This issue may occur if the MSI software update registration has become corrupted, or if the .NET Framework installation on the computer has become corrupted (source: Microsoft, KB976982). 

Repair or reinstall .NET framework. 

PowerShell path not set as a system variable

How to troubleshoot:

Option 1

1. Start "Advanced System Settings" ->  "Advanced" tab -> "Environment Variables"
2. Select "Path" from "System variables" and Click Edit
3. Check if the PowerShell path is listed
4. If it is missing, add "%SYSTEMROOT%\System32\WindowsPowerShell\v1.0\" and save the change
   
   Please make sure powershell.exe can be evoked successfully from %SYSTEMROOT%\System32\WindowsPowerShell\v1.0\ before adding the path.
   
   

1. Run the batch file in the article Windows Commands To Retrieve Environment Information v2.14
2. Open the result file and find the "Environment Variables" part
   

   ********************************************        
   *********  Environment Variables  **********        
   ********************************************    
   .......
   
   Path=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Users\administrator\AppData\Local\Microsoft\WindowsApps;C:\WINDOWS\system32\inetsrv\​

3. If the path of PowerShell is missing, refer to option 1 to add into the system variables. 

Web Platform Installer Extensions required

Install the Web Platform Installer Extension that includes the latest components of the Microsoft Web Platform, including Internet Information Services (IIS), SQL Server Express, .NET Framework and Visual Studio.

More information about the tool on the Microsoft page. 

Windows Updates are pending

Verify if there are pending Windows updates. Complete them and try again.

The Installer is not hosted on a local drive

The installation may fail if the installer is being executed from a network drive. Copy the installer to your local drive.

Environment:

Qlik Sense Desktop 

The Qlik Cloud DynamoDB connector is not able to process and list tables on the Data Preview screen, when DynamoDB tables have secondary indexes added. 

Resolution

The issue is caused by an index being applied on non-existent columns or by indexes which are wrongly defined. It cannot be reproducible with correctly applied indexes on valid columns and valid column data types. 

A new Qlik Cloud DynamoDB connector will be released with an updated DynamoDB driver, which intends to fix the issue when setting up a connection between Qlik Cloud and AWS DynamoDB.

Information provided on this defect is given as is at the time of documenting. For up to date information, please review the most recent Release Notes, or contact support with the ID QB-26946 for reference.

Workaround:

Remove the index is applied on non-existent columns or indexes that are wrongly defined.

Fix Version:

Qlik team is actively working with DynamoDB and will release the new DynamoDB connector with updated DynamoDB driver.

Cause

Product Defect ID: QB-26949

Environment

• Qlik Cloud

Cause

• Improper execution of the migration scripts.
• When the destination database is not empty or freshly initialized.
• Multiple executions of the migration Scripts: - If you've run the same migration script more than once, it might attempt to create the table again.

Note: Quartz can store Job and scheduling information in a relational database and quartz can automatically create tables with initialize-schema

Resolution

Verify the list of database tables and rename or drop any tables that contain the prefix "qrtz" in their names.

Related Content 

Migrating database X to database Y

What does each table for quartz scheduler signify?

Environment

• Talend Administration Center 8.0.1

Talend Cloud platform provides computational capabilities that allow organizations to securely run data integration processes natively from cloud to cloud, on-premises to cloud, or cloud to on-premises environments.

These capabilities are powered by compute resources, commonly known as Engines. This article covers the four basic types.

Content:

• Cloud Engine (CE)
• Remote Engine (RE)
• Remote Engine Gen2 (REG2)
• Cloud Engine for Design (CE4D)
• Cloud Engine versus Remote Engine
• Cloud Engine for Design versus Remote Engine Gen 2
• Need for additional engines
• Cloud Engine - usage considerations
• Remote Engine – recommendations
• Summary

Cloud Engine (CE)

A Cloud Engine is a compute resource managed by Talend in Talend Cloud that executes Job tasks.

• You can allocate Cloud Engines to environments in proportion to the number of concurrent task executions, workloads, and Job designs you plan to run.
• All environments can use unassigned Cloud Engines. If Cloud Engines are not allocated to specific environments, you may not be able to run certain tasks because other tasks might keep all the unassigned Cloud Engines occupied.
• Cloud Engines can handle parallel execution of three tasks. That means a maximum of three different tasks can run in parallel on a single Cloud Engine. (A task cannot run more than once concurrently on a single Cloud Engine.) So, if three different tasks are already running on a Cloud Engine or if the same task is already running on that engine, another Cloud Engine is selected to execute the task.
• If you run your task in Cloud Exclusive mode, you cannot execute other tasks on that Cloud Engine. You can only use Cloud Exclusive engines in environments that do not have Cloud Engines assigned to them.
• Cloud Engines have limited system resources – memory usage: 8 GB, disk usage: 200 GB.
• Only standard Data Integration Batch Jobs can run on Cloud Engines.
• You cannot group Cloud Engines together to form clusters.
• Cloud Engines are hosted on AWS or Azure Cloud.
• Talend manages Cloud Engines.
• Cloud Engines employ TCP communication.

Remote Engine (RE)

A capability in Talend Cloud platform that allows you to securely run data integration Jobs natively from cloud to cloud, on-premises to cloud, or cloud to on-premises environments completely within your environment for enhanced performance and security, without transferring the data through the Cloud Engines in Talend Cloud platform.

Java-based runtime (similar to a Cloud Engine) to execute Talend Jobs on-premises or on another cloud platform that you control.

• Remote Engines allow you to run Jobs, Routes, and Data Service tasks.
• Data Service and Route Microservice tasks can only be deployed on Remote Engines. OSGi type deployments require that Talend Runtime version 7.1.1 or higher is installed and running on the same machine as the Talend Remote Engine.
• Remote Engines support configurable max parallel execution: by default, a maximum of three different tasks can run in parallel on the same Remote Engine. However, this is a modifiable configuration.
• Remote Engines can be grouped to form clusters called Remote Engine Cluster. Remote Engines added to a cluster cannot be used to execute tasks directly from Talend Studio.
• Remote Engines are hosted on-premises or on the cloud.
• You manage Remote Engines.
• Remote Engines employ HTTPS communication.

Remote Engine Gen2 (REG2)

A Remote Engine Gen2 is a secure execution engine on which you can safely execute data pipelines (that is, data flows designed using Talend Pipeline Designer). It allows you to have control over your execution environment and resources because you can create and configure the engine in your own environment (Virtual Private Cloud or on-premises). Previously referred to as Remote Engines for Pipelines, this engine was renamed Remote Engine Gen2 during H1/2020. It is a Docker-based runtime to execute data pipelines on-premises or on another cloud platform that you control.

A Remote Engine Gen2 ensures:

• Data processing in a safe and secure environment, because Talend never has access to your pipelines' data and resources
• Optimal performance and security by increasing the data locality instead of moving large data to computation

Cloud Engine for Design (CE4D)

Cloud Engine for Design is a built-in runner that allows you to easily design pipelines without setting up any processing engines. With this engine you can run two pipelines in parallel. For advanced processing of data, Talend recommends installing the secure Remote Engine Gen2.

• CE4Ds have limited system resources – memory usage: 8 GB
• CE4Ds support a maximum of two pipelines that can run in parallel on a single CE4D
• CE4Ds should be used only for design purposes; that is, you shouldn’t use them to execute data pipelines in a Production environment

Cloud Engine versus Remote Engine

The following table lists a comparative perspective between the two engines:

Cloud Engine for Design versus Remote Engine Gen 2

Need for additional engines

Additional engines (CE or RE) may be required if you have one or more of the following use cases:

• Continuous delivery – for example, Dev and QA separate from UAT and Production environments
• Data access - data is in two different private locations where an engine is needed in each site (or a mix of Cloud and Remote Engines)
• Scalability - concurrent Job volume requires additional engines, Jobs are complex and require significant memory or CPU

These use cases depend on the deployment architecture in the specific customer environment and layout of the Remote Engine at the environment or workspace level configurations. This would need proper capacity planning and automatic horizontal and vertical scaling of the compute Engines.

Cloud Engine - usage considerations

Remote Engine – recommendations

If a source or target system is not accessible through the internet:

If one of the systems is not accessible using the internet, then a Remote Engine is needed.

When single flow requirements exceed the capacity of a Talend Cloud Engine:

If the Cloud Engine is too small (for example, the maximum memory of 5.25 GB, temporary space of 200 GB, two vCPU, or the maximum of 225 GB per hour) then, a Remote Engine is needed.

If a native driver is required:

If the solution requires a native driver, which is not part of the Talend action or Job generated code, a typical case for this is SAP with the JCO v3 Library, MS SQL Server Windows Authentication, then a Remote Engine is needed.

Data jurisdiction, security, or compliance reasons:

It may be desirable or required to retain data in a particular region or country for data privacy reasons. The data being processed may be subject to regulations such as PCI or HIPAA, or it may be more efficient to process the data within a single data center or public cloud location. These are all valid reasons to use a Remote Engine.

Summary

References

Talend Help Center documentation:

• REG2 and CE4D Architecture

• REG2 and CE4D

• REG2

• Execution engines in Talend Cloud

When synchronizing Qlik Sense with Active Directory, you may encounter an error message saying "the User Directory Connector (UDC) is not configured, because the following error occurred: Setting up connection to LDAP root node failed. Check log file"

This often indicates a log on failure, i.e. the username and/or password is wrong.
 

Cause:

A common cause for this is wrong username and/or password.

Resolution:

1. Attempt to set up the UDC and receive the error message
2. Go to the the following location to get user management log file: %ProgramData%\Qlik\Sense\Log\Repository\Trace
3. Locate the QLIKSERVER_UserManagement_Repository_<Date/timestamp>.txt file and open it
4. Scroll to the bottom of the file
5. Read the content of the line marked by ERROR
6. If the error message is related to username and password related, move to the next steps, otherwise file a support case with us
7. Verify that the user and password actually exists, and that they are in the domain
8. In the username field, enter the domain name with a "back-slash", it should look like this domain_name\<username>
9. Enter the password normally.

The HubSection_Home resource filter in Qlik Sense refers to the button which allows a user to navigate back to the Hub from inside of an application.

Default ruleset:

Resolution:

If an administrator should want to disable this functionality for their users, for example, if the application is embedded into another page. Then they will want to disable the default rule named HubSections.

The result with this rule disabled is as follows for the end user:

The result of this change will disable this functionality for all users. If an administrator wants to provide this functionality to a select set of users then the administrator can create a new rule in this schema:

• Resource filter: HubSection_*
• Actions: Read
• Conditions: Some User Condition
  • Example: user.roles like "*Admin*"
    • This will show the Open Hub button for any user who has an Admin role assigned to their user account.
• Context: Both Hub and QMC

Using a Salesforce source endpoint, especially while using the Incremental Load source endpoint, all UPDATE operations are treated as INSERT operations for the table "UserRole". This leads to duplicate IDs found from the target table in the CDC processing stage.

Environment

• Qlik Replicate, all versions
• Salesforce source, all versions

Resolution

Set the task to UPSERT mode with Apply Conflicts set to Update the existing target record and Insert the missing target record. For more information see Apply Conflicts.

Note: this WA applied to Apply Change Mode, if the 'store changes' are enabled, duplicate ID is presented in __ct table still.

Cause

71 objects are missing the "CreatedDate" system field (including tables "AccountShare", "UserLogin", "UserRole", and similar). This is why Qlik Replicate can't identify if the change is inserted or updated, leading to both INSERT and UPDATE being converted to INSERT operation for these tables.

Internal Investigation ID(s)

00294581

A task containing tMysqlOutput component, which performs insert/update operations, has been blocked/suspended due to a PAGEIOLATCH_SH wait type status and has been pending for several hours.

Cause

Based on the SQL query check we figured out the avg_fragmentation_in_percent showing  90%+ , which means the index is maintenance badly. 

USE DBName;
GO
-- Find the average fragmentation percentage of all indexes
-- in the HumanResources.Employee table.
SELECT a.index_id, name, avg_fragmentation_in_percent
FROM sys.dm_db_index_physical_stats (DB_ID(N'DBName'),
OBJECT_ID(N'dbo.TableName'), NULL, NULL, NULL) AS a
JOIN sys.indexes AS b
ON a.object_id = b.object_id AND a.index_id = b.index_id;
GO

==Detecting Fragmentation==

The first step in deciding which defragmentation method to use is to analyze the index to determine the degree of fragmentation. By using the system function sys.dm_db_index_physical_stats, you can detect fragmentation in a specific index, all indexes on a table or indexed view, all indexes in a database, or all indexes in all databases. For partitioned indexes, sys.dm_db_index_physical_stats also provides fragmentation information for each partition.

The result set returned by the sys.dm_db_index_physical_stats function includes the following columns.

Resolution

Environment

• Talend Data Integration 7.3.1, 8.0.1

A task using ODBC as a target fails after an upgrade to 2023.5 or later versions:

Errors 00001888: 2024-09-09T10:31:56:332057 [TARGET_LOAD ]E: Failed (retcode -1) to execute statement: CREATE SCHEMA "CLILIBF" [1022502] (ar_odbc_stmt.c:5082) 

00001888: 2024-09-09T10:31:56:332057 [TARGET_LOAD ]E: RetCode: SQL_ERROR SqlState: 42000 NativeError: -552 Message: [IBM][System i Access ODBC Driver][DB2 for i5/OS]SQL0552 - Not authorized to CREATE DATABASE. [1022502] (ar_odbc_stmt.c:5090) 

Before the upgrade, the tasks may have encountered the same error but could be run using a Full Load.

Environment

• Qlik Replicate
• ODBC 
• DB2 iSERIES as target

Resolution

To resolve the issue, we will modify an Internal Parameter to the endpoints settings, as well as set a provider syntax.

1. Add the Internal Parameter:
   
   
   1. Go to the  Endpoint connection
      
      
   2. Switch to the Advanced tab
      
      
   3. Click Internal Parameters
      
      
   4. Modify the Parameter: additionalConnectionProperties 
      
      The old value should be DSN=TGT_PPCPROD
      If no parameter exists, create it.
      
      
   5. Add the new value: DSN=TGT_PPCPROD; DBQ=CLILIBF
      
      
      
      This tells the ODBC driver that any references to table names without an explicit schema name should be resolved in schema CLILIBF. 
      
      
2. Add the provider syntax
   
   
   1. Go to the  Endpoint connection
      
      
   2. Switch to the Advanced tab
      
      
   3. Locate Provider Syntax and add the value GenericWithoutSchema 
      
      
      
      This indicates to the endpoint to disregard schema information, which is the cause of the error.

Cause

Replicate Upgrade. Oracle to ODBC(iSeries) on 2023.5 will not do a Full Load. 

Internal Investigation ID(s)

QB-29117

Question I
What are the compatible operating systems?
The compatible operating systems can be found in the article: compatible-operating-systems.
Please check the section: Talend Remote Engine.

Question II
What are the compatible Java environments?
Java 8, Java 11, Java 17 can be used for task executions. By default, Talend Remote Engine uses Java 17 to run tasks. The compatible Java environments can be found in the article: launching-talend-cloud-remote-engine-for-aws-via-cloudformation.
Please check the section: Procedure ⇒ Step 10. 
Furthermore, there are no differences whether Talend Cloud Remote Engine for AWS is launched using Cloud Formation or AMI.

Question III
Where is the Remote Engine installed?
It is installed in the following directory.
/opt/talend/ipaas/remote-engine-client/

Question IV
Where can I find the settings file to make parameter changes?
The settings file is located in the following directory.
/opt/talend/ipaas/remote-engine-client/etc/

Question V
Where can I find further details on Talend Cloud Remote Engine for AWS?
Please refer to Qlik Talend Documentation: talend-cloud-remote-engine-for-aws.

Environment

Talend Cloud 

For general advice on how to troubleshoot Qlik Replicate latency issues, see Troubleshooting Qlik Replicate Latency and Performance Issues.

If your task shows latency issues, one of the first things to do is to set the logging component performance to trace and run the task you identified for five to 10 minutes and review the resulting task log.

We advise you to:

1. Open the task log in a text editor (such as Notepad++),
2. then search source latency 
3. and choose Find all in current document.

This will list all available latency information. We can now identify a trend. 

Remember, Target latency = Source latency + Handling latency.

No latency

[PERFORMANCE     ]T:  Source latency 0.00 seconds, Target latency 0.00 seconds, Handling latency 0.00 seconds (replicationtask.c:3703)

The source, target, and handling latency are all at 0 seconds.

Source latency

[PERFORMANCE     ]T:  Source latency 7634.89 seconds, Target latency 7634.89 seconds, Handling latency 0.00 seconds  (replicationtask.c:3793)

 [PERFORMANCE     ]T: Source latency 7663.00 seconds, Target latency 7663.00 seconds, Handling latency 0.00 seconds (replicationtask.c:3793)

[PERFORMANCE     ]T:  Source latency 7690.12 seconds, Target latency 7693.12 seconds, Handling latency 3.00 seconds  (replicationtask.c:3793)

[PERFORMANCE     ]T:  Source latency 7710.25 seconds, Target latency 7723.25 seconds, Handling latency 13.00 seconds  (replicationtask.c:3793)

The source latency is higher than the handling latency. The key point is to look at handling latency, it must be lower than the source latency.

Cause: 

• Source database busy
• Lack of network bandwidth to the source
• There are a lot of CDC changes to capture (the task has stopped for some time and only just resumed)
• There are a lot of CDC changes to capture (the source database has maintenance or a sudden influx of DML operations occurred)

If the source latency decreases during your monitoring, it is a good sign that the latency will recover; if it increases, review the causes mentioned above and resolve any outstanding source issues. You will want to consider reloading the task. 

Target latency

[PERFORMANCE     ]T:  Source latency 2.05 seconds, Target latency 7116.05 seconds, Handling latency 7114.00 seconds  (replicationtask.c:3793)

[PERFORMANCE     ]T:  Source latency 2.77 seconds, Target latency 7150.77 seconds, Handling latency 7148.00 seconds  (replicationtask.c:3793)

[PERFORMANCE     ]T:  Source latency 2.16 seconds, Target latency 7182.16 seconds, Handling latency 7180.00 seconds  (replicationtask.c:3793)

The target latency is higher than the source latency. 

Cause:

• Target is busy
• Lack of network bandwidth to the target 
• A bottleneck in the target database computing power, especially for cloud endpoints

If the target latency continues to increase, consider reloading the task. 

Handling latency

Identifying whether or not you are looking at handling latency or target latency can be tricky. When the task has target latency, the queue is blocked, so the handling latency will automatically be higher as well (remember: Target latency = Source latency + Handling latency).

The key point to decide if it is handling latency is to check if there are a lot of swap files saved in the sorter folder inside the task folder of the Qlik Replicate server. 

In addition, if the task log shows when the task is resumed, the handling latency increases dramatically from 0 seconds (or a low number) to a much higher value in a very short time. This can then be clearly identified as a handling latency:

2023-05-10T08:21:02:537595 [PERFORMANCE     ]T: Source latency 5.54 seconds, Target latency 5.54 seconds, Handling latency 0.00 seconds  (replicationtask.c:3788)

2023-05-10T08:21:32:610230 [PERFORMANCE     ]T: Source latency 4.61 seconds, Target latency 55363.61 seconds, Handling latency 55359.00 seconds (replicationtask.c:3788)

This log shows handling latency increased from 0 seconds to 55359 seconds after only 30 seconds of a task's runtime. This is because Qlik Replicate will read all the swap files into memory when the task is resumed. In this situation, you need to reload the task or resume the task from a timestamp or stream position.

Related Content

• Troubleshooting Qlik Replicate Latency and Performance Issues 

Environment

• Qlik Replicate

After a DB2 LUW upgrade from 11.1 to 11.5, Qlik Replicate tasks which read from the upgraded DB2 LUW Source Endpoint fail with the error:

[SOURCE_CAPTURE ]I: Error reading log buffer (db2luw_endpoint_proc.c:679)
[SOURCE_CAPTURE ]E: Error at 'Reading log records': Unexpected Error. Original SQLCODE -1263: ' message SQL1263N The archive log file "S0534570.LOG" is not a valid log file for
database "R3P" on database partition "0" and log stream "0".

Environment

• Qlik Replicate

Resolution

Perform a reload of the Qlik Replicate Task(s) and Table(s) which use the DB2 LUW Source Endpoint.

Cause

Upgrading a DB2 LUW environment changes the DB2 Transaction log format.