Loading data from Oracle may fail on a full load with the error:

ORA-01555: snapshot too old: rollback segment number string with name "string" too small

Resolution

This is an Oracle configuration issue which must be resolved for the task to be able to continue.

In Automatic Undo Management mode, increase the setting of UNDO_RETENTION. Otherwise, use larger rollback segments.

You can verify your current settings:

SHO PARAMETER UNDO;
SELECT SUM(BYTES)/1024/1024 "MB", TABLESPACE_NAME FROM DBA_FREE_SPACE GROUP BY TABLESPACE_NAME

Verify how large the problematic table is and what the current settings are. Then increase the sizes as per your findings. 

Oracle references: 

http://www.dba-oracle.com/t_ora_01555_snapshot_old.htm
https://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:275215756923
http://www.dba-oracle.com/oracle_tips_rollback_segments.htm

Cause

It caused by rollback records needed by a reader being overwritten by other writers.

Environment:

Qlik Replicate 

A log stream from SQL was being loaded into Microsoft Azure Databricks Delta Target, but the process failed.  The error occurred during a full load of a table:

00007080: 2023-02-20T19:32:08 [TARGET_LOAD ]E: Failed (retcode -1) to execute statement: 'COPY INTO `raw_iso_les`.`text_translation` FROM(SELECT cast(_c0 as INT) as `TextID`, cast(_c1 as INT) as `LanguageID`, _c2 as `Micro`, _c3 as `Short`, _c4 as `Medium`, _c5 as `Extended`, _c6 as `Text`, _c7 as `MicroAudioURL`, _c8 as `ShortAudioURL`, _c9 as `MediumAudioURL`, _c10 as `LongAudioURL`, _c11 as `TextAudioURL`, _c12 as `SmallIconURL`, _c13 as `MediumIconURL`, _c14 as `LargeIconURL`, cast(_c15 as INT) as `OrgChecksum`, cast(_c16 as INT) as `CustChecksum`, cast(_c17 as INT) as `ReferenceID`, cast(_c18 as TIMESTAMP) as `LastUpdateOn`, _c19 as `LastUpdatedBy`, cast(_c20 as TIMESTAMP) as `CreatedOn`, _c21 as `CreatedBy`, cast(_c22 as BOOLEAN) as `Active`, cast(_c23 as TIMESTAMP) as `LastDeleteOn`, _c24 as `LastDeletedBy`, cast(_c25 as TIMESTAMP) as `LastReactivateOn`, _c26 as `LastReactivatedBy`, cast(_c27 as INT) as `ArchiveID`, cast(_c28 as TIMESTAMP) as `LastArchiveOn`, _c29 as `LastArchivedBy`, cast(_c30 as TIMESTAMP) as `LastRestoreOn`, _c31 as `LastRestoredBy`, cast(_c32 as INT) as `RowVersionStamp` from 'abfss://XXXX') FILEFORMAT = CSV FILES = ('/X/X/X/X/12/X.csv.gz') FORMAT_OPTIONS('nullValue' = 'attrep_null', 'multiLine'='true') COPY_OPTIONS('force' = 'true')' [1022502] (ar_odbc_stmt.c:4906)
00007080: 2023-02-20T19:32:08 [TARGET_LOAD ]E: RetCode: SQL_ERROR SqlState: HY000 NativeError: 35 Message: [Simba][Hardy] (35) Error from server: error code: '0' error message: 'org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3805190.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3805190.0 (TID 32173534) (10.139.64.145 executor 2621): com.univocity.parsers.common.TextParsingException: java.lang.ArrayIndexOutOfBoundsException - 20480
Identified line separator characters in the parsed content. This may be the cause of the error. The line separator in your parser settings is set to '[lf]'. Parsed content: ... [1022502] (ar_odbc_stmt.c:4913)
00007080: 2023-02-20T19:32:08 [TARGET_LOAD ]E: Failed to copy data of file x\x.csv to database [1022502] (cloud_imp.c:6497)

Resolution

To resolve the issue, the user updated Qlik Replicate to the November 2022 version and reloaded the table. This will resolve the current issue and prevent the table from failing again in the future. 

Environment

Qlik Replicate 
Microsoft Azure Databricks Delta Target

To connect to Snowflake using ODBC with keypair authentication, you can use the following connection string format:

Driver={SnowflakeDSIIDriver};Server=<account_name>.<region_name>.snowflakecomputing.com;User=<username>;PrivateKey=<path_to_private_key>;Db=<database_name>;Schema=<schema_name>

Each part of the connection string represents:

• Driver: The ODBC driver name for Snowflake.
• Server: The name of your Snowflake account with the region where your account is located (e.g. us-west-2).
• User: Your Snowflake username.
• PrivateKey: The path to the private key file on your local machine.
• Db: The name of the Snowflake database you want to connect to.
• Schema: The name of the Snowflake schema you want to use.

Note that you may also need to include additional connection parameters, such as Warehouse, Role, and SSL. Here's an example connection string that includes these parameters:

Driver={SnowflakeDSIIDriver};Server=<account_name>.<region_name>.snowflakecomputing.com;User=<username>;PrivateKey=<path_to_private_key>;Db=<database_name>;Schema=<schema_name>;Warehouse=<warehouse_name>;Role=<role_name>;SSL=on

Replace the placeholders in the connection string with the appropriate values for your Snowflake account and keypair authentication setup.

Environment

Qlik Replicate 
Snowflake Target

Due a product defect, users deleted in mid-March 2023 might still keep a license assigned to them.

Environment

• Qlik Cloud 

Resolution

The issue was solved on March 21st, and deleting users correctly deallocates licenses.

Qlik is working on finding a solution for removing licenses still allocated to users that were deleted during this period for both Qlik Sense Enterprise and Qlik Sense Business subscriptions.
To speed up things, customers on Qlik Sense Enterprise can import and run the attached automation.

Qlik Replicate tasks error out, and tables are suspended.

RetCode: SQL_ERROR  SqlState: 22007 NativeError: 100035 Message: Timestamp '0000-00-00 00:00:00.000000' is not recognized Failed (retcode -1) to execute statement: 'INSERT  INTO "XXXXX_CTRL"."public_refer__ct"

Environment:

Qlik Replicate 
Snowflake Endpoint

Resolution:

'0000-00-00 00:00:00' is not a valid timestamp in Snowflake. A transformation will need to be added to change this to your desired value.

Here is an example, in this case we are replacing the timestamp to '1970-01-01 00:00:00':

coalesce(
CASE WHEN substr($AR_M_SOURCE_COLUMN_DATA,1,4) = '0000'
THEN '1970-01-01 00:00:00'
WHEN substr($AR_M_SOURCE_COLUMN_DATA,1,4) = ''
THEN '1970-01-01 00:00:00'
ELSE
$AR_M_SOURCE_COLUMN_DATA
END
,$AR_M_SOURCE_COLUMN_DATA)

Update operations to the Nvarchar(max) data types will cause the values to become 'NULL' values when the target is Microsoft Azure SQL Database.

Environment

Qlik Replicate 2021.11.0.165
Microsoft Azure SQL Database

Resolution

The behavior was resolved in Replicate 2022.11.0.475

Upgrade to Qlik Replicate sp03-2022.11.0.475 or later.

Internal Investigation ID(s) 

Case 00072614

Working with an Amazon RDS for Oracle endpoint as a source, we might encounter an issue where CDC doesn't start and fails with the following error:

Cannot create Oracle directory name 'ATTUREP_XXXXXX_XXX' with path '/XXXXX/log/arch'

Resolution

This is because of undocumented access request required which is not available in the User Guide yet. We have to execute the following grant statements to provide read only access to Qlik user to the online and archive redo logs:

grant read on directory ONLINELOG_DIR to qlik; 

grant read on directory ARCHIVELOG_DIR to qlik;

or

exec rdsadmin.rdsadmin_util.grant_sys_object('ONLINELOG_DIR','QLIK','READ');
exec rdsadmin.rdsadmin_util.grant_sys_object('ARCHIVELOG_DIR','QLIK','READ');

Where qlik/QLIK is the user specified in the source endpoint connection.

Environment

Qlik Replicate 

The Apply Exceptions table is a Qlik Replicate Control Table created on the target endpoint when the corresponding table is selected in the Control Tables tab.

You may notice that you cannot turn off the Apply Exceptions in the Qlik Replicate control table settings. The option to do so is greyed out and cannot be unchecked.  See img 01.

img 1

Change Processing errors are recorded in the attrep_apply_exceptions table. The data in this table is never deleted. For more information on the Apply Exceptions table please review attrep_apply_exceptions.

Additional tables and data take up space and resources in the database, so some people prefer to opt out of having this table for a Replicate task. This can be done with some additional configuration.

Disabling the Apply Exceptions table

1. Stop the Replicate task and then export a JSON of the task with the Export Task option under the Designer tab
   
   
   
   
2. Open the JSON exported and add the following line of information to the common_settings object and save it.
   
   "exception_table_enabled": false,
   
   You may find that your task JSON contains the following line of information. In this case, the word true can be changed to false without needing to add the information a second time.
   
   "exception_table_enabled": true,
   
   Before change:
   

   "common_settings":	{
   						"support_lobs":	false,
   						"change_table_settings":	{
   							"handle_ddl":	false,
   							"header_columns_settings":	{
   							}
   						},
   						"full_load_enabled":	false,
   						"audit_table_settings":	{
   						},
   						"dr_settings":	{
   						},
   						"statistics_table_settings":	{
   						},
   						"bidi_table_settings":	{
   						},
   						"task_uuid":	"c24fe2e3-9d04-334c-80c4-35f4b349f9e4",
   						"status_table_settings":	{
   						},
   						"status_table_enabled":	true,
   						"suspended_tables_table_settings":	{
   						},
   						"suspended_tables_table_enabled":	true,
   						"history_table_settings":	{
   						},
   						"history_table_enabled":	true,
   						"exception_table_settings":	{
   						},
   						"recovery_table_settings":	{
   						},
   						"data_batching_settings":	{
   						},
   						"data_batching_table_settings":	{
   						},
   						"log_stream_settings_depricated":	{
   						},
   						"ddl_history_table_settings":	{
   						},
   						"customized_charset_settings":	{
   							"validation":	{
   								"sub_char":	0
   							}
   						}
   					}
   				},

   
   After change:
   

   "common_settings":	{
                           "exception_table_enabled": false,
   						"support_lobs":	false,
   						"change_table_settings":	{
   							"handle_ddl":	false,
   							"header_columns_settings":	{
   							}
   						},
   						"full_load_enabled":	false,
   						"audit_table_settings":	{
   						},
   						"dr_settings":	{
   						},
   						"statistics_table_settings":	{
   						},
   						"bidi_table_settings":	{
   						},
   						"task_uuid":	"c24fe2e3-9d04-334c-80c4-35f4b349f9e4",
   						"status_table_settings":	{
   						},
   						"status_table_enabled":	true,
   						"suspended_tables_table_settings":	{
   						},
   						"suspended_tables_table_enabled":	true,
   						"history_table_settings":	{
   						},
   						"history_table_enabled":	true,
   						"exception_table_settings":	{
   						},
   						"recovery_table_settings":	{
   						},
   						"data_batching_settings":	{
   						},
   						"data_batching_table_settings":	{
   						},
   						"log_stream_settings_depricated":	{
   						},
   						"ddl_history_table_settings":	{
   						},
   						"customized_charset_settings":	{
   							"validation":	{
   								"sub_char":	0
   							}
   						}
   					}
   				},

   
   Notes: Each object within the common_settings is separated by a comma. The last object does not end with a comma. In this scenario, the extra line of information to be added is added as the first object. The order of the objects does not matter so this line of information can be placed in any order as long as the objects are separated by commas in the correct syntax format. Spacing does need to be kept in its original format.
   
   
3. Import the task JSON that was saved and resume the task. If the attrep_apply_exceptions table was already created previously, you may drop the table. The new settings in the task import will prevent the task from creating an attrep_apply_exceptions table. This is not reflected in the UI, but will continue to be reflected in the task JSON with the line of information stating  "exception_table_enabled": false, 
   

Environment

• Qlik Replicate 

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

The Qlik Replicate server service in Windows used to be named Attunity Replicate Server. 

This name has changed to Qlik Replicate Server. The change was introduced in the 2022.11 release.

If you are running a Windows Cluster and are monitoring Qlik Replicate through it, or if you have deployed third-party applications monitoring the service uptime, these applications must be modified to look for the correct service name. 

Environment

2021.11 and later

In this article, we address best practices through a questionnaire for Qlik Replicate administrators and architects before an environment goes live in production.  Having answers to these questions is helpful for a successful deployment and sustainable implementation. 

Content:

• Pre-deployment
• Qlik Professional Services
• Source
• Source Sizing
• Sizing
• Testing
• Additional Administration
• Table and data level
• Go live
• Support
• Security
• Related Content

Pre-deployment 

1. Is is possible to perform a POC? If a POC already occurred, what would you like to repeat or avoid?
2. Are the source and target endpoint versions in compliance with Replicate Support Matrix?
3. Which role is in charge of installing correct drivers and pre-requisites? DBAs, Replicate administrators, systems engineers, etc.
4. What type of replication will you be requiring (Store changes, Apply changes, both, bi-directional, full load only etc.)?
5. Although driven by your Replicate license, it is helpful to define the following:

1. Number and types of sources
2. Number and types of targets

Qlik Professional Services

1. Is there a process or budget to engage  Qlik Professional services for the following?
   1. Implementing new endpoints
   2. Performance tuning task settings

Source 

1. Is there a source DBA available to help configure source connection? Are they available for troubleshooting?
2. What are the source log retention limitations?
3. Does  Qlik Replicate have proper permissions configured to source DB?
4. Test opportunity: Work with Source DBA to confirm security policy and test permissions access to tables
5. If three or more tasks connecting to a source database, have you considered implementing Log Stream?

Source Sizing

1. Please work with Qlik Replication project owners and administrators to define values for the following: 
   1. Rate of source changes
   2. Number of source schemas
   3. Number of source tables
   4. Size of all tables (rows, data)
2. Do tables have LOB columns? If so, how many tables and what size?

Sizing 

1. Define the location of Servers (source, targets, Qlik Replicate) type (on premise, cloud, on premise virtual machine)
2. Is there hardware already procured to work with or can Qlik help size the hardware (Professional Services)?
3. What is the network bandwidth available between endpoints and Qlik Replicate?
4. Qlik Replicate disk storage size (disk requirements if changes are written to disk)
5. Is your \data folder location in a separate location from the Qlik Replicate install folder? 
6. If using log stream, is your storage path on a separate drive from your /data directory?  This can help with disk contention by having separate drives. 
7. What is your acceptable latency requirement (time from source commit to target commit)?

Testing

1. Define number of environments DEV, QA/TEST, PROD
2. Do you have at least 1 non-production environment that mirrors production for data volume, data types, and processes?
3. Define a testing plan, scripts, queries for:
   
   1. Validating row counts between source and target
   2. Validating any filters or transformations
   3. Validating failover if a HA/failover environment
   4. Validating query times on source and target. If needed, reduce the batch size or increase timeout parameters.
   5. Validating row size limits 
   6. Practicing a Advanced Run Options -> Restart from time or SCN/LSN (and when to use these options)
   7. Adding or removing a table from a task
   8. Apply Conflicts settings such as UPSERT mode 
4. How do we handle maintenance or upgrades on Source DB and target DB? How often do we expect these to occur?

Additional Administration 

1. How many users will be engaging with Qlik Replicate at one time? Who is responsible for managing adding, removing or configuring users?
2. Are notifications configured on server or task errors? For Disk/CPU limits? What emails are included to receive notifications?
3. Is there a naming convention established for sources, targets, and tasks?
4. Is there a change control strategy in place? Qlik Enterprise Manager has audit trail capabilities if also installed with Qlik Replicate environment. 

Table and data level

1. Do tables have PKs? Are PKs changing on any tables? Tables with a PK which is not changing will have the best performance with bulk apply mode. 
2. For change data capture: are you using Transactional apply vs Bulk apply?

Go live

1. Do we bring all task online at once or can they be started sequentially or in batches?

Support

1. How long will it take to perform a full reload on each task?
2. Are there any task/tables where a Full load is not an acceptable option?
3. Do you know how to increase Task and Server logging and/or download diagnostic package?
4. Do we have a team who can manage 24 hour Severity 1 Support?
5. What is process for applying Qlik Replicate service packs with fixes? (i.e Dev then prod, or direct on Prod, how quickly can these fixes be applied?)
6. Are you familiar with decrypting log files?
7. Does your team have access to the Case Portal? 

Security

1. Have you tested access to the system with set password? 
2. Has openSSL been set up and configured for Qlik Replicate UI Console and Server?

Related Content

How To Get Started with Qlik Replicate

Environment

Qlik Replicate 

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

This Techspert Talks session addresses:

• Monitoring multi-node activity
• Centralized metrics collection
• Demo of available tools

• 00:00 – Intro
• 01:19 - Overview of multi-node architecture
• 02:04 - Butler SOS architecture
• 03:24 - Grafana Loki architecture
• 05:20 - Butler SOS demo
• 11:32 - Loki with Grafana demo
• 16:30 - Selenium IDS tool
• 17:53 - New Relic Demo
• 19:49 - Q&A: What about data model and app optimization?
• 20:16 - Q&A: What security issues are there?
• 23:04 - Q&A: What metrics should we be monitoring?
• 24:06 - Q&A: Can this be done with the Operations Monitor app?
• 25:03 - Q&A: How to set this up?
• 25:51 - Q&A: Are these tools free and publicly available?
• 26:26 - Q&A: Will these tool work with QlikView?
• 26:50 - Q&A: Can you measure App response times?

Resources:

Optimizing Performance for Qlik Sense Enterprise

Qlik Sense Enterprise deployment examples

Butler SOS

Setting up Butler SOS

Grafana Loki | Grafana Loki documentation

Docker + Compose

Portainer for Docker management

Selenium IDE for user session replay

Webmin for server admin

Click here to see video transcript

When using Qlik Cloud with a custom IdP and a user is removed from the IdP, will that change be propagated into Qlik Cloud? 

Resolution

The answer to the question depends on the IdP used and the if the provisioning has been enabled or not.

If the current IdP used does not support SCIM then the changes done on your IdP side will not be propagated into Qlik Cloud. So, the user will not be flagged as inactive or removed on your tenant.

If you are using Azure AD* and the provisioning has been enabled, then the changes will be propagated as explained in Provisioning users and groups using SCIM | Qlik Cloud Help.

SCIM provides a standardized way for IT systems to communicate and exchange user identity data. With SCIM, you can:

• Create a new user in one system and automatically provision the same user in another system.

• Update user information in one system and reflect that change in all other systems.

• Delete a user in one system and automatically deprovision the user in all other systems.

By standardizing the user management process, SCIM makes managing users across different IT systems easier and reduces the likelihood of errors or discrepancies in user information.

Related Content 

Provisioning users and groups using SCIM - Qlik | Help
SCIM: System for Cross-domain Identity Management (simplecloud.info)
SCIM synchronization with Azure Active Directory - Microsoft Entra | Microsoft Learn

*SCIM connectors that work with Qlik Cloud (as of the date of creation of this article - 15th March 2023) :

1. Qlik Cloud supports Microsoft Azure Active Directory (Azure AD).

Environment

Qlik Cloud 

A newly designed Replication task (with default settings) reported errors in the first run:

2023-03-15T21:02:28 [STREAM_COMPONEN ]E: Replication task cannot support lobs, since log stream staging task does not [1020480] (ar_cdc_channel.c:261)
2023-03-15T21:02:28 [METADATA_MANAGE ]E: Allocating utility stream component failed [1020480] (metadatamanager.c:837)
2023-03-15T21:02:28 [METADATA_MANAGE ]E: Cannot create the source utility component [1020480] (metadatamanager.c:753)
2023-03-15T21:02:28 [TASK_MANAGER ]E: Creating Metadata Manager's utility components failed [1020480] (replicationtask.c:3893)

In general this error comes with "Replicate LOB columns" is deselected in Log Stream Staging task :

Workaround:

1. Enable "Replicate LOB columns" in Log Stream Staging task if LOB columns are needed, or see (2), (3) below.
2. If no need to replicate LOB columns, or there are no LOB columns in the table(s) selected, then Edit Replication task and SAVE (any change is good enough, eg change Task setting --> Logging --> PERFORMANCE from INFO to Trace, SAVE, and change it back to INFO again). This operation will add the missing components implicitly.
   
   
   
3. Or export/import the  Replication task by adding the missing property explicitly:
   
   "common_settings": {
   "support_lobs": false,
   "change_table_settings": {
   
   

Internal Investigation ID(s):

RECOB-6036, #00075812, #00054094

Environment:

Replicate 2022.5   (eg PR02, PR03, PR04)
Replicate 2022.11 (eg PR03)

 When Data mart has referenced dimensions with one or more local dimensions, generating instructions for DataMart fails with the below error message:

SYS,GENERAL_EXCEPTION,Object reference not set to an instance of an object

Information provided on this defect is given as is at the time of documenting. For up-to-date information, please review the most recent Release Notes with RECOB-6639 for reference

Environment

• Qlik Compose 2022.5.0.403

Fix Version:

2022.05-SP08 (build 2022.5.583)

Cause

Jira issue: RECOB-6639

The Qlik Compose will result in the below error when deleting a workflow or task that was created using a non-printable character or one of these following characters: /\,&"#$@=^*+'`~?|<>:;[]{} as well as percentage sign.

REPO-E-OBJNOTFND, Object not found. name <TASK NAME>, type FlowDto.

Cause

Error handling in Qlik Compose version 2021.8.0.569 is different to how it is handled in version 2021.8.0.734 and higher.  

In version 2021.8.0.569, attempting to create a workflow with a special character, Qlik Compose will create a workflow but with the below message. As the message says workflow can not be edited or deleted once created.

"REPO-E-OBJNOTFND, Object not found" and workflow will be created, the workflow cannot be edited or deleted

However, in version 2021.8.0.734 and higher, attempting to create a workflow with a special character will fail with the below error message and the workflow will NOT be created:

"COMPOSE-E-TASKINV, Invalid task name “Workflow/1”. Task names cannot contain the following characters: /\,&"#$@=^*+'`~?|<>:;[]{} as well as percentage sign and all non-printable characters (below 0x20)"

So, the workflows that were created in version 2021.8.0.569 with a special character will never be able to delete from the UI even after upgrading to later versions. The fix for this issue would be to delete the special character in the SQLite file manually. See below for the workaround.

 Workaround

1. Back up the current project SQLite file.
2. Open the project’s SQLite file in the DBbrowser and rename the workflow by replacing the unsupported characters with supported characters in both the NAME & JASON columns.
3. Save the changes and restart the Qlik Compose service.
4. Check the workflow in UI, and now it shouldn’t present the above error when deleting the workflow.

Related Content

For more information on the unsupported characters please check this link: Qlik User guide.

Environment

Qlik Compose 2021.8.0.569

From Qlik Sense February 2023 onwards, apps listed in the Qlik Sense Management Console (Apps view) are now represented with clickable links. Clicking them will open the app directly on the hub.

This means the app names are no longer plain grey text but are now formatted as links (blue, underline). See Apps for details.

Example: 

Environment

Qlik Sense Enterprise on Windows February 2023 and above

External program tasks in Qlik Sense are simply another task type. With external program tasks, you can trigger external processes, such as scripts or .exe files. Task chaining is supported and you can combine reload tasks with external program tasks.

The functionality is exposed in the Qlik Sense Management Console beginning with the May 2021 release. See Creating and editing external program tasks.

Previous versions allow you to create external tasks in a command shell process behind the scenes; which in turns means you can run more or less anything you can run on the command prompt. Once you define your command and create a new task, you can both trigger and chain these types of tasks in the QMC.

In this example, we will be using a Powershell Script which means that powershell.exe needs to be in your PATH variable (By default C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe). Again, you can run any command that you can run in the windows command prompt, so these examples are only for demo purposes.

Before we get started, we need to understand that IF you are calling scripts in your command, the script needs to be accessible by the user running the Qlik services. In this example, there’s a folder on the root C: drive called ‘externalTasksExample’, in this folder, there is a script.ps1 file which simply creates a new file in the same folder.

The file looks like:

Start-Transcript C:\externalTasksExample\transcript.log
$date = Get-Date
Write-Host $date
Stop-Transcript

You can run the file from the command prompt by doing the following:

C:\externalTasksExample>C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -File script.ps1

When the command runs successfully, you should have a new file created in the externalTasksExample folder called “transcript.log”.

Now we will delete this transcript.log and try to run that same PowerShell file using an external program task in Qlik Sense. The first thing we need to do is actually create the task.

The endpoint is:

POST /qrs/externalprogramtask

With the body being:

{
  "path": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
  "parameters": "C:\\externalTasksExample\\Script.ps1",
  "name": "Task Name",
  "taskType": 1,
  "enabled": true,
  "taskSessionTimeout": 1440,
  "maxRetries": 0,
  "privileges": null,
  "schemaPath": "ExternalProgramTask"
}

Here is a sample script:

$hdrs = @{}
$hdrs.Add("X-Qlik-xrfkey","12345678qwertyui")
$hdrs.Add("X-Qlik-User","UserDirectory=DOMAIN;UserId=Administrator")
$hdrs.Add("content-type","application/json; charset=UTF-8")
$body = '{
  "path": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
  "parameters": "C:\\externalTasksExample\\Script.ps1",
  "name": "Task Name",
  "taskType": 1,
  "enabled": true,
  "taskSessionTimeout": 1440,
  "maxRetries": 0,
  "privileges": null,
  "schemaPath": "ExternalProgramTask"
}'

$cert = Get-ChildItem -Path "Cert:\CurrentUser\My" | Where {$_.Subject -like '*QlikClient*'}
$url = "https://qlikserver1.domain.local:4242/qrs/externalprogramtask?xrfkey=12345678qwertyui"
Invoke-RestMethod -Uri $url -Method Post -Headers $hdrs -Body $body -Certificate $cert

This generates a response like:

id                 : 1353eb7c-44c7-422b-9e73-c2667011a1b0
createdDate        : 2022-06-13T19:35:40.382Z
modifiedDate       : 2022-06-13T19:35:40.382Z
modifiedByUserName : DOMAIN\administrator
customProperties   : {}
path               : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
parameters         : C:\externalTasksExample\Script.ps1
qlikUser           : 
operational        : @{id=5ececc2a-dc6c-4769-b424-c325faddf74c; lastExecutionResult=; nextExecution=1753-01-01T00:00:00.000Z; privileges=}
name               : Task Name
taskType           : 1
enabled            : True
taskSessionTimeout : 1440
maxRetries         : 0
tags               : {}
privileges         : 
schemaPath         : ExternalProgramTask

Once you post this into Qlik, you should be able to go to the QMC and see the following task in the task list.

You should be able to select the task, click 'Start', and the same file should appear in 'C:\externalTasksExample'. You are now running an external task. Remember you don't have to use powershell.exe, you can trigger anything you want as long as it can be run from the command line.

Related Content:

How to configure Postman (desktop app) to connect to Qlik Sense 
 

An automation will not automatically rerun or retry if it fails. You can, however, rerun a failed automation by using an additional automation. 

Caution should be taken when implementing this solution to prevent running endless loops or reruns. 

• Considerations
• Retry Automation Run
• Run Automation
• Rerun example
• Full Automation:
• Environment

Considerations

Before a solution can be implemented you must decide if the rerun should:

1. Retry the failed automation run using the same, if any, input.
2. Initiate an entirely new run of the automation.

Retry Automation Run

The block Retry Automation Run can be used to retry a specific run of the automation using the same, if any, inputs. This block is useful if the monitored automation has its run mode set to Webhook or Triggered. It is also possible to manually re-run these runs using the Retry button in the automations run history.

Run Automation

The block Run Automation can be used to initiate a new run of the automation. This block is useful if the automation that fails has the run mode set to Scheduled, and the inputs of the failed run wouldn’t be unique.

Rerun example

The following example shows how a failed automation can be retried using the Retry Automation Run. Depending on the specifics of the automation you want to rerun, you may want to use the Run Automation block instead.

1. Use the block List My Automations to get all of your automations.
2. Then use the block List Automation Runs to only get the runs that have failed by filtering on status:
   
   
   
   
3. Use the block Transform List to create a list of failed runs. Add the two fields id and stopTime to this list and configure them such as this:
   
   
   
   
   
   

   The value for stopTime must be transformed using the Date formula so that the value may be used for comparisons. The output format must be changed to Unix format (U):

   
   
   
4. Now use the block Filter List to only get the most recently failed runs.
   
   
   
   The condition that controls how recent failed runs you would like to retry can be configured according to your needs by modifying the "Date" formula. Ensure that you configure the output format to be in Unix format (U). In this case runs that has failed within 30 minutes will be rerun:
   
   
   
   
5. The final step is to include one of the blocks Retry Automation Run or Run Automation, depending on what you are trying to achieve. This is an example of how the Retry Automation Run should be configured:
   
   
   
   In case the Run Automation block is used, only the "Automation Id" is needed as input.

Full Automation:

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

Environment

Qlik Application Automation