Environment

Qlik Enterprise Manager 2022.5.0.402
Microsoft SQL Server (MS-CDC) as a source endpoint

The Windows authentication and Database name fields are missing in Enterprise Manager when attempting to create the Microsoft SQL Server (MS-CDC) as a source endpoint. You will not be able to create the endpoint from this version of Enterprise Manager.

Workaround

Create the Microsoft SQL Server (MS-CDC) endpoint from the Qlik Replicate console

Fix Version

To be fixed in Qlik Enterprise Manager version 2023.5.

Internal Investigation ID(s) 

RECOB-6602, RECOB-6154

If a Qlik Sense Desktop installation fails due to a software restriction policy, install it as a portable installation instead.

Remember to get authorization or approval from your organization's administration teams where necessary prior to installing software if the default installation failed due to policy restrictions.

1. Installing and Moving the Qlik Sense Desktop Installation:
   1. Install Qlik Sense Desktop on an unrestricted client machine
   2. Go to the Qlik folder location, by default %AppData%\Local\Programs
   3. Zip up the Qlik folder to create Qlik.zip
   4. Move the Qlik.zip file to the machine where you want to use use Qlik Sense Desktop but the installation is restricted
   5. On the restricted computer, Unzip Qlik.zip to your desired location
      1. Examples: C:\Users\<user name>\Software or %AppData%\Local\Programs, etc.
   6. After unzipping, go to the location, into the Qlik\Sense folder, and find QlikSense.exe to open Qlik Sense Desktop after configuring the Authentication Link
      • Optional, right click on QlikSense.exe, and 'Send to' Desktop (create shortcut) if you would like to create a shortcut on the desktop to resemble a standard installation
2. Using the Authentication Link in Qlik Sense Desktop portable installation via hubs.ini file:
   1. Generate an Authentication Link in the QMC via QMC > Virtual proxies > Edit > Client authentication link
   2. Enter the desired host URI and the friendly name and select Generate as shown in the Qlik Sense Desktop Help
   3. Copy the Authentication link
   4. Create a new text file called hubs.ini
   5. Open the file in a text editor such as Notepad
   6. Paste the Authentication link into the file, save and close
   7. The output should look similar to the following:
      
      
      
      
   8. Move, copy, or save this file in the Documents location
      • example: C:\Users\<user name>\Documents\Qlik\Sense\Hubs\
      • Note that you will need to manually create the Hubs folder if it does not exist
   9. Launch Qlik Sense Desktop via the QlikSense.exe in step 1.6, or the shortcut you created in the optional step
   10. Select the button with the friendly name for the Authentication link and sign in using your credentials

Environment

• Qlik Sense Desktop

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

Related Content 

Qlik Sense Desktop Help

A Replicate task is entering a stop state on its own once every few hours. When the task is resumed manually, the same behavior is noticed in the next few hours. The log has no information on why the task has stopped.

Environment

• Qlik Replicate version 2022.5
• Kafka target endpoint

Resolution

Issue was due to an application defect which is fixed in version 2022.5.0.652 (SP3).

Component/Process: Kafka Target

Description: In rare scenarios, the task would end abnormally.

An environment may experience failing tasks after cloud disconnects or related communication issues. Reviewing the Qlik Replicate logs shows the error found more than one matching file for pattern even if no files with those names are found in the directory.  

 Found more than one matching file for pattern ^CDC00000008.tmp(#[0-9]+)?$ [1022406]  (filestream.c:640)
00027860: 2022-11-04T20:30:01 [TARGET_APPLY    ]E:  Failed to init Target File producer. [1022406]  (file_imp.c:4794)
00027860: 2022-11-04T20:30:01 [TARGET_APPLY    ]E:  Could not create CSV writer on table id: 201. [1022406]  (file_apply.c:218)
00027860: 2022-11-04T20:30:01 [TARGET_APPLY    ]E:  create and insert file writer failed. [1022406]  (file_apply.c:229)
00027860: 2022-11-04T20:30:01 [TARGET_APPLY    ]E:  Could not insert and create file writer for table ID: 201. [1022406]  (file_apply.c:571)
00027860: 2022-11-04T20:30:01 [TARGET_APPLY    ]E:  Failed to create file writers for on directory: /attadhdata/data/attunity/AWS_PROD_OSO_PART1. [1022406]  (file_apply.c:636)
00027860: 2022-11-04T20:30:01 [TARGET_APPLY    ]E:  Could not init data file writers. [1022406]  (file_apply.c:688)
00027860: 2022-11-04T20:30:01 [TARGET_APPLY    ]E:  Error executing command [1022406]  (streamcomponent.c:2024)
00027860: 2022-11-04T20:30:01 [TASK_MANAGER    ]E:  Stream component failed at subtask 0, component st_0_TGT_PROD_OSO_PART1  [1022406]  (subtask.c:1396)
00027860: 2022-11-04T20:30:01 [TARGET_APPLY    ]E:  Stream component 'st_0_TGT_PROD_OSO_PART1' terminated [1022406]  (subtask.c:1565)
00027857: 2022-11-04T20:30:01 [TASK_MANAGER    ]W:  Task 'PT_AWS_OSO_PART1' encountered a fatal error  (repository.c:5794)

Resolution

Start the task from timestamp to resolve the issue.

Patch to version 2021.5 to resolve the issue permanetly. 

Environment

Qlik Replicate version 2021.5
File target

A query timeout expired error is displayed.

Example query run that resulted in the error:

select bmf.physical_device_name, bs.position, [dbo].[attrep_fn_NumericLsnToHexa](bs.first_lsn), [dbo].[attrep_fn_NumericLsnToHexa](bs.last_lsn), bs.backup_set_id from msdb.dbo.backupmediafamily bmf, msdb.dbo.backupset bs where bmf.media_set_id = bs.media_set_id and bs.backup_set_id > 0 and bs.database_name=db_name() and bs.type='L' and ( cast('00042ead:00024eb8:0001' collate SQL_Latin1_General_CP1_CI_AS as varchar(24)) >= cast([dbo].[attrep_fn_NumericLsnToHexa](bs.first_lsn) collate SQL_Latin1_General_CP1_CI_AS as varchar(24)) and cast('00042ead:00024eb8:0001' collate SQL_Latin1_General_CP1_CI_AS as varchar(24)) < cast([dbo].[attrep_fn_NumericLsnToHexa](bs.last_lsn) collate SQL_Latin1_General_CP1_CI_AS as varchar(24)) ) and bmf.device_type in(2, 102, 0

Environment

• Qlik Replicate 
• SQL Server Source Endpoint

Resolution

This timeout can happen when at least one of the tables msdb.dbo.backupmediafamily and msdb.dbo.backupset has grown in size and delays the query that locates backup files to load. You can purge backup data that is above your retention by using sp_delete_backuphistory with @oldest_date set with a value that should be enough for your retention policy to work fine.

We have data that is not being replicated from DB2 LUW DPF (database partitioning facility). Data during a time frame will be replicated for some update jobs but not other.  

Environment

• Qlik Replicate 2021.11 (probably affects other versions too)
• DB2 LUW Source with DPF in use

Resolution

Root cause is due to a documented limitation on adding partitions. Found a viable work around: Stop all the tasks, add the partition, start all the tasks from timestamp just before the tasks were stopped. A feature request will be submitted to see if we can better handle this use case in the software.

Related Content

Limitations and considerations #Limitations and considerations | Qlik Replicate Help

Internal Investigation ID(s)

7449

Issue:

May 2022 (2022.5.0.291) version needs to be updated because of Java SE vulnerability.    Plugin Output: Path              : /opt/attunity/replicate/jvm/  Installed version : 11.0.14 Fixed version     : Upgrade to version 11.0.16 or greater CVE-2022-21426,CVE-2022-21434,CVE-2022-21443,CVE-2022-21449,CVE-2022-21476,CVE-2022-21496 CVE-2022-21540,CVE-2022-21541,CVE-2022-21549,CVE-2022-25647,CVE-2022-34169

Environment

• Qlik Replicate version May 2022
• Linux

Resolution

Qlik Data Integration products use JVM version 11 for QEM/Replicate and also JVM version 8 for Compose. This issue does not apply. You can upgrade Java SE to 11.0.17 for the Qlik products/versions you are using.

Internal Investigation ID(s) 

7345

Our Qlik Replicate instances are non-PCI compliant due to weak SSL ciphers on ports 443, 3389 and 3552. Ports 443 and 3552 are used by Qlik. Port 3389 is RDP.

Environment

• Qlik Replicate 
• Windows

Resolution

Qlik cannot offer advice on how to configure Windows to disable certain ciphers which customer security guidelines forbid. Customer security staff folks should be able to apply their policy on any Windows system, whether it is managed by AD or not (see for example https://learn.microsoft.com/en-us/windows-server/security/tls/tls-registry-settings). Lucky 13 and Sweet 32 are the versions which are not compliant for port 3552. Luck-13 (https://crashtest-security.com/prevent-ssl-lucky13/) and Sweet-32 (https://crashtest-security.com/prevent-ssl-sweet32/) are not versions, nor are they ciphers that Replicate uses on port 3552. Those are old vulnerabilities (5Y+) that are either mitigated in the version of OpenSSL currently used in Replicate (with its cipher selection) or is otherwise impractical or irrelevant in the way Replicate works. In short, there is no security issue here from a Replicate perspective.

Task error: JAVA_EXCEPTION, message: 'io.swagger.client.ApiException: Operations per second is over the account limit.'

Environment

• Qlik Replicate 
• Databricks Target

Resolution

This is a Databricks issue with the storage account for this Replicate user. Please check with your Databricks Admin to confirm the limits on data per second for the storage account for the Replicate user.

Environment

• Qlik Replicate 2022.5
• DB2 iSeries Source

Resolution

Issue was resolved after installing the Qlik UDTF

When using the Snowflake ODBC driver as a source, the source endpoint is returning columns from multiple tables of the same name from different databases/schemas.

Environment

• Qlik Replicate version 2022.5 (possibly other versions too)
• Snowflake ODBC Source

Resolution

Environment

• Qlik Cloud 
• Qlik Sense Business 

This video will demonstrate how to install and configure Qlik-CLI for SaaS editions of Qlik Sense.

Transcript

Installation method 1:

1. Download Qlik-CLI from GitHub
2. Copy the executable to a local directory
3. Add the Qlik-CLI as an Environment Variable Path. 
   1. Open your Windows Control Panel
   2. Navigate to System 
   3. Open Advanced system settings in the leftmost menu
   4. Click Environment Variables 
   5. Locate Path in the System variables 
   6. Click Edit
   7. Click New
   8. Add the path to Qlik-CLI for example C:\Tools\Qlik-QLI\
   9. Confirm with OK until all windows are closed
4. Confirm the Qlik-CLI location path to verify it the installation completed successfully 
   1. Open PowerShell
   2. Execute:
      

      get-command qlik​

   3. Closer PowerShell

Installation Method 2:

1. Download and install Chocolatey as documented on chocolatey.org.
2. Open PowerShell
3. Execute:
   

   choco install qlik-cli --source=https://www.nuget.org/api/v2​ --version 2.1.0

   Where --version 2.1.0 needs to be replaced by the latest package version. Locate the latest version on Nuget.org. You can also download them from there directly. 

4. This completes the installation.

Enable the Completion Feature (optional, but useful)

1. Verify if a PowerShell profile exists or create a new one
   1. Open PowerShell
   2. Execute:
      

      if ( -not (Test-Path $PROFILE) ) {
         echo "" > $PROFILE
      }​

2. Open the profile with the command notepad $PROFILE:
   

   qlik completion ps > "./qlik_completion.ps1" # Create a file containing the powershell completion.
   . ./qlik_completion.ps1                      # Source the completion.​

3. Save it. 
4. Restart PowerShell

Use Qlik-CLI

Advanced and additional instructions as seen in the video can be found at Qlik-CLI on Qlik.Dev. Begin with Get Started.

Changing QlikView to a new serial number license requires you to use Apply License in the Management Console, rather than Update License from Server. 

Environment

QlikView using a LEF and Serial Number

Resolution

To change the license, you must use Apply License.

It is recommended to backup all license related files on the QlikView server prior to updating license details. This allows for reverting to the previous license if the update causes any problems. QlikView license information is stored in two locations. Files from both locations, as specified below, must be copied to a safe location prior to updating or applying a new license.

1. Root document folder. The location of this folder is defined in the QMC > System > Setup > QlikView Server > Folders tab > Root folder.
• BorrowedCalDat.pgo
• CalData.pgo
• IniData.pgo
• ServerCounters.pgo
• TicketData.pgo
2. QlikViewServer application folder. Default path: C:\ProgramData\QlikTech\QlikViewServer 
• CalData.pgo
• IniData.pgo
• ServerCounters.pgo
• BorrowedCalData.pgo

Cause

When clicking update from server, QlikView will update the current license and not replace the old one.

Related Content

How to License a QlikView Server or Update the License  

Loading files larger than 50MB from either Dropbox or SharePoint with their respective connectors leads to: 

Data file size exceeded file streaming buffer size. Adjust the StreamingBufferSizeMB setting.

Environment

Qlik Sense Enterprise on Windows 
Qlik Cloud 

Resolution

Qlik Sense Enterprise on Windows (client-managed):

The solution requires us to modify the Qlik Sense Engine settings.ini. See How to modify Qlik Sense Engine settings.ini for details on the procedure.

1. Stop the Engine Server
2. Go to C:\ProgramData\Qlik\Sense\Engine
3. Create a backup of the Settings.ini 
4. Open the Settings.ini file
5. Add StreamingBufferSizeMB=1024 line  (higher values are not currently supported by connectors)
6. Add an empty line after the above parameter if the parameter is the last one to be added
7. Save
8. Restart the Engine Service

Qlik Cloud

The value cannot be configured in Qlik Cloud. A workaround is to split the big data files by e.g. months.

Internal Investigation ID(s) 

QLIK-83207
CP-4120
QLIK-82097

In this article, we will explain how to set up one automation as a source and keep multiple target automations in other tenants synced with the changes made to the original. We will be using an intermediary tool for this process, in our case Github, to store the changes made to the source automation so you could also say this article doubles as a version control article for your future automations.

Setting up the source automation and syncing changes to Github

To get the initial automation data inside a Github repository, all we need to do is construct a quick automation that contains the "Get Automation" block and run it in order to receive the much-needed data in the block history. We can then copy that data and paste it all into a file inside our working repository. For example purposes, our working file is called 'automation.json' and is situated in the main directory of the demo repository.

Now that our initial set-up is complete, we will list the steps needed to keep our repository up to date with all the changes to our automation. This is how the workflow will look:

You can see that our initial start block is replaced with a Qlik Cloud Services webhook that listens to all automations being updated in our tenant. We have also created a variable to keep the Github branch name in memory and keep it as a version number for the cases where it might be needed:

The formula used takes the date at which the automation was modified, transforming it to a UTC format, and attachs it to the branch name: version-{date: {$.AutomationUpdated.data.updatedAt}, 'U'}

We also need to attach a condition block to our flow to make sure only our source automation gets sent to our repository:

In this case, we are hardcoding the id of our automation in the condition input parameter. Next, on the 'yes' branch of the condition, comes the 'Get automation' block that will return the latest version of our source automation, followed by the flow that sends that data to a new branch in our Github repository. First, we create a new branch:

Secondly, we update the automation.json file we had in our main repository with the new data we received from the 'Get automation' block:

As you can see, Github platform only accepts BASE64 encoded file contents, so the JSON content received from 'Get automation' will be transformed to that format. To create the commit and pull request, we have one more step, which is to find out the branch TREE SHA information:

With this information, we can go ahead and create the commit:

The final step in our flow is to create the pull request :

Keeping your target automation up to date with the latest version

To do this, we now move to our target tenant and create a separate automation there. The workflow involved is a simple one:

Again, the start block will get replaced by the Github webhook that is listening to all new pull requests done in our repository:

We also create an empty object variable that will be later used to save the incoming information from the repository. We need to create a condition block as well with the following two conditions:

This lets the automation flow continue only if the pull request has been closed and its contents have been merged into the main branch of our repository. It tells us that the pull request has been approved and we can sync all changes into our target automations. For that, we query the file contents of the repository and save that information in our object variable after we have transformed it from a BASE64 encoding format to a text-based format:

The formula used when assigning the value to the variable is {object: '{base64decode: {$.getFileContent}}'} which just removes the base64 formatting and turns the string into a JSON object to better handle it in the next block. Now all that is left is to use the 'Update Automation' block to bring the new changes to our target automation:

As you can see, we hardcoded the GUID of our target automation, but the do lookup functionality can be used as well to select one of the automations in our tenant. Finally, we need to send the correct parameters from our object to the input parameters of the block. This can easily be done by selecting the Name, Description, Workspace and Schedule parts of the object.

This should let you keep your automations synced over multiple tenants. Your target automation history should not be impacted by the syncs in any way.

Please be aware that if the target automation is open during the sync process, the changes to it will not be reflected in real-time and a reopening of the automation will be needed in order to see them.

Attached to this article you can also find the JSON files that contain the automation flows presented.

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

Related Content

• https://community.qlik.com/t5/Knowledge/Qlik-Application-Automation-How-to-use-the-create-update/ta-p/1978622

Executing tasks or modifying tasks (changing owner, renaming an app) in the Qlik Sense Management Console and refreshing the page does not update the correct task status. Issue affects Content Admin and Deployment Admin roles.

The behavior began after an upgrade of Qlik Sense Enterprise on Windows. 

Should the issue persist after applying the workaround/fix, contact Qlik Support.

Environment

• Qlik Sense Enterprise on Windows 

Fix version:

This issue can be mitigated beginning with August 2021

• Enable QMCCachingSupport Security Rule

Workaround for earlier versions:

Upgrade to the latest Service Release and disable the caching functionality: 

To do so:

1. Navigate to C:\Program Files\Qlik\Sense\CapabilityService\
2. Locate and open the capabilities.json
3. Modify the flag for these values to false
   • QmcCacheEnabled
   • QmcDirtyChecking
   • QmcExtendedCaching
4. Restart the Qlik Sense services
   
   

NOTE: Make sure to use lower case when setting values to true or false as capabilities.json file is case sensitive.

Internal Investigation ID(s): 

QB-2096
QB-5168
QB-7655

This is part 3 of 4 in a series of articles with information about migrating Hive HDS projects from Compose for Data Lakes 6.6 (C4DL 6.6) to Qlik Compose 2021.8 (Gen2)

Different Migration Modules for customers

• Migration Pre-Requisites
  • Pre-Requisites for Migrating Projects from Compose for Data Lakes to Qlik Compose
• Databricks to Databricks
  • Migrate Databricks projects from Compose for Data Lakes 6.6 to Qlik Compose 2021.8
• Spark to Spark supported version
  • Migrate Spark projects from Compose for Data Lakes 6.6 to Qlik Compose 2021.8

Module 3: Migrate Hive HDS projects from C4DL 6.6 to Gen2

There are two paths for Hive HDS project, you can choose one of the paths to finish migration.

Path 1: You can clean up storage database and replicate landing database (including underlaying files for attrep_cdc_partition). Reload replicate task and start the compose tasks as it is new project. It was covered in first demo.

Path 2: You can follow migration path, and we explained all the required steps in this document.

Here are the migration steps (Path 2) if you don’t want to reload the replicate task. You are going to completely migrate your project definition and data to Gen2.

• Install Qlik Compose 2021.8 on a dedicated machine\the same machine where FDL 6.6 exist.

• Select a project in C4DL 6.6 and run the CDC task last time with verbose.

1. Create the deployment package   à Suresh_Hive_HDS_deployment_<datetime>.zip
2. Run the CDC task last time before migrate data and disable if there are any schedules on the tasks.

• Create new project in 2021.8 with the same name and project type as it is in FDL 6.6. you have to migrate data also so create a new storage database with different name (we can’t use same database which is using in C4DL project) and provide it in Data Lake connection. Do the test connection to make sure you have correct drivers installed for your data lake. Now we have an empty project in gen2 with storage connection.

• Deploy the projects on Qlik Compose 2021.8 and migrate the data

• Migrate the project definitions

1. Login to Qlik Compose 2021.8 cli and run below command to create compatible package.

ComposeCli.exe adjust_cfdl_project --project Suresh_Hive_HDS --infile “C:\Program Files\Qlik\C4DL66\ Suresh_Hive_HDS_deployment_<datetime>.zip”

- C:\ProgramFiles\Qlik\Compose\data\projects\Suresh_Hive_HDS\deployment_package\ Suresh_Hive_HDS__<datetime>__QlikComposeDLMigration.zip

1. Deploy adjusted deployment package on gen2 project.
2. Validate model and create the storage tables.

• Migrate the data

1. Generate tasks then execute a data migration SQL script from notebook.
2. Generate 'create data migration script’ using below cli (Qlik Compose 2021.8 cli) command and run it on storage database using External tool.

ComposeCli.exe create_cfdl_data_migration_script --project Suresh_Hive_HDS --infile       “C:\Program Files\Qlik\C4DL66\ Suresh_Hive_HDS_deployment_<datetime>.zip”

1. verify the data is correct

NOTE:  

• We must use new storage database in path 2 because we are going to run select * from C4DL 6.6 storage database and insert into Qlik Compose 2021.8 storage database in the migrate data step (3 (b)).
• If you want, you can run C4DL project and Gen2 project parallelly for few days.

After migration you can check\run

• ODS and HDS live views before running CDC
• CDC with updates\inserts\deletes
• CDC with reload event
• schema evolution - you should run schema evolution and skip all detected changes - to set the context

also set

• workflow
• scheduler
• notification
• work with multiple replicate servers

You can watch demo video How to Migrate Hive HDS projects.

Environment

• From Compose for Data Lakes 6.6
• To Qlik Compose 2021.8

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

Related Content 

Migration paths for Qlik Compose

Description

When either preview and NPrinting Report containing a straight table or pivot table, the following error is generated:

• ERROR: Newtonsoft.Json.JsonReaderException: The reader's MaxDepth of 64 has been exceeded
• Example of error when seen in the NPrinting Designer (Note that a similar error will appear in the NPrinting Publish Task Execution log which contains this same NPrinting Report).

Environment

• NPrinting May 2022 SR 3 and earlier versions

Resolution

• Upgrade to May 2022 SR 4 (scheduled for release Mid February 2023)

Fix Version:

• May 2022 SR 4

Cause

• Product Defect ID: OP-95796 

Going to retrieve archived REDO log with sequence 2056016, thread 1 but fails REDO log with a sequence not found error. The Archived Redo Log has Primary Oracle DB as DEST_ID 1 and the Standby DEST_ID 32 pointing to the correct location.

Environment

Qlik Replicate 

Resolution

Cannot use DEST_ID greater than 31 for Primary or Standby Oracle Redo Log locations

Cause

Qlik Replicate only supported the DEST_ID 0 through 31 

Related Content 

https://docs.oracle.com/en/database/oracle/oracle-database/19/refrn/V-ARCHIVE_DEST.html

Internal Investigation ID(s)

JIRA RECOB-6332 case 00058856