Qlik DataTransfer Release Notes - November 2022 Initial Release

[suhas_sutariya]

Table of Contents

• What's new in Qlik DataTransfer November 2022
• Resolved Defects
• Improvements
• Upgrade notes
• System requirements notes
• Known issues and limitations

The following release notes cover the versions of Qlik DataTransfer released in November 2022.

What's new in Qlik DataTransfer November 2022

Qlik DataTransfer November 2022 release does not have any new feature, but it has fixes for the issues described below and few other improvements which can help customer to avoid any issues.

Resolved Defects

Qlik DataTransfer November 2022 comes with fixes for the issues described below.

 

November 2022

File Upload error if Folders with same name is configured.

Jira issue ID: DTB-61

Description: Qlik DataTransfer attempts to move files to a temporary folder created based on parent directory name. If two or more folders had the same name and two separate files in both folders had the same name, then the temporary file copies would conflict with each other, potentially causing local file and upload operations to fail. This also potentially left orphaned files in the temporary files folder, interfering with uploads already in progress and subsequent upload of files having the same name.

File is deleted from Source if failed to upload from Data Folder.

Jira issue ID: DTB-62

Description: File(s) are deleted from source folder when an error occurs during the manual upload of a Data folder.

QDT UI hangs indefinitely and cannot handle large lists of tables in data preview.

Jira issue ID: DTB-63

Description: Data Preview screen hangs while fetching tables when there is a large list of tables (around 100k).

App Reload Delay Too Short and Not Configurable.

Jira issue ID: DTB-64

Description: In some cases, app reload is triggered too quickly during automatic folder upload.

Improvements

Qlik DataTransfer November 2022 comes with below improvements in current features.

Improved upload process by adding extra validation, more configurable settings, and better error handling.

Description:

• There is a predefined TTL (Time to Live) for a file in QCS while it is being uploaded. The default TTL in QCS is 10000 seconds (02:46:40) which was used in prior releases of Qlik DataTransfer. If the file size is large and the upload speed is low, it might take more than the default TTL to upload and subsequently fail. To handle this scenario, Qlik DataTransfer now calculates the estimated TTL.
  
  Qlik DataTransfer will calculate the TTL considering the below parameters which are also configurable under [data-upload-service.parameters] in services.conf file.
  • Current file size
  • Minimum upload bandwidth (in kbps). The default is 25600 kbps / 25 mbps
    ex. --minBandwidth 25600
  • Maximum number of concurrent uploads (default is 10) ex. --maxConcurrentUploads 10
  • Retry delays. If a file chunk fails to upload to, this setting controls how many times to retry and the time to wait between each retry. (Default is 2000,5000,10000,20000,30000)
    ex. --tusRetryDelays 2000,5000,10000,20000

Qlik DataTransfer calculates the time to upload based on the assumption of bandwidth being equally divided among concurrent uploads. It also adds a 20% safety margin and the sum of all retry delays to estimate overall worst case TTL.

The minimum TTL is 600 secs (10 minutes), and the maximum is 259200 secs (3 days).

• A concurrent upload queue mechanism has been added. This manages the maximum number of uploads allowed at the same time. This may result in a delay of several seconds to build up the queue before uploading of files begins. The user can configure this wait time; we call it queueing delay. Default is 30000 mSec, ex. --queueingDelay 30000
• A maximum file size override (in GB) has been added. The default is 0 which uses the QCS tenant’s maxLargeFizeSize quota. This can be set to a smaller size to prevent users from accidentally uploading very large files, ex. --maxFileSize 6.0
• The default debounce delay (--debounceDelay) value has been changed from 0 to 1000 mSec.
• If QCS returns a resource busy (423) error during upload, then it will wait for a specific amount of time before it retries. The default value is 60000 mSec. ex. --resourceBusyDelay 60000 
• Qlik DataTransfer reloads apps in QCS once all files in a particular Data folder have been uploaded. By default, Qlik DataTransfer waits 10 seconds before it requests that QCS reload an associated app to make sure there isn't any pending upload. Users can now configure this delay. This prevents the reload of associated apps in QCS from being triggered too quickly during automatic uploads. The default value is now 30 seconds, ex. --appReloadDelay 30000 (mSec)

Upgrade notes

 

Qlik DataTransfer older versions

Backup recommendations:

• It is important that you make a backup of the folder C:\ProgramData\Qlik\Sense\DataTransfer  and C:\Program Files\Qlik\DataTransfer\services.conf  before upgrading. This will make it possible to recover the installation if the upgrade fails.
• If you have installed Qlik DataTransfer to a folder other than C:\Program Files\Qlik\DataTransfer, then please make a copy of the services.conf file from that folder.
• If you have changed the default StoragePath in the services.conf file, then make a backup of that folder instead.
• The services.conf file is reset to the latest release’s defaults during upgrades so the backup is helpful for reapplying any custom user settings.
• If you are installing or upgrading on machine where Qlik Sense Enterprise for Windows is already installed, please make sure you install/upgrade Qlik DataTransfer using Qlik Sense service(s) user account.

System requirements notes

Please refer to the online help for information about the requirements for Qlik DataTransfer.

System requirements for Qlik DataTransfer

Known issues and limitations

The following issues and limitations were identified at release time. The list is not comprehensive; it does however list all known major issues and limitations.

General

• Maximum allowed file size is controlled by QCS platform and considered automatically by Qlik DataTransfer. 
• You can upload a maximum of 200 tables or files per dataset. If your dataset has more than 200 tables or files, only the first 200 are uploaded. For example, if you select more than 200 tables in a data connection or Qlik app, or if you add a data folder with more than 200 files.
• When reloading apps created in Qlik Sense Desktop, Qlik DataTransfer does not support connections to Essbase.
• Files uploaded using Qlik DataTransfer must be manually deleted from SaaS editions of Qlik Sense. Qlik DataTransfer does not delete files from SaaS.
• While Qlik DataTransfer can extract data from any .qvf file, it can only reload the data in that .qvf if it was created using Qlik Sense Desktop and all of the data sources for that .qvf are accessible from the Qlik DataTransfer machine.
• Qlik Sense Desktop cannot be installed on server operating systems. Therefore, if you install Qlik DataTransfer on a server operating system, and wish to use the QVF refresh functionality, you must create the app on a desktop operating system and then copy your QVF files to the server. All data sources that are used in the QVF file must be accessible to the QVF load script when it is run on the server by Qlik DataTransfer.
• Qlik DataTransfer should not be used to upload data from Qlik apps that use section access or have load scripts that control user access to data. Protected data might be exposed to unauthorized users.
• Incremental load is not supported.
  Work around: Use Qlik Sense Desktop to create an app including incremental load logic, and reload it using the Qlik Apps.
• You cannot edit the query or data sources of the Data Connection.
• There is no notification of upload and reload failures for scheduled tasks. To determine if your data has uploaded successfully, please check the last execution details or consult the log files.
• Mapped drives are not supported. You can use a shared path instead: \\<machine-name>\folder.
• If you change the service log on account after configuring Qlik DataTransfer with your tenant, you must reconfigure the tenant and connections.
• QlikView apps are not automatically reloaded by Qlik DataTransfer when data sources in apps are updated with new data. QlikView apps must be manually reloaded and saved for Qlik DataTransfer to upload data to a SaaS edition of Qlik Sense tenant.
• If write access to configured space is revoked, the upload will be failed and results in errors which can be seen in the logs but not in the UI.
• In Data Folder, if unsupported files i.e..png, .bmp, etc. are resided along with supported files and QDT tries to upload them it fails to upload unsupported files. Hence it will show status as ‘Failed’.
• On upgrade from the Oct2020 release or later, if a user chose a different location for storing configuration files or chose different ports to run QDT from configuration file - Service.conf – the installation will override it. The mitigation to this known issue is to backup this file prior to the upgrade and update the Service.conf file with earlier changes after the upgrade completed.
• You changed the configuration folder location from C:\ProgramData\Qlik to some another location in Service.conf file and when you uninstall Qlik DataTransfer, it is not deleting the configuration folder.
• While the Daylight Saving is happened, Qlik DataTransfer changes the scheduled hour. So schedules in Qlik DataTransfer are updated on new DST time.
• Qlik DataTransfer does not support the use of web proxies. If your system uses proxies, you must add a proxy bypass rule specifically for Qlik DataTransfer.
• Special characters from Japanese language are not supported if they are used in the table name or file name or path. Ex. 【】

About Qlik

Qlik converts complex data landscapes into actionable insights, driving strategic business outcomes. Serving over 40,000 global customers, our portfolio provides advanced, enterprise-grade AI/ML, data integration, and analytics. Our AI/ML tools, both practical and scalable, lead to better decisions, faster. We excel in data integration and governance, offering comprehensive solutions that work with diverse data sources. Intuitive analytics from Qlik uncover hidden patterns, empowering teams to address complex challenges and seize new opportunities. As strategic partners, our platform-agnostic technology and expertise make our customers more competitive.

qlik.com