Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Search our knowledge base, curated by global Support, for answers ranging from account questions to troubleshooting error messages.
Qlik offers a wide range of channels to assist you in troubleshooting, answering frequently asked questions, and getting in touch with our technical experts. In this article, we guide you through all available avenues to secure your best possible experience.
For details on our terms and conditions, review the Qlik Support Policy.
Index:
We're happy to help! Here's a breakdown of resources for each type of need.
Support | Professional Services (*) | |
Reactively fixes technical issues as well as answers narrowly defined specific questions. Handles administrative issues to keep the product up-to-date and functioning. | Proactively accelerates projects, reduces risk, and achieves optimal configurations. Delivers expert help for training, planning, implementation, and performance improvement. | |
|
|
(*) reach out to your Account Manager or Customer Success Manager
Your first line of support: https://community.qlik.com/
Looking for content? Type your question into our global search bar:
Leverage the enhanced and continuously updated Knowledge Base to find solutions to your questions and best practice guides. Bookmark this page for quick access!
Subscribe to maximize your Qlik experience!
The Support Updates Blog
The Support Updates blog delivers important and useful Qlik Support information about end-of-product support, new service releases, and general support topics. (click)
The Qlik Design Blog
The Design blog is all about product and Qlik solutions, such as scripting, data modelling, visual design, extensions, best practices, and more! (click)
The Product Innovation Blog
By reading the Product Innovation blog, you will learn about what's new across all of the products in our growing Qlik product portfolio. (click)
Q&A with Qlik
Live sessions with Qlik Experts in which we focus on your questions.
Techspert Talks
Techspert Talks is a free webinar to facilitate knowledge sharing held on a monthly basis.
Technical Adoption Workshops
Our in depth, hands-on workshops allow new Qlik Cloud Admins to build alongside Qlik Experts.
Qlik Fix
Qlik Fix is a series of short video with helpful solutions for Qlik customers and partners.
Suggest an idea, and influence the next generation of Qlik features!
Search & Submit Ideas
Ideation Guidelines
Get the full value of the community.
Register a Qlik ID:
Incidents are supported through our Chat, by clicking Chat Now on any Support Page across Qlik Community.
To raise a new issue, all you need to do is chat with us. With this, we can:
Log in to manage and track your active cases in the Case Portal. (click)
Please note: to create a new case, it is easiest to do so via our chat (see above). Our chat will log your case through a series of guided intake questions.
When creating a case, you will be prompted to enter problem type and issue level. Definitions shared below:
Select Account Related for issues with your account, licenses, downloads, or payment.
Select Product Related for technical issues with Qlik products and platforms.
If your issue is account related, you will be asked to select a Priority level:
Select Medium/Low if the system is accessible, but there are some functional limitations that are not critical in the daily operation.
Select High if there are significant impacts on normal work or performance.
Select Urgent if there are major impacts on business-critical work or performance.
If your issue is product related, you will be asked to select a Severity level:
Severity 1: Qlik production software is down or not available, but not because of scheduled maintenance and/or upgrades.
Severity 2: Major functionality is not working in accordance with the technical specifications in documentation or significant performance degradation is experienced so that critical business operations cannot be performed.
Severity 3: Any error that is not Severity 1 Error or Severity 2 Issue. For more information, visit our Qlik Support Policy.
If you require a support case escalation, you have two options:
When other Support Channels are down for maintenance, please contact us via phone for high severity production-down concerns.
A collection of useful links.
Qlik Cloud Status Page
Keep up to date with Qlik Cloud's status.
Support Policy
Review our Service Level Agreements and License Agreements.
Live Chat and Case Portal
Your one stop to contact us.
CICD build Job fails with the below message:
Failed to execute goal on project: Could not resolve dependencies for project
Solution 1
Solution 2
Qlik Talend Data Integration: CICD build fails and shows an error in the Maven log
This article provides a guide on installing Qlik Replicate on a docker image.
sudo rpm -qa|grep dockersudo yum remove docker docker-client docker-client-latest docker-common docker-latest docker-laster-logrotate docker-logrotate docker-engine
sudo yum install -y yum-utils device-mapper-persistent-data lvm2 sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo sudo yum install --allowerasing docker-ce docker-ce-cli containerd.io sudo systemctl start docker sudo systemctl enable docker
/kit
" The /kit directory is a temporary folder used to store installation files. In the example below, the RPM file is areplicate-2024.5.0-357.x86_64.rpm
. Once the installation is complete, the entire /kit folder can be safely deleted.mkdir -p /kit/ar_docker
cd /kit
rpm2cpio areplicate-2024.5.0-357.x86_64.rpm | cpio -iv --make-directories --no-absolute-filenames -D ar_docker/ ./opt/attunity/replicate/addons/samples/docker/*
mv ./ar_docker/opt/attunity/replicate/addons/samples/docker/* ./ar_docker
rm -rf ./ar_docker/opt
cd /kit/ar_docker
cp ../areplicate-2024.5.0-357.x86_64.rpm .
./create-dockerfile.sh
/kit/ar_docker/Dockerfile
, from:
RUN yum -y install /tmp/areplicate-*.rpmto
RUN systemd=no yum -y install /tmp/areplicate-*.rpm
NOTE!
The parameter systemd=no is used to solve the below error you may hit during the docker build stage: This rpm is not supported on your system (no systemctl), exiting. error: %prein(areplicate-2024.5.0-357.x86_64) scriptlet failed, exit status 43
NOTE!
The password should not be empty and must be strong enough.
WARNING!
If you want to install ODBC Drivers, please make the corresponding ODBC Drivers rpm files are ready in this folder. If you want to skip the ODBC Drivers installation at present, rename or delete the file "drivers" in this folder.
docker build --no-cache -t johnw/replicate:2024.5 .where
/johnw/replicate:2024.5
is the image tag.Do not forget the last period "."
docker run -d --name ar --hostname cdc2 -e ReplicateRestPort=3552 -p 3552:3552 -v /dockermount/data/replicate/data:/replicate/data johnw/replicate:2024.5
Now Qlik Replicate is running in the docker and can be accessed from Qlik Replicate Console GUI.
By default, QlikView Publisher sets the number of "Max number of QlikView engines to use for distribution" to 4. This value determines how many QlikView Publisher tasks can be run in parallel and can be freely adjusted to suit any environment. Sometimes you see the error/warning Failed to allocate engine before you hit the limit you have set.
A more detailed best practice guide is available in Scaling the QlikView Publisher.
The image below depicts where to increase the number of engines for distribution or administration. (default 4 and 20 respectively)
When determining how many Reload Engines to allow per QlikView Publisher, there are a number of factors to take into consideration.
The first and fundamental factor that affects how many Reloads that can safely take place simultaneously, is that the Reload Engine, the QVB.exe process, is a fully MultiThreaded process. This means that a single executing task using a Reload Engine WILL try to use as much resources from the hardware as it can, without limit. The limiting factors are only Script Efficiency and Data transfer rates. A reload script that does lots of data aggregation at script level will have a big impact on the CPU utilization for that Reload Task for example. Tasks that run in parallel on a server will compete for the same pool of resources containing CPU cycles and RAM.
With this basic fundamental factor in mind, we can start discussing the possible settings
The first consideration is that you should never allow for more Engines than the Server has CPU cores -1. So for a Server with 8 CPU cores, we should never sat a value higher than 7.
Note also, that the QlikView Distribution Service has been equipped with a load protection mechanism since the 12.10 release. See Tasks are not executed as expected and are shown in status Queued for details.
The second consideration is that even on big servers with lots of CPU cores (16+) it rarely pays to go higher than approximately 13 simultaneous reload engines allowed. This is partly due to the first fundamental factor (every running task will try to consume all the resources) and partly due to other bottlenecks, where for example multiple data connections using the same drive will start becoming slower as the concurrent connections goes up. Going for a high number also normally increases the failure rate significantly. Qlik recommends starting with a low number and then increasing gradually until you hit the sweet-spot for your own environment, where you are getting a high throughput without introducing a high failure rate. If your reload schedule calls for a high number of simultaneous tasks to execute, you will be better off clustering the Publisher role over more servers than increasing the number of engines on a single node.
A third consideration is Windows settings that affect the number of concurrent processes of the same kind. If you enable a high number of reload engines and start seeing windows errors related to Windows running out of Desktop Heap Size. This may be visible in the Windows System Event log of the server running Qlikview Distribution Service, with errors saying "Desktop Heap exhausted". If so, you will need to adjust Windows Desktop Heap Size to allow for a higher server load. Allowing for a bigger Desktop Heap may also allow a highly loaded server to run for longer periods of time without the need for server restarts.
Messages like this,
Failed to create QlikView Engine.. Exception=System.Runtime.InteropServices.COMException (0x80080005)
Shows that there is an issue with desktop heap size.
IMPORTANT: Windows registry changes are made at your own risk and are not supported by Qlik Support. Please make a backup of the Windows registry before applying changes to enable rollback if the setting change fails or introduces Windows instability.
Desktop Memory Heap size is control in the following Windows registry key on the server running Qlikview Distribution Service:
Change the desktop heap setting by setting the SharedSection value to 1024,20480,2048. After change the registry key should look something like the following (all on one line):
%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,2048 ...
Notice that the default value is 1024,3072,512 in x86 or 1024,20480,768 in x64 environments.
Also, please note that the skipped part (marked as "..." ) might differ according to the operating system version and should not be modified.
Depending on size of deployment and complexity of tasks/scripts you might need to increase the Non-interactive Desktop Heap even further, the following list gives the next possible setting for the value.
%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,4096 ...
OR
%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,6144 ...
OR
%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,8192 ...
Microsoft reference regarding Windows heap and Server 2012: https://support.microsoft.com/en-us/kb/947246
A more detailed best practice guide is available in Scaling the QlikView Publisher.
This document is a general guide and is provided as is. Modifications to the process may be necessary depending on your individual database setup.
If you have installed a standalone PostgreSQL database, or if you have used the Qlik PostgreSQL Installer (QPI) to upgrade and decouple your previously bundled database, then you can upgrade PostgreSQL at any time. This means you control maintenance and can immediately react to potential PostgreSQL security concerns by upgrading to a later service release or a later major version.
Content
This document covers the following scenario:
Run a complete backup of Qlik Sense Enterprise on Windows site as described in Backup and restore Qlik Sense Enterprise on Windows.
These steps apply if you are upgrading within a major PostgreSQL release (example: 14.5 to 14.8).
No further steps are required.
If you are moving to a higher major version, an in-place upgrade will not be possible. Instead, we will install the the new version in parallel, then then migrate the old database and eventually uninstall the old version. Our example is written using PostgreSQL 12 to 14.
Upgrading and unbundling the Qlik Sense Repository Database using the Qlik PostgreSQL Installer
How to manually upgrade the bundled Qlik Sense PostgreSQL version to 12.5 version
Changing the Database Superuser Password without Qlik Sense Installed
The option Apply Exceptions (b) is not enabled in the Control Tables (a) menu. This setting is typically enabled by default:
Further troubleshooting reveals the option Apply batched changes to multiple tables concurrently (b) has been enabled in the Change Processing Tuning (a) options:
This setting affects the option to set a Global Error Handling policy, disabling it, and resets task-specific error handling options to their default settings.
This caused the Apply Exceptions check box to be unticked.
Disable Apply batched changes to multiple tables concurrently and enable the Apply Exceptions table.
Qlik Replicate
Snowflake on Azure
When working with the IBM DB2 for z/OS source endpoint, the trailing spaces are trimmed in CHAR datatype columns, especially when the Log Stream is involved in the tasks. The complete replication is from DB2z to Log Stream, then from Log Stream to S3 target endpoint.
To address this issue, add the internal parameter keepCharTrailingSpaces and set its value to TRUE.
This step has to be taken on these task endpoints:
1: Log Streaming task (parent task)
2: Replication task (child task)
Steps:
#00146985
Data values in Qlik Sense are stored as dual values, which means that they have an underlying numerical value and a textual presentation value. This is important to understand before (re)formatting values.
Num(number, format) format function can be used to load a numeric value, number, and store it as a dual value where the presentation is stored according to the format.
For example, Num(1050.75, ‘#,##0.0’) will be stored as numeric value 1050.75 with the presentation rounded to one decimal value as 1,050.8.
The general numeric precision in Qlik Sense is 14 digits. By using a format string with 14 # characters, the engine will interpret the number as a generic 14-digit value.
For example, Num(1050.75, ‘##############’) will be stored as numeric value 1050.75 with the presentation sustained as the input value 1050.75.
The num #(text, format) function allows reverse formatting, where a text is parsed according to the format string as a numeric value. The text value becomes the presentation value and the interpreted numeric value is stored as the underlying value.
In Qlik Cloud, the reporting service requests the visualization data from the Qlik Engine. In this data set, each column has one format stored by the engine, which is used by the reporting service to format columns in the exported Excel sheet.
Columns with mixed formats (numbers and text) will be treated as undefined, and can be expected as text in the exported Excel.
Columns with mixed numeric formats will be exported in a uniform format. Columns formatted with ############## are interpreted as generic numeric values by the reporting service. The output format in Excel in this case becomes General.
There are two ways to apply the generic format to mixed numeric fields or chart columns:
There is naturally a maintenance benefit in controlling formats as much as possible in the script so that object-level formatting can be avoided to the most possible extent.
For data processed through the QVD layer, it may be a further optimization to format the data closer to the source to benefit from the preformatted data as it is ingested through an ETL flow.
You can apply styling to customize the app based on your company standards. These styles are managed through custom themes.
To create your own custom theme, you’ll need to compile a compressed zip folder with three key components:
See Themes overview.
Once you’ve created the theme, you can upload it to your Qlik Cloud Management Console.
If you utilize the aws-java-sdk to customize utility classes under Repository-->Code -> Routines and use this customized class within the components such as tJava / tJavaRow etc, please be aware that the aws-java-sdk is an all-in-one JAR. You might overlook the significance of SDK modularity and opt for an all-in-one JAR, potentially leading to the adverse effects detailed in the article.
1. Increased Application Size:
The all-in-one JAR bundles all AWS service SDKs together, significantly increasing the size of your application. This results in larger deployment artifacts, slower download times, and increased storage consumption, due to the increased size of the Job design (aws-java-sdk.jar: 100-400MB), thereby intensifying the network transfer burden for deployment.
2.Higher Memory Usage:
During application startup, loading the all-in-one JAR into memory demands additional memory, as all AWS service classes are loaded, including those not in use. This can result in increased memory consumption and footprint, and may lead to out-of-memory errors in resource-constrained environments.
3.Reduced Modularity and Flexibility:
Using the all-in-one JAR reduces the modularity of your application. It limits your ability to customize or optimize the inclusion of specific AWS services, leading to a less flexible and maintainable codebase.
Instead of including the entire aws-java-sdk JAR, opt for individual service modules. Include only the specific AWS SDK modules your application needs. For instance, if your application interacts only with S3 and DynamoDB, include only aws-java-sdk-s3 and aws-java-sdk-dynamodb dependencies.
This approach helps minimize the application size, reduces memory footprint, and enhances overall performance by loading only the necessary classes at runtime.
Qlik Sense Enterprise on Windows creates a subfolder on app import from the Qlik Sense Management Console. The folder uses the hostname of the server machine and will include a subdirectory named after the user carrying out the import.
The folder is emptied after the import has been completed. Unless the imported app is large and requires a long time to process, users will not see the temporary file being created.
In this example, we imported a 2 GB large file:
This article provides step-by-step instructions for implementing Azure AD as an identify provider for Qlik Cloud. We cover configuring an App registration in Azure AD and configuring group support using MS Graph permissions.
It guides the reader through adding the necessary application configuration in Azure AD and Qlik Sense Enterprise SaaS identity provider configuration so that Qlik Sense Enterprise SaaS users may log into a tenant using their Azure AD credentials.
Content:
Throughout this tutorial, some words will be used interchangeably.
The tenant hostname required in this context is the original hostname provided to the Qlik Enterprise SaaS tenant.
Copy the "value of the client secret" and paste it somewhere safe.After saving the configuration the value will become hidden and unavailable.
In the OpenID permissions section, check email, openid, and profile. In the Users section, check user.read.
Failing to grant consent to GroupMember.Read.All may result in errors authenticating to Qlik using Azure AD. Make sure to complete this step before moving on.
In this example, I had to change the email claim to upn to obtain the user's email address from Azure AD. Your results may vary.
While not hard, configuring Azure AD to work with Qlik Sense Enterprise SaaS is not trivial. Most of the legwork to make this authentication scheme work is on the Azure side. However, it's important to note that without making some small tweaks to the IdP configuration in Qlik Sense you may receive a failure or two during the validation process.
For many of you, adding Azure AD means you potentially have a bunch of clean up you need to do to remove legacy groups. Unfortunately, there is no way to do this in the UI but there is an API endpoint for deleting groups. See Deleting guid group values from Qlik Sense Enterprise SaaS for a guide on how to delete groups from a Qlik Sense Enterprise SaaS tenant.
Qlik Cloud: Configure Azure Active Directory as an IdP
Users are not able to see their alerts after upgrading Qlik Alerting. Alerts are visible only to the Qlik Alerting Admin.
This issue can be diagnosed by reviewing the Qlik Alerting logs (How to collect the Qlik Alerting log files).
ERROR Wrong password for user QVSERVICE.
ERROR Access denied
ERROR User qvservice@domain.local not found.
ERROR Wrong password for user User qvservice@domain.local.
ERROR History not found
To resolve this issue, follow these steps to restore the Qlik Alerting database:
Reference articles:
During the upgrade, Qlik Alerting was unable to locate the admin account, resulting in the need for the admin to reset the password. Consequently, all user accounts were affected, leading to the removal of user passwords. As a result, each user will need to recover their password using the "Forgot Password" link.
QB-28235
This article provides an example of how to retrieve user attributes from Excel.
For more information on the Qlik Sense User Directory Connector and its options for reading user attributes from different sources, see User directory connectors.
Synchronizing user data and attributes does not carry out authentication. It is intended to enable the use of additional attributes which may not otherwise be synchronized or to pre-emtively distribute licenses. Authentication must be handled by your choice of identity provider. In the Excel example, this would typically be Windows.
Each data source has a different configuration and the following example (Excel) of adding an ODBC user directory connector.
Do the following:
Verify that the Microsoft Excel Driver is installed.
Set up an ODBC source on the server.
You need to store the data in two separate sheets in the same excel file, for example, for example you can place it in this location: %ProgramData%\Qlik\Sense\temp.
The temp folder is not included in the default installation. You need to create the temp folder, if not already done by another QMC administrator.
SheetA contains the users and SheetB the user attributes.
Example:
SheetA
userid | name |
1 |
John |
2 |
Bill |
3 |
Tom |
SheetB
userid | type | value |
1 | jd@email.com | |
1 | lastname | Doe |
2 | bg@email.com | |
2 | lastname | Gates |
3 | th@email.com | |
3 | lastname | Hanks |
Select User directory connectors on the QMC start page or from the Start drop-down menu to display the overview. Create a new user directory connector (ODBC) and edit the properties.
Identification
All fields are mandatory and must not be empty.
Property DescriptionName | The name of the UDC configuration, defined from the QMC. |
Type |
The UDC type. |
User sync settings
Property Description Default valueSync user data for existing users |
|
Selected |
Connection
Property Description Default valueUser directory name |
The name of the user directory. Must be unique, otherwise the connector will not be configured. The name must not contain spaces. |
- |
Users table name | The name of the table containing the users. Include the file extension in the table name, for example: [SheetA$]. | - |
Attributes table name | The name of the table containing the user attributes. Include the file extension in the table name, for example: [SheetB$]. | - |
Visible connection string |
The visible part of the connection string that is used to connect to the data source. |
- |
Encrypted connection string |
The encrypted part of the connection string that is used to connect to the data source. Typically, this string contains user name and password. The two connection strings are concatenated into a single connection string when making the connection to the database. |
- |
Synchronization timeout (seconds) | The timeout for reading data from the data source. | 240 |
Example:
User table name: [SheetA$]
Attributes table name: [SheetB$]
Visible connections string: Driver={Microsoft Excel Driver (*.xls, *.xlsx, *.xlsm, *.xlsb)};Dbq=%ProgramData%\Qlik\Sense\temp
Click Apply to apply your changes.
Go to the User directory connectors overview and check if the user directory is displayed as Configured and Operational.
If the User directory name is not unique the connector will not be configured. If not operational, check the repository system log in: %ProgramData%\Qlik\Sense\Log\Repository\Trace.You have added an ODBC data source and initial synchronization will be performed within five minutes (by default).
Qlik ODBC connector package (database connector built-in Qlik Sense) fails to reload with error Connector reply error:
Executing non-SELECT queries is disabled. Please contact your system administrator to enable it.
The issue is observed when the query following SQL keyword is not SELECT, but another statement like INSERT, UPDATE, WITH .. AS or stored procedure call.
See the Qlik Sense February 2019 Release Notes for details on item QVXODBC-1406.
By default, non-SELECT queries are disabled in the Qlik ODBC Connector Package and users will get an error message indicating this if the query is present in the load script. In order to enable non-SELECT queries, allow-nonselect-queries setting should be set to True by the Qlik administrator.
To enable non-SELECT queries:
As we are modifying the configuration files, these files will be overwritten during an upgrade and will need to be made again.
Only apply !EXECUTE_NON_SELECT_QUERY if you use the default connector settings (such as bulk reader enabled and reading strategy "connector"). Applying !EXECUTE_NON_SELECT_QUERY to non-default settings may lead to unexpected reload results and/or error messages.
More details are documented in the Qlik ODBC Connector package help site.
Feature Request Delivered: Executing non-SELECT queries with Qlik Sense Business
Execute SQL Set statements or Non Select Queries
Qlik Replicate can use SAP HANA as the Backend Database, which you can define on the SAP Application (DB) Source Endpoint setting with Triggers in SAP HANA.
PM-13722
It may be necessary to migrate, backup, or otherwise move a Qlik Sense Enterprise on Windows app with its community and personal sheets included. This option is not available when using the default Export of the app in the Qlik Sense Enterprise Management Console, as this only experts a version of the app without personal and community sheets.
Note:
Community sheets can be promoted to public (base) sheets. See Adding sheets to the public sheets of an app.
Community and Personal sheets are kept in the binary file (.qvf) file stored in the Qlik Sense share storage.
This location is stated under Qlik Management Console (QMC) > Service Cluster > App folder:
The binary file is given the app ID (32 hexadecimal digit string) as the naming convention without the extension .qvf, you can find the ID of the app under QMC > Apps, there you can activate ID as a column and look for the ID of the app you need to import:
In order to backup community and private sheets (besides the base sheets), this binary file needs to be backed up following these steps:
Copy the file in any other location, e.g. desktop, downloads, etc.
All objects in the app (including bookmarks, snapshots, and so on) will be assigned to the App owner (admin importing the app), and all prior ownership information and publish states will be lost.
Are you looking to download your purchased Qlik Products or download a Trial? All supported on-premise Qlik Products can be downloaded from Qlik's Product Download Site.
To access the Download Site, you need an active QlikID. You will be able to see all products your account is eligible for.
You can access the Download Page directly here, or navigate to it from the Community Home page:
This shows the download page with the Latest release and patch preselected.
Unsupported versions are not available for download. See Product Lifecycle for details on what versions have reached end of support.
Alternatively, instead of selecting your product directly, you can also search all available columns:
If you encounter issues with the download site, start a chat with us and we will be able to help you right away.
Click here for video transcript
When working with the Oracle source endpoint, there is consistently a 30-minute latency. After an update is performed on the source, it takes 30 minutes for the changes to be reflected in the target Kafka. No configuration in the task explicitly sets this delay.
To address this issue, consider one of the following options:
Qlik Replicate captures changes from archived redo logs only. If changes on the source are still stored in Online Redo Logs during a given period, Qlik Replicate cannot retrieve them from the archived redo logs until approximately 30 minutes have passed.
In the Oracle source endpoint advanced property, Use archived redo logs only is ticked:
The task log reads:
2024-08-18T11:22:33 [SOURCE_CAPTURE ]I: Oracle CDC will access Archived Redo logs only
00293052
For almost all services or APIs running in production, the requirement exists to secure access to the service. In the rest of this article, we will look at how to ensure the following security requirements:
This article shows you how to implement these concepts when running data services or Routes in the Talend Runtime. The focus is on the cloud runtime, but also applies to on-premises environments. Microservices are not handled in this article.
The attached security.zip file contains a Talend Studio v7.0 project with a sample service and Route to experiment with. It can be imported into any Talend Studio v7.x. In addition, it contains sample configuration files for the LDAP login module described later.
For REST-based services, in a cloud environment, Talend provides support only for basic authentication, where the user name and password are sent in an HTTP header alongside the request in clear text. Therefore it is mandatory to use HTTPS instead of HTTP as the protocol to ensure that the password is encrypted. Contrary to on-premises, for cloud environments, SAML token and OAuth are not supported. If more sophisticated solutions are required, Talend recommends using an API Gateway to secure and control access to the services and APIs. Talend Help provides information on how to integrate with API Gateways and examples for deployment on AWS API Gateway and Azure API Management.
Activating basic authentication for a service or a Route is done in Talend Studio in the corresponding REST component (tRESTRequest in a data service, or cREST in a Route). A data service where authentication is enabled is shown below:
Figure 1: Authentication enabled data service
Enforcing authentication information when sending the request to the service is only half the story. The provided identity must be validated and compared to the set of accepted identities. Talend Runtime provides an authentication framework based on Java Authentication and Authorization Service (JAAS), which allows you to plug in different modules targeting a specific backend to validate identity information. Out of the box, Talend Runtime is configured to support a file-based backend, but among others LDAP can also be used to validate. For a complete list of supported backends, see the Apache Karaf documentation.
The PropertiesLoginModule login module is the one configured by default in the Talend Runtime. It is based on the ${Runtime_Home}/etc/users.properties file, which uses the properties file format. Each property represents an identity. The format of the property is as follows:
user=password[,role][,role]...
Several roles may also be grouped together and the group can be referenced in the user entry:
_g_\:group=role[,role]... user=password[,role][,_g_\:group]...
The figure below shows the users.properties file, which is part of the Talend Runtime installation. In addition to the standard users tadmin, tesb, and karaf, two users alice and bob were added.
Figure 2: Sample users file
Passwords in the properties file are in clear text by default, but can be hashed to ensure protection. To automatically hash passwords after restarting the runtime, open and edit the <Runtime_Home>/etc/org.apache.karaf.jaas.cfg file.
The encryption.enabled property must be set to true. Furthermore, the encryption algorithm in the property encryption.algorithm should be set to something like SHA-256. The configuration file should look like this:
Figure 3: org.apache.karaf.jaas.cfg
The properties file solution is fine to quickly get some identity information defined, for example, when testing, but does not scale well. Furthermore, in many cases, identity information is stored in some LDAP server or Active Directory. Talend runtime provides an LDAP module to validate identity information using the information stored in LDAP or Active Directory.
The security.zip file includes the OSGi Blueprint bundle file authentication/LDAP-login-config.xml, which activates the LDAP login module when deployed into the runtime, and the authentication/org.talend.esb.jaas.ldap.cfg file to configure the login module and adapt it to the specific environment.
Table 1: LDAP login parameters for authentication
Name |
Description |
connection.url |
The LDAP URL, for example, ldap://ldap-host.example.com:389 |
connection.username |
Username to connect to LDAP, for example, cn=admin,dc=example,dc=com. The user requires read access to the part of the Directory Information Tree (DIT) where the user information is stored. |
connection.password |
User password to connect to LDAP. |
user.base.dn |
The LDAP base DN used to look up users, for example, ou=users,dc=example,dc=com. |
user.filter |
The LDAP filter used to locate the user applied to the subtree specified in user.base.dn, for example, (uid=%u) where %u will be replaced by the username. |
user.search.subtree |
If “true”, the user lookup will be recursive (sub). If “false”, the user lookup will be performed only at the first level (one). |
authentication |
Specifies the authentication method used when binding to the LDAP server. The default is simple, where a username and password is required. To enable anonymous, set to none and leave username and password blank. |
Edit the configuration file and adapt the settings to your environment. Table 1 provides a list of the properties to be configured and their meaning. The configuration file also contains the description for each property. The sample values in Table 1 and in the configuration file correspond to the Directory Information Tree (DIT) structure shown in Figure 4.
When finished with editing, copy the configuration file to the directory <Runtime_Home>/etc.
Deploy the authentication/LDAP-login-config.xml file into the Talend Runtime. The easiest way to deploy it is to copy the file to the deploy folder of the Talend Runtime, <Runtime_Home>/deploy.
Often the LDAP server or Active Directory requires the use of Secure LDAP (LDAPS) to ensure the communication is encrypted. The LDAP login module also supports LDAPS, but in addition requires a keystore containing the X.509 certificate of the LDAP server. The location of the keystore and required passwords are specified in the OSGi Blueprint bundle file, which contains the specification of the LDAP login module.
security.zip includes an extended version of the OSGi Blueprint bundle file ldaps/LDAP-login-config.xml with an additional section for the keystore, and ldaps/org.talend.esb.jaas.ldap.cfg with additional properties to configure the keystore-related parameters. Table 2 provides a list of the additional properties that need to be configured.
Name |
Description |
truststore.path |
Absolute path to a trust store containing required Active Directory certificates, for example, /opt/talend/7.2.1/runtime/etc/keystores/ldaptruststore |
truststore.password |
Password for the specified trust store. |
The article Authorization for REST service based routes with HTTP Basic Authentication describes how to enable authorization for a Route, and how to configure the user properties file. If LDAP is used for identity validation, the user information is stored in the roles for authorization. Figure 5 shows a container object groups and three child objects that represent the different groups or roles. Users belonging to a group are referenced through the member attribute, where the fully qualified distinguished name of the user is specified.
Figure 5: Sample LDAP for groups/roles
security.zip includes an extended version of the OSGi Blueprint bundle file authorization/LDAP-login-config.xml with additional configuration parameters to retrieve roles, and authorization/org.talend.esb.jaas.ldap.cfg with additional properties to configure the parameters. Table 3 provides a list of the additional properties that need to be configured.
Table 3: LDAP login parameters for authorization
Name |
Description |
role.base.dn |
The LDAP base DN used to looking for groups/roles, for example, ou=groups,dc=example,dc=com |
role.filter |
The LDAP filter used to look for user’s role, for example, (member=%fqdn) where %fqdn will be replaced by the user's full qualified distinguished name |
role.name.attribute |
The LDAP role attribute containing the group/role string used by Talend Runtime, for example, cn |
Currently, the sample Route provided in article Authorization for REST service based routes with HTTP Basic Authentication and also included in security.zip does not work for Talend version 7.1, 7.2, and 7.3.
In the default configuration of the Talend Runtime, both HTTP and HTTPS are enabled, and the private key used for HTTPS is a well-known standard key provided by Talend. If you use HTTPS, replacing the key is a must.
org.osgi.service.http.enabled=false
The file should now look like:
The Server HTTP Configuration section of the Talend ESB Container Administration Guide provides documentation for the HTTP settings in org.ops4j.pax.web.cfg.
After the Remote Engine microservice is invoked over 100+ times intensely, the microservice becomes unresponsive, with threads getting stuck and not progressing. There are a total of 12 microservices deployed.
The customer has to un-deploy the affected microservice and redeploy it to restore functionality. However, after another period of intensive service usage, the same problem reoccurs.
Log trace and thread dump
http-nio-5081-exec-1
Stack Trace is:
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(java.base@11.0.24/Native Method)
at java.net.SocketOutputStream.socketWrite(java.base@11.0.24/SocketOutputStream.java:110)
at java.net.SocketOutputStream.write(java.base@11.0.24/SocketOutputStream.java:150)
at org.apache.logging.log4j.core.net.TcpSocketManager.writeAndFlush(TcpSocketManager.java:253)
at org.apache.logging.log4j.core.net.TcpSocketManager.write(TcpSocketManager.java:219)
- locked <0x00000006c72c6ed0> (a org.apache.logging.log4j.core.net.TcpSocketManager
As Microservice is an always-on daemon process that will occupy a dedicated log collector worker thread, with the Microservice deployment increasing and considering there will be other concurrent task executions that consuming more log collector workers, the default ms.worker.thread.number=10
will not meet the performance requirement and it should be adapted with a bigger size to avoid the race condition of the log collection process.
Please review the thread dump and make some adjustments to the configuration:
ms.worker.thread.number
in /etc/org.talend.ipaas.rt.dsrunner.log4jsocket.collector.cfg
file from 10 to 20.ms.worker.thread.number
should consider the total number of microservice deployed + concurrent tasks in high-load situations. For more information about Data Service Runner configuration files, please refer to Talend Help Documentation
Configuring-data-service-runner