Recent versions of Qlik connectors have an out-of-the-box value of 255 for their DefaultStringColumnLength setting. 

This means that, by default, any strings containing more than 255 characters is cut when imported from the database.

To import longer strings, specify a higher value for DefaultStringColumnLength.

This can be done in the connection definition and the Advanced Properties, as shown in the example below. 

The maximum value that can be set is 2,147,483,647.

Environment

• Qlik Connectors
• Built-in Connectors Qlik Sense Enterprise on Windows November 2024 and later

This article outlines how to handle DDL changes on a SQL Server table as part of the publication. 

Resolution

The steps in this article assume you use the task's default settings: full load and apply changes are enabled, full load is set to drop and recreate target tables, and DDL Handling Policy is set to apply alter statements to the target.

To achieve something simple, such as increasing the length of a column (without changing the data type), run an ALTER TABLE command on the source while the task is running, and it will be pushed to the target.

For example: alter table dbo.address alter column city varchar(70) 

To make more complicated changes to the table, such as: 

• Changing the Allow Nulls setting for a column
• Reordering columns in the table
• Changing the column data type
• Adding a new column
• Changing the filegroup of a table or its text/image data

Follow this procedure: 

1. Stop the task
2. Remove the table from the task
3. Remove the table from the publication.
   Some changes do not require this, and some do. Removing it from the publication will work in all cases. 
   1. Right-click the publication (starts with AR_) and select Properties
   2. Click to select the Articles tab
   3. Uncheck the box next to the table and click OK
   4. Copy all the data to a temp table
4. Recreate the table with the needed modifications
5. Copy the data back into the table
6. Add the table back to the task and resume. This will automatically add the table back to the publication and reload the table on the target. Reference: Saving changes is not permitted error message in SSMS

Environment

• Qlik Replicate

When connecting to Microsoft OneDrive using either Qlik Cloud Analytics or Qlik Sense Enterprise on Windows, shared files and folders are no longer visible.

Resolution

While the endpoint may intermittently work as expected, it is in a degraded state until November 2026. See drive: sharedWithMe (deprecated) | learn.microsoft.com. In most cases, the API endpoint is no longer accessible due to the publicly documented degraded state.

Qlik is actively reviewing the situation internally (SUPPORT-7182).

However, given that the MS API endpoint has been deprecated by Microsoft, a Qlik workaround or solution is not certain or guaranteed.

Suggested Workaround

Use a different type of shared storage, such as mapped network drives, Dropbox, or SharePoint, to name a few. 

Cause

Microsoft deprecated the /me/drive/sharedWithMe API endpoint.

Internal Investigation ID(s)

SUPPORT-7182

Environment

• Qlik Cloud
• Qlik Sense Enterprise on Windows

To check the location where your tasks are running on, access your task and refer to the right-hand side where artifact details can be found under Configuration.

In this instance, the Binary type is displayed as "Talend Runtime" because it is a REST type artifact, indicating that it will be deployed and executed on Talend Runtime.

If you are using Remote Engine to execute your task, it will be displayed in the "Processor" section under Configuration. In this instance, it demonstrates that Remote Engine version 2.13.13 will be employed to run the task.

Environment

• Talend Cloud 

After upgrading from Talend 7.3 to Talend 8 (R2025-10), it includes the Camel 4 update, and a change in the cSetHeader component. In previous Talend 7.x, JSONPath evaluation generated a correctly formatted JSON header, however, in Talend 8, the same configuration produces a non-JSON representation where double quotes are stripped from the resulting object.

For example

Source Input

{ "META": {"Correlation_ID": "B","BGM_ACTION_CODE": “A”},
 "STATUS": {"ADD_R_UPDATE_SUCCEEDED": “Y”},
 "ERR": [ {"field": "ADD","keyword": "custOrgMustHaveAdd", "entity": “ORGANI” }]
}
 

cSetHeader Component Setting

Output in  Talend 7 (Correct)

[{"field":"ADD","keyword":"custOrgMustHaveAdd","entity":"ORGANI"}]

cSetHeaderOutputTalend7

Output in Talend 8 (Invalid JSON format with out Double Quotes)

[{field=ADD, keyword=custOrgMustHaveAdd, entity=ORGANI}]

cSetHeaderOutputTalend8

Resolution

Install the latest R2025-11 (or newer) update for both Talend Studio and Talend Runtime, where the corrected behavior has been implemented.

Cause

This behavior stems from changes in the underlying JSON processor, specifically, the shift from json-smart to JacksonJsonProvider, which modifies how JSONPath results are serialized.

The issue aligns with the upstream Camel defect jira issue reported here:
CAMEL-16389 | issues.apache.org 

Environment

Talend ESB 

Extracting data from SAP BW InfoProvider / ASDO with two values in a WHERE clause returns 0 lines.

Example:

The following is a simple standard ADSO with two infoObjects (‘0BPARTNER’,  ‘/BIC/TPGW8001’,) and one field (‘PARTNUM’). All are CHAR data type. 

In the scripts we used [PARTNUM] = A, [PARTNUM] = B in the WHERE clause.

However 

• If you use only one field value in the WHERE clause, it works as expected:
  From GTDIPCTS2
Where ([PARTNUM] = A); 
• If you use two values infoObject [TPGW8001] instead of the field [PARTNUM] in the WHERE clause, it also works as expected:
  From GTDIPCTS2
Where ([TPGW8001] = “1000”,  [TPGW8001] = “1000”); 

Environment

• Qlik Cloud / Direct Data Gateway version 1.7.4
• SAP BW Connector

Resolution

Upgrade to Direct Data Gateway version 1.7.8.

Cause

Defect SUPPORT-5101.

Internal Investigation ID(s)

SUPPORT-5101

When using Google Cloud Pub/Sub as a target and configuring Data Message Publishing to Separate topic for each table, the Pub/Sub topic may be unexpectedly dropped if a DROP TABLE DDL is executed on the source. This occurs even if the Qlik Replicate task’s DDL Handling Policy When source table is dropped is set to Ignore DROP.

Environment

• Qlik Replicate versions before 2025.5 SP3

Resolution

This issue has been fixed in the following builds:

• 2025.5.0.454 (SP3)
• 2025.11 GA

To apply the fix, upgrade Qlik Replicate to one of the listed versions or any later release.

Cause

A product defect in versions earlier than 2025.5 SP3 causes the Pub/Sub topic to be dropped despite the DDL policy configuration.

Internal Investigation ID(s)

• 00401331

Qlik Talend Administration Center shows Process Message Port WARNING and JobServer is not listening on port 8555 (default port).

ProcessMessagePortWarning

Resolution

1. Locate the file $Job_Server_Home/agent/conf/TalendJobServer.properties
2. Change the value for parameter org.talend.remote.jobserver.server.TalendJobServer.ENABLED_PROCESS_MESSAGE= true  

   org.talend.remote.jobserver.server.TalendJobServer.ENABLED_PROCESS_MESSAGE= true

   PROCESSMESSAGETRUE

3. Re-start the JobServer and the Process message port should show as green

   ProcessMessagePortGreen

Cause

This issue is caused by that with Talend Administraion Center Patch 8.0.2.20250129_1317, the values for the parameter org.talend.remote.jobserver.server.TalendJobServer.ENABLED_PROCESS_MESSAGE is set as false in JobServer configuration file TalendJobServer.properties

PROCESSMESSAGEFalse

Environment

Talend Administration Center 

Qlik Talend Administration Center may report Jobserver with TPS-6012 installed as unavailable or misconfigured status.

MisconfiguredStatus

PatchVersion

In  Error Log

Exception in thread "t_ForceUpdateCacheForServer_11" java.lang.NullPointerException: Cannot invoke "org.talend.utils.ProductVersion.compareTo(org.talend.utils.ProductVersion)" because "productVersion" is null
        at org.talend.administrator.scheduler.business.wrapper.jobserver.JobServerWrapper.isSudo(JobServerWrapper.java:316)

Resolution

1. Copy the branding.properties from the patch directory into <jobserver_home> where you have previously copied the bat/sh files from patch.  

   BrandingProperties

2. Update branding properties and change 

   patchName=Patch_20251028_TPS-6012_v1-8.0.1

   to

   patchName=8.0.1.Patch_20251028_TPS-6012_v1-8.0.1

3. Restart Talend JobServer to take effort

Cause

The format of the version information provided by JobServer has changed.

Internal Investigation ID(s) 

SUPPORT-7132

Environment

• Talend Administration Center 
• Talend Data Integration 

This article explains how to extract changes from a Change Store by using the Qlik Cloud Services connector in Qlik Automate and how to sync them to a database.

The example will use a MySQL database, but can easily be modified to use other database connectors supported in Qlik Automate, such as MSSQL, Postgres, AWS DynamoDB, AWS Redshift, Google BigQuery, Snowflake.

The article also includes:

• Two automation examples you can download and import (see Qlik Automate: How to import and export automations):
  • Automation Example to Extract Change Store History to MySQL Incremental.json
  • Automation Example to Bulk Extract Change Store History to MySQL Incremental.json
• Configuration instructions for the examples

Content

• Prerequisites
• Creating the automation
• Insert changes in MySQL one by one
• Making this incremental
• Bulk updates
• Attachment configuration instructions
• Bonus!
• Replace field names
• User email instead of user id
• Triggering the automation from a sheet

Prerequisites

1. A working Write Table with a set of editable columns and some example values already stored in it. More information about the Write Table chart can be found in Write Table | help.qlik.com.
2. A MySQL (or similar) database table with columns that match your editable columns.
3. Setting up the database, extend your database with metadata fields:
   • userId: to store the user ID of the user who made a change
   • updatedAt: to store the datetime when a change was saved

Here is an example of an empty database table for a change store with:

• a single PK “productId”
• editable columns are “AmountToOrder”, “Priority”, and “Note”.
• additional columns “userId” and “updatedAt”, which will be used to log user activity and merge separate changes into the same record

Creating the automation

1. Create a new automation. See Qlik Automate for details.
2. Add the List Change Store History block from the Qlik Cloud Services connector.
3. Configure this block with the change store ID. You can copy this from the write table chart.
4. Perform a manual run of the automation to make sure some records are returned. The List Change Store History block will return a list of objects for each cell that contains one or more changes. Every object includes the combination of primary key(s), the editable column for that cell, and a list of all values belonging to that cell.
5. You have two options on how to perform this sync:
   • Insert changes one by one
   • Insert changes in bulk (this option is more complex but also more performant)

Insert changes in MySQL one by one

1. Add a Loop block (A)inside the List Change Store History block and configure it to loop over the list of changes inside each object returned by the List Change Store History block (B) :
   
   
   

   
   

2. Search for the MySQL connector (A) in the automation block library and drag the Upsert Record block inside the Loop block (B) :
   
   
   
   

   

3. Create a new connection (C) to your MySQL database in the Connection tab of the Upsert Record block and activate it for the block by clicking it once created.
4. Configure the Upsert Record block as follows
   1. Table: the database table you have created for the Write Table
   2. Where: the key:value mappings for the granularity on which you want to save the changes. In this example, it is a combination of the primary key (productId) with userId and updatedAt.
      
      
      1. productId (primary key): this comes from the cellKey in the List Change Store History block.
         
         
         

          

      2. userId and createdAt: these are defined for each change, so they should be retrieved from the item in the Loop block.
         
         The userId maps to the createdBy parameter.
         
         
         

          

         
         
         
   3. Record: this is the key:value mapping for the individual change that should get updated. The Key maps to the columnName that is returned by the List Change Store History block, the Value maps to the cellValue parameter that is returned in the Loop block:
      
      
      
      
5. This is what your automation will look like now:
   
   
   
   

Run the automation manually by clicking the Run button in the automation editor and review that you have records showing in the MySQL table:

Making this incremental

Currently, there is no incremental version yet for the Get Change Store History block. While this is on our roadmap, the automation from this article can be extended to do incremental loads, by first retrieving the highest updatedAt value from the MySQL table. The below steps explain how the automation can be extended:

1. Add a Do Query block from the MySQL connector to the automation and configure the query as follows:
   

   SELECT MAX(updatedAT) FROM <your database table>

   
   
   
   
2. Run a test run of the automation without the other blocks attached to verify the result in the Do Query block’s History tab:
   
   
   
   
3. Add a Condition block (A) to the automation and configure it to evaluate the MAX(updatedAt) (B) field from the Do Query block.
   
   Because the Do Query block returns the value as part of a list, the automation editor prompts you to specify which item of the list you want to use. Use the default version option Select first item from list (C).
   
   
   
   

   

4. Configure the operator in the Condition block to is not empty. If an updatedAt timestamp is found, the Yes part of the Condition block will be executed. If no timestamp is found, the No part will be executed.
   
   
   
   
5. Add a Variable block (A)to the Yes part of the Condition block and create a new variable of type String named Filter. The setting is accessed from the Manage variables (B) button in the Variables block.
   
   
   

   
   

6. Add an operation to the Variable block to Set value of Filter and type updatedAt gt “ in the input field.
   
   
   
   Click in the input field to add a reference to the timestamp in the Do Query block, mirroring the Condition block configuration.
   
   
   
   After the mapping is added, append it with an additional double quote character:
   
   
   
   
7. Right-click the Variable block and duplicate it, then add the duplicated block to the No part of the Condition (A).
   
   Remove the Set value of Filter step and replace it with the Empty Variable (B) operation.
   
   Right-click each Variable block and add a comment explaining the respective function. 
   
   
   

    

8. Reattach the original automation after the Condition block. 
   
   Verify that it is attached after the block and not inside the Yes or No sections. 
   
   
   

    

9. In the List Change Store History block (A), map the Filter variable to the Filter parameter (B) :
   
   
   

    

10. Run the automation and confirm it only picks up new changes on new runs.

Bulk updates

The solution documented in the previous section will execute the Upsert Record block once for each cell with changes in the change store. This may create too much traffic for some use cases. To address this, the automation can be extended to support bulk operations and insert multiple records in a single database operation.

The approach is to transform the output of the List Change Store History block from a nested list of changes into a list of records that contains the changes grouped by primary key, userId, and updatedAt timestamp.

See the attached automation example:  Automation Example to Bulk Extract Change Store History to MySQL Incremental.json. 

1. Drag the Loop and Upsert Record block outside the List Change Store History loop, but do not delete them:
   
   
   
   
2. Add a Variable block inside the List Change Store History block loop (A) and create a new variable partialChangeRecord of type Object (B).
   
   This variable will be used to map each cell value to the primary key(s), userId and updatedAt timestamp.
   
   
   
   
3. Add the first operation to the Variable to Empty it (A).
   
   This is important to ensure that for every item in the loop, we start with an empty partialChangeRecord variable.
   
   Next, add a Set key/values operation to set the primary key(s) (B).
   
   In our example, we have a single primary key, productId, but if you have multiple fields, you should add them one by one.
   
   
   
   Set it to the cellKey.rowKey parameter returned in the output from List Change Store History
   
   
   

    

4. Drag the Loop block back into the automation and attach it to the Variable block.
   
   Disconnect the Upsert Record block.
   
   
   
   
5. Add a new Variable block inside the Loop block and configure the Variable parameter to the partialChangeRecord object.
   
   Now set additional Key/values in the variable for the primary userId, updatedAt, and the cellValue:
   
   
   1. userId: createdBy parameter returned in the loop
   2. updatedAt: updatedAt parameter returned in the loop
   3. For the Cell value, set the key to the columnName returned by the List Change Store History block and set the Value to the cellValue returned from the Loop block:
      
      
      

       

   4. uniqueKey: add a fourth keyValue pair uniqueKey.
      
      This will be used later to merge various cell changes into a single record.
      
      This combines the primary key(s), userId, and updatedAt timestamp, each separated by a pipe (|) symbol.
      
      Click the input field, map the first parameter, then type a | -symbol, and click the input field again to map the next parameter:
      
      
      

       

      
      

       

6. Add another Variable block inside the Loop block (A) and create a new list type variable listOfPartialChanges.
   
   Add an Add item operation to add the partialChangeRecord variable (B) to this list.
   
   
   

    

7. Add a Merge Lists block after the List Change Store History block. 
   
   The Merge Lists block will be used to merge the partial change records into full records for each change.
   
   Configure both List parameters in the Merge Lists block to use the listOfPartialChanges variable.
   
   
   

    

8. Go to the Settings tab of the Merge Lists block and apply this configuration:
   
   
   1. Item merge strategy: Merge list 1 item and list 2 item in one new item (default)
   2. List 1 unique key: uniqueKey
   3. List 2 unique key: uniqueKey
   4. On duplicate unique key: Merge item from list 2 into item from list 1
   5. When a property exists on both lists: Keep value from list 1 (default) 
      
      Due to an ongoing defect, this parameter is only available after refreshing the automation editor. As the parameters use the default values, this should not impact you.
      
      
      
      
9. Perform a manual run of the automation to verify that the Merge Lists block output is merging.
   
   You will notice that this list contains many duplicates.
   
   
10. Add a Deduplicate List block (A) and configure the List parameter to use the output from the Merge Lists block, then set the key to uniqueKey (B).
    
    
    

     

11. Perform another manual run and confirm that the Deduplicate List block now only contains unique entries.
12. The output still contains the uniqueKey parameter that is not compatible with the database.
    
    There are two options: either extend the database or remove uniqueKey.
    
    To remove it, add a Transform List block and set the Input List parameter to the Deduplicate list block:
    
    
    

     

13. Configure the Fields in output list parameter from the Transform List block.
    
    To make this easier, it is best to have example data from a manual run. If you haven’t performed a manual run yet, do one now.
    
    Click the Add Field button and configure new fields:
    
    
    1. For the Field parameter, click the input field and start by mapping your primary keys
       
       

       

    2. For the Value parameter, do the same and select the corresponding parameter in the Item in input list option
       
       
       
       
    3. Repeat this for all other items in the list except for the uniqueKey
       
       

       

14. Add a Loop batch block (A) to the automation and configure it to loop over the Transform List block output in batches of 50 (B).
    
    Adjust this batch size depending on your database and the number of editable columns.
    
    

    

15. Add an Insert Bulk block from the MySQL connector within the Loop Batch block loop.
    
    Configure it to use your database table variable (or hardcode your table name) and set the Values parameter to the Batch in Loop Batch parameter.
    
    
    
    
16. Run the automation and confirm that the database gets updated.
    
    
    
    
17. Optionally, collapse all empty loop blocks to clean up the automation and provide comments to explain what the blocks and functions do.
    
    This will help you understand this automation when you revisit it in the future.
    
    To add a comment, right-click on the block and click Edit comment.
    
    
    

     

Attachment configuration instructions

The provided automations will require additional configuration after being imported, such as changing the store, database, and primary key setup. 

Automation Example to Extract Change Store History to MySQL Incremental.json

1. Variable - databaseTable -> configure with the name of your database table
2. Variable - changeStoreId -> configure with your change store ID
3. Upsert Record - MySQL -> replace the productId with your primary key, add additional primary keys if necessary

Automation Example to Bulk Extract Change Store History to MySQL Incremental.json

1. Variable - databaseTable -> configure with the name of your database table
2. Variable - changeStoreId -> configure with your change store id
3. Variable - partialChangeRecord -> replace the productId with your primary key, add additional primary keys if necessary
4. Variable - partialChangeRecord in Loop block -> Update the uniqueKey field by replacing the productId with your primary key, add additional primary keys if necessary
5. Transform List -> replace the productId with your primary key, add additional primary keys if necessary

Bonus!

Replace field names

If field names in the change store don't match the database (or another destination), the Replace Field Names In List block can be used to translate the field names from one system to another.

1. Search the Replace Field Names List block and add it to your automation.
   
   
   
   
2. Provide the translations for the field names that need to be changed to match the destination system.
   
   
   
   

User email instead of user id

To add a more readable parameter to track the user who made changes, the Get User block from the Qlik Cloud Services connector can be used to map User IDs into email addresses or names.

1. Search the Get user block (A) in the Qlik Cloud services connector and add it to your automation.
   
   Configure it to use the createdBy parameter (B).
   
   
   
   
2. Update the Upsert Record block to use the output from Get User.

   A user's name might not be sufficient as a unique identifier. Instead, combine it with a user ID or user email.

   
   

    

Triggering the automation from a sheet

Add a button chart object to the sheet that contains the Write Table, allowing users to start the automation from within the Qlik app. See How to run an automation with custom parameters through the Qlik Sense button for more information.

Environment

• Qlik Cloud Analytics
• Qlik Automate

Usually, to generate a key file for key-pair authentication to the snowflake, it needs to use key pair authentication and key pair rotation.

For more information, please refer to documentation about: key-pair-auth | docs.snowflake.com

How to create a key file for Qlik Talend Data Catalog that use for key-pair authentication to the snowflake?

How To

Since Qlik Talend Data Catalog only support PKCS#8 version 1 encryption with PBE-SHA1-3DES (-v1 option) for the moment, please use a sample command below to generate the keystore file via OpenSSL:

openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -v1 PBE-SHA1-3DES -out rsa_key.p8

Internal Investigation ID(s)

TALMM-6182

Environment

#Talend Data Catalog

It encounters the following loading error with Snowflake destination:

Insufficient privileges to operate on database '<DATABASE_NAME>'

Resolution

Follow up documentation about  connecting-a-snowflake-data-warehouse-to-stitch#create-database-and-use(Qlik Stitch Documentation) to create the database and user for Snowflake.

Further Troubleshooting

If you still see the error after following the setup guide,  please try to use below steps 

1. Verify Database Ownership

   SHOW DATABASES LIKE '<DATABASE_NAME>';

2. Check Current Grants

   SHOW GRANTS TO ROLE STITCH_ROLE;

3. Ensure Default Role

   ALTER USER STITCH_USER SET DEFAULT_ROLE = STITCH_ROLE;

If any required privileges are missing, grant them using an ACCOUNTADMIN role:

GRANT USAGE ON DATABASE <DATABASE_NAME> TO ROLE STITCH_ROLE;
GRANT CREATE SCHEMA ON DATABASE <DATABASE_NAME> TO ROLE STITCH_ROLE;

Cause

This error occurs because the Stitch Snowflake user does not have the required privileges on the target database.

Snowflake access control is role-based. Even if a database exists and is accessible under another role (e.g., ACCOUNTADMIN), the Stitch role must explicitly be granted privileges to use and create objects within that database.

When the Stitch role does not own the database and lacks privileges such as USAGE, CREATE SCHEMA, or CREATE TABLE, the Stitch integration fails to load data and raises an insufficient privileges error.

Environment

Stitch 

A Job design is shown below, using a tSetKeystore component to set the keystore file in the preJob, followed by using a tMysqlConnection to establish a MYSQL connection. However, MYSQL fails to connect. 

Nevertheless, by changing the order of the components as demonstrated below, the MYSQL connection is successful.

Resolution

To address this issue, you can choose from the following solutions without altering the order of the tSetKeyStore and tMysqlConnection components.

Disable useSSL parameter

Isolate the MYSQL connection in a separate child Job

To isolate the MYSQL connection, create a separate subJob. Use a tRunJob to execute the subJob and ensure the 'Use an independent process to run subJob' option is checked. The Job design is outlined below. 

Cause

tSetKeyStore sets values for javax.net.ssl properties, thereby affecting the subsequent components. Most recent MySQL versions use SSL connections by default. Since the Java SSL environment has been modified, the MySQL JDBC driver inherits these changes from tSetKeyStore, which can potentially impact the connection.

Environment

• Talend Data Integration 

This article briefly introduces how to config on audit.properties to generate audit log locally.

How To

1. Stop running Talend Administration Center
2. Change audit.properties setting under <talend>/tac/apache-tomcat/webapps/tac/WEB-INF/classes 
3. Talend Administration Center provided 3 sink methods per audit.properties: socket, file and console, so you can use comma to combine 2 more sink methods, in the meanwhile, please make sure the appender.file.path using correct path

   ==audit.properties==
   log.appender=file,console
   appender.file.path=c:/tmp/audit/audit.json

4. Start Talend Administration Center to take effect

Environment

Talend Administration Center 

Users assigned the Operator and Integration Developer roles may encounter the following error when attempting to connect to Talend Studio and fetch the license from Talend Management Console:

401 Authentication credentials were missing or incorrect

This issue occurs even though the user has the necessary roles to access Talend Studio. The root cause is typically missing Studio-specific permissions within the Operator role configuration.

Resolution

To resolve the issue, update the Operator role permissions in Talend Management Console:

1. Log in to Talend Management Console.
2. Navigate to: Users & Security > Roles
3. Select the Operator role.
4. Go to Permissions and click Edit Item.
5. Ensure the following permissions are selected:
   1. Studio
   2. Studio - Develop
6. Save the changes.
7. Retry connecting to Talend Studio. The 401 Authentication error should no longer appear, and the license will be fetched successfully.

Cause 

The Operator role does not have the required Studio and Studio - Develop permissions enabled. Without these permissions, authentication fails when Talend Studio attempts to validate credentials against Talend Management Console.

Symptoms

• Error message: 401 Authentication credentials were missing or incorrect
• Unable to connect to Talend Studio
• License cannot be fetched from Talend Management Console
• User has both Operator and Integration Developer roles assigned

Environment

• Talend Studio 
• Talend Cloud 

In Talend ESB Runtime environment,  it fails to deploy new routes after nexus/jrog password update, although deploying and undeploying existing routes still work.

Resolution

Ensure the correct login/password credentials are configured in ${M2_HOME}/conf/settings.xml

Cause

When updated the Nexus/JFrog credentials and modified the configuration at the known locations, the issue still persists.

It indicates that additional locations are still managing the authentication for JAR downloads.
Since org.ops4j.pax.url.mvn.cfg relies on ${M2_HOME}/conf/settings.xml for Maven authentication, that file also needed to be reviewed.

Check locations:

runtime\data\cache\org.eclipse.osgi\19\data\state.json

runtime\etc\org.ops4j.pax.url.mvn.cfg (org.ops4j.pax.url.mvn.repositories)

Location initially missed:

${M2_HOME}/conf/settings.xml and ${M2_HOME} is the path to the Maven installation directory.

Environment

Stitch 

A Job design is presented below:

tSetKeystore: set the Kafka truststore file.

tKafkaConnection, tKafkaInput: connect with Kafka Cluster as a Consumer and transmits messages.

However, while running the Job, an error exception occurred under the tKafkaInput component.

org.apache.kafka.common.KafkaException: Failed to construct kafka consumer 

Resolution

Make sure to execute the tSetKeyStore component prior to the Kafka components to enable the Job to locate the certificates required for the Kafka connection. To achieve this, connect the tSetKeystore component to tkafkaConnection using an OnSubjobOK link, as demonstrated below:

For more detailed information on trigger connectors, specifically OnSubjobOK and OnComponentOK, please refer to this KB article: What is the difference between OnSubjobOK and OnComponentOK?.

Environment

• Talend Data Integration 7.3.1, 8.0.1

We often see new users implement Qlik Stitch in under five minutes. While the user experience for adding a data source and a destination is simple, there is a lot of complexity behind the scenes.

Let’s pull the curtains back and see how each record gets from its source to its destination through Qlik Stitch’s data pipeline.

The journey typically begins in a SaaS application or database, where it’s either pushed to Stitch via our API or through a webhook, or pulled on a schedule by the Singer-based replication engine that Qlik Stitch runs against data sources like APIs, databases, and flat files.

The image below outlines the internal architecture. For a full view, click on the image or download it.

The data’s next stop is the Import API, a Clojure web service that accepts JSON and Transit, either point-at-a-time or in large batches. The Import API does a validation check and an authentication check on the request’s API token before writing the data to a central Apache Kafka queue.

At this point, Qlik Stitch has accepted the data. Qlik's system is architected to meet our most important service-level target: don’t lose data. To meet this goal, we replicate our Kafka cluster across three different data centers and require each data point to be written to two of them before it is accepted. Should the write fail, the requester will try again until it’s successful.

Under normal conditions, data is read off the queue seconds later by the streamery, a multithreaded Clojure application that writes the data to files on S3 in batches, separated according to the database tables the data is destined for. We have the capacity to retain data in Kafka for multiple days to ensure nothing is lost in the event that downstream processing is delayed. The streamery cuts batches after reaching either a memory limit or an amount of time elapsed since the last batch. Its low-latency design aims to maximize throughput while guarding against data loss or data leaking between data sets.

Batches that have been written to S3 enter the spool, which is a queue of work waiting to be processed by one of our loaders. These Clojure applications read data from S3 and do any processing necessary (such as converting data into the appropriate data types and structure for the destination) before loading the data into the customer’s data warehouse. We currently have loaders for Redshift, Postgres, BigQuery, Snowflake, and S3. Each is a separate codebase and runtime because of the variation in preprocessing steps required for each destination. Operating them separately also allows each to scale and fail independently, which is important when one of the cloud-based destinations has downtime or undergoes maintenance.

All of this infrastructure allows us to process more than a billion records per day, and allows our customers to scale their data volumes up or down by more than 100X at any time. Qlik Stitch customers don’t need to worry about any of this, however. They just connect a source, connect a destination, and then let Qlik Stitch worry about making sure the data ends up where it needs to be.

For additional information, reference the official Stitch documentation on Getting Started.

Environment

• Qlik Stitch