Executing the Qlik Migration Tools leads to the following error on line 185 of 7_migrateapps.ps1:

Error: operation unsupported for server-type "cloud" (Qlik Cloud). Supported server-types are: ["windows"]

Example:

qlik context use $QlikWindowsContext | out-null

qlik qrs app export create $appid --skipdata $SS --exportScope all --output-file $exportAppFile

Resolution

Verify you have followed the required step-by-step configuration documented in Using qlik-cli with Qlik Sense Enterprise client-managed Repository API (QRS).

Specifically, do not forget to execute this cmd to add the Qlik Sense Enterprise on Windows server to qlik-cli. See Configure a context in qlik-cli for details.

Example:

##qlik context create QSEoW --server <server name> --server-type windows --api-key <JWT Token>

Cause

Possible causes:

• failed to specify --server-type windows
• executed the cmd against Qlik Cloud, not Qlik Sense Enterprise on Windows

Lines 184 and 185 test the connection as documented in Qlik-CLI: Test the connection and point to a misconfiguration of the context. 

Environment

• Qlik Cloud
• Qlik Sense Enterprise on Windows

From R2024-05, Java 17 will become the only supported version to start most Talend modules, enforcing the improved security of Java 17 and eliminating concerns about Java's end-of-support for older versions. In 2025, Java 17 will become the only supported version for all operations in Talend modules.

Starting from v2.13, Talend Remote Engine requires Java 17 to run. If some of your artifacts, such as Big Data Jobs, require other Java versions, see Specifying a Java version to run Jobs or Microservices.

Content

• Prerequisites
• Procedure
• Windows
• Linux
• MAC OS
• Multiple JDK versions
• Studio
• Remote Engine
• ESB - Runtime
• Studio
• Talend Administration Center (TAC)
• Talend JobServer
• CICD
• Windows Users
• Linux Users
• Jenkins Users
• Additional Notes
• Specifying a Java version to run Jobs or Microservices

Prerequisites

Procedure

Windows

For Windows users, please follow the JDK installation guide (docs.oracle.com).

Linux

For Linux users, please follow the JDK installation guide (docs.oracle.com).

MAC OS

For MAC OS users, please follow the JDK installation guide (docs.oracle.com).

Multiple JDK versions

When working with software that supports multiple versions of Java, it's important to be able to specify the exact Java version you want to use. This ensures compatibility and consistent behavior across your applications. Here is how you can specify a specific Java version on the following products (such as build servers, shared application server, and similar):

Studio

For Studio users who are using multiple JDKs, please follow the appropriate instructions listed above and follow the proceeding additional steps:

1. Backup and edit the <Studio Home>\Talend-Studio-win-x86_64.ini file
2. Prepend:
   

   -vm
   <JDK17 HOME>\bin\server\jvm.dll

Remote Engine

For Remote Engine (RE) users who are using multiple JDKs, please follow the appropriate instructions listed above and follow the proceeding additional steps.

1. Backup and edit the <RE HOME>/etc/talend-remote-engine-wrapper.conf file
2. Modify the set.default.JAVA_HOME= property to point to the <JDK 17 HOME> path.
   
   Note 1: If Remote Engine is not installed as a service, the JDK file will be set in the <RE HOME>/bin/setenv file.
   
   Note 2: When it comes to running Jobs or Microservices, you retain the flexibility to either use the default Java 17 version or choose older Java versions, through straightforward configuration of the engine.
   
   How to modify?
   
   Check the following etc folder based configuration and change it to installed jdk/jre path: 
   
   {
   org.talend.ipaas.rt.dsrunner.cfg--> ms.custom.jre.path
   org.talend.remote.jobserver.server.cfg--> org.talend.remote.jobserver.commons.config.JobServerConfiguration.JOB_LAUNCHER_PATH
   }

ESB - Runtime

For Runtime users who are using multiple JDKs, please follow the appropriate instructions listed above and follow the proceeding additional steps.

1. Backup and edit the <Runtime home>/etc/<TALEND-8-CONTAINER service>-wrapper.conf
2. Modify the set.default.JAVA_HOME=C:\<JDK 17 HOME> path
   

If Runtime is not running as a service:

1. Backup and edit the <Runtime home>/bin/setenv.sh
2. Modify the SET JAVA_HOME= <JDK 17 HOME> path
   
   

Studio

• Data Integration (DI):  After installing the 8.0 R2023-10 Talend Studio monthly update or a later one, if you switch the Java version to 17 and relaunch your Talend Studio with Java 17, you must enable your project settings for Java 17 compatibility.
  
  
  1. Go to Studio
  2. Go to File
  3. Edit Project properties
  4. Go to Build
  5. Go to Java Version
  6. Activate "Enable Java 17 compatibility"
     

     With the Enable Java 17 compatibility option activated, any Job built by Talend Studio cannot be executed with Java 8. For this reason, verify the Java environment on your Job execution servers before activating the option.

• Big Data Users:  Do not enable Java 17 compatibility unless your Spark Cluster supports Java 17.  

Talend Administration Center (TAC)

To use Talend Administration Center with Java 17, you need to open the <tac_installation_folder>/apache-tomcat/bin/setenv.sh file and add the following commands:

# export modules 
export JAVA_OPTS="$JAVA_OPTS --add-opens=java.base/sun.security.x509=ALL-UNNAMED --add-opens=java.base/sun.security.pkcs=ALL-UNNAMED"

Windows users use <tac_installation_folder>\apache-tomcat\bin\setenv.bat

Talend JobServer

Follow the steps below to configure the JobServer to use the new Java version.

1. Navigate to the JobServer Configuration:
   Go to the <JobServerRootDir>\conf directory, where <JobServerRootDir> is the path to your Talend JobServer installation.

2. Open the Configuration File for Editing:
   Locate the TalendJobServer.properties file and open it with a text editor of your choice.

3. Set the Java 17 Executable Path:
   Find the line dedicated to the Job launcher path within the file. You will modify this line to point to the Java 17 executable.

   • For Windows, if your Java installation path contains spaces, ensure to enclose the path in quotes.
     
     

     org.talend.remote.jobserver.commons.config.JobServerConfiguration.JOB_LAUNCHER_PATH="C:\\Program Files\\Java\\jdk-17\\bin\\java.exe"
     

   • For Linux or Mac OS, the path doesn’t require quotes.
      

     org.talend.remote.jobserver.commons.config.JobServerConfiguration.JOB_LAUNCHER_PATH=/usr/lib/jvm/java-17-openjdk/bin/java
     

   Replace the example paths with the actual path where Java 17 is installed on your system. Ensure to point directly to the Java executable within the bin directory of your JDK installation.

4. Save Your Changes:
   After editing, save the TalendJobServer.properties file.

5. Restart Talend JobServer:
   For the changes to take effect, restart your Talend JobServer.

After completing these steps, Talend JobServer will utilize Java 17 for executing Jobs, ensuring compatibility with the latest Java version supported by Talend modules.

CICD

Windows Users

For Java 17 users, Talend CICD process requires the following Maven options:

1. Backup and edit <Maven_home>\bin\mvn.cmd
2. Modify to:

   set "MAVEN_OPTS=%MAVEN_OPTS% --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/sun.security.x509=ALL-UNNAMED --add-opens=java.base/sun.security.pkcs=ALL-UNNAMED"

Linux Users

For Java 17 users, Talend CICD process requires the following Maven options:

1. Backup and edit <Maven_home>/bin/mvn
2. Modify to:

   export MAVEN_OPTS="$MAVEN_OPTS \ 
   --add-opens=java.base/java.net=ALL-UNNAMED \ 
   --add-opens=java.base/sun.security.x509=ALL-UNNAMED \ 
   --add-opens=java.base/sun.security.pkcs=ALL-UNNAMED"

Jenkins Users

1. Backup and edit the jenkins_pipeline_simple.xml 
2. Include the following in the Talend_CI_RUN_CONFIG parameter:
   

   <name>TALEND_CI_RUN_CONFIG</name>
   <description>Define the Maven parameters to be used by the product execution, such as:
       - Studio location
       - debug flags
   These parameters will be put to maven 'mavenOpts'.
   If Jenkins is using Java 17, add:
   --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/sun.security.x509=ALL-UNNAMED --add-opens=java.base/sun.security.pkcs=ALL-UNNAMED
   </description>

Additional Notes

Specifying a Java version to run Jobs or Microservices

Overview

Enable your Remote Engine to run Jobs or Microservices using a specific Java version.

By default, a Remote Engine uses the Java version of its environment to execute Jobs or Microservices. With Remote Engine v2.13 and onwards, Java 17 is mandatory for engine startup. However, when it comes to running Jobs or Microservices, you can specify a different Java version. This feature allows you to use a newer engine version to run the artifacts designed with older Java versions, without the need to rebuild these artifacts, such as the Big Data Jobs, which reply on Java 8 only.

When developing new Jobs or Microservices that do not exclusively rely on Java 8, that is to say, they are not Big Data Jobs, consider building them with the add-opens option to ensure compatibility with Java 17. This option opens the necessary packages for Java 17 compatibility, making your Jobs or Microservices directly runnable on the newer Remote Engine version, without having to go through the procedure explained in this section for defining a specific Java version. For further information about how to use this add-opens option and its limitation, see Setting up Java in Talend Studio.

Procedure

1. Stop the engine.
   
   
2. Browse to the <RemoteEngineInstallationDirectory>/etc directory.
   
   
3. Depending on the type of the artifacts you need to run with a specific Java version, do the following:
   
   For both artifact types, use backslashes to escape characters specific to a Windows path, such as colons, whitespace, and directory separators, while keeping in mind that directory separators are also backslashes on Windows.
   
   Example:
   

   c:\\Program\ Files\\Java\\jdk11.0.18_10\\bin\\java.exe

   • For Jobs, in the <RemoteEngineInstallationDirectory>/etc/org.talend.remote.jobserver.server.cfg file, add the path to the Java executable file.
     
     Example:
     

     org.talend.remote.jobserver.commons.config.JobServerConfiguration.JOB_LAUNCHER_PATH=c:\\jdks\\jdk11.0.18_10\\bin\\java.exe

   • For Microservices, in the <RemoteEngineInstallationDirectory>/etc/org.talend.ipaas.rt.dsrunner.cfg, add the path to the Java executable file.
     
     Example:
     

     ms.custom.jre.path=C\:/Java/jdk/bin

     Make this modification before deploying your Microservices to ensure that these changes are correctly taken into account.
     
     
4. Restart the engine.

For Java Option Command --add-opens, using SPACE or = depends on OS, JDK version or the place where you setup, there are 3 cases:
1. support both SPACE and =
2. support SPACE only
3. support = only
Example:
 --add-opens=java.base/java.net=ALL-UNNAMED
 --add-opens java.base/java.net=ALL-UNNAMED

Problem

Our Customer has over existing 200 big data spark jobs migrated from 7.3.1 to 8.0.1 and it gives "IndexOutOfBoundsException" after "use Dataset API in migrated components" checkbox is enabled in Spark Configuration.

Cause  

UsedatasetAPIinMigratedComponents

Error Log

!ENTRY org.eclipse.ui.workbench 4 2 2024-03-28 18:01:50.038 !MESSAGE Problems occurred when invoking code from plug-in: "org.eclipse.ui.workbench". !STACK 0 java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0 at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) at java.base/java.util.Objects.checkIndex(Objects.java:372) at java.base/java.util.ArrayList.get(ArrayList.java:459) at org.talend.designer.bigdata.avro.AvroRecordStructGenerator.firstSchemaFieldHasAnAsterisk(AvroRecordStructGenerator.java:256) at org.talend.designer.bigdata.avro.AvroRecordStructGenerator.generateRecordStructForDataset(AvroRecordStructGenerator.java:242) at org.talend.repository.spark.editor.SparkDataProcess.generateRecordStructForConnection(SparkDataProcess.java:207) at org.talend.designer.bigdata.common.BigDataDataProcess.buildRecordStructsForConnections(BigDataDataProcess.java:612) at org.talend.designer.bigdata.common.BigDataDataProcess.buildRecordStructsForConnections(BigDataDataProcess.java:588) at org.talend.repository.spark.editor.SparkDataProcess.buildFromGraphicalProcess(SparkDataProcess.java:156) at org.talend.designer.bigdata.common.BigDataProcess.getGeneratingNodes(BigDataProcess.java:93) at org.talend.designer.core.ui.editor.process.Process.getNodesOfType(Process.java:3949) at org.talend.designer.core.ui.editor.nodes.Node.checkParameters(Node.java:3029) at org.talend.designer.core.ui.editor.nodes.Node.checkNode(Node.java:4406) at org.talend.designer.core.ui.editor.process.Process.checkProblems(Process.java:3801) at org.talend.designer.core.ui.editor.process.Process.checkProcess(Process.java:3792) at org.talend.designer.core.ui.views.problems.Problems.initCurrentProblems(Problems.java:265) at org.talend.designer.core.ui.views.problems.Problems.addProcess(Problems.java:257) at org.talend.designer.core.ui.ActiveProcessTracker.addJobInProblemView(ActiveProcessTracker.java:169) at org.talend.designer.core.ui.ActiveProcessTracker.partOpened(ActiveProcessTracker.java:294)

After checked with R&D  and made some investigation and analysis on it,  this is a Plugin code generation logic bug from: plugins/org.talend.designer.bigdata/src/org/talend/designer/bigdata/avro/AvroRecordStructGenerator.java

Resolution

Fixed patch: V8.0.1-R2024-04 patch 

Internal Investigation ID(s)

Internal defect ID, TBD-16238

Environment

Talend Big Data

Talend Studio 

This Techspert Talks session covers:

- What to plan for
- Migration Pathways
- Cloud Best Practices

Chapters:

• 01:20 - Cloud Tenant Locations
• 01:47 - Qlik Cloud Architecture
• 03:08 - Planning the migration
• 03:50 - Migration Journey
• 05:13 - Licensing Guardrails
• 06:39 - Migrating users, groups, and permissions
• 11:11 - Migrating apps
• 14:44 - Migrating the data connections
• 21:07 - Completed Migration
• 21:35 - 5 Key Take Aways
• 22:48 - Q&A: How to license sync with hybrid?
• 23:59 - Q&A: Best practices for 1 app migration?
• 24:47 - Q&A: What's included with license cost?
• 26:03 - Q&A: How to customize the UI?
• 26:45 - Q&A: How safe is the data in Cloud?
• 27:37 - Q&A: What are common stumbling blocks?
• 28:41 - Q&A: How are reload monitored?
• 29:28 - Q&A: What is the data size cap?
• 30:15 - Q&A: What cannot be migrated?

Resources:

• Qlik Cloud Best Practices
• Troubleshooting Qlik Data Gateway Direct Access
• Qlik Cloud Migration Center
• Qlik Professional Services
• Space-aware data source syntax examples
• Trust and Security at Qlik
• The Reload Analyzer for Qlik Cloud Customers
• Qlik Licensing Service Reference Guide
• Migrating NPrinting to Qlik Cloud Reporting

Introduction

In the realm of software integration and data management, ensuring compatibility and functionality across different versions is crucial.

Talend, a leading provider of data integration and integrity solutions, offers a straightforward process for users to test migration from existing active licenses to newer versions.

This guide outlines how to request and effectively utilize a Talend Test Migration License, whether for cloud or on-premises deployments.

Understanding Talend Test Migration License

A Talend Test Migration License allows users to transition from their current active license to a newer version temporarily for testing purposes.

This enables organizations to assess the new features, functionalities, and compatibility with their existing systems before committing to a full migration.

Steps to Request a Talend Test Migration License

1. Gather Necessary Information

Before requesting a Talend test migration license, ensure you have the following details ready:

• New User Email ID: The email address where the test migration license will be sent.
• Existing Active License Version: The current version of Talend Data Integration or other products.
• Target Test Migration Version: Specify the exact version to which you intend to migrate for testing.
• Purchase Order (PO) Number: Include the PO number associated with your existing active license.
• Existing Email ID: The email address to which the original license was sent.
• Deployment Type: Specify if your deployment is on-premises or cloud-based.
• Description of Request: Briefly explain why you need the test migration license and the objectives of the migration.

2. Create a new case in Qlik Customer Support Portal

For Cloud Deployments

Please create a new case in Qlik Customer Support Portal with the following case template.

Case Subject: Request for Talend Test Migration License

Dear Support Team,

I hope this message finds you well.

I am writing to request a Talend test migration license for our cloud deployment. Below are the required details:

• New User Email ID: [Insert User Email]
• Existing Active License Version: [Insert Existing Active License Version]
• Target Test Migration Version: [Insert Target Test Migration Version]
• PO Number: [Insert PO Number]
• Existing Email ID: [Insert Existing Email ID]
• Deployment Type: Cloud License
• Description: [Briefly explain your request]

Your prompt assistance in providing this test migration license would be greatly appreciated. Thank you for your support and consideration.

Best regards, [Your Name]

Example Request (Cloud)

Case Subject: Request for Talend Test Migration License (Cloud)

Dear Support Team,

I hope this message finds you well.

I am writing to request a Talend test migration license for our cloud deployment. Below are the required details:

• New User Email ID: example@domain.com
• Existing Active License Version: Talend Data Integration Version 7.3.1
• Target Test Migration Version: Talend Data Integration Version 8.0.1
• PO Number: PO-123456
• Existing Email ID: original.recipient@domain.com
• Deployment Type: Cloud License
• Description: Due to our project requirements, we need to test migration to the latest version of Talend Data Integration. Attached is a screenshot of the Talend Management Console (TMC) Subscription page, confirming our current cloud deployment.

Your prompt assistance in providing this test migration license would be greatly appreciated. Thank you for your support and consideration.

Best regards,

John Doe

For On-Premises Deployments:

Please create a new case in Qlik Customer Support Portal with the following case template.

Case Subject: Request for Talend Test Migration License

Dear Support Team,

I hope this message finds you well.

I am writing to request a Talend test migration license for our on-premises deployment. Below are the required details:

• New User Email ID: [Insert User Email]
• Existing Active License Version: [Insert Existing Active License Version]
• Target Test Migration Version: [Insert Target Test Migration Version]
• PO Number: [Insert PO Number]
• Existing Email ID: [Insert Existing Email ID]
• Deployment Type: On-Premises License
• Description: [Briefly explain your request]

Your prompt assistance in providing this test migration license would be greatly appreciated. Thank you for your support and consideration.

Best regards, [Your Name]

Example Request (On-Premises)

Case Subject: Request for Talend Test Migration License (On-Premises)

Dear Support Team,

I hope this message finds you well.

I am writing to request a Talend test migration license for our on-premises deployment. Below are the required details:

• New User Email ID: example@domain.com
• Existing Active License Version: Talend Data Integration Version 7.3.1
• Target Test Migration Version: Talend Data Integration Version 8.0.1
• PO Number: PO-123456
• Existing Email ID: original.recipient@domain.com
• Deployment Type: On-Premises License
• Description: Due to our project requirements, we need to test migration to the latest version of Talend Data Integration.

Your prompt assistance in providing this test migration license would be greatly appreciated. Thank you for your support and consideration.

Best regards, John Doe

Important Notes

• Ensure all details provided are accurate and complete before sending your request.
• Use the appropriate email address based on your user type (general user or Qlik partner).
• Test migration licenses are provided for migrating from an older active version to a newer version for testing purposes.
• The test migration license can be generated for a period of 60 days (2 months) at a time.
• For cloud license requests, include a screenshot of the Talend Management Console (TMC) Subscription page for verification.
• For further inquiries or assistance, please contact the respective support team.

Conclusion

Effectively utilizing a Talend test migration license is essential for organizations looking to evaluate new Talend software versions without disrupting existing operations.

By following the outlined steps and using the provided templates, users can streamline the process of requesting and deploying a test migration license tailored to their specific deployment needs.

If you are looking to extend a Test Migration License, see How to extend Talend Test Migration License 

If you are looking for Qlik Support Case Portal, see How to contact Qlik Support 

Environments

Talend Cloud 

Talend Studio

Talend Administration Center 

The Oracle source endpoint occasionally encounters an error during the CDC stage. 

The error message in task log file:

[SOURCE_CAPTURE ]T: OCI error 'ORA-12592: TNS:bad packet' [1022307] (oradcdc_bfileapi.c:312)

The issue specifically arises during the CDC stage; The problem is irrelevant to the Oracle instant client version, and it's not a timeout issue (parameter SQLNET.SEND_TIMEOUT).

Cause:

Oracle SQLNet packet inspection feature in Firewall (sometimes called as Advanced SQL feature in Firewalls) has been causing the TNS bad packet ORA-12592 error. 

Internal Investigation ID(s):

#00156619, QB-26584

Environment:

• Qlik Replicate All versions
• Oracle All versions

Qlik Replicate  

The problem occurs on a Windows 2019 server, generally after an upgrade or a new installation of NPrinting.
It is possible to access the NPrinting Web console and click on the start button of a task or on "Reload Metadata" for a connection, but then nothing happens. The execution is not started and no error message is shown.
This error message can be found on Scheduler and Engine logs:

ERROR: RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> System.AggregateException: One or more errors occurred. ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> System.ComponentModel.Win32Exception: The client and server cannot communicate, because they do not possess a common algorithm↓↓   --- End of inner exception stack trace

Resolution

In order to fix the problem, it is necessary to force NPrinting to work with TLS 1.2. This can be done in the RabbitMQ configuration.

1.  Log in to the NPrinting Server machine as an administrator.
2.  Stop the NPrinting services.
3.  Do a backup of "C:\ProgramData\NPrinting\RabbitMQ\advanced.config". It is sufficient to rename it "advanced.config_OLD".
4. Paste the "advanced.config" file attached here in the "C:\ProgramData\NPrinting\RabbitMQ\" folder.
5.  Restart the NPrinting services.
   

The "advanced.config" file contains a setting forcing NPrinting to use TLS 1.2 for the services communication.
Notice that it may be necessary to repeat this process after an upgrade, because configuration files can be replaced in some cases.

Cause 

The problem is due to a defect in the Server Operating System.
Some Windows 2019 machines erroneously advertise TLSv 1.3 as a supported version, so the .NET executables shipped with NPrinting May 2023 try to use TLS 1.3 to connect to the messaging service. But TLS 1.3 is not supported by Windows Server 2019 and the connection fails for this reason.
Some communications between services are then not occurring. For example, the task execution requests are not reaching the Database and can't start.

Environment

• Qlik NPrinting 2023 and higher versions.

This article outlines the best practices for moving from on-premise Talend Data Services to Talend API services in Talend Cloud.

Content:

• Why move to Talend API Services in the cloud?
• On-premises data services versus Talend in the cloud
• Talend on-premises architecture
• Talend in the cloud architecture
• Talend APIs on AWS
• Monitoring and management
• Other considerations when moving to Talend API Services in the cloud
• Choice of deployment
• CI/CD
• Usage of external libraries in Talend Routes and data services
• Talend Studio patch

Why move to Talend API Services in the cloud?

Talend on-premises ESB implementations have two methods of implementing REST APIs:

• Data Services: simple orchestrations, not suitable for complex orchestrations
• Route: more complex orchestrations, also suitable for simple orchestrations

Both approaches can be implemented in Talend Runtime, in that both methods are built as OSGi bundles and deployed in Talend Runtime. You can deploy both services as a standalone microservice or as a microservice on a Docker container. This article focuses on the Remote Engine and Remote Runtime for deployment as an OSGi bundle.

As organizations are modernizing their IT landscape, most of the infrastructure and services are moving to the cloud. As part of that trend, the emergence of microservice architecture data services and APIs are taking center stage. These data services and APIs need flexibility, scalability, and manageability built around common and company-wide standards for infrastructures and services.

On-premises data services versus Talend in the cloud

A move to Talend Cloud platform is typically part of a larger organizational strategy to move the data environment to the cloud. Therefore, many auxiliary components for data services that were necessary for Talend on-premises ESB and API Services are no longer needed and are, thus, not included in Talend API Services in the cloud.

For example, Talend Cloud platform doesn't require the following services:

• Service Activity Monitoring (SAM) and Service Locator
  Monitoring and management using Talend supplied components like Service Locator and SAM.

• Monitoring and logging
  Logging from the OSGi bundles were also supporting advanced MDC event logging, which was integrated into the ELK stack for log monitoring. The on-premises and ESB implementation would use the ELK stack deployed and managed on-premises to provide logging functionalities.

• Authentication and Authorization Service
  Security (Authentication and Authorization) was provided in Talend on-premises and ESB using Talend Identity and Access Management based on Syncope, with authorization based on XACML policies. Also, JAAS based configurations were supported.

• Artifact repository
  Typically, it was necessary to configure an artifact repository on Talend Runtime to resolve any dependencies that may have been defined in Talend built OSGi resources.

While these services are beneficial in enabling sophisticated functionality for REST APIs on-premises, they are not necessary on Talend Cloud platform because most cloud-based deployments are architected to use cloud vendor-provided services as a best practice.

As a result, these services, included within Talend on-premises platform, are no longer required for Talend in the cloud API deployments in Talend because more suitable options are available on AWS or Azure, on which Talend Cloud platform runs.

Talend on-premises architecture

Talend in the cloud architecture

The on-premises only services discussed above are no longer needed or included in the architecture because you can use services from the cloud provider (AWS or Azure) to achieve the same functionalities. The end result is a reduced Talend footprint and a simpler to manage data services platform. The following diagram illustrates a typical architecture for deploying Talend services in a cloud environment:

Talend APIs on AWS

Taking a deeper look at using an AWS cloud to deploy a full stack for data services deployment. An AWS API Gateway is used to create APIs. The Endpoint type can also be decided at this point. 

Talend recommends using Talend API Designer to create an OAS or Swagger definition, and the API that is defined can be used for importing to an Amazon API Gateway. If you haven’t implemented the APIs, you can import the same specification into Talend Studio to help build tight APIs. Because all aspects of an API build is on a Technology Standard definition, you can deploy an API on any vendor supplied API Gateway.

Monitoring and management

The monitoring and logging aspects of the APIs, previously handled by SAM for on-premises deployments, can be enabled on API Gateway.

Amazon CloudWatch provides a specific dashboard for APIs hosted on an API Gateway. You can use CloudWatch to gather runtime logs, and it provides most of the features provided by an ELK stack on-premises.

You can enable API security on an API Gateway as well. There is a complete set of AWS services that provide various functionality to enable the necessary API security. Apart from the Authentication and Authorization features, they include SSL certificate management needed for enterprise-wide security. Also, API key management and usage plans can help build modern data marketplace setups for your customers.

For more information, see the Amazon web page, Controlling and managing access to a REST API in API Gateway.

Other considerations when moving to Talend API Services in the cloud

Choice of deployment

There are three types of deployments:

• OSGi bundle
• Microservice
• Docker

The overall cloud strategy can guide your choice.

CI/CD

The CI/CD pipelines setup for on-premises data services and Routes can be enhanced and extended to handle CI/CD in the cloud. As a result, the SDLC for the Talend projects will, for the most part, remain the same. The only enhancements would be the usage of Talend Promotion pipelines, which can be executed from Talend Cloud platform or through provider APIs.

Usage of external libraries in Talend Routes and data services

The corporate artifact repository used for managing dependencies in an on-premises deployment can be reused for Talend in the cloud. You can leverage the artifact repository to host plug-ins needed for CI/CD.

Talend Studio patch

Talend recommends that you use the latest Talend Studio cumulative patch for migration.

Dependency issues with cConfig on Routes and tLibraryLoad for data service type Jobs noticed in builds are solved with new patches.