Best practices for moving from Talend Data Services to Talend API Services in the cloud

[TalendSolutionExpert]

This article outlines the best practices for moving from on-premise Talend Data Services to Talend API services in Talend Cloud.

Content:

• Why move to Talend API Services in the cloud?
• On-premises data services versus Talend in the cloud
• Talend on-premises architecture
• Talend in the cloud architecture
• Talend APIs on AWS
• Monitoring and management
• Other considerations when moving to Talend API Services in the cloud
• Choice of deployment
• CI/CD
• Usage of external libraries in Talend Routes and data services
• Talend Studio patch

Why move to Talend API Services in the cloud?

Talend on-premises ESB implementations have two methods of implementing REST APIs:

• Data Services: simple orchestrations, not suitable for complex orchestrations
• Route: more complex orchestrations, also suitable for simple orchestrations

Both approaches can be implemented in Talend Runtime, in that both methods are built as OSGi bundles and deployed in Talend Runtime. You can deploy both services as a standalone microservice or as a microservice on a Docker container. This article focuses on the Remote Engine and Remote Runtime for deployment as an OSGi bundle.

As organizations are modernizing their IT landscape, most of the infrastructure and services are moving to the cloud. As part of that trend, the emergence of microservice architecture data services and APIs are taking center stage. These data services and APIs need flexibility, scalability, and manageability built around common and company-wide standards for infrastructures and services.

On-premises data services versus Talend in the cloud

A move to Talend Cloud platform is typically part of a larger organizational strategy to move the data environment to the cloud. Therefore, many auxiliary components for data services that were necessary for Talend on-premises ESB and API Services are no longer needed and are, thus, not included in Talend API Services in the cloud.

For example, Talend Cloud platform doesn't require the following services:

• Service Activity Monitoring (SAM) and Service Locator
  Monitoring and management using Talend supplied components like Service Locator and SAM.

• Monitoring and logging
  Logging from the OSGi bundles were also supporting advanced MDC event logging, which was integrated into the ELK stack for log monitoring. The on-premises and ESB implementation would use the ELK stack deployed and managed on-premises to provide logging functionalities.

• Authentication and Authorization Service
  Security (Authentication and Authorization) was provided in Talend on-premises and ESB using Talend Identity and Access Management based on Syncope, with authorization based on XACML policies. Also, JAAS based configurations were supported.

• Artifact repository
  Typically, it was necessary to configure an artifact repository on Talend Runtime to resolve any dependencies that may have been defined in Talend built OSGi resources.

While these services are beneficial in enabling sophisticated functionality for REST APIs on-premises, they are not necessary on Talend Cloud platform because most cloud-based deployments are architected to use cloud vendor-provided services as a best practice.

As a result, these services, included within Talend on-premises platform, are no longer required for Talend in the cloud API deployments in Talend because more suitable options are available on AWS or Azure, on which Talend Cloud platform runs.

Talend on-premises architecture

Talend in the cloud architecture

The on-premises only services discussed above are no longer needed or included in the architecture because you can use services from the cloud provider (AWS or Azure) to achieve the same functionalities. The end result is a reduced Talend footprint and a simpler to manage data services platform. The following diagram illustrates a typical architecture for deploying Talend services in a cloud environment:

Talend APIs on AWS

Taking a deeper look at using an AWS cloud to deploy a full stack for data services deployment. An AWS API Gateway is used to create APIs. The Endpoint type can also be decided at this point. 

Talend recommends using Talend API Designer to create an OAS or Swagger definition, and the API that is defined can be used for importing to an Amazon API Gateway. If you haven’t implemented the APIs, you can import the same specification into Talend Studio to help build tight APIs. Because all aspects of an API build is on a Technology Standard definition, you can deploy an API on any vendor supplied API Gateway.

Monitoring and management

The monitoring and logging aspects of the APIs, previously handled by SAM for on-premises deployments, can be enabled on API Gateway.

Amazon CloudWatch provides a specific dashboard for APIs hosted on an API Gateway. You can use CloudWatch to gather runtime logs, and it provides most of the features provided by an ELK stack on-premises.

You can enable API security on an API Gateway as well. There is a complete set of AWS services that provide various functionality to enable the necessary API security. Apart from the Authentication and Authorization features, they include SSL certificate management needed for enterprise-wide security. Also, API key management and usage plans can help build modern data marketplace setups for your customers.

For more information, see the Amazon web page, Controlling and managing access to a REST API in API Gateway.

Other considerations when moving to Talend API Services in the cloud

Choice of deployment

There are three types of deployments:

• OSGi bundle
• Microservice
• Docker

The overall cloud strategy can guide your choice.

CI/CD

The CI/CD pipelines setup for on-premises data services and Routes can be enhanced and extended to handle CI/CD in the cloud. As a result, the SDLC for the Talend projects will, for the most part, remain the same. The only enhancements would be the usage of Talend Promotion pipelines, which can be executed from Talend Cloud platform or through provider APIs.

Usage of external libraries in Talend Routes and data services

The corporate artifact repository used for managing dependencies in an on-premises deployment can be reused for Talend in the cloud. You can leverage the artifact repository to host plug-ins needed for CI/CD.

Talend Studio patch

Talend recommends that you use the latest Talend Studio cumulative patch for migration.

Dependency issues with cConfig on Routes and tLibraryLoad for data service type Jobs noticed in builds are solved with new patches.