Talend Cloud Platform Engines: Cloud Engine, Remote Engine, and Remote Engine Gen 2

TalendSolutionExpert · Jan 22, 2024 9:35:30 PM

Talend Cloud platform provides computational capabilities that allow organizations to securely run data integration processes natively from cloud to cloud, on-premises to cloud, or cloud to on-premises environments.

These capabilities are powered by compute resources, commonly known as Engines. This article covers the four basic types.

Content:

Cloud Engine (CE)
Remote Engine (RE)
Remote Engine Gen2 (REG2)
Cloud Engine for Design (CE4D)
Cloud Engine versus Remote Engine
Cloud Engine for Design versus Remote Engine Gen 2
Need for additional engines
Cloud Engine - usage considerations
Remote Engine – recommendations
Summary

Cloud Engine (CE)

A Cloud Engine is a compute resource managed by Talend in Talend Cloud that executes Job tasks.

You can allocate Cloud Engines to environments in proportion to the number of concurrent task executions, workloads, and Job designs you plan to run.
All environments can use unassigned Cloud Engines. If Cloud Engines are not allocated to specific environments, you may not be able to run certain tasks because other tasks might keep all the unassigned Cloud Engines occupied.
Cloud Engines can handle parallel execution of three tasks. That means a maximum of three different tasks can run in parallel on a single Cloud Engine. (A task cannot run more than once concurrently on a single Cloud Engine.) So, if three different tasks are already running on a Cloud Engine or if the same task is already running on that engine, another Cloud Engine is selected to execute the task.
If you run your task in Cloud Exclusive mode, you cannot execute other tasks on that Cloud Engine. You can only use Cloud Exclusive engines in environments that do not have Cloud Engines assigned to them.
Cloud Engines have limited system resources – memory usage: 8 GB, disk usage: 200 GB.
Only standard Data Integration Batch Jobs can run on Cloud Engines.
You cannot group Cloud Engines together to form clusters.
Cloud Engines are hosted on AWS or Azure Cloud.
Talend manages Cloud Engines.
Cloud Engines employ TCP communication.

Remote Engine (RE)

A capability in Talend Cloud platform that allows you to securely run data integration Jobs natively from cloud to cloud, on-premises to cloud, or cloud to on-premises environments completely within your environment for enhanced performance and security, without transferring the data through the Cloud Engines in Talend Cloud platform.

Java-based runtime (similar to a Cloud Engine) to execute Talend Jobs on-premises or on another cloud platform that you control.

Remote Engines allow you to run Jobs, Routes, and Data Service tasks.
Data Service and Route Microservice tasks can only be deployed on Remote Engines. OSGi type deployments require that Talend Runtime version 7.1.1 or higher is installed and running on the same machine as the Talend Remote Engine.
Remote Engines support configurable max parallel execution: by default, a maximum of three different tasks can run in parallel on the same Remote Engine. However, this is a modifiable configuration.
Remote Engines can be grouped to form clusters called Remote Engine Cluster. Remote Engines added to a cluster cannot be used to execute tasks directly from Talend Studio.
Remote Engines are hosted on-premises or on the cloud.
You manage Remote Engines.
Remote Engines employ HTTPS communication.

Remote Engine Gen2 (REG2)

A Remote Engine Gen2 is a secure execution engine on which you can safely execute data pipelines (that is, data flows designed using Talend Pipeline Designer). It allows you to have control over your execution environment and resources because you can create and configure the engine in your own environment (Virtual Private Cloud or on-premises). Previously referred to as Remote Engines for Pipelines, this engine was renamed Remote Engine Gen2 during H1/2020. It is a Docker-based runtime to execute data pipelines on-premises or on another cloud platform that you control.

A Remote Engine Gen2 ensures:

Data processing in a safe and secure environment, because Talend never has access to your pipelines' data and resources
Optimal performance and security by increasing the data locality instead of moving large data to computation

Cloud Engine for Design (CE4D)

Cloud Engine for Design is a built-in runner that allows you to easily design pipelines without setting up any processing engines. With this engine you can run two pipelines in parallel. For advanced processing of data, Talend recommends installing the secure Remote Engine Gen2.

CE4Ds have limited system resources – memory usage: 8 GB
CE4Ds support a maximum of two pipelines that can run in parallel on a single CE4D
CE4Ds should be used only for design purposes; that is, you shouldn’t use them to execute data pipelines in a Production environment

Cloud Engine versus Remote Engine

The following table lists a comparative perspective between the two engines:

Cloud Engine (CE)	Remote Engine (RE)
Consumes 45,000 engine tokens	Consumes 9,000 engine tokens
Runs within Talend Cloud platform – no download required	Downloadable software from Talend Cloud platform
Managed by Talend, run on-demand as needed to execute Jobs	Managed by the customer
No customer resources required	Customer can run on Windows, Linux, or OS X
Set physical specifications (Memory, CPU, Temp Disk Space)	Unlimited Memory, CPU, and Temp Space
Require data sources/targets to be visible through the internet to the Cloud Engine	Hybrid cloud or on-premises data sources
Restricted to three concurrent Jobs	Unlimited concurrent Jobs (default three)
Available within Talend Cloud portal	Available in AWS and Azure Marketplace
Runs natively within Talend Cloud iPaaS infrastructure	Uses HTTPS calls to Talend Cloud service to get configuration information and Job definition and schedules

Cloud Engine for Design versus Remote Engine Gen 2

Cloud Engine for Design (CE4D)	Remote Engine Gen 2 (REG2)
Consumes zero engine tokens	Consumes 9000 engine tokens
Build upon a Docker-compose stack	Build upon a Docker-compose stack
Available as Cloud Image and Instantiated in Talend Cloud platform on behalf of the customer	Available as an AMI Cloud Formation Template (for AWS) and Azure Image (for Azure)
Not available as downloadable software as this type of engine is only suitable for design using Pipeline Designer in Talend Cloud portal	Available as .zip or .tar.gz (for local deployment)
A Cloud Engine for Design is included with Talend Cloud platform, to offer a serverless experience during design and testing. However, it is not meant for production (that is, not for running pipelines in non-development environments). It won’t scale for prod-size volumes and long-running pipelines. It should be used for design teams to get a preview working and test execution during development. This engine should not be used for production execution.	It is used to run artifacts, tasks, preparations, and pipelines in the cloud, as well as creating connections and fetching data samples.
Static IPs cannot be enabled for CE4D within Talend Management Console	Not applicable as REG2 runs outside Talend Management Console (that is, in Customer Data Center)

Need for additional engines

Additional engines (CE or RE) may be required if you have one or more of the following use cases:

Continuous delivery – for example, Dev and QA separate from UAT and Production environments
Data access - data is in two different private locations where an engine is needed in each site (or a mix of Cloud and Remote Engines)
Scalability - concurrent Job volume requires additional engines, Jobs are complex and require significant memory or CPU

These use cases depend on the deployment architecture in the specific customer environment and layout of the Remote Engine at the environment or workspace level configurations. This would need proper capacity planning and automatic horizontal and vertical scaling of the compute Engines.

Cloud Engine - usage considerations

Question	Guideline
How much data must be transferred per hour?	Each Cloud Engine can transfer 225 GB per hour.
How many separate flows can run in parallel?	Each Cloud Engine can run up to three flows in parallel.
How much temporary disk space is needed?	Each Cloud Engine has 200GB of temp space.
How CPU and memory intensive are the flows?	Each Cloud Engine provides 8 GB of memory and two vCPU. This is shared among any concurrent flows.
Are separate execution environments required?	Many users desire separate execution for QA/Test/Development and Production. If this is needed, additional Cloud Engines should be added as required.

Remote Engine – recommendations

If a source or target system is not accessible through the internet:

If one of the systems is not accessible using the internet, then a Remote Engine is needed.

When single flow requirements exceed the capacity of a Talend Cloud Engine:

If the Cloud Engine is too small (for example, the maximum memory of 5.25 GB, temporary space of 200 GB, two vCPU, or the maximum of 225 GB per hour) then, a Remote Engine is needed.

If a native driver is required:

If the solution requires a native driver, which is not part of the Talend action or Job generated code, a typical case for this is SAP with the JCO v3 Library, MS SQL Server Windows Authentication, then a Remote Engine is needed.

Data jurisdiction, security, or compliance reasons:

It may be desirable or required to retain data in a particular region or country for data privacy reasons. The data being processed may be subject to regulations such as PCI or HIPAA, or it may be more efficient to process the data within a single data center or public cloud location. These are all valid reasons to use a Remote Engine.

Summary

Cloud Engine (CE)	Remote Engine (RE)	Remote Engine Gen 2 (REG2)
Cloud Engines allow you to run batch tasks that use on-premises or cloud applications and datasets (sources, targets)	Remote Engines allow you to run batch tasks or microservices (APIs or Routes) that use on-premises or cloud applications and datasets (sources, targets)	The Remote Engine Gen2 is used to run artifacts, tasks, preparations, and pipelines in the cloud, as well as creating connections and fetching data samples
Consumes 45,000 engine tokens	Consumes 9,000 engine tokens	Consumes 9,000 engine tokens
No download required - Runs within Talend Cloud platform	Downloadable software from Talend Cloud platform	Downloadable software from Talend Cloud platform
Managed by Talend, run on-demand as needed to execute Jobs	Managed by the customer	Managed by the customer
No customer resources required	Can run on Windows, Linux, or OS X	Require compatible Docker and Docker compose versions for Linux, Mac, and Windows
Set physical specifications (Memory, CPU, and Temp Disk Space)	Unlimited Memory, CPU, and Temp Space	Unlimited Memory, CPU, and Temp Space
Require data sources/targets to be visible through the internet to the Cloud Engine	Hybrid cloud or on-premises data sources	Hybrid cloud or on-premises data sources
Restricted to three concurrent Jobs	Unlimited concurrent Jobs (default three)	Unlimited concurrent pipelines (configurable)
Available within Talend Cloud portal	Available in AWS and Azure Marketplace	Available as an AMI Cloud Formation Template (for AWS) and Azure Image (for Azure)
Runs natively within Talend Cloud iPaaS infrastructure	Uses HTTPS calls to Talend Cloud service to get configuration information and Job definition and schedules	Uses HTTPS calls to Talend Cloud service to get configuration information and pipeline definition and schedules

References

Talend Help Center documentation:

MoeE · ‎2024-04-19

Hi,

Regarding the following table, does "flow" mean pipeline? Are the words Flow and Pipeline interchangeable?

e.g Cloud engine can run three flows/pipelines in parallel.

Regards,

Mohammed

Xiaodi_Shi · ‎2024-04-19

Hello,

The word "flow" mentioned in this Talend Cloud Platform Engines article refers to data flow not specific to pipeline tasks.

About Talend Cloud architecture, could you please refer to these online documentations for more information?

https://help.talend.com/r/en-US/Cloud/development-operational-management/operations-in-talend-cloud

https://help.talend.com/r/en-US/Cloud/development-operational-management/simple-talend-cloud-archite...

Best regards

Sabrina

C_Kneeland · ‎2024-09-19

Hello,

I noticed that the Remote Engine Gen2 is mentioned as being available as an Azure Image, but I haven't found detailed instructions on its installation specifically for Azure. Could you clarify whether the Remote Engine Gen2 can be installed directly within an Azure environment, or is it restricted to on-premises/virtual machine installations?

I found the following Azure Marketplace link for the Remote Engine, but I am unsure if this applies to the Gen2 version:
Azure Marketplace - Talend Remote Engine

Any guidance or clarification on this would be greatly appreciated!

Best regards,
Connor

Xiaodi_Shi · ‎2024-09-19

Hello @C_Kneeland

We have an installation specific guide for RE Gen2 on Qlik Talend Documentation site for any installation/pairing concern.

https://help.qlik.com/talend/en-US/remote-engine-gen2-quick-start-guide/Cloud/engine-script-install

The usage of images for gen2 has been deprecated (engine gen 2 build by Data Processing Platform team for Pipeline workloads -> no more aws amis and azure).

If you need additional assistance, I recommend posting directly in our Formerly Talend Forum, where active agents and our active user base are able to give more assistance.

Best regards

Sabrina

Talend Cloud Platform Engines: Cloud Engine, Remote Engine, and Remote Engine Gen 2