topic Re: Talend Remote Engine in Talend Studio

Talend Remote Engine

Anonymous — Sat, 16 Nov 2024 04:02:59 GMT

Hello,

We are configuring Talend Cloud and have few questions regarding Remote Engines. We have both DI & Big data job,

Hence should we install remote engine in both EC2 (DI) and EMR (Big Data) ?

Or Just remote engine in EC2 can support both DI & Big Data Jobs?

Also any best practice for configuring Remote engines - like sizing, numbers etc.

Re: Talend Remote Engine

Anonymous — Fri, 22 Nov 2019 04:38:34 GMT

Hi,

Talend reference architecture website will help you here. Could you please refer the below details from the link? Recommended server sizes are at the bottom of the page.

https://talendpnp.github.io/architecture-center/reference-architecture/summer-2018-7.0/products/commercial/entry/talend-big-data/index

If you have plans to use both DI and BD jobs, I would recommend to keep them separate.

Hope I answered your query. Please spare a second to mark the topic as resolved 🙂

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: Talend Remote Engine

Anonymous — Fri, 22 Nov 2019 14:00:04 GMT

@nthampi

Thanks for the response. Is there any specific reason to have different remote engine for DI and Big data?.

Is it not possible to have same remote engine for both types of Job?

What is the advantages and disadvantages on having same or different remote engines for different types of job.

Re: Talend Remote Engine

Anonymous — Fri, 22 Nov 2019 15:58:49 GMT

Hi,

You do not have to use the system resources of a BD cluster for a simple DI job. For BD jobs, you normally create the Remote Engine over an edge node of the cluster. But for DI job, you can use it over a simple standalone server or EC2 instance.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: Talend Remote Engine

Anonymous — Sun, 24 Nov 2019 00:28:51 GMT

@Gopik

Hi Gopi,

Hope I have answered your query. Could you please mark the topic as answered by selecting the posts which helped you to reach resolution? It will help other Talend community members during their reference.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

Re: Talend Remote Engine

Anonymous — Thu, 05 Mar 2020 21:01:20 GMT

if you are using AWS EMR, At the start of the EMR, I would bootstrap to install/pair remote engine dynamically on one of the EMR node. This could avoid having a separate EC2 for edge node(saves money). However, if you already have an Edge node, you could install on the remote engine on the edge node.

I agree that its recommended to have separate remote engines for DI and Big data jobs.