Talend Remote Engine

[Anonymous]

Hello,

We are configuring Talend Cloud and have few questions regarding Remote Engines. We have both DI & Big data job,

Hence should we install remote engine in both EC2 (DI) and EMR  (Big Data) ?

Or Just remote engine in EC2 can support both DI & Big Data Jobs?

Also any best practice for configuring Remote engines - like sizing, numbers etc.

[Anonymous]

Hi,

    Talend reference architecture website will help you here. Could you please refer the below details from the link? Recommended server sizes are at the bottom of the page.

https://talendpnp.github.io/architecture-center/reference-architecture/summer-2018-7.0/products/comm...

If you have plans to use both DI and BD jobs, I would recommend to keep them separate. 

Hope I answered your query. Please spare a second to mark the topic as resolved 🙂

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[Anonymous]

@nthampi 

Thanks for the response. Is there any specific reason to have different remote engine for DI and Big data?.

Is it not possible to have same remote engine for both types of Job?

What is the advantages and disadvantages on having same or different remote engines for different types of job.

[Anonymous]

Hi,

    You do not have to use the system resources of a BD cluster for a simple DI job. For BD jobs, you normally create the Remote Engine over an edge node of the cluster. But for DI job, you can use it over a simple standalone server or EC2 instance.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[Anonymous]

@Gopik 

Hi Gopi,

     Hope I have answered your query. Could you please mark the topic as answered by selecting the posts which helped you to reach resolution? It will help other Talend community members during their reference.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[Anonymous]

if you are using AWS EMR, At the start of the EMR, I would bootstrap to install/pair remote engine dynamically on one of the EMR node. This could avoid having a separate EC2 for edge node(saves money). However, if you already have an Edge node, you could install on the remote engine on the edge node. 

I agree that its recommended to have separate remote engines for DI and Big data jobs.