Deployment options for big data streaming job

Anonymous — Sat, 16 Nov 2024 08:14:14 GMT

Hi,

I have managed to build my first big data streaming job that consumes a kineses stream. I have installed a jobserver on an aws emr cluster and I am able to successfully deploy and run the job on that job server.

My only concern is that we would need an emr cluster running 24/7 just for this one job. Is there any other ways of deploying / "productionizing" a big data streaming job without running a whole cluster just for that?

Re: Deployment options for big data streaming job

Anonymous — Tue, 29 May 2018 10:01:00 GMT

Hello,

This is the whole purpose of a streaming processing running on top of a big data cluster.

If you do not need such computation power, could you please check 4 options :

- Set the spark configuration to run locally. It will only require an EC2 instance where the jobserver is deployed.
- Use Talend ESB / Camel
- Leverage the latest 7.0 feature with Cloudera Altus distribution (acting as Hadoop as a service)
- Leverage the new serverless distribution we shared on Talend Marketplace based on Qubole Saas offering (Hadoop as a service too).

Let us know if it is what you are looking for.

Best regards

Sabrina

topic Re: Deployment options for big data streaming job in Talend Studio

Deployment options for big data streaming job

Re: Deployment options for big data streaming job