topic Re: Route Messages VS Kafka Messages in Talend Studio

Route Messages VS Kafka Messages

Anonymous — Sat, 16 Nov 2024 06:45:55 GMT

Good afternoon,

I just open this topic to confirm that what I think is correct.

I would like to send a message from a Route to another one, using Kafka.

My message contains a body and some headers (only 1 in my test).

In my second job, I catch my Kafka message and log it.

But I only have my message body in my second route, and I have nothing on my header :

I imagine that because this is not the same kind of message :

- from route side, a message contains 0,* headers, a body and some attachment ;

- from kafka side, a message is only a body (a content)

If I want to send a route message to another route via Kafka, I have to package my route message in an XML for instance ?

Is there documentation on this or best practice ?

Thanks for your time

Re: Route Messages VS Kafka Messages

vapukov — Fri, 25 Jan 2019 01:48:19 GMT

HI,

check this for feature proposals - https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers

so for now - if you need additional information you need to encapsulate it into the body, XML or JSON is good approach

Re: Route Messages VS Kafka Messages

Loko — Fri, 25 Jan 2019 15:08:11 GMT

An other solution would be to use ActiveMQ messaging system instead of Kafka. It is really simple to install and seeded into the Talend ESB stuff.

AMQ will keep all your CamelMessage headers , no need to manage them in your routes.

Re: Route Messages VS Kafka Messages

vapukov — Sat, 26 Jan 2019 11:59:40 GMT

@Loko wrote:

An other solution would be to use ActiveMQ messaging system instead of Kafka. It is really simple to install and seeded into the Talend ESB stuff.

AMQ will keep all your CamelMessage headers , no need to manage them in your routes.

did you sure it is correct to compare these two brokers?

Re: Route Messages VS Kafka Messages

Loko — Mon, 28 Jan 2019 07:43:43 GMT

I don't know Kafka, its advantages and drawbacks vs AMQ, but what I know is : with AMS you don't have to bother with headers it handles them very good, both sending and consuming.

Re: Route Messages VS Kafka Messages

vapukov — Mon, 28 Jan 2019 08:05:47 GMT

@Loko wrote:

I don't know Kafka, its advantages and drawbacks vs AMQ, but what I know is : with AMS you don't have to bother with headers it handles them very good, both sending and consuming.

sure, ActiveMQ is good

but if Kafka was chosen not because of "it is a popular trend" - ActiveMQ will not replace it

Re: Route Messages VS Kafka Messages

Loko — Mon, 28 Jan 2019 08:15:37 GMT

so what ? Neither you nore me know why Kafka has been chosen. Only Bouga will know if AMQ can replace Kafka in his project. It is a possibility, So, why does it bother you when I talk about this possibility ?

I won't waste my time in such a boring and useless discussion. Just wanted to help, and ...

Re: Route Messages VS Kafka Messages

vapukov — Mon, 28 Jan 2019 08:19:07 GMT

😉

@Loko wrote:

I don't need a job to parse messages. I can do it very easily with a Camel route. And mixing routes and jobs is a really bad idea because of lacks of performance, Talend said.

Please keep in my purpose : scheduling JMS consuming. I don't need help for parsings messages, I've been doing it for years now.

Thank you 😕

this is your answer (and you look unhappy) when wrong advice, suggest to use ActiveMQ instead of Kafka (without know why) - it similar

Re: Route Messages VS Kafka Messages

Loko — Mon, 28 Jan 2019 08:22:25 GMT

The very big difference is that you can't say AMQ is a wrong advice as it is not your project. Maybe this person can use it, or not.

Are you just here to bother people, really ?

You never ever helped this person, you are just here to ennoy people.

I'm done with you.

Re: Route Messages VS Kafka Messages

Anonymous — Mon, 28 Jan 2019 08:51:33 GMT

Thanks everybody for your answers !

We have somebody who is actually working on an evaluation grid between Kafka and ActiveMQ, and which one will be the best appropriated in our project.

I just work in parralel to try one and using one on our POC to see how it work.

So it may be a valid argument to use ActiveMQ instead of Kafka, but we will wait the end of the grid !

I just would like to be sure that there is no possibility to send key/value in Kafka message without using a XML in the body

Have a nice week and thanks again for your answers !