topic Query about Consumers in Talend Studio

Query about Consumers

RP1629548713 — Fri, 15 Nov 2024 23:37:51 GMT

Hi All,

I have a question regarding kafka usage within talend data fabric.

How can we increase the number of consumers from talend batch job/streaming job?

Any help on this appreciated. Thanks !

Re: Query about Consumers

Anonymous — Fri, 15 Oct 2021 03:40:43 GMT

Hello,

Are you trying to do Huge data processing? What is your batch job/streaming job doing? How are you managing connections in there for each message?

More information about your use case scenario and job background will be helpful for us to give you a precise answer.

Best regards

Sabrina

Re: Query about Consumers

RP1629548713 — Sun, 17 Oct 2021 06:09:11 GMT

Yes we have a huge data processing.

As of now we are only ingesting the Kafka records into Hbase.

But we frequently run into the issue of consumer while dealing with Kafka through talend.

Hence we would like to know if we have any approach to increase the number of consumers from talend

Re: Query about Consumers

RP1629548713 — Mon, 25 Oct 2021 12:08:39 GMT

hi ,

Any input on this?

Re: Query about Consumers

Anonymous — Thu, 28 Oct 2021 04:18:47 GMT

Hello,

A Talend Job being left to continually receive Kafka messages may not be the best method for consuming theses messages. Talend ESB may be a better option for this.

Best regards

Sabrina