Pushing data to Kafka

[Anonymous]

Hi,

I'm currently trying to push database data to Kafka in a near real-time manner.

Basically, what this job does is it checks a SQL Server table (i.e. Talend CDC table) for any new rows. If yes, the data is converted into a JSON, then serialized into byte[] type.

I've added a tReplicate and tLogRow to verify that the data output is correct.

All other components are successfully executed except tKafkaOutput, which returns the following error:

org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
[ERROR]: testing_kafka.talendcdc_consumecdctable_0_1.TalendCDC_ConsumeCDCTable - tKafkaOutput_1 - Failed to update metadata after 60000 ms.
[ERROR]: testing_kafka.talendcdc_consumecdctable_0_1.TalendCDC_ConsumeCDCTable - tKafkaOutput_1 - Failed to update metadata after 60000 ms.

Also, I noticed that there are only several options for Kafka versions in Talend 7.0:

• 0.8.2.0
• 0.9.0.1
• 0.10.0.1

But the version configured on our Hadoop cluster is version 2.0. Is this a possible cause of the problem?

Regards,

Veronica

[vapukov]

Hi Veronica,

from one side - Kafka is critical to a version number, but with 0.10.0.1 it must work

first of all check - could you connect from Talend host to Kafka? because your error looking like connection timeout

[Anonymous]

Hi vapukov,

Thanks for responding.

I am able to connect to Kafka through Talend as I was able to create a Kafka topic with Talend. But loading data to the topic doesn't seem to work from Talend.

[vapukov]

Hi,

I use Confluent 5.1 which is Kafka 2.0
And Kafka 1.?? From Apache

In both cases - Talend and create topics and send data to it

Try with any other Kafka client, but from Talend machine and with same settings - is it work or not

add. what kafka settings you use in tKafkaOutput?

[Anonymous]

Hi,

I'm using Apache Kafka version 2.0 under the HDF distribution.

I'm using only broker list and topic name configurations in tKafkaOutput.

Do you have any other configurations set in your tKafkaOutput?

[Anonymous]

If it helps, I managed to setup a local zookeeper and kafka to test out kafka 1.10 to match the kafka version supported by Talend 7.1. The data loads into the topic just fine this way.

So I'm guessing it really is a compatibility issue after all.

[vapukov]

I use this - https://docs.confluent.io/current/release-notes.html

5.1.0 is a major release of Confluent Platform that provides you with Apache Kafka 2.1.0, the latest stable version of Kafka.

The technical details of this release are summarized below.

and in my case, it works.

it could depend on many moments, I don't have installed Kafka from HDP, so I don't know - what definitely wrong in your case

but you could test your job with soft above - same install it locally. because testing different version of local instance - it not a proper test.

try to publish from command line from your machine to HDP Kafka, is it work or not?

try to connect to HDP Kafka from tools like - www.kafkatools.com and check do you have any issues or not?

[Anonymous]

Hi,

I have tried sending a message through Kafka CLI, and this works. The consumer prints out the payload just fine.

I initially thought that it might be a compatibility issue, but it seems to work without problems now... So it's definitely not a compatibility issue. Though I might have to check why it wasn't working before.

Thanks for the help!

[KB_Parpally]

Hello @veee/ @Vladimir Apukov​ ,

I am facing the same issue with talend when i try to connect Kafka output. could you please let me know if you figure out with any resolution for the above issue. Thanks in advance.

Below is the error what I am facing.

( [ERROR] 09:48:55 eim_talend_projects.eventhub_connection_0_1.Eventhub_connection- tKafkaOutput_1 - Failed to update metadata after 60000 ms.

[ERROR] 09:48:55 eim_talend_projects.eventhub_connection_0_1.Eventhub_connection- tKafkaOutput_1 - Failed to update metadata after 60000 ms.)