topic Re: tKafkaInput payload is empty in Talend Studio

tKafkaInput payload is empty

rnathan2020 — Sat, 16 Nov 2024 07:45:01 GMT

I am doing a POC on tKafka* components. I have one standard job generating data and dropping messages into a kafka topic as byte []. Another job is reading it as string and printing on the console as well as outputting to a file via tFileOutputRaw.

The first job is creating the messages and the second job appears to be reading the messages. I see the metrics. However, when I dump it to the console, I am not seeing any data and the file does not get created. If I replace the tFileOutputRaw with tFileOutputDelimited, the file is created but it is empty.

All of this in standard job environment. Eventually I want to run the second job in BigData Streaming environment. I remember testing this on my laptop a while ago, when kafka was running locally. This is a different environment and I don't know the version and any server configuration information.

Re: tKafkaInput payload is empty

rnathan2020 — Wed, 29 Aug 2018 22:39:23 GMT

BTW, this is Talend 6.5.1

Re: tKafkaInput payload is empty

Anonymous — Thu, 30 Aug 2018 17:45:00 GMT

tKafkaInput reads from the end of the topic by default. You won't see any messages until some other process writes some new ones to the topic. You can change this in the component settings under "New consumer group starts from":

I also prefer to have the component stop after a small number of messages while testing. This way I don't have to try and produce messages before it exits. It will wait forever for 10 messages to appear on the topic.

Re: tKafkaInput payload is empty

rnathan2020 — Fri, 31 Aug 2018 17:12:43 GMT

Hi, I was retrieving the messages but they contain nothing, null. I did further testing, by retrieving the messages from kafka shell client. That was also returning null content messages.

I traced this down to tConvertType component, where I was trying to convert from string to byte[]. Instead I used java function getBytes() on the string and it worked! To get the string back from byte[], I used the cast (byte[]).

Now it is working.

Thanks all for pitching in.

Re: tKafkaInput payload is empty

Anonymous — Fri, 31 Aug 2018 19:17:41 GMT

You can also change the output type to String on tKafkaInput: