tKafkaInput payload is empty

[rnathan2020]

I am doing a POC on tKafka* components. I have one standard job  generating data and dropping messages into a kafka topic as byte []. Another job is reading it as string and printing on the console as well as outputting to a file via tFileOutputRaw. 

The first job is  creating the messages and the second job appears to be reading the messages. I see the metrics. However, when I dump it to the console, I am not seeing any data and the file does not get created. If I replace the tFileOutputRaw with tFileOutputDelimited, the file is created but it is empty.

All of this in standard job environment. Eventually I want to run the second job in BigData Streaming environment. I remember testing this on my laptop a while ago, when kafka was running locally. This is a different environment and I don't know the version and any server configuration information. 

[rnathan2020]

BTW, this is Talend 6.5.1

[Anonymous]

tKafkaInput reads from the end of the topic by default. You won't see any messages until some other process writes some new ones to the topic. You can change this in the component settings under "New consumer group starts from":

I also prefer to have the component stop after a small number of messages while testing. This way I don't have to try and produce messages before it exits. It will wait forever for 10 messages to appear on the topic.

[rnathan2020]

Hi, I was retrieving the messages but they contain nothing, null. I did further testing, by retrieving the messages from kafka shell client. That was also returning null content messages.

I traced this down to tConvertType component, where I was trying to convert from string to byte[]. Instead I used java function getBytes() on the string and it worked! To get the string back from byte[], I used the cast (byte[]).

Now it is working.

Thanks all for pitching in.

[Anonymous]

You can also change the output type to String on tKafkaInput: