Get Message create Timestamp from Kafka Topic while reading

[vasan212]

Hi,

I'm trying to download a Json file from Kafka topic and I want to retrieve message create timestamp from kafka topic while reading.

Used below configs in the tkafkainput component, but they are not working.

I Need help here to capture the timestamp while reading message from Kafka topic.

configs tried:

print.timestamp=true

message.timestamp.type=CreateTime

data.timestamp=CreateTime

all of them are throwing a warning message while running as "The configuration 'print.timestamp' was supplied but this isn't a known config."

Note - While providing the same config in the backend I can see the createtime .

ex: bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topicName --property print.timestamp=true --from-beginning

CreateTime:1609917197764. is printed in the beginning of the message.

[Anonymous]

Hello @Thanigaivasan Dayalan​ ,

To get the message create timestamp of kafka topic, please follow the below steps:

1. setup the Output type to "ConsumerRecord" in tKafkaInput Basic settings
2. using tJavaRow to retrieve the record timestamp and values by the code like

org.apache.kafka.clients.consumer.ConsumerRecord record = (org.apache.kafka.clients.consumer.ConsumerRecord) input_row.record;

output_row.timestamp = record.timestamp();

output_row.topic = record.topic();

output_row.key = (byte[]) record.key();

output_row.value = (byte[]) record.value();

3 using tLogRow to output the rows

I tested the above steps, it's working as expected.

Please refer to the below article

https://help.talend.com/r/en-US/8.0/kafka/kafka-avro-read-avro-data-from-consumerrecord

Best regards

Aiming

[vasan212]

@Aiming Chen​  - Thank you for the response. But, I don't see 'ConsumerRecord' in the output type. All I can see is 'String' and 'byte[]'. Is there any setting I need to enable?

I'm using Talend Real time BigData Platform 7.3.1 and job type is standard job.

[Anonymous]

Unfortunately, the 'ConsumerRecord' output type for tKafkaInput is only available in Talend studio 8.0 but not in 7.3.1. thanks

[vasan212]

Is there any other approach in older version of Talend to get the Timestamp from the Kafka topic, like using java components?

[vasan212]

@Xiaodi Shi​ - Hello Sabrina... Any help here is much appreciated. TIA

[Anonymous]

Hello @Thanigaivasan Dayalan​ 

tKafka* to handle Avro data in DI jobs (Avro functionality) is a new feature and not available in V 7.3.1 I'm afraid.

Please also note that this feature was implemented in V 8.0.1 for the tKafkaInput/Output components available in Standard jobs:

https://help.talend.com/r/en-US/8.0/kafka/tkafkainput-standard-properties

You should upgrade to Talend 8 to achieve it and feel free to let me know if it is OK with you.

Best regards

Sabrina

[vasan212]

@Xiaodi Shi​  - I'm not using Avro. I'm consuming Json message from Kafka Topic and I want to extract the message create timestamp(Time when the file was placed in Kafka Topic).