topic Re: Get Message create Timestamp from Kafka Topic while reading in Talend Studio

Get Message create Timestamp from Kafka Topic while reading

vasan212 — Fri, 15 Nov 2024 21:33:49 GMT

Hi,

I'm trying to download a Json file from Kafka topic and I want to retrieve message create timestamp from kafka topic while reading.

Used below configs in the tkafkainput component, but they are not working.

I Need help here to capture the timestamp while reading message from Kafka topic.

configs tried:

print.timestamp=true

message.timestamp.type=CreateTime

data.timestamp=CreateTime

all of them are throwing a warning message while running as "The configuration 'print.timestamp' was supplied but this isn't a known config."

Note - While providing the same config in the backend I can see the createtime .

ex: bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topicName --property print.timestamp=true --from-beginning

CreateTime:1609917197764. is printed in the beginning of the message.

Re: Get Message create Timestamp from Kafka Topic while reading

Anonymous — Thu, 20 Jul 2023 08:01:36 GMT

Hello @Thanigaivasan Dayalan ,

To get the message create timestamp of kafka topic, please follow the below steps:

setup the Output type to "ConsumerRecord" in tKafkaInput Basic settings
using tJavaRow to retrieve the record timestamp and values by the code like

org.apache.kafka.clients.consumer.ConsumerRecord record = (org.apache.kafka.clients.consumer.ConsumerRecord) input_row.record;

output_row.timestamp = record.timestamp();

output_row.topic = record.topic();

output_row.key = (byte[]) record.key();

output_row.value = (byte[]) record.value();

3 using tLogRow to output the rows

I tested the above steps, it's working as expected.

Please refer to the below article

https://help.talend.com/r/en-US/8.0/kafka/kafka-avro-read-avro-data-from-consumerrecord

Best regards

Aiming

Re: Get Message create Timestamp from Kafka Topic while reading

vasan212 — Thu, 20 Jul 2023 14:16:59 GMT

@Aiming Chen - Thank you for the response. But, I don't see 'ConsumerRecord' in the output type. All I can see is 'String' and 'byte[]'. Is there any setting I need to enable?

I'm using Talend Real time BigData Platform 7.3.1 and job type is standard job.

Re: Get Message create Timestamp from Kafka Topic while reading

Anonymous — Fri, 21 Jul 2023 03:09:40 GMT

Unfortunately, the 'ConsumerRecord' output type for tKafkaInput is only available in Talend studio 8.0 but not in 7.3.1. thanks

Re: Get Message create Timestamp from Kafka Topic while reading

vasan212 — Fri, 21 Jul 2023 16:31:40 GMT

Is there any other approach in older version of Talend to get the Timestamp from the Kafka topic, like using java components?

Re: Get Message create Timestamp from Kafka Topic while reading

vasan212 — Tue, 25 Jul 2023 13:37:04 GMT

@Xiaodi Shi - Hello Sabrina... Any help here is much appreciated. TIA

Re: Get Message create Timestamp from Kafka Topic while reading

Anonymous — Wed, 26 Jul 2023 04:19:49 GMT

Hello @Thanigaivasan Dayalan

tKafka* to handle Avro data in DI jobs (Avro functionality) is a new feature and not available in V 7.3.1 I'm afraid.

Please also note that this feature was implemented in V 8.0.1 for the tKafkaInput/Output components available in Standard jobs:

https://help.talend.com/r/en-US/8.0/kafka/tkafkainput-standard-properties

You should upgrade to Talend 8 to achieve it and feel free to let me know if it is OK with you.

Best regards

Sabrina

Re: Get Message create Timestamp from Kafka Topic while reading

vasan212 — Thu, 27 Jul 2023 12:22:53 GMT

@Xiaodi Shi - I'm not using Avro. I'm consuming Json message from Kafka Topic and I want to extract the message create timestamp(Time when the file was placed in Kafka Topic).