Populating the 'timestamp' column in a Kafka data message during Full Load

[NakulanR]

Hi Support,

When a Replicate task writes to Kafka, the data message contains a column called 'timestamp'. As per the Replicate User guide, the column gets populated with the original change timestamp and is only relevant for CDC operations. 

Is there anything in the task settings, Kafka endpoint internal parameters or Replicate configuration file which allows for that 'timestamp' column to be populated during the full load, such that the value reflects the time at which data was loaded into Kafka?

Thanks,

Nak

[SushilKumar]

Hello @NakulanR 

Full load. Done having the Timestamp ID because of the way it functions. AS a full load, create a table along with metadata and sequentially perform an insert. However CDC have a timestamp at the because they have timestamp/Checkpoint attached to it. its the way it works.

Regards,
Sushil Kumar 

[john_wang · Accepted solution]

Hi Nak, @NakulanR 

I hope this message finds you well.

In the task configuration, the "timestamp" column has been enabled for inclusion in the Kafka messages. However, its value appears to be empty for records processed during the Full Load phase.

As far as I can tell, there is no built-in parameter to automatically populate this field (e.g., with the current timestamp) during Full Load.

A possible workaround is to add a transformation column, for example, named "ts", using the following expression:

strftime('%Y-%m-%d %H:%M:%f', 'now', 'localtime')

Please note that this field will be included in the data section of the Kafka message, not in the headers section, as shown below:

If you would like this value to appear in the headers section, you are welcome to submit a Feature Request for consideration in a future release.

Regards,

John.