topic Re: QR: Log stream vs header__timestamp in Qlik Replicate

QR: Log stream vs header__timestamp

Jon_Donker — Mon, 08 Jan 2024 22:38:29 GMT

Hi,

Just need a conformation of header__timestamp in regards to reading from the Log stream task:

Is the header__timestamp in the S3 json file the timestamp of the change on the source?

Or the time the change was written to the log stream?

Re: QR: Log stream vs header__timestamp

Alan_Wang — Tue, 09 Jan 2024 17:37:49 GMT

Hi @Jon_Donker

Header timestamp is the timestamp of the Change operation, which is represented as the local time on the Replicate Server. Header commit timestamp is the source database commit timestamp, according to Replicate Server time. For an extensive explanation of all our header variables, please refer to the following link:

https://help.qlik.com/en-US/replicate/May2022/Content/Global_Common/Content/SharedEMReplicate/Customize%20Tasks/tasks_headColumns.htm#Headers

Re: QR: Log stream vs header__timestamp

Jon_Donker — Tue, 09 Jan 2024 21:16:32 GMT

Thanks Alan.

So those values are not impacted by the fact that the data is reading from a Log stream? They still relate to the source DB2 database?

Re: QR: Log stream vs header__timestamp

Alan_Wang — Tue, 09 Jan 2024 23:08:04 GMT

Header timestamp is when the change is applied for the child log stream task. No relation to the DB2 timestamps. Header commit timestamp is the one that logs DB2 commit timestamps.

Re: QR: Log stream vs header__timestamp

Jon_Donker — Tue, 07 May 2024 23:26:14 GMT

Hi - just coming back to this problem.

The downstream users would like the "Header DB Commit Timestamp".

When we shift to log stream; is it possible to overwrite the header__timestamp with H_DB_COMMIT_TIMESTAMP so they don't have to modify their ingestion patterns to use a new field?

Re: QR: Log stream vs header__timestamp

Alan_Wang — Fri, 10 May 2024 15:51:32 GMT

Change Tables have some prebuilt headers that can be turned on or off in task settings.

Change Processing -> Stored Changes Settings

Headers that do not come prebuilt into the table will need to be added as a new column through transformations. You will use the expression builder to add the header in as the value. You can name the column anything you like.

$AR_H_DB_COMMIT_TIMESTAMP

These changes will need a reload to rebuild the table.