Compose Hive project: header__from_date in final tables

[Abrie_M]

Good day

In Hive projects, how is the value of the column header__from_date determined in tables/views in the Storage Zone?

Is it the timestamp that the record was changed on the source? Or when it was handled by Compose?

The only reference I could get in the documentation is: The name of the "From Date" column. This column is added to tables that contain attributes (columns) with history. The column is used to delimit the range of dates for a given record version.

Still waiting for feedback from Qlik support. Will post comments here if/when they reply.

Thanx

[TimGarrod · Accepted solution]

Hi Abrie_M - the default behavior by Compose is to populate the header__from_date using the header__timestamp value that Replicate delivers to the __ct tables in the landing area, but you can over-ride this. 

hen you look at a Compose mapping - if the from_date column in the mapping is NOT mapped - Compose will use the header__timestamp from Replicate.  (For a full load, since Compose doesn't know when the records where written to your source it will use the Lowest Date as defined in the project settings).

If you wish to use data values that are in the data (for example your source provides type 2 data) or a different value - you simply need to map the from_date column in the mapping. 

FYI - The header__timestamp  is (according to Replicate user guide) - the original change UTC timestamp (the value may be approximate)

Hope that helps!

[Abrie_M]

Hi @TimGarrod 

Thanx for your detailed feedback.

Qlik support also replied: When the change occurs in the source DB, the Replicate CDC task adds the change to the __ct table and writes in this table the time-stamp at which the change was committed on the source database.

Regards

Abrie