Qlik Community

Ask a Question

Qlik Compose for Data Lakes Discussions

Discussion board for collaboration on Qlik Compose for Data Lakes.

Announcements
QlikWorld Online 2021, May 10-12: Our Free, Virtual, Global Event REGISTER TODAY
cancel
Showing results for 
Search instead for 
Did you mean: 
Abrie_M
Contributor II
Contributor II

Compose Hive project: header__from_date in final tables

Good day
 
In Hive projects, how is the value of the column header__from_date determined in tables/views in the Storage Zone?
 
Is it the timestamp that the record was changed on the source? Or when it was handled by Compose?
 
The only reference I could get in the documentation is: The name of the "From Date" column. This column is added to tables that contain attributes (columns) with history. The column is used to delimit the range of dates for a given record version.
 
Still waiting for feedback from Qlik support. Will post comments here if/when they reply.
 
Thanx
Labels (1)
1 Solution

Accepted Solutions
TimGarrod
Employee
Employee

Hi Abrie_M - the default behavior by Compose is to populate the header__from_date using the header__timestamp value that Replicate delivers to the __ct tables in the landing area, but you can over-ride this. 

hen you look at a Compose mapping - if the from_date column in the mapping is NOT mapped - Compose will use the header__timestamp from Replicate.  (For a full load, since Compose doesn't know when the records where written to your source it will use the Lowest Date as defined in the project settings).

If you wish to use data values that are in the data (for example your source provides type 2 data) or a different value - you simply need to map the from_date column in the mapping. 

FYI - The header__timestamp  is (according to Replicate user guide) - the original change UTC timestamp (the value may be approximate)

 

Hope that helps!

View solution in original post

2 Replies
TimGarrod
Employee
Employee

Hi Abrie_M - the default behavior by Compose is to populate the header__from_date using the header__timestamp value that Replicate delivers to the __ct tables in the landing area, but you can over-ride this. 

hen you look at a Compose mapping - if the from_date column in the mapping is NOT mapped - Compose will use the header__timestamp from Replicate.  (For a full load, since Compose doesn't know when the records where written to your source it will use the Lowest Date as defined in the project settings).

If you wish to use data values that are in the data (for example your source provides type 2 data) or a different value - you simply need to map the from_date column in the mapping. 

FYI - The header__timestamp  is (according to Replicate user guide) - the original change UTC timestamp (the value may be approximate)

 

Hope that helps!

View solution in original post

Abrie_M
Contributor II
Contributor II
Author

Hi @TimGarrod 

Thanx for your detailed feedback.

Qlik support also replied: When the change occurs in the source DB, the Replicate CDC task adds the change to the __ct table and writes in this table the time-stamp at which the change was committed on the source database.

Regards

Abrie