Skip to main content

Qlik Compose for Data Warehouses

Discussion board for collaboration on Qlik Compose for Data Warehouses.

cancel
Showing results for 
Search instead for 
Did you mean: 
garyclark
Contributor
Contributor

Compose Inconsistency

Hi Team,

We are seeing inconsistency while using compose, below is the scenario:

Inserted new record in source, replicate captured the change in CT tables and inserted new recording into our landing db. However when we ran the incremental work flow in compose the change in CT table is not picked up hence data is not loaded into dwh. I still see the record in CT table even after the incremental work flow ran successfully .

Repeated the same process but this time compose picked by new record and also the previous record which was not picked by earlier and loaded both the records into DWH. This is just one example there were many instances of these issues. Can you please let us know how this can be avoided? 

2 Solutions

Accepted Solutions
aj96
Contributor III
Contributor III

Compose uses max(seq_id) for each ct table before running the staging query. Maybe at the time of run the changes might not have been committed in ct resulting in not picking any records for the process.  

View solution in original post

TimGarrod
Employee
Employee

Hi @garyclark , as AJ96 said - the first thing Compose does in a CDC ETL Task is grab the max(seq) from the __CT table.   (When we process change data, Compose needs to grab a static set of data and load to the TSTG table). 

TimGarrod_1-1639673337607.png

It then uses this to grab only those values that existed when the process started - 

TimGarrod_2-1639673377148.png

 So its likely that data wasn't fully committed, or wasn't in the CT table when the Compose job ran. 

If you need to validate the value that Compose captured, you can enable  DEBUG level logging for the CDC task and view the ETL logs to get the number. 
To enable DEBUG logging - under the Designer, drill into the Data Warehouse / ETL management section of the UI.

Highlight the task , select Settings and change the log level to DEBUG (in the example, I'm changing the settings for just the Sales_Source_CDC task to DEBUG).  (This does not require generation of code).

TimGarrod_4-1639674087305.png

Run the ETL. 

When complete - in the Monitor select the Task > History - select the run and Click"View Log"

TimGarrod_5-1639674159051.png

 

Search for the appropriate MaxSeq_# value - and look for the log message that provides the seq number parameter

TimGarrod_6-1639674293244.png

Review that number with the value of the records in the CT table.   I expect you'll find it will be lower than the CT records.

header__change_seq description - (https://help.qlik.com/en-US/replicate/November2021/Content/Replicate/Main/Change%20Tables/read_chang...

 

 

Hope that helps!!

 

Tim.

 

View solution in original post

2 Replies
aj96
Contributor III
Contributor III

Compose uses max(seq_id) for each ct table before running the staging query. Maybe at the time of run the changes might not have been committed in ct resulting in not picking any records for the process.  

TimGarrod
Employee
Employee

Hi @garyclark , as AJ96 said - the first thing Compose does in a CDC ETL Task is grab the max(seq) from the __CT table.   (When we process change data, Compose needs to grab a static set of data and load to the TSTG table). 

TimGarrod_1-1639673337607.png

It then uses this to grab only those values that existed when the process started - 

TimGarrod_2-1639673377148.png

 So its likely that data wasn't fully committed, or wasn't in the CT table when the Compose job ran. 

If you need to validate the value that Compose captured, you can enable  DEBUG level logging for the CDC task and view the ETL logs to get the number. 
To enable DEBUG logging - under the Designer, drill into the Data Warehouse / ETL management section of the UI.

Highlight the task , select Settings and change the log level to DEBUG (in the example, I'm changing the settings for just the Sales_Source_CDC task to DEBUG).  (This does not require generation of code).

TimGarrod_4-1639674087305.png

Run the ETL. 

When complete - in the Monitor select the Task > History - select the run and Click"View Log"

TimGarrod_5-1639674159051.png

 

Search for the appropriate MaxSeq_# value - and look for the log message that provides the seq number parameter

TimGarrod_6-1639674293244.png

Review that number with the value of the records in the CT table.   I expect you'll find it will be lower than the CT records.

header__change_seq description - (https://help.qlik.com/en-US/replicate/November2021/Content/Replicate/Main/Change%20Tables/read_chang...

 

 

Hope that helps!!

 

Tim.