Skip to main content
Announcements
Introducing Qlik Answers: A plug-and-play, Generative AI powered RAG solution. READ ALL ABOUT IT!
cancel
Showing results for 
Search instead for 
Did you mean: 
Al_gar
Contributor III
Contributor III

Some NULL values on key columns

Hi,

We're getting NULL values on key columns for some records replicated from HANA with CDC logs.

We're using using Apply changes and incremental load filter pattern. There's only one global rule to set the ChangeSeq  value for the incremental load.

Can you please help determine the possible issue and solution? 

Labels (2)
1 Solution

Accepted Solutions
Al_gar
Contributor III
Contributor III
Author

For anybody experiencing the same issue, I met with the Support team and it was found that the cause of NULL values on Primary Key columns  was the Global Error Handling  "Apply conflict" set up for the task is not supported for the source endpoint. When using SAP HANA CDC log-based the "Apply Conflicts > for the corresponding task need to be set to an option other than "INSERT the missing target record".

View solution in original post

3 Replies
Heinvandenheuvel
Specialist III
Specialist III

>> We're getting NULL values on key columns for some records replicated from HANA with CDC logs.

Those are always 'fun' problems to tackle and more often than not require detailed investigation of the data stream.

You'll need to be a lot, whole lot, more precise and concise with your information here to have any hope that anyone can magically provide a solution. We are NOT mind readers. More than likely you'll need a support ticket for this, which also needs all the data so you may as well start here.

"key columns" - as in IMPORTANT columns or columns which are part to the Primary Key (the only key Replicate cares about). What are the (common) attributes for those 'key' columns? Dates? Numbers? Precision? Name?

"from HANA "  - Where is the data going to ? What versions for Replicate? Source DB, Target DB, Drivers?

"incremental load filter pattern." - What does that mean? The opening suggests CDC activity? 

"ChangeSeq" - Do you mean  'changeSequence'. It's unlikely to be relevant but  what do you do to it?

"Can you please help determine the possible issue and solution? " - No Way -  Look back at the information you provided - how on earth can anyone help with that - UNLESS they had a possibly similar issue. 

We can help you for you to determine the cause and perhaps a solution.

Have you checked the REPTASK_xxx.LOG files and APPLY_EXCEPTION tables for any Warnings?

"some records" - Here is the biggest clue. Some, but not all. Correct? So all YOU need to do is determine what make some records special. Since the method used (implicit conversions) works some of the time the most likely cause is that some specific data values cannot be handled.  One classic example is an invalid date on the source. It might even be valid on the source, but not on the target.   The best approach to this is to 'find' a handful of rows with the issue on the target, and then find the corresponding rows on the source and see what the data values for the bad columns look like. Do you see a pattern? Look at some rows 'next' to the bad rows which were valid. What is significantly  difference? very high, very low? 0 vs value? SPACE vs values? Accented/UTF-16 very special characters versus 'regular' ascii? 

If you cannot figure it out you may want to  add a reply here (and to the support case once you have that) to attach a CSV file with SOURCE and TARGET values - with rows reduced to PK + the suspect column(s) and some good, and some bad rows.

Good luck!

Hein.

 

Al_gar
Contributor III
Contributor III
Author

Oooh I'm so sorry Heins. My intention was to draw attention to the people that might have had the same issue we have, but thanks for the reply...it's quite extensive (TL;DR) so I'll just cut out the weed and add some details in case someone can reply with valuable content:

We're replicating tables from HANA S/4 (UTF-8) to SQL Server 2019.
About the key column data type, as mentioned earlier we're getting NULL on 'key columnS' (i.e. many columns) across many tables replicated. But nvarchar(3) is one of the data types. Nothing special or crazy going with the column data types...otherwise I would have note it.

Incremental load filter pattern...it's a workaround from a Qlik Whitepaper to be able to capture changes from HANA CDC logs. You may search around and read on the matter.

Only errors we have seen are related to insert errors due to dups.

Thanks!

Al_gar
Contributor III
Contributor III
Author

For anybody experiencing the same issue, I met with the Support team and it was found that the cause of NULL values on Primary Key columns  was the Global Error Handling  "Apply conflict" set up for the task is not supported for the source endpoint. When using SAP HANA CDC log-based the "Apply Conflicts > for the corresponding task need to be set to an option other than "INSERT the missing target record".