Compose Metadata Transformation

[Abrie_M]

Good day

Are there any examples available of an expression to edit an attribute in the Compose for Data Lakes "Manage Metadata" screen?

We have datetime columns with the value '0001-01-01', which we need to transform to a date of '1902-0101'

Is this even possible ?

Thanx!

Abrie Marais

[Abrie_M · Accepted solution]

Hi

Looks like we got it working with adding the following expression in the mapping:

case `ens3257_e_dt` < '1902-01-01 00:00:00:000'
when true then '1902-01-01 00:00:00:000'
else `ens3257_e_dt`
end

We've first tried adding the expression to the attribute on the Manage Metadata screen, and it worked for the full load, but for the CDC task, it failed. From the task sql, when an expression is applied to a column, the relevant column is not selected from the __ct table, so the task fails with a sql error, column not found.

Just to add, we are running on Cloudera CDP 7.1, Replicate 7.0 and Compose 6.6.