topic Re: delta.columnMapping.mode in Talend Studio

delta.columnMapping.mode

MBourassa1682971203 — Fri, 15 Nov 2024 21:10:07 GMT

Hi,

I have created a big data batch in spark 3.3.

I use a TDeltaLakeOutPut to write in a Delta files.

How can I add this option "delta.columnMapping.mode" ?

Thank you

Anonymous — Thu, 21 Dec 2023 02:33:06 GMT

Hello,

Please try to spark properties in Spark Configuration UI and feel free to let us know if it works.

Best regards

Sabrina

MBourassa1682971203 — Thu, 21 Dec 2023 19:17:13 GMT

Hi,

it works !!

Thank you very much

I'm looking for this since a long time.

Can you tell me where did you find theses properties ?

Thank you

MBourassa1682971203 — Thu, 21 Dec 2023 20:41:01 GMT

It works but there is a litle problem with the column partition.

This is my configuration:

and this is the result:

instead of this:

MBourassa1682971203 — Thu, 21 Dec 2023 20:42:25 GMT

this appen only if I add "spark.databricks.delta.properties.defaults.columnMapping.mode" "name" to the spark properties.

Anonymous — Mon, 25 Dec 2023 02:21:04 GMT

Hello,

I will check it with our big data experts and then come back to you as soon as possible.

Best regards

Sabrina