adding partition if not exists

badri-nair — Sat, 16 Nov 2024 02:50:44 GMT

Hello,

one of the extract BigData jobs we have writes hourly partition to the s3 output in parquet .

The component only supports overwrite or create mode.

In the hadoop configuration, I have tried setting the below

spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")

This is to actually to prevent from overwriting e everything in

the folder to only overwrite if it exists

But this does not seem to work.

Can you please suggest a way where we can get to overwrite partitions only if it exists instead of overwriting everything under the folder.

Thanks

Badri Nair

Re: adding partition if not exists

RBussi — Fri, 11 Jun 2021 15:00:03 GMT

Hi Team,

I am also facing same issue, anyone got the solution that could be great.

Thanks in advance