adding partition if not exists

[badri-nair]

Hello, 

one of the extract BigData jobs we have writes hourly partition to the s3 output in parquet .

The component only supports overwrite or create mode.

In the hadoop configuration, I have tried setting the below

spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")

This is to actually to prevent from overwriting e everything in

the folder to only overwrite if it exists 

But this does not seem to work.

Can you please suggest a way where we can get to overwrite partitions only if it exists instead of overwriting everything under the folder.

Thanks

Badri Nair

[RBussi]

Hi Team,

I am also facing same issue, anyone got the solution that could be great.

Thanks in advance