Hello,
one of the extract BigData jobs we have writes hourly partition to the s3 output in parquet .
The component only supports overwrite or create mode.
In the hadoop configuration, I have tried setting the below
spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")
This is to actually to prevent from overwriting e everything in
the folder to only overwrite if it exists
But this does not seem to work.
Can you please suggest a way where we can get to overwrite partitions only if it exists instead of overwriting everything under the folder.
Thanks
Badri Nair
