S3 Endpoint Target - Dynamic Date/Time Folder?

[mp-sf]

Hello!

I have an S3 Replication task setup to copy a set of data once a day, every day, and write to a Parquet file on my S3 Endpoint.  I'm trying to figure out a way to generate a dynamic folder under my S3 Endpoint so I can retain each days load without it overwriting/replacing the previous days load.

IE:

• Today the file would write to the static S3 Endpoint Bucket "s3:mybucket", the static S3 Endpoint Folder "sub1/sub2/sub3" but the task would create a folder named "12-23-2022" and write to "Load000000001.parquet"  (s3:mybucket/sub1/sub2/sub3/12-23-2022/Load000001.parquet)
• Tomorrow the file would write to the same static S3 Endpoint Bucket/Folder, but it would create a folder named "12-24-2022" and write to it's Load*.parquet file (s3:mybucket/sub1/sub2/sub3/12-24-2022/Load000001.parquet)

Any ideas/thoughts on how to accomplish that?

Thanks,

Mike

[john_wang]

Hello @mp-sf ,

Seems no straight setting to accomplish that, per my test (in the past) the  DATE can be a part of the folder name only (see my screen copy below), I'd like to suggest to:

1- open a support cases to get more investigation, or Feature Request involved

2- Utilizes  Post Upload Processing  to process the uploaded files.

I hope this helps.

Regards,

John.