Skip to main content
Announcements
UPGRADE ADVISORY for Qlik Replicate 2024.5: Read More
cancel
Showing results for 
Search instead for 
Did you mean: 
mp-sf
Contributor II
Contributor II

S3 Endpoint Target - Dynamic Date/Time Folder?

Hello!

I have an S3 Replication task setup to copy a set of data once a day, every day, and write to a Parquet file on my S3 Endpoint.  I'm trying to figure out a way to generate a dynamic folder under my S3 Endpoint so I can retain each days load without it overwriting/replacing the previous days load.

IE:

  • Today the file would write to the static S3 Endpoint Bucket "s3:mybucket", the static S3 Endpoint Folder "sub1/sub2/sub3" but the task would create a folder named "12-23-2022" and write to "Load000000001.parquet"  (s3:mybucket/sub1/sub2/sub3/12-23-2022/Load000001.parquet)
  • Tomorrow the file would write to the same static S3 Endpoint Bucket/Folder, but it would create a folder named "12-24-2022" and write to it's Load*.parquet file (s3:mybucket/sub1/sub2/sub3/12-24-2022/Load000001.parquet)

Any ideas/thoughts on how to accomplish that?

Thanks,

Mike

Labels (1)
1 Reply
john_wang
Support
Support

Hello @mp-sf ,

Seems no straight setting to accomplish that, per my test (in the past) the  DATE can be a part of the folder name only (see my screen copy below), I'd like to suggest to:

1- open a support cases to get more investigation, or Feature Request involved

2- Utilizes  Post Upload Processing  to process the uploaded files.

john_wang_0-1672039177079.png

 

I hope this helps.

Regards,

John.

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!