S3 Endpoint Target - Dynamic Date/Time Folder?

mp-sf — Fri, 23 Dec 2022 16:35:12 GMT

Hello!

I have an S3 Replication task setup to copy a set of data once a day, every day, and write to a Parquet file on my S3 Endpoint. I'm trying to figure out a way to generate a dynamic folder under my S3 Endpoint so I can retain each days load without it overwriting/replacing the previous days load.

IE:

Today the file would write to the static S3 Endpoint Bucket "s3:mybucket", the static S3 Endpoint Folder "sub1/sub2/sub3" but the task would create a folder named "12-23-2022" and write to "Load000000001.parquet" (s3:mybucket/sub1/sub2/sub3/12-23-2022/Load000001.parquet)
Tomorrow the file would write to the same static S3 Endpoint Bucket/Folder, but it would create a folder named "12-24-2022" and write to it's Load*.parquet file (s3:mybucket/sub1/sub2/sub3/12-24-2022/Load000001.parquet)

Any ideas/thoughts on how to accomplish that?

Thanks,

Mike

Re: S3 Endpoint Target - Dynamic Date/Time Folder?

john_wang — Mon, 26 Dec 2022 07:20:21 GMT

Hello @mp-sf ,

Seems no straight setting to accomplish that, per my test (in the past) the DATE can be a part of the folder name only (see my screen copy below), I'd like to suggest to:

1- open a support cases to get more investigation, or Feature Request involved

2- Utilizes Post Upload Processing to process the uploaded files.

I hope this helps.

Regards,

John.

topic Re: S3 Endpoint Target - Dynamic Date/Time Folder? in Qlik Replicate

S3 Endpoint Target - Dynamic Date/Time Folder?

Re: S3 Endpoint Target - Dynamic Date/Time Folder?