Good morning Qlik Community!
For once-daily Full Load operations, it would be *VERY* nice to have the capability of adding a dynamic folder name to the Target Folder field of an S3 Target Endpoint, or even to a Replication Task type that is targeted for S3 replication tasks.
For example, adding a dynamic DATE function so that each day the Replication Task will generate a new subfolder based on the date, and upload the daily parquet file to that subfolder.
IE:
- Target Endpoint S3 Bucket Name = my-s3-bucket-name
- Target Endpoint S3 Target Folder = dir1/dir2/dir3/subfolder/{DATE_VAR}
Would result in:
- Full Path (from 12/25/2022) = my-s3-bucket-name/dir1/dir2/dir3/subfolder/12-25-2022/table_name/fileload00001.parquet
- Full Path (from 12/26/2022) = my-s3-bucket-name/dir1/dir2/dir3/subfolder/12-26-2022/table_name/fileload00001.parquet
The DATE capability is the need for my current use-case, but I can think of at least a couple more instances where a dynamic folder name would be incredibly worthwhile, especially in an environment as large as ours.
An ALTERNATE suggestion (See Here: Qlik Replicate - S3 Replication Task Type (plus dy... - Qlik Community - 2020407) would be an S3 Replication Profile option during New Task creation, where the Target Endpoint data would be treated more like files instead of Database Schema/Table/Column/etc... fields, and the Target Endpoint itself would simply be a pointer to a top level directory.
For example, in an S3-Replication-Profile task, I would select the Source Schema/Tables/Columns - but for the S3 Task type, instead of "Map to target table" it would have "Map to Folder" options (that would include the capability to add a dynamic formula field- ie: DATE).
IE:
- Target Endpoint S3 Bucket Name = my-s3-bucket-name
- Target Endpoint S3 Target Folder = dir1/dir2/dir3/subfolder
- S3 Replication Task Settings Option: Target Folder = {DATE_VAR}
- -or, less desirable, a setting within each Table Mapping of each Replication Task-
- Target Subfolder Name (instead of Table Schema/Table Name fields) = {DATE_VAR}
Would result in:
- Full Path (from 12/25/2022) = my-s3-bucket-name/dir1/dir2/dir3/subfolder/12-25-2022/table_name/fileload00001.parquet
- Full Path (from 12/26/2022) = my-s3-bucket-name/dir1/dir2/dir3/subfolder/12-26-2022/table_name/fileload00001.parquet
Thanks for your time!
-mp