idea Re: Qlik Replicate - S3 Replication Task Type (plus dynamic folder vars) in Suggest an Idea

Qlik Replicate - S3 Replication Task Type (plus dynamic folder vars)

mp-sf — Wed, 28 Dec 2022 17:05:24 GMT

Create an S3 Replication Profile option during New Task creation, where the Target Endpoint data would be treated more like files instead of Database Schema/Table/Column/etc... fields, and the Target Endpoint itself would simply be a pointer to a top level directory.

For example, in an S3-Replication-Profile task, I would select the Source Schema/Tables/Columns - but in the S3 Task type, instead of "Map to target table" it would have "Map to Folder" options (that would include the capability to add a dynamic formula field- ie: DATE).

IE:

Target Endpoint S3 Bucket Name = my-s3-bucket-name
Target Endpoint S3 Target Folder = dir1/dir2/dir3/subfolder
S3 Replication Task Settings Option: Target Folder = {DATE_VAR}
-or, less desirable, a setting within each Table Mapping of each Replication Task-
- Target Subfolder Name (instead of Table Schema/Table Name fields) = {DATE_VAR}

Would result in:

Full Path (from 12/25/2022) = my-s3-bucket-name/dir1/dir2/dir3/subfolder/12-25-2022/table_name/fileload00001.parquet
Full Path (from 12/26/2022) = my-s3-bucket-name/dir1/dir2/dir3/subfolder/12-26-2022/table_name/fileload00001.parquet

Thanks for your time!

-mp

From now on, please track this idea from the Ideation por...

Meghann_MacDonald — Wed, 02 Aug 2023 16:01:42 GMT

From now on, please track this idea from the Ideation portal.

Link to new idea

Meghann

NOTE: Upon clicking this link 2 tabs may open - please feel free to close the one with a login page. If you only see 1 tab with the login page, please try clicking this link first: Authenticate me! then try the link above again. Ensure pop-up blocker is off.

Re: Qlik Replicate - S3 Replication Task Type (plus dynamic folder vars) - Status changed to: Closed - Archived

Ideation — Wed, 02 Aug 2023 16:01:43 GMT

Re: Qlik Replicate - S3 Replication Task Type (plus dynamic folder vars)

john_wang — Wed, 23 Apr 2025 15:51:39 GMT

This can be achieved by transformation, see:

https://community.qlik.com/t5/Qlik-Replicate/Qlik-Replicate-can-we-create-different-runtime-folder-for-change/td-p/2515457