-Task Settings -> Change Processing Tunning -> Transaction Offload Tuning -> Offload transactions in progress to disk if -> Total transactions memory size exceeds (MB)
Thanks for your contribution to Qlik Community. I agree, at first consideration the differences between these settings is not readily apparent.
The Amazon S3 target endpoint is file based as you know. Qlik Replicate uses separate files for the full load phase and the change processing/store changes phase of a task. As such, the maximum file size setting on the target endpoint relates to the full load files, and the Apply/store changes when: File size reaches(KB) relates to the change processing/store changes files (after the full load completes). These settings allow to choose different file sizes for the full load phase versus the change processing/store changes phase of the task if you so desire.
Here's a link to our documentation on these settings:
Conversely, the transaction offload tuning is a memory management setting. During times of high activity on the source or performance/connection issues to the target, this setting determines at what point changes are cached to disk on the Replicate server as files so as not to completely exhaust system memory which may cause the application or server to fail. These cached files are stored until they can be sent to the target to ensure data read from the source is not lost. These cached files are stored in the task's sorter folder in a binary format; they are not 1:1 or the same as the files sent to the target.
Thanks for your contribution to Qlik Community. I agree, at first consideration the differences between these settings is not readily apparent.
The Amazon S3 target endpoint is file based as you know. Qlik Replicate uses separate files for the full load phase and the change processing/store changes phase of a task. As such, the maximum file size setting on the target endpoint relates to the full load files, and the Apply/store changes when: File size reaches(KB) relates to the change processing/store changes files (after the full load completes). These settings allow to choose different file sizes for the full load phase versus the change processing/store changes phase of the task if you so desire.
Here's a link to our documentation on these settings:
Conversely, the transaction offload tuning is a memory management setting. During times of high activity on the source or performance/connection issues to the target, this setting determines at what point changes are cached to disk on the Replicate server as files so as not to completely exhaust system memory which may cause the application or server to fail. These cached files are stored until they can be sent to the target to ensure data read from the source is not lost. These cached files are stored in the task's sorter folder in a binary format; they are not 1:1 or the same as the files sent to the target.
