Skip to main content
Announcements
Have questions about Qlik Connect? Join us live on April 10th, at 11 AM ET: SIGN UP NOW

Replicate Sorter Files

No ratings
cancel
Showing results for 
Search instead for 
Did you mean: 
lyka
Support
Support

Replicate Sorter Files

Last Update:

May 11, 2022 5:27:09 AM

Updated By:

Sonja_Bauernfeind

Created date:

Dec 17, 2021 7:31:28 PM

Sorter files are located on the data directory of the Replicate server.

 

Sample location: 

C:\Program Files\Attunity\Replicate\data\tasks\<task name>\sorter

 

The twsp files are transactions that have not been applied to target yet. They are automatically purged after use. 

 

Sorter files will accumulate if there if either:

  • Big transactions from the source are not committed yet
  • Target transactions are not applied yet
  • Target transactions have been applied but no confirmation was returned

 

You cannot control the size of the sorter file however you can configure Replicate to use more memory and a longer transaction timeout.

 

Increasing the Transaction Offload Tuning values can help, especially the Transactions duration exceeds the default values. 

 

Locate the settings in:

 

  1. Designer --> Task Settings --> Change Processing --> Change Processing Tuning -->
  2. Transaction Offload Tuning Total transactions memory size exceeds (MB): 
    Set this to a larger memory allocation, such as 4G, 8G etc, depending on the total memory of the AR server.
  3. Transactions duration exceeds (seconds): 
    Set to a longer time, eg 1 hour etc, depending on the transaction requirements. 

It is not advisable to delete the tswp files in the sorter file as this will impact Replicate's ability to process the changes. It acts as a landing zone for changes which have not been applied to the target, so deleting those files can cause unexpected behaviour and possibly data loss.

 

Labels (1)
Comments
JrQ
Contributor II
Contributor II

Does Sorter files cleaned up after they are not needed?

Sonja_Bauernfeind
Digital Support
Digital Support

Hello @JrQ 

We can confirm that the .TWSP files are automatically purged by Replicate once consumed or have otherwise become invalid.

cdc
Partner - Contributor III
Partner - Contributor III

Hi Sonja,

Thank you before for the explanation, if our twsp file consist of not used task and not yet purged by Replicate, can we just delete them and will this action affect the Qlik Replicate?

Thank you,

Sonja_Bauernfeind
Digital Support
Digital Support

Hello @cdc 

Let me get clarification for you on this.

All the best,
Sonja 

KellyHobson
Support
Support

Hey @cdc ,

It is not advisable to delete tswp files in the sorter file as this will impact Replicate's ability to process the changes. It acts as a landing zone for changes which have not been applied to the target, so deleting those files can cause unexpected behavior and possibly data loss.

If your .twsp files are sitting in the sorter path of a non-use or non-active task, it is possible to delete them to clean up space. You need to ensure the task is completely stopped first. When you want to restart the task in the future you will either need to 1. Perform a reload of the task or 2. Restart from timestamp / or SCN;LSN depending on your source type, task settings, what data is already in your target.

So short answer: it is best practice to leave sorter path alone.

If you are in dire need of cleaning up space, you can do the manual deletes, but you will need to be prepared for a reload of the task. Additional note: If you stop the task and perform a reload or a restart from timestamp it will clean up the .twsp files and start "fresh" since it is doing a new query of the metadata.

Please let me know if you have any additional questions.

Best,

Kelly

cdc
Partner - Contributor III
Partner - Contributor III

Hi Kelly,

Thank you before,
Since it is not advisable to delete the tswp files, so if we are in a position of running out of memory, is there any recommendation to clean the space up?

Thank you,


cdc
Partner - Contributor III
Partner - Contributor III

Hi Kelly,

This question is a little out of the topic, but in case we are going to delete the tswp file, but the position of our task is disable which mean the task cannot be stopped.
Can we delete the tswp file in posisition of disable task?

Thank you,

KellyHobson
Support
Support

Hey @cdc 

For first question:

If you are running out of memory, you can stop the task and perform a restart from timestamp to clear out .twsp files.

See more information in this article:

https://community.qlik.com/t5/Knowledge/How-to-Start-Replicate-task-from-timestamp-or-bin-log-positi...

Additionally, the comments on this post are helpful about the sorter files:

https://community.qlik.com/t5/Qlik-Replicate/Qlik-replicate-task-folder-sorter/m-p/1918634#M2422

For second question:

It seems like there may be something going on if you are unable to properly stop the task. Do you plan to resume the task again if it is disabled? If you delete the files it will cause some missing data, so you will need to do a reload. I recommend opening a support case or posting on the community forum specific to why task getting disabled: https://community.qlik.com/t5/Qlik-Replicate/bd-p/qlik-replicate-discussions

Best,

Kelly

MoeyE
Partner - Creator II
Partner - Creator II

Hi  @KellyHobson ,

How is it decided whether swap files or attrep_changes tables are used ? 

Kind regards,

Mohammed

Sonja_Bauernfeind
Digital Support
Digital Support

Hello @MoeyE 

I gathered up an answer for you:

These two are entirely different mechanisms, so there is no real "or". 

Swap files are used on the source side and are temporary files used by Qlik Replicate. Qlik Replicate will not begin applying anything to the target until a commit arrives for a specific transaction - and while it will usually keep things in memory, when the transaction is too large, it is first written to disk.

attrep_changes, on the other hand, is a table which Qlik Replicate creates on any target when it is used in batch apply mode (default for Qlik Replicate). 

All the best,
Sonja 

Version history
Last update:
‎2022-05-11 05:27 AM
Updated by: