Adding a transform to an existing task/ prevent table triggering a FULL load

[Michael_Litz]

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

Field Transformation

When adding a field level transformation to an existing table the table will be flagged by the task for reload. This is especially important if the transformation will change the table structure, like adding a field.

In this case the transformation will be a simple data obfuscation. The topic of obfuscation would command and entire article to itself, and this article is just using it as an example since it would trigger a table reload.

 

The general form of the transformation is as follows:

'No Reload' a simple string that will replace the actual data.

As simple as this transformation is, it will cause the table to be reloaded.

Resolution - Prevent table Reload

The way to prevent the table from reloading is to use the advance run option and select the option:
Tables are already loaded. Start processing changes from:

With this option, the table metadata will be refreshed and the table will not Reload.

NOTE: Full Load monitor screen - looks like table reload

It is normal behavior for the task to go through the Full Load phase as this is where the task metadata will be refreshed and the table transformation will be recognized going forward.

When the Full Load phase kicks off during the restart with timestamp it does not actually move any data (do the full load) as indicated by the transferred record count which stays at 0 records.

Environment

• Qlik Replicate 
• Table/Field level transformation

[RichardP]

The popup describes an "Apply Changes Only" task. Is that a solution too or something else?

[Michael_Litz]

Hi Richard,

Apply Changes Only in the pop up is referring to the phase of replicate task and the type of task.

Replicate task can do a Full Load phase (initial copy of the entire table from source to target) and CDC (apply changes) where the changes are read from the transaction log and applied tot he target.

The Apply Changes Only task in the pop up refers to a task with only the CDC phase enabled. In fact even a Full Load with CDC task will have the same behavior with the adding of a transformation. Which is to want to do a reload of the table - to apply the transformation to all records.

When we do the restart with timestamp we are telling the task that we want the new transformation for future records read form source logs, but we will not apply it to existing records in the target table. So that whatever the transformation does you would need to apply it manually to the existing records in the target.

Thanks,
Michael