Attunity Replicate - Batch Tuning - Force apply batch memory

[randomtask]

Can someone explain what the following setting really means? Is this setting an aggregate of all the data Attunity Replicate collects from multiple tables? 

I am assuming it's an aggregate and that the resulted files (AWS S3) for a single batch all add up to the below limit within the timeframe?

Batch Tuning- "Force apply a batch when processing memory exceeds (MB):"

See setting in screen below:

Any help is appreciated.

Cheers!

Emilio

[Alan_Wang · Accepted solution]

Hi Emilio,

Replicate will apply changes on the target only when the transaction is closed, if a commit arrives the transaction is applied, if a rollback arrives, the events for this transaction are removed. Batch Optimized Apply mode commits the changes in batches. In this case, a pre-processing action occurs to group the transactions into batches in the most efficient way.  

In Batch Optimized Apply mode when a commit arrives, events are transferred in bulks of many events to a special table (created by the task), the events are transferred as row events, and from that table, they are applied to the target tables, first the deletes, then the inserts and finally the updates. In addition, operations on the same row are being optimized, for example, if an insert is encountered and then an update or delete is encountered for the same row, those operations will be reduced to just one operation.

The Batch tuning settings apply to the single task. It allows you to change the amount of memory you allocate to a single batch process. The more memory you allocate, the more events a single batch process will be able to apply. Target endpoints such as Teradata, Vertica, and Pivotal Greenplum are optimized for processing large batches. This setting allows you to tune the size of the batch process for better performance.

[randomtask]

Hi Alan,

Thank you for your detailed reply. So, you are saying that the total amount of data that ends up at the target is no more than what's configured in the "Force apply a batch when processing memory exceeds (MB):" setting?

Thank you for your kind help.

Emilio

[Alan_Wang]

Hi Emilio,

The limit is only for a single batch process. Replicate will continuously apply batch processes until all the transactions are replicated over to the target endpoint.

[Kohila]

Hi Alan,

Could you please give more insight on the statement  "

if an insert is encountered and then an update or delete is encountered for the same row, those operations will be reduced to just one operation."

This happend because of the setting 'Batch Optimized Apply mode".

We are in the scenario we lost our inserts to the target or change table which comes to the source before we resume our replicate task. i.e only updates has entry but no inserts. In these case these records are entered into apply_exception instead target. 

[Michael_Litz · Accepted solution]

Hi Kohila,

One of the features replicate will do when running in batch apply mode is to consolidate multiply operations that occur on a single record in the source. An example would be if the source record was Inserted, then updated and updated again, all within the same transaction then replicate would consolidate those three operations into a single insert statement. This consolidation of operations servers to reduce the number of operations on the target to a single transaction.

1) Insert statement (with all changes)

To contrast this with Transactional apply mode the same scenario from above would result in three discreate operations on the target.

1) Insert statement

2) Update statement

3) Update statement

This is one reason why running in Batch Apply Mode is usually faster then running in transactional Mode.

I hope this helps out.

Thanks
Michael

[Kohila]

Thanks Michael_Litz  for detailed explanation..

[Kohila]

@Michael_Litz  - One other scenario as follow up, When we have delete and insert event for the same record in batch optimized, will the Delete will be applied or insert event?

[Heinvandenheuvel]

>> When we have delete and insert event for the same record in batch optimized

Well, it wouldn't be the same record, would it now? It would just a a row with the same PK, likely with a different DBkey (DB specific, rarely exposed).

Replicate _could_ replace that by a single update, but I don't think it does.

But why don't you try? Seriously! 

Just create a test task with just a single test table. Switch to target apply - trace and run your experiments.

Check the logs ('tail -f') to see what happens for the various scenarios depending on order and commit points.

Hein