Global transaction

[Anonymous]

Hi,
I have a scenario where I want to get different resources (file in/output, AMQ, DB) involved in a single global transaction. Is there a way to achieve that?
Adrie

[Anonymous]

Hi Adrie
Maybe you can use Run-if to trigger different subjobs.
Regards,
Pedro

[Anonymous]

Ok, thanks, will have a look at that.
Does that imply that a commit/rollback has to be explicitly added to a job instead of being executed automatically by a transaction manager?

[Anonymous]

Hi
Which version of TOS do you use, DI, DQ, ESB or MDM?
Could you make it more clear? Transaction manager?
Regards,
Pedro

[Anonymous]

I'm using TOS for ESB 5.1.0.
The (simplified) scenario I want to implement is as follows:
1. read file from disk
2. write to AMQ queue
3. read from AMQ queue
4. write as file to disk
I want step 1 & 2 and 3 & 4 be reliable so if for some reason the write to AMQ fails (2) the file shouldn't be deleted from disk and if the write to disk fails (4) the message should stay in the AMQ queue.
The file pickup of step 1 could also be a read from a JMS queue e.g. WMQ.
So because of the different resources (file, AMQ, WMQ) to be involved in one single transaction I was thinking about a global (XA) transaction with a transaction manager taking care of the commit/rollback of the work.
Hope this clarifies my question a bit.

[Anonymous]

Hi
For topics related to ESB, please move to section.
Regards,
Pedro

[Anonymous]

Any thoughts how I can implement the above scenario?

[Anonymous]

Hi ahr,

I want step 1 & 2 and 3 & 4 be reliable so if for some reason the write to AMQ fails (2)
the file shouldn't be deleted from disk and if the write to disk fails (4) the message
should stay in the AMQ queue.

I believe you've chose a good approach to do transactions only between AMQ and another component. You can do transaction only from parts using AMQ. Anyway - file system is not transactional, so you cannot have "full transactional" support
There are two ways you can do it in Talend ESB
1. use a Camel route - if is faster, more lightweight, but to do transactional operations, it's a little trickier to set up. But to be simple - try to use file->queue with acknowledgementModeName parameter on the jms/activemq endpoint ( http://camel.apache.org/jms.html) and delete option in the file component.
2. use a talend job
- tWaitForFile -> tMomOutput -> iMomCommit
|
| on Sobjob Ok
\/
tFileDelete

- tMomInput -> tFileOutput
|
| on Sobjob Ok
\/
tMomCommit
and for tMomInput and tMomOuput you can look into the "Advanced" folder and set Acknowledgement mode and Transacted flag. (don't forget about tMomRollback on error )
Hope it helped a little.
Gabriel

[Anonymous]

Does your system need to be fault tolerant and support failover? If so, then you will need to take a few other things into account in order to handle the edge cases.
First, if you have the option of using a transacted resource in step 1 then you have three options. If both JMS receive and send are with the same JMS provider then you can just use JMS transactions which are made to span the very common receive-send use case. This is best if it is available. But it sounds like you may well have two different JMS providers. In this case XA is an option.
A third option, is to use the JMS receiver along with an idempotent consumer pattern. This will handle the edge case for failures. If the received message is successfully written to the target queue you are all set. If it fails, it will roll back to the receiving queue and be reprocessed x times before flowing to the dead letter queue. When it is resubmitted you will be able to check the JMS headers and take appropriate action. If for some reason your system has crashed so that the message queue thinks the message needs to be resubmitted when in fact it has been processed, the idempotent consumer will protect you.
In general the only option for handling non-transactional resources such as the file system is compensating transactions. In Camel these are implemented with the Unit of Work with the OnCompletion interface. See http://camel.apache.org/oncompletion.html .
I would recommend the Camel / ESB approach over the Job approach if this is a HA scenario. The use case is more naturally implemented in ESB IMO.
Also, see the semantics of the Camel cFile component here http://camel.apache.org/file.html . You will want to use the endpoint and header settings to manage exactly when / how a file is removed so that it is processed exactly once under all conditions.
Your use case may sound simple, but that doesn't mean it is easy. But it is fundamental so once you have it down you will get a lot of mileage out of it. Good luck.

[Anonymous]

Thanks for the responses. I will have a look.