ESB or ETL Message Flow Tracking / Monitoring

[Jeeva]

Is there any inbuilt way in talend to monitor message flow which passes through each steps or components in the ESB route or ETL job? Some of the ESB products will have JMS topic and  if you listen then you can read the messages flowing through the each step or components. In my search i could see only the Service Monitoring option which can be used only in case of the cfx components. 

Please let me know this kind of features are available to talend. ? however we can implement manually implement this feature by adding logging component. but that wont be good to manage all the time. 

[Théo_Cap]

Maybe you can use tFlowMeterCatcher component ?

https://help.talend.com/reader/NNO~fmVQU4rlkF9Depfdxw/0oEBI6YCTFXSgc5Z9Gpr6Q

[Jeeva]

Thanks Aman. 

If we start  including a tFlowMeterCatcher component in between each step this will become too clumsy to handle the flow.  Instead writing the actual solution to the problem, this will lead to write a defect handling code. 

Is there any similar to this in ESB route..?

Thanks

[Anonymous]

@Jeeva the tFlowMeterCatcher is not used like that. I have written a tutorial showing how the AMC functionality can be used without any significant interference to the design of a job. You can find it here (https://www.rilhia.com/tutorials/job-using-talend-activity-monitoring-console-amc). I have also written some tutorials on the AMC in general. The first can be found here (https://www.rilhia.com/tutorials/talend-activity-monitoring-console-amc).

With regard to the ESB, take a look at the Spring tab (Window --> Show View --> Spring). You can see this described here (https://help.talend.com/reader/wfhJDzvjUeTs~09i4kMf7A/y0Us7J_ukdgxhe9Jx_o_NQ). This is a nice place to use some Spring code to intercept errors, etc. It can also be easily reused across multiple routes. This requires a bit of research and isn't quite as easy as the DI stuff.

[Jeeva]

@rhall Thanks for your suggestions.

Looks like active monitoring which you have mentioned mainly used as auditing... 

Mainly I'm looking for an option in ESB to see step by step tracking mechanism so that we can spot the issues directly by looking at the each step how the messages has been been transformed. 

However thanks for your reply and those links will be useful different scenarios... 

[Jeeva]

Im looking for something as in atttachment.. This mechanism used in Sonic ESB

Tracking.png

[Anonymous]

Ah.....you can seldom do things in the same way when using different tools. There are plenty of logging opportunities within Talend which you can implement once you know it well. An easy start would be to switch your Log4j functionality to DEBUG in your Karaf. That will give you a lot of data. If you are looking for something which is a bit more tailored, you can make use of cProcessor components and use Camel to add to what you get with the Log4j logging. Then, as mentioned before, there is the Spring functionality.  

Unfortunately (or fortunately for developers who want to keep themselves employed after finishing computer science degrees  🙂 ), none of this is easy without first spending time learning the product. The DI application is a lot easier to get used to than the ESB application.

[Jeeva]

You are right!! @rhall

We've tried in both DEBUG with Karaf and Adding cProcessor with Log4j mechanisms. As you said too many information in case of enabling them. In prod servers sometimes due to load sometimes very hard to go through the each files. Talend takes very less time to develop things. but takes longtime to locate which component causes the issue. We've written custom java code to read both message body along with its headers to read. But that makes code clumsy sometimes. However we are still using the custom cProcessor component

Other products i felt long time to develop but fast to locate the issue. Here development is made simple but takes relatively long time to locate the issue. 

[Anonymous]

The problem you have with an Enterprise Service Bus is that literally thousands of messages can be processed in seconds. These messages can be very verbose. This kind of prohibits and "easy" logging system which will both allow for the performance and make it easy to find that data. What you could try is making use of the Spring functionality (only really of any use within the Karaf) and send logging messages (or just known errors) to a message queue. Then you could read from that queue into a database to make it easier to pinpoint your your issue.

With Data Integration it is pretty easier to produce a generic logging system (the AMC is a good example....and you can very easily add to it in a generic way). With ESB it really isn't as straight forward. I find it a good idea to build your routes in a way which supports a framework which suits your requirements for logging. Sometimes this impacts performance a little, but it is certainly worth it if you are experiencing issues like I suspect you are.