Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Overview
This article expands on the Talend Knowledge Base (KB) article, Improve Camel Flow Monitoring, by demonstrating how to use the monitoring solution for routes created in Talend Studio, and how to minimize changes in the routes. It also provides a sample of how to extend the information written to the sinks by Karaf Decanter.
The article includes:
- A Java project that you can deploy in Talend Runtime
- A configuration for the Decanter Camel collector to monitor all routes deployed to the runtime and avoid the need to configure and monitor each route separately
- A sample class that shows you how to extend the information to be stored
- A Studio project enabled for monitoring
Prerequisites
-
Read the Knowledge Base (KB) article Improve Camel Flow Monitoring
-
Basic Java knowledge
-
Experience building bundles for Talend Runtime using Maven
Configuring Decanter Camel
Opposite to the configuration file in the Improve Camel Flow Monitoring KB article, which is a Spring file, the one provided in the Java project is an OSGi blueprint file, as Spring is not supported in version 7.0. In addition, it registers the DecanterEventNotifier bean and Tracer bean as a service in the OSGi registry. When running in an OSGi environment, Camel includes the OSGi registry to lookup beans. So, registering the beans in the OSGi registry ensures they are found by the Camel components dealing with tracing and event notification. The Tracer bean is class provided by Camel allowing you to configure how the traces are logged. The Apache Camel documentation page, Camel Tracer, provides information about the available configuration properties.
Without having to build the project, you can copy the configuration file below, or use the decanter-camel.xml file attached to this article, into the deploy directory. The logLevel property of the Tracer bean is set to OFF, that disables tracing. If tracing is enabled, the property must have one of the following values: FATAL, ERROR, WARN, INFO, DEBUG, TRACE.
Blueprint configuration for Decanter Camel components
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
xmlns:cm="http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.0.0">
<cm:property-placeholder persistent-id="org.talend.ps.decanter-camel">
<cm:default-properties>
<cm:property name="camel.event.includeBody" value="false" />
<cm:property name="camel.event.includeProperties" value="true" />
<cm:property name="camel.event.includeHeaders" value="true" />
<cm:property name="camel.event.silent" value="true" />
<cm:property name="camel.trace.enabled" value="false"/>
<cm:property name="camel.trace.includeBody" value="false"/>
<cm:property name="camel.trace.includeProperties" value="true" />
<cm:property name="camel.trace.includeHeaders" value="true" />
</cm:default-properties>
</cm:property-placeholder>
<reference id="eventAdmin" interface="org.osgi.service.event.EventAdmin"/>
<bean id="decanterEventNotifier"
class="org.apache.karaf.decanter.collector.camel.DecanterEventNotifier">
<property name="eventAdmin" ref="eventAdmin"/>
<!-- Optional to add custom properties to the event -->
<!--<property name="extender" ref="customExtender" />-->
<property name="includeBody" value="${camel.event.includeBody}"/>
<property name="includeProperties" value="${camel.event.includeProperties}"/>
<property name="includeHeaders" value="${camel.event.includeHeaders}"/>
<property name="ignoreCamelContextEvents" value="true" />
<property name="ignoreRouteEvents" value="true" />
<property name="ignoreServiceEvents" value="true" />
<property name="ignoreExchangeEvents" value="false" />
<property name="ignoreExchangeCreatedEvent" value="false" />
<property name="ignoreExchangeCompletedEvent" value="false" />
<property name="ignoreExchangeFailedEvents" value="false" />
<property name="ignoreExchangeRedeliveryEvents" value="false" />
<property name="ignoreExchangeSendingEvents" value="${camel.event.silent}" />
<property name="ignoreExchangeSentEvents" value="${camel.event.silent}" />
</bean>
<service id="eventNotifierService" ref="decanterEventNotifier"
interface="org.apache.camel.spi.EventNotifier" />
<bean id="traceHandler" class="org.apache.karaf.decanter.collector.camel.DecanterTraceEventHandler">
<property name="eventAdmin" ref="eventAdmin"/>
<!-- Optional to add custom properties to the event -->
<!--<property name="extender" ref="customExtender" />-->
<property name="includeBody" value="${camel.trace.includeBody}"/>
<property name="includeProperties" value="${camel.trace.includeProperties}"/>
<property name="includeHeaders" value="${camel.trace.includeHeaders}"/>
</bean>
<bean id="tracer" class="org.apache.camel.processor.interceptor.Tracer">
<property name="traceHandler" ref="traceHandler"/>
<property name="enabled" value="${camel.trace.enabled}"/>
<property name="traceOutExchanges" value="true"/>
<property name="logLevel" value="OFF"/>
</bean>
<service id="tracerService" ref="tracer" interface="org.apache.camel.processor.interceptor.Tracer" />
</blueprint>
Decanter Extender
Decanter allows you to enrich the events generated by the Camel Collector through additional properties. It provides the DecanterCamelEventExtender interface, that must be implemented by a custom extender. An example of an implementation of an extender, is shown below:
Implementation of DecanterCamelEventExtender
public class ActivityMonitoringExtender implements DecanterCamelEventExtender{
@Override
public void extend(Map<String, Object> decanterData, Exchange exchange) {
String eventType = (String) decanterData.get("eventType");
if (ExchangeCreatedEvent.class.getName().equals(eventType)) {
Endpoint ep = exchange.getFromEndpoint();
if (ep instanceof JmsEndpoint) {
JmsEndpoint jmsEp = (JmsEndpoint) ep;
extendForJmsEndpoint(decanterData, exchange, jmsEp);
} else if (ep instanceof FtpEndpoint) {
FtpEndpoint<?> ftpEp = (FtpEndpoint<?>) ep;
extendForFtpEndpoint(decanterData, exchange, ftpEp);
}
}
}
...
}
To activate the extender, inject an instance of this extender into the DecanterEventNotifier Bean or the DecanterTraceEventHandler. An example of the relevant snippet of the blueprint configuration of the Java project, is shown below:
Injecting a handler
...
<bean id="customExtender" class="org.talend.ps.monitoring.ActivityMonitoringExtender"/>
<bean id="decanterEventNotifier" class="org.apache.karaf.decanter.collector.camel.DecanterEventNotifier">
<property name="eventAdmin" ref="eventAdmin"/>
<!-- Optional to add custom properties to the event -->
<property name="extender" ref="customExtender" />
<property name="includeBody" value="${camel.event.includeBody}"/>
...
Building and installing a project
Install a recent version of Maven, 3.3 or higher. To build the project, invoke the command mvn install from within the root directory of the project, where the pom.xml file is located. To install the project, the Decanter Camel Collector and at least one Decanter appender must be installed. For test purposes, the Decanter File appender is best suited. If you already have an Elasticsearch instance, you may use the Elasticsearch specific appenders. The Decanter documentation lists the available appenders and their configuration options. The installation must be executed in the console of the Talend Runtime. The required commands are shown below:
Decanter installation in the Runtime
/_ __/__ _/ /__ ___ ___/ /
/ / / _ `/ / -_) _ \/ _ /
/_/ \_,_/_/\__/_//_/\_,_/
(version 7.0.1)
Hit '<tab>' for a list of available commands
and '[cmd] --help' for help on a specific command.
Hit '<ctrl-d>' or 'system:shutdown' to shutdown the TRUN.
karaf@trun()> feature:repo-add decanter 2.0.0
Adding feature url mvn:org.apache.karaf.decanter/apache-karaf-decanter/2.0.0/xml/features
karaf@trun()> feature:install decanter-collector-camel
karaf@trun()> feature:install decanter-appender-file
karaf@trun()>
Install the bundle by executing the following command in the runtime console, or copy the corresponding JAR file, camel-decanter-EventNotifier.zip attached to this article, to the deploy folder of the runtime installation.
install -s mvn:org.talend.ps/camel-decanter-eventnotifier/1.0.0-SNAPSHOT
Configuring routes in Studio
Enable Studio routes for tracing and event notification. Figure 1 shows the sample routes from the monitored-routes.zip file, that is attached to this article. It includes a cConfig component where the CamelContext containing all the routes is retrieved, and the tracing property is set. Unfortunately, even if you're only interested in event notifications, tracing must be activated, and the corresponding tracer and traceHandling bean, in the blueprint file, must be configured. Otherwise, the Decanter collector throws exceptions instead of creating events.
Before using the sample routes, complete the properties for the FTP, ActiveMQ server, and the data directory.
Sample output
A sample Decanter event created when running the above routes, is shown below:
Sample Decanter event
{
"@timestamp": "2018-09-13T19:33:50,101Z",
"hostName": "sopmac23",
"camelContextName": "activitymonitoring.monitored_0_1.Monitored",
"exchangePattern": "InOnly",
"eventType": "org.apache.camel.management.event.ExchangeCreatedEvent",
"type": "camelEvent",
"shortExchangeId": "ID-sopmac23-1536867218901-0-2",
"exchangeId": "ID-sopmac23-1536867218901-0-2",
"jms": {
"destinationName": "monitored",
"destinationType": "queue",
"messageID": "ID:sopmac23-56922-1536867229058-1:1:2:1:1"
},
"routeId": "Monitored_cJMS_1",
"inHeaders": {
"breadcrumbId": "ID-sopmac23-1536867218901-0-1",
"CamelFileAbsolute": false,
"CamelFileAbsolutePath": "message/message_ID:sopmac23-51225-1536237210202-4:2:1:1:3.txt",
"CamelFileLastModified": 1536493380000,
"CamelFileLength": 44,
"CamelFileName": "message_ID:sopmac23-51225-1536237210202-4:2:1:1:3.txt",
"CamelFileNameConsumed": "message_ID:sopmac23-51225-1536237210202-4:2:1:1:3.txt",
"CamelFileNameOnly": "message_ID:sopmac23-51225-1536237210202-4:2:1:1:3.txt",
"CamelFileParent": "message",
"CamelFilePath": "message/message_ID:sopmac23-51225-1536237210202-4:2:1:1:3.txt",
"CamelFileRelativePath": "message_ID:sopmac23-51225-1536237210202-4:2:1:1:3.txt",
"CamelFtpReplyCode": 226,
"CamelFtpReplyString": "226 Transfer complete.\r\n",
"JMSDeliveryMode": 2,
"JMSExpiration": 0,
"JMSMessageID": "ID:sopmac23-56922-1536867229058-1:1:2:1:1",
"JMSPriority": 4,
"JMSRedelivered": false,
"JMSTimestamp": 1536867230089
},
"fromEndpointUri": "cMQConnectionFactory1://queue:monitored",
"karafName": "trun",
"hostAddress": "127.0.0.1",
"properties": {
"CamelExternalRedelivered": false
},
"timestamp": 1536867230101,
"inBodyType": "byte[]",
"event_topics": "decanter/collect/camel/event"
}
Note: Apache Decanter is officially supported starting v7.1.1. It has been introduced as a new feature with Event Monitoring.
