Qlik Talend ESB: Improve Camel flow monitoring

Problem Description There is no request/exchange specific logging information out-of-the-box from Camel without increasing the log level to DEBUG.... Show More

Problem Description

There is no request/exchange specific logging information out-of-the-box from Camel without increasing the log level to DEBUG.
It must be possible to monitor requests without having to add a lot of manual log statements.

Solution

Overview of Best Practice

Camel provides two ways to gather information about every message (Camel Exchange) being processed:

Camel Trace provides detailed information on messages being processed within the route. Every process step triggers an event, and you can configure what content should be added to the event (content, headers, properties, custom data).
Camel Events are triggered when an exchange is created, completed, failed, and so on. You can configure what content should be added to the event.

Apache Decanter is a flexible monitoring solution for Talend ESB (Apache Karaf) that allows you to gather monitoring information by using their collector architecture, like JMX, which is out-of-scope for this best practice.

Decanter provides the collector Camel Tracer & Notifier, which is notified whenever a Camel Event or Trace occurs. Decanter creates an OSGi Event, which can then be fed to any kind of appender to store the monitoring data centrally in Elasticsearch or MongoDB, or you can create a log event with the log appender. This solution will not work with Spring Boot, because Apache Decanter uses the OSGi EventAdmin service to bridge between the Camel events and the Decanter appenders.

Installation

Log in to the Talend ESB shell and install Decanter:

feature:repo-add decanter 2.0.0
feature:install decanter-appender-kafka decanter-collector-camel decanter-collector-eventadmin

Configuration

The configuration must be done with the Spring configuration file. You must configure several beans provided by Camel and Decanter to build the bridge between the two, and enable Camel Trace and Event features.

<!--
  Example property definition
-->

<osgix:cm-properties id="cmProps" persistent-id="com.example.myapplication>">
  <prop key="application">myapplication</prop>
  <prop key="camel.context.trace">false</prop>
  <prop key="camel.context.useMDCLogging">false</prop>
  <prop key="camel.trace.enabled">false</prop>
  <prop key="camel.trace.includeBody">false</prop>
  <prop key="camel.trace.includeProperties">true</prop>
  <prop key="camel.trace.includeHeaders">true</prop>
  <prop key="camel.event.includeBody">false</prop>
  <prop key="camel.event.includeProperties">true</prop>
  <prop key="camel.event.includeHeaders">true</prop>
  <prop key="camel.event.verbose">false</prop>    
</osgix:cm-properties>

<osgi:reference id="eventAdmin" interface="org.osgi.service.event.EventAdmin" />

<!--
  Decanter bean definitions
  Documentation: https://karaf.apache.org/manual/decanter/latest-2/#_camel_tracer_notifier
-->

<bean id="decanterEventNotifier" class="org.apache.karaf.decanter.collector.camel.DecanterEventNotifier">
  <property name="eventAdmin" ref="eventAdmin"/>
  <!-- Optional to add custom properties to the event -->
  <!--<property name="extender" ref="customExtender" />-->
  <property name="includeBody" value="${camel.event.includeBody}"/>
  <property name="includeProperties" value="${camel.event.includeProperties}"/>
  <property name="includeHeaders" value="${camel.event.includeHeaders}"/>
  <property name="ignoreCamelContextEvents" value="false" />
  <property name="ignoreRouteEvents" value="false" />
  <property name="ignoreServiceEvents" value="false" />
  <property name="ignoreExchangeEvents" value="false" /> 
  <property name="ignoreExchangeCreatedEvent" value="#{!${camel.event.verbose}}" />
  <property name="ignoreExchangeCompletedEvent" value="false" />
  <property name="ignoreExchangeFailedEvents" value="false" />
  <property name="ignoreExchangeRedeliveryEvents" value="false" />
  <property name="ignoreExchangeSendingEvents" value="#{!${camel.event.verbose}}" />
  <property name="ignoreExchangeSentEvents" value="#{!${camel.event.verbose}}" />
</bean>

<bean id="traceHandler" class="org.apache.karaf.decanter.collector.camel.DecanterTraceEventHandler">
  <property name="eventAdmin" ref="eventAdmin"/>
  <!-- Optional to add custom properties to the event -->
  <!--<property name="extender" ref="customExtender" />-->
  <property name="includeBody" value="${camel.trace.includeBody}"/>
  <property name="includeProperties" value="${camel.trace.includeProperties}"/>
  <property name="includeHeaders" value="${camel.trace.includeHeaders}"/>
</bean>

<!--
  Camel Trace documentation
  Documentation: http://camel.apache.org/tracer.html
-->

<bean id="tracer" class="org.apache.camel.processor.interceptor.Tracer">
  <property name="traceHandler" ref="traceHandler"/>
  <property name="enabled" value="${camel.trace.enabled}"/>
  <property name="traceOutExchanges" value="true"/>
  <property name="logStackTrace" value="true"/>
  <property name="logLevel" value="OFF"/>
</bean>

<camelContext xmlns="http://camel.apache.org/schema/spring"
  trace="{{camel.trace.enabled}}"
  useMDCLogging="{{camel.context.useMDCLogging}} ">
</camelContext>

Add custom context data to every monitoring event

The two Decanter classes, DecanterEventNotifier and DecanterTraceEventHandler, provide a property to set a custom extender class, which implements the interface org.apache.karaf.decanter.collector.camel.DecanterCamelEventExtender. You can add properties to the passed map, and have full access to the Camel Exchange object.

For more information, see the Apache Karaf Decanter documentation, Camel Tracer & Notifier page.

Example: monitoring in Elasticsearch

Camel Event

{
  "hostName": "MYHOST",
  "customContext": {
    "customerId": "61601601",
    "orderId": "300018631"
  },
  "type": "camelEvent",
  "topic_type": "camel",
  "shortExchangeId": "ID-MYHOST-35588-1520542853557-2-2",
  "routeId": "myroute",
  "inHeaders": {},
  "cap": "tesb",
  "fromEndpointUri": "cxf://bean:proxyOrderEndpoint",
  "@version": "1",
  "topic_name": "event",
  "inBodyType": "org.apache.camel.converter.stream.InputStreamCache",
  "outHeaders": {},
  "@indexer": "myhost",
  "camelContextName": "mycamel-context",
  "exchangePattern": "InOut",
  "eventType": "org.apache.camel.management.event.ExchangeCompletedEvent",
  "tags": [
    "decanter"
  ],
  "exchangeId": "ID-MYHOST-35588-1520542853557-2-2",
  "@timestamp": "2018-03-09T13:57:28.228Z",
  "application": "orderApp",
  "outBodyType": "org.apache.camel.converter.stream.InputStreamCache",
  "topic": "decanter/collect/camel/event",
  "hostAddress": "192.157.11.11",
  "category": "trace",
  "properties": {
    "CamelCharsetName": "UTF-8",
    "mtom-enabled": "false",
    "CamelSkipGzipEncoding": true,
    "CamelToEndpoint": "cxf://bean:OrderEndpoint"
  }
}

Camel Trace

{
  "hostName": "MYHOST",
  "type": "camelTracer",
  "topic_type": "camel",
  "shortExchangeId": "ID-MYHOST-46763-1518450121529-21-872",
  "routeId": "ws-route",
  "inHeaders": {
    "CamelHttpResponseCode": 202,
    "Accept": "*/*",
    "Server": "Jetty(9.2.21.v20170120)",
    "CamelHttpCharacterEncoding": "UTF-8",
    "operationName": "do",
    "Date": "Tue, 20 Feb 2018 06:57:41 GMT",
    "operationNamespace": "http://webservices.test/OrderService",
    "CamelFileNameProduced": "work/order/ID-MYHOST-46763-1518450121529-21-871",
    "messageType": "Health",
    "breadcrumbId": "ID-MYHOST-46763-1518450121529-21-871",
    "CamelHttpMethod": "POST",
    "CamelAcceptContentType": "*/*",
    "CamelHttpResponseText": "Accepted",
    "Content-Length": "0",
    "CamelHttpUri": "/services/orderService",
    "flowid": "d08f54bd-44a6-48ae-94d6-099763159402",
    "CamelHttpPath": ""
  },
  "cap": "tesb",
  "fromEndpointUri": "cxf://bean:proxyOrderEndpoint",
  "@version": "1",
  "topic_name": "tracer",
  "inBodyType": "String",
  "outHeaders": {
    "CamelHttpResponseCode": 202,
    "Accept": "*/*",
    "Server": "Jetty(9.2.21.v20170120)",
    "CamelHttpCharacterEncoding": "UTF-8",
    "operationName": "do",
    "Date": "Tue, 20 Feb 2018 06:57:41 GMT",
    "operationNamespace": "http://webservices.test/OrderService",
    "CamelFileNameProduced": "work/ID-MYHOST-46763-1518450121529-21-871",
    "messageType": "ORDER",
    "breadcrumbId": "ID-MYHOST-46763-1518450121529-21-871",
    "CamelHttpMethod": "POST",
    "CamelAcceptContentType": "*/*",
    "CamelHttpResponseText": "Accepted",
    "Content-Length": "0",
    "CamelHttpUri": "/services/orderService",
    "flowid": "d08f54bd-44a6-48ae-94d6-099763159402",
    "CamelHttpPath": ""
  },
  "@indexer": "myhost",
  "camelContextName": "mycamel-context",
  "exchangePattern": "InOut",
  "outBody": "<removed>",
  "toNode": "setBody[Simple: OK]",
  "tags": [
    "decanter"
  ],
  "exchangeId": "ID-MYHOST-46763-1518450121529-21-872",
  "@timestamp": "2018-02-20T06:57:41.131Z",
  "application": "orderApp",
  "outBodyType": "String",
  "topic": "decanter/collect/camel/tracer",
  "hostAddress": "192.157.11.11",
  "category": "trace",
  "nodeId": "to616",
  "previousNode": "",
  "inBody": "<removed>",
  "properties": {
    "CamelCharsetName": "UTF-8",
    "mtom-enabled": "false",
    "CamelSkipGzipEncoding": true,
    "CamelToEndpoint": "http://localhost:10001/order?bridgeEndpoint=true"
  }
}

Show Less

0

858

Official Support Articles

Changing the logLevel produced by LogServer

Problem Description You want to produce smaller log files by configuring them to contain only error and warning messages, and not info messages. ... Show More

Problem Description

You want to produce smaller log files by configuring them to contain only error and warning messages, and not info messages.

Solution

Edit the logging level in the logging.yml file, located in the C:\Talend\x.x.x\logserv\elasticsearch-2.4.0\config folder, and change the log level from INFO to WARN or ERROR.

Your updated file should look similar to this:

#es.logger.level: INFO 
es.logger.level: WARN 
rootLogger: ${es.logger.level}, console, file 
logger: 
# log action execution errors for easier debugging 
action: DEBUG 

# deprecation logging, turn to DEBUG to see them 
deprecation: INFO, deprecation_log_file 

# reduce the logging for aws, too much is logged under the default INFO 
com.amazonaws: WARN 
# aws will try to do some sketchy JMX stuff, but its not needed. 
com.amazonaws.jmx.SdkMBeanRegistrySupport: ERROR 
com.amazonaws.metrics.AwsSdkMetrics: ERROR 

org.apache.http: WARN

Show Less

0

185

Official Support Articles

Security issues of Elasticsearch v1.1.1 used in Talend Log Server

Overview Elasticsearch version 1.1.1 used in Talend Log Server has a security vulnerability issue, and this article proposes a solution to this i... Show More

Overview

Elasticsearch version 1.1.1 used in Talend Log Server has a security vulnerability issue, and this article proposes a solution to this issue that impacts Talend products versions 5.4.x to 5.6.1.

For more information on this issue, please read the Security Issues page on the Elastic official website.

Environment

This article only applies to Talend products version between 5.4.x to 5.6.1.

Resolution

If you are running a Talend product with a version between 5.4.x and 5.6.1, and you have the Talend Log Server installed, you need to complete the following configuration changes to properly secure your system:

Create a file called elasticsearch.yml in your Talend Log Server installation directory (/Talend/version/Talend_logServer_dir).

Edit the file and add the following lines to disable dynamic scripting:

script.disable_dynamic: true
http.cors.allow-origin: "http://TAC_SERVER_HOST:TAC_SERVER_PORT"

Restart the Talend Log Server to incorporate your changes.

Show Less

0

161

Official Support Articles

Failed to Query Elasticsearch error when starting Logserver

Symptoms start_logserver.sh will not start, and displays the following behavior. [user@system logserv]$ pwd /apps/Talend-6.3.1/logserv [user@sy... Show More

Symptoms

start_logserver.sh will not start, and displays the following behavior.

[user@system logserv]$ pwd
/apps/Talend-6.3.1/logserv
[user@system logserv]$ ./start_logserver.sh
Failed to query elasticsearch

The Elasticsearch service will appear to be running when the services are checked, but the Kibana and Logserver services will not start.

Diagnosis

The data/node that is actively being worked on has become corrupt.

Solution

Delete the node number by removing the corresponding folder located at \elasticsearch\nodes. After deleting the number from under nodes, Elasticsearch will recreate the nodes folder when starting the stack of ELK.

Example of the path that needed to be deleted:

Talend\6.3.1\logserv\elasticsearch-2.4.0\data\elasticsearch\nodes\0

Show Less

0

145

Official Support Articles

Official Support Articles

Recent Documents

Qlik Talend ESB: Improve Camel flow monitoring

Problem Description

Solution

Overview of Best Practice

Installation

Configuration

Add custom context data to every monitoring event

Example: monitoring in Elasticsearch

Camel Event

Camel Trace

Changing the logLevel produced by LogServer

Problem Description

Solution

Security issues of Elasticsearch v1.1.1 used in Talend Log Server

Overview

Environment

Resolution

Failed to Query Elasticsearch error when starting Logserver

Symptoms

Diagnosis

Solution