How to connect to Microsoft Azure using cMQConnectionFactory

Talend Version 6.1.1 Summary Connecting to Microsoft Azure using cMQConnectionFactory.Additional Versions ProductTalend ESBComponentcMQConnectio... Show More

Talend Version	6.1.1
Summary	Connecting to Microsoft Azure using cMQConnectionFactory.
Additional Versions
Product	Talend ESB
Component	cMQConnectionFactory and cAMQP
Problem Description
Problem root cause
Solution or Workaround	In order to send messages to the Microsoft Azure queue using the cAMQP component, you have to configure connection factory. Following are the steps to set up a connection to Azure in the cMQConnectionFactory component if the Azure endpoint is something like: Endpoint=sb://mytest.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=1alxVTE9xxxxGFB+E7D7pt7Q74BU0lAzxcvbU+RlJSLs= In the cMQConnectionFactory component, set the Host and Port as follows: Host: "RootManageSharedAccessKey:1alxVTE9xxxxGFB%2BE7D7pt7Q74BU0lAzxcvbU%2BRlJSLs%3D@mytest.servicebus.windows.net" Note: the special characters in the key need to be substituted with their hex characters. In this example, the plus sign was replaced with %2B and the equals sign was replaced with %3D. Port: "5671" Select Use SSL.
JIRA ticket number

Show Less

2

238

Official Support Articles

Talend ESB: Use tHash components in ESB Runtime

Question Is it safe to use the tHashOutput and tHashInput components in SOAP/REST services or in Jobs called by routes? Answer tHashxxx components a... Show More

Question

Is it safe to use the tHashOutput and tHashInput components in SOAP/REST services or in Jobs called by routes?

Answer

tHashxxx components are not thread-safe and are not supported in Data Services. They are designed to store/keep data/objects in memory and are not recommended for use in the JVM process of the runtime.

Show Less

2

326

Official Support Articles

'Talend ESB Administration' Training Course

Talend ESB is a reliable and scalable enterprise service bus (ESB) that lets development teams manage integration projects in a holistic manner, combi... Show More

Talend ESB is a reliable and scalable enterprise service bus (ESB) that lets development teams manage integration projects in a holistic manner, combining the integration of applications and data management in complex and heterogeneous computing environments.

This learning plan helps administrators better understand Talend server in the context of ESB deployments. It prepares you to use ESB Conductor, ESB Publisher, and ESB Runtime with Routes and data services.

Target Audience: Software architects and system administrators
Prerequisites: Completion of Talend Data Integration Administration, or significant experience with Talend Server.
Availability: This learning plan is available as part of a Talend Academy Learning Subscription, and as a private virtual or private onsite instructor-led course.

If you are already a Talend Academy subscriber or want to access the publicly available content on the platform, go to the Talend Academy Welcome page to log in or create an account.

Show Less

0

263

Official Support Articles

Managing Talend microservices with Istio service mesh on Azure Kubernetes Servic...

Content: Environment SetupLaunching a standard Azure Kubernetes Service (AKS)Using Azure CLI or Cloud shellUsing the Azure portalInstalling Istio with... Show More

Content:

The Talend Community Knowledge Base (KB) article, Managing Talend microservices with Istio service mesh on Kubernetes, shows you how to connect, secure, control, and observe Talend microservices leveraging Istio.

This article describes the installation and platform-specific steps for managing Talend microservices using Istio service mesh on Azure Kubernetes Service (AKS).

For more information on installing Talend microservices using Istio on other cloud providers, see the following Talend Community KB articles:

Environment Setup

Launching a standard Azure Kubernetes Service (AKS)

Create an Azure Kubernetes Service by using Azure CLI or Azure portal.

Using Azure CLI or Cloud shell

Set the Azure subscription ID.

az account set -s REPLACE_WITH_SUBSCRIPTION_ID
# for example, az account set -s d93xxxxxxxxxxxxxx

Create a resource group.

az group create --name AKS_RESOURCE_GROUP --location REPLACE_WITH_AZURE_LOCATION
# for example, az group create --name rchinta_csa_resource_group --location "West Europe"

Create an AKS Cluster.

az aks create --resource-group REPLACE_WITH_AKS_RESOURCE_GROUP --name AKS_CLUSTER_NAME --node-count 2 --kubernetes-version 1.14.8  --node-vm-size DS2_v2
#for example, az aks create --resource-group rchinta_csa_resource_group --name talend-bonn-Az-aks-cluster --node-count 2 --kubernetes-version 1.14.8  --node-vm-size DS2_v2

Using the Azure portal

From the Azure portal search for Kubernetes services. Click the Add button.
In the Create Kubernetes cluster > Basic settings, enter the details for Resource group, Kubernetes cluster name, Region, Kubernetes version, and Node size. Click Next.
In the Scale settings, Enable the VM scale sets option to allow auto-scaling of the nodes. Click Next.
In the Authentication settings, set Service principal to use default service principal, then select Yes to Enable RBAC.

Note: If you want to create/assign a service principal, click the Configure service principal link.
Click Next until Review + Create, validate the settings, then click Create.

Note: The cluster creation might take up to 5 or 10 minutes.
In the Azure portal, click the Cloud Shell icon, initialize storage if required, then launch the bash shell.

In the cloud shell, create a kubeconfig file in the ~/.kube folder by executing the following command:

az aks get-credentials --resource-group REPLACE_WITH_AKS_RESOURCE_GROUP  --name  REPLACE_WITH_AKS_CLUSTER_NAME
# for example, az aks get-credentials --resource-group rchinta_aks_resource_group --name talend-bonn-Az-aks-cluster

Verify that the worker nodes joined with the cluster, by executing the following command:
```
kubectl get nodes
```

Installing Istio with Helm on AKS

Helm is a package manager for Kubernetes. Helm Charts help you define, install, and upgrade even the most complex Kubernetes application.

Istio provides customizable Helm templates for installation on a Kubernetes cluster. For more information on other installation options, see the Istio Installation Guides.

Install Helm client

Install Helm client using one of these three approaches, for example, Azure Cloud shell / Linux / Windows OS.

Using Cloud Shell (AKS)

Helm client is preinstalled on Azure Cloud Shell.

helm version

Using shell script (Linux)

Skip this step if you use Azure Cloud Shell. Helm client can be installed on Linux OS using shell commands.

Connect to a Cloud shell, then install Helm client.

curl -LO https://git.io/get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh

From Chocolatey (Windows)

Skip this step if you use Azure Cloud shell. Helm package can be installed on Windows using Chocolatey

Launch a command prompt and execute the below command.
```
choco
```
Note: If you get an error command not found choco, then follow the instructions in this link and Install Chocolatey on Windows.

Install Helm using the choco install command:

#Install helm client
choco install kubernetes-helm

Customizing the Istio installation with Helm

From the cloud shell or command prompt.

Create an istio_install directory.

# create directory istio_install
mkdir istio_install (Windows / Linux)

# move into the istio_install
cd istio_install

Download an Istio release and then execute the steps in the guide Customizable Install with Helm.

# Download the installation file using the curl command
curl -Ls -O https://github.com/istio/istio/releases/download/1.3.4/istio-1.3.4-linux.tar.gz  ( Linux)
curl -Ls -O https://github.com/istio/istio/releases/download/1.3.4/istio-1.3.4-win.zip (Windows)

# gunzip and untar the file
gunzip istio-1.3.4-linux.tar.gz 
tar -xvf istio-1.3.4-linux.tar

# move into the directory istio-1.3.4
cd istio-1.3.4

# Configure the PATH environment variable
export PATH=$PWD/bin:$PATH (Linux)
set PATH=%CD%/bin;%PATH% (Windows)

# Initialize helm
helm init

#Add Istio to the helm repository
helm repo add istio.io https://storage.googleapis.com/istio-release/releases/1.3.4/charts/

#Create namespace istio-system
kubectl create namespace istio-system

#Install all the istio CRDs
helm template install/kubernetes/helm/istio-init --name istio-init --namespace istio-system | kubectl apply -f -

Install Istio choosing one of the configuration profiles.

Note: This article uses the demo_auth profile.

# Execute the helm template with the predefined settings in the demo_auth profile

helm template install/kubernetes/helm/istio --name istio --namespace istio-system \
--values install/kubernetes/helm/istio/values-istio-demo-auth.yaml | kubectl apply -f -

Verify that all of the Istio components in the demo-auth configuration profile are in the status Running / Completed, by executing the following command:
```
kubectl get pods -n istio-system
```
Verify an External IP is assigned to the istio-ingressgateway service.
```
kubectl get svc -n istio-system
```
Note:
1. Ingress traffic to application pods or microservices is only made possible using the External / Public IP assigned to the istio-ingressgateway service.
2. An external Load-balancer is launched on AKS automatically, and its IP is assigned to the istio-ingressgateway service.

Launching Istio monitoring dashboards for AKS

With Istio demo-auth profile, add-ons for dashboards like Prometheus, Grafana, Kiali, and Jaeger are enabled and installed along with Istio.

Prometheus is a web-based graphical user interface for querying the Istio metric values.
Grafana is a web-based graphical user interface that provides a global view of the mesh along with services and their workloads.
Kiali is a web-based graphical user interface to view service graphs of the mesh and Istio configuration objects. Different graph types such as App, Versioned App, Workload, Service are available to view the services in the mesh.

Open a Windows command prompt in Administrator mode, then execute the following commands, which creates the kubeconfig file in the Windows USER_HOME/.kube folder.

# login to windows from command prompt
az login

az aks get-credentials --resource-group REPLACE_WITH_AKS_RESOURCE_GROUP --name REPLACE_WITH_AKS_CLUSTER_NAME
# for example, az aks get-credentials --resource-group rchinta_aks_resource_group --name talend-bonn-Az-aks-cluster

Launch the dashboards using local port forwarding.

# Execute the below port-forward command and access Prometheus using local port forwarding
kubectl port-forward svc/prometheus 9090:9090 -n istio-system

# Execute the below port-forward command and access Grafana using local port forwarding
kubectl port-forward svc/grafana 3000:3000 -n istio-system

# Execute the below port-forward command and access Kiali using local port forwarding
kubectl port-forward svc/kiali 20001:20001 -n istio-system

# Execute the below port-forward command and access Jaeger using local port forwarding
kubectl port-forward svc/jaeger-query 16686:16686 -n istio-system

Publish Talend microservices to Azure Container Registry

Creating an Azure Container Registry (ACR)

From the Azure portal search for container registries. Click Add.
Enter the registry details such as Registry name and Resource group as shown below, then click Create.
After the registry is created, open the registry, then click Access keys.
Obtain the ACR credentials from the fields username and password.

Publishing microservices from Talend Studio to ACR

From Talend Studio, publish the Customers microservice to ACR, as shown below. Click Finish.

Note: In the Publish window, enter the usernameand password fields with the ACR credentials. For more information, follow the steps in Creating an Azure Container Registry (ACR) section.
Repeat Step 1 and publish the Orders microservice. Make a note of the URLs of the microservice images published to ACR.

Authentication with Azure Container Registry from AKS using a service principal

Skip this section if you are not using Azure Container Registry as a container registry.

An AKS cluster, by default, doesn't have read access to the images in ACR. The application images are pulled from the registry after a successful Authenticate with Azure Container Registry from Azure Kubernetes Service.

Obtaining a service principal

When creating an AKS cluster, in the Authentication settings by default, a service principal is created and assigned to the AKS cluster.

Obtain the Client ID of the default service principal by executing the following command:

az aks show --name  REPLACE_WITH_AKS_CLUSTER_NAME  --resource-group  REPLACE_WITH_AKS_CLUSTER_RESOURCE_GROUP --query servicePrincipalProfile.clientId -o tsv 
# for example,  az aks show --name talend-bonn-Az-aks-cluster --resource-group rchinta_aks_resource_group --query servicePrincipalProfile.clientId -o tsv
# command outputs clientId : 197bdxxx-xxxx............

Obtaining the ACR resource ID

Obtain the ACR resource ID by executing the following command:

az acr show --name  REPLACE_WITH_AZURE_CONTAINER_REGISTRY_NAME --query id --output tsv
# for example, az acr show --name rchinta --query id --output tsv
# command outputs ACR resource ID : /subscriptions/xxxxxx/resourceGroups/rchinta_aks_resource_group/providers/Microsoft.ContainerRegistry/registries/rchinta

Granting role-based access to ACR from AKS

Assign the acrpull role to the AKS service principal by executing the following command:

az role assignment create --assignee  REPLACE_WITH_AKS_SERVICE_PRINCIPAL_ID  --scope  REPLACE_WITH_ACR_RESOURCE_ID  --role acrpull
# for example, /subscriptions/xxxxxx/resourceGroups/rchinta_aks_resource_group/providers/Microsoft.ContainerRegistry/registries/rchinta

Note: If you get an Insufficient privileges to complete the operation error, contact your Azure administrator.

Create a Kubernetes secret with service principal

Create a Kubernetes secret that contains the AKS service principal's Client ID and Client Secret.

Using the Azure CLI / Cloud shell execute the following command:

# Fill your registry details in the below command before executing.
kubectl create secret docker-registry acr-auth --docker-server REPLACE_WITH_YOUR_AZURE_REGISTRY_URL --docker-username REPLACE_WITH_YOUR_SERVICE_PRINCIPAL_CLIENT_ID --docker-password REPLACE_WITH_YOUR_SERVICE_PRINCIPAL_CLIENT_SECRET --docker-email REPLACE_WITH_YOUR_EMAIL_ID

# for example
kubectl create secret docker-registry  acr-auth  --docker-server rchinta.azurecr.io --docker-username ff420.... --docker-password 5CQHrss.... --docker-email rchinta@talend.com

Using Secret in the Kubernetes resource files

Azure Kubernetes service authenticates with the Azure Container Registry using the Secret configuration in the resource files and downloads the application images.

In the Demo Kubernete resource files for AKS attached to this article, observe the usage of the property imagePullSecrets with name acr-auth.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: order-service-v2
spec:
replicas: 2
template:
metadata:
labels:
app : order-service
version: v2
spec:
containers:
- name: order-service
image: rchinta.azurecr.io/k8s_microservice/customer/microservices/orders:0.5.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
command: ["/bin/sh", "/maven/Orders/Orders_run.sh"]
args: ["--spring.config.location=classpath:config/contexts/PROD.properties"]
imagePullSecrets:
- name: acr-auth

Conclusion

This article showed you how to launch an Azure Kubernetes Cluster, install Istio manually using Helm, and how to publish Talend microservices on Azure Container Registry.

Show Less

0

234

Official Support Articles

Routes deployed on dual runtime servers

When a Route (which has a cFtp component) is deployed on multiple runtime servers, it consumes the file from the FTP at the same time and creates a... Show More

When a Route (which has a cFtp component) is deployed on multiple runtime servers, it consumes the file from the FTP at the same time and creates a duplicate. This is expected behavior.

Best practice

Talend suggests deploying a Route on a single runtime server. If there is a requirement to deploy it on multiple runtime servers, you must redesign the logic accordingly, as FTP Route consumers are not thread-safe.

This is a limitation of Apache Camel > FTP components.

You can use the readLock option to force Camel to not consume files currently being written. However, this option is turned off by default, as it requires that you have write access.

There are other solutions for avoiding consuming files currently being written over FTP; for example, you can write to a temporary destination, and move the file after it has been written.

When moving files using the move or preMove options, the files are restricted to the FTP_ROOT folder. That prevents you from moving them outside of the FTP area. If you want to move files to another area, you can use soft links and move them into a soft-linked folder.

For more information, see Apache Camel.

Show Less

0

89

Official Support Articles

Qlik Talend ESB: Improve Camel flow monitoring

Problem Description There is no request/exchange specific logging information out-of-the-box from Camel without increasing the log level to DEBUG.... Show More

Problem Description

There is no request/exchange specific logging information out-of-the-box from Camel without increasing the log level to DEBUG.
It must be possible to monitor requests without having to add a lot of manual log statements.

Solution

Overview of Best Practice

Camel provides two ways to gather information about every message (Camel Exchange) being processed:

Camel Trace provides detailed information on messages being processed within the route. Every process step triggers an event, and you can configure what content should be added to the event (content, headers, properties, custom data).
Camel Events are triggered when an exchange is created, completed, failed, and so on. You can configure what content should be added to the event.

Apache Decanter is a flexible monitoring solution for Talend ESB (Apache Karaf) that allows you to gather monitoring information by using their collector architecture, like JMX, which is out-of-scope for this best practice.

Decanter provides the collector Camel Tracer & Notifier, which is notified whenever a Camel Event or Trace occurs. Decanter creates an OSGi Event, which can then be fed to any kind of appender to store the monitoring data centrally in Elasticsearch or MongoDB, or you can create a log event with the log appender. This solution will not work with Spring Boot, because Apache Decanter uses the OSGi EventAdmin service to bridge between the Camel events and the Decanter appenders.

Installation

Log in to the Talend ESB shell and install Decanter:

feature:repo-add decanter 2.0.0
feature:install decanter-appender-kafka decanter-collector-camel decanter-collector-eventadmin

Configuration

The configuration must be done with the Spring configuration file. You must configure several beans provided by Camel and Decanter to build the bridge between the two, and enable Camel Trace and Event features.

<!--
  Example property definition
-->

<osgix:cm-properties id="cmProps" persistent-id="com.example.myapplication>">
  <prop key="application">myapplication</prop>
  <prop key="camel.context.trace">false</prop>
  <prop key="camel.context.useMDCLogging">false</prop>
  <prop key="camel.trace.enabled">false</prop>
  <prop key="camel.trace.includeBody">false</prop>
  <prop key="camel.trace.includeProperties">true</prop>
  <prop key="camel.trace.includeHeaders">true</prop>
  <prop key="camel.event.includeBody">false</prop>
  <prop key="camel.event.includeProperties">true</prop>
  <prop key="camel.event.includeHeaders">true</prop>
  <prop key="camel.event.verbose">false</prop>    
</osgix:cm-properties>

<osgi:reference id="eventAdmin" interface="org.osgi.service.event.EventAdmin" />

<!--
  Decanter bean definitions
  Documentation: https://karaf.apache.org/manual/decanter/latest-2/#_camel_tracer_notifier
-->

<bean id="decanterEventNotifier" class="org.apache.karaf.decanter.collector.camel.DecanterEventNotifier">
  <property name="eventAdmin" ref="eventAdmin"/>
  <!-- Optional to add custom properties to the event -->
  <!--<property name="extender" ref="customExtender" />-->
  <property name="includeBody" value="${camel.event.includeBody}"/>
  <property name="includeProperties" value="${camel.event.includeProperties}"/>
  <property name="includeHeaders" value="${camel.event.includeHeaders}"/>
  <property name="ignoreCamelContextEvents" value="false" />
  <property name="ignoreRouteEvents" value="false" />
  <property name="ignoreServiceEvents" value="false" />
  <property name="ignoreExchangeEvents" value="false" /> 
  <property name="ignoreExchangeCreatedEvent" value="#{!${camel.event.verbose}}" />
  <property name="ignoreExchangeCompletedEvent" value="false" />
  <property name="ignoreExchangeFailedEvents" value="false" />
  <property name="ignoreExchangeRedeliveryEvents" value="false" />
  <property name="ignoreExchangeSendingEvents" value="#{!${camel.event.verbose}}" />
  <property name="ignoreExchangeSentEvents" value="#{!${camel.event.verbose}}" />
</bean>

<bean id="traceHandler" class="org.apache.karaf.decanter.collector.camel.DecanterTraceEventHandler">
  <property name="eventAdmin" ref="eventAdmin"/>
  <!-- Optional to add custom properties to the event -->
  <!--<property name="extender" ref="customExtender" />-->
  <property name="includeBody" value="${camel.trace.includeBody}"/>
  <property name="includeProperties" value="${camel.trace.includeProperties}"/>
  <property name="includeHeaders" value="${camel.trace.includeHeaders}"/>
</bean>

<!--
  Camel Trace documentation
  Documentation: http://camel.apache.org/tracer.html
-->

<bean id="tracer" class="org.apache.camel.processor.interceptor.Tracer">
  <property name="traceHandler" ref="traceHandler"/>
  <property name="enabled" value="${camel.trace.enabled}"/>
  <property name="traceOutExchanges" value="true"/>
  <property name="logStackTrace" value="true"/>
  <property name="logLevel" value="OFF"/>
</bean>

<camelContext xmlns="http://camel.apache.org/schema/spring"
  trace="{{camel.trace.enabled}}"
  useMDCLogging="{{camel.context.useMDCLogging}} ">
</camelContext>

Add custom context data to every monitoring event

The two Decanter classes, DecanterEventNotifier and DecanterTraceEventHandler, provide a property to set a custom extender class, which implements the interface org.apache.karaf.decanter.collector.camel.DecanterCamelEventExtender. You can add properties to the passed map, and have full access to the Camel Exchange object.

For more information, see the Apache Karaf Decanter documentation, Camel Tracer & Notifier page.

Example: monitoring in Elasticsearch

Camel Event

{
  "hostName": "MYHOST",
  "customContext": {
    "customerId": "61601601",
    "orderId": "300018631"
  },
  "type": "camelEvent",
  "topic_type": "camel",
  "shortExchangeId": "ID-MYHOST-35588-1520542853557-2-2",
  "routeId": "myroute",
  "inHeaders": {},
  "cap": "tesb",
  "fromEndpointUri": "cxf://bean:proxyOrderEndpoint",
  "@version": "1",
  "topic_name": "event",
  "inBodyType": "org.apache.camel.converter.stream.InputStreamCache",
  "outHeaders": {},
  "@indexer": "myhost",
  "camelContextName": "mycamel-context",
  "exchangePattern": "InOut",
  "eventType": "org.apache.camel.management.event.ExchangeCompletedEvent",
  "tags": [
    "decanter"
  ],
  "exchangeId": "ID-MYHOST-35588-1520542853557-2-2",
  "@timestamp": "2018-03-09T13:57:28.228Z",
  "application": "orderApp",
  "outBodyType": "org.apache.camel.converter.stream.InputStreamCache",
  "topic": "decanter/collect/camel/event",
  "hostAddress": "192.157.11.11",
  "category": "trace",
  "properties": {
    "CamelCharsetName": "UTF-8",
    "mtom-enabled": "false",
    "CamelSkipGzipEncoding": true,
    "CamelToEndpoint": "cxf://bean:OrderEndpoint"
  }
}

Camel Trace

{
  "hostName": "MYHOST",
  "type": "camelTracer",
  "topic_type": "camel",
  "shortExchangeId": "ID-MYHOST-46763-1518450121529-21-872",
  "routeId": "ws-route",
  "inHeaders": {
    "CamelHttpResponseCode": 202,
    "Accept": "*/*",
    "Server": "Jetty(9.2.21.v20170120)",
    "CamelHttpCharacterEncoding": "UTF-8",
    "operationName": "do",
    "Date": "Tue, 20 Feb 2018 06:57:41 GMT",
    "operationNamespace": "http://webservices.test/OrderService",
    "CamelFileNameProduced": "work/order/ID-MYHOST-46763-1518450121529-21-871",
    "messageType": "Health",
    "breadcrumbId": "ID-MYHOST-46763-1518450121529-21-871",
    "CamelHttpMethod": "POST",
    "CamelAcceptContentType": "*/*",
    "CamelHttpResponseText": "Accepted",
    "Content-Length": "0",
    "CamelHttpUri": "/services/orderService",
    "flowid": "d08f54bd-44a6-48ae-94d6-099763159402",
    "CamelHttpPath": ""
  },
  "cap": "tesb",
  "fromEndpointUri": "cxf://bean:proxyOrderEndpoint",
  "@version": "1",
  "topic_name": "tracer",
  "inBodyType": "String",
  "outHeaders": {
    "CamelHttpResponseCode": 202,
    "Accept": "*/*",
    "Server": "Jetty(9.2.21.v20170120)",
    "CamelHttpCharacterEncoding": "UTF-8",
    "operationName": "do",
    "Date": "Tue, 20 Feb 2018 06:57:41 GMT",
    "operationNamespace": "http://webservices.test/OrderService",
    "CamelFileNameProduced": "work/ID-MYHOST-46763-1518450121529-21-871",
    "messageType": "ORDER",
    "breadcrumbId": "ID-MYHOST-46763-1518450121529-21-871",
    "CamelHttpMethod": "POST",
    "CamelAcceptContentType": "*/*",
    "CamelHttpResponseText": "Accepted",
    "Content-Length": "0",
    "CamelHttpUri": "/services/orderService",
    "flowid": "d08f54bd-44a6-48ae-94d6-099763159402",
    "CamelHttpPath": ""
  },
  "@indexer": "myhost",
  "camelContextName": "mycamel-context",
  "exchangePattern": "InOut",
  "outBody": "<removed>",
  "toNode": "setBody[Simple: OK]",
  "tags": [
    "decanter"
  ],
  "exchangeId": "ID-MYHOST-46763-1518450121529-21-872",
  "@timestamp": "2018-02-20T06:57:41.131Z",
  "application": "orderApp",
  "outBodyType": "String",
  "topic": "decanter/collect/camel/tracer",
  "hostAddress": "192.157.11.11",
  "category": "trace",
  "nodeId": "to616",
  "previousNode": "",
  "inBody": "<removed>",
  "properties": {
    "CamelCharsetName": "UTF-8",
    "mtom-enabled": "false",
    "CamelSkipGzipEncoding": true,
    "CamelToEndpoint": "http://localhost:10001/order?bridgeEndpoint=true"
  }
}

Show Less

0

478

Official Support Articles

NullPointerException instead of Timeout when calling web service

Talend Version (Required) 6.x Summary ESB Web Service call throws a NullPointerException when the web service takes a long time to send a repl... Show More

Talend Version (Required)	6.x
Summary	ESB Web Service call throws a NullPointerException when the web service takes a long time to send a reply.
Additional Versions
Product (Required)	ESB
Component (Required)	Web Services
Problem Description	Assume that a route is exposed as web service provider using a cCXF component, and that it also calls an external web service using another cCXF. If this external web service takes longer to respond, then the first cCXF throws a NullPointerException instead of a Timeout: java.lang.NullPointerException at org.apache.camel.impl.DefaultConsumer.doneUoW(DefaultConsumer.java:107)[158:org.apache.camel.camel-core:2.16.3] at org.apache.camel.component.cxf.CxfConsumer$1.asyncInvoke(CxfConsumer.java:112)[159:org.apache.camel.camel-cxf:2.16.3] at org.apache.camel.component.cxf.CxfConsumer$1.invoke(CxfConsumer.java:75)[159:org.apache.camel.camel-cxf:2.16.3] ...
Problem root cause	Camel issue
Solution or Workaround	Upgrade to Talend version 6.4.1. Also add the continuationTimeout argument in the first cCXF component ("Advanced settings"), and set it to a value greater than the ReceiveTimeout. For example, set it to 70000 (70 seconds) if you are using the default ReceiveTimeout of 60000 (60 seconds).
JIRA ticket number	TESB-19757

Show Less

0

90

Official Support Articles

cFTP error: Error writing file

Problem Description When a cFTP component is used to put files to an FTP location, the route creates an empty file with 0 bytes. The same P... Show More

Problem Description

When a cFTP component is used to put files to an FTP location, the route creates an empty file with 0 bytes.

The same Put operation can transfer the files as expected to an FTP location using a tFTPPut component accordingly.

The stack trace captured in the console is shown below:

org.apache.camel.component.file.GenericFileOperationFailedException: Error writing file [EDI/in/TEST2]
	at org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:279)
	at org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:165)
	at org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:58)
	at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
	at org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:173)
	at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:341)
	at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:168)
	at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:91)
	at test_equ.test_cfile_ftp_0_1.TEST_CFILE_FTP$1$1$1.process(TEST_CFILE_FTP.java:130)
	at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:91)
	at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
	at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
	at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:442)
	at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:214)
	at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:178)
	at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:174)
	at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:101)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Solution

The route test cases do not work with JRE. Instead, you should install JDK.

Show Less

0

94

Official Support Articles

'TESB_ENV_PASSWORD system variable is not specified' Runtime error

Problem Description For encrypting clear text parameters and passwords, the environment variable TESB_ENV_PASSWORD is set in the Runtime/bin/sete... Show More

Problem Description

For encrypting clear text parameters and passwords, the environment variable TESB_ENV_PASSWORD is set in the Runtime/bin/setenv file.

However, running the Runtime command tesb:encrypt-text tadmin command fails with the following error:

TESB_ENV_PASSWORD system variable is not specified.
Second parameter should be used to specify password.

Root Cause

Runtime started as a system service. The Runtime/bin/setenv file is only taken into account if Runtime starts with the Runtime/bin/start or Runtime/bin/trun script.

Solution

Edit the Runtime/etc/****-wrapper.conf file and add the following variable:
```
set.TESB_ENV_PASSWORD=tadmin
```
Restart the service.

Show Less

0

95

Official Support Articles

Route configured with Apache Camel split xtokenize fails in Studio

Problem Description A route configured to use the split xtokenize method, as described on the Apache Camel Splitter page, runs as expected outsid... Show More

Problem Description

A route configured to use the split xtokenize method, as described on the Apache Camel Splitter page, runs as expected outside of Talend Studio, but fails to run in Studio, with the following exception:

javax.xml.stream.XMLStreamException: reader not supporting Location
    at org.apache.camel.support.XMLTokenExpressionIterator$XMLTokenIterator.getNextToken(XMLTokenExpressionIterator.java:470)
    at org.apache.camel.support.XMLTokenExpressionIterator$XMLTokenIterator.<init>(XMLTokenExpressionIterator.java:233)
    at org.apache.camel.support.XMLTokenExpressionIterator$XMLTokenIterator.<init>(XMLTokenExpressionIterator.java:185)
    at org.apache.camel.support.XMLTokenExpressionIterator.createIterator(XMLTokenExpressionIterator.java:92)
...

Root Cause

Camel split xtokenize uses StAX Location API and requires a StAX Reader implementation (for example, woodstox). These require stax2-api and woodstox-core libraries.

Solution

Add a cConfig to the route, then select the following JAR files from the drop-down list of libraries:

stax2-api-3.1.4
woodstox-core-asl-4.4.1

Show Less

0

167

Official Support Articles

SAM not logging FAULT_IN event when a service call fails in tESBConsumer

Question Sometimes, if tESBConsumer receives a fault response because it fails to call a web service provider, the SAM (Service Activity Monitori... Show More

Question

Sometimes, if tESBConsumer receives a fault response because it fails to call a web service provider, the SAM (Service Activity Monitoring) doesn't add a FAULT_IN event to its database. Is this expected?

Answer

This is expected behavior if the service provider is not available.

There will be a FAULT_IN event only for Business Faults. For example, the consumer sends a request with empty/wrong parameters, and the service provider sends a Fault as a response.
There will not be a FAULT_IN event if a service provider is not available (it cannot generate a FAULT_IN event, because this service is not available).

So, the only way to get a FAULT_IN event in the SAM database is to enable the SAM feature for the service consumer.

Show Less

0

73

Official Support Articles

Context issue when running route as ESB Microservice

Talend Version 6.3.1 Summary Context values are null when built as Spring Boot.Additional Versions Product ComponentStudioProblem Descripti... Show More

Talend Version	6.3.1
Summary	Context values are null when built as Spring Boot.
Additional Versions
Product
Component	Studio
Problem Description	Assume a route that uses a context variable called My_Var (set to some value). It runs successfully from Studio. But if this route is built as an ESB Microservice (Spring Boot), then the execution fails because the value of the context variable My_Var is null.
Problem root cause	There is a limitation from Spring Boot that a combination of underscores and upper-case in context variable names does not work as a property name.
Solution or Workaround	Rename the context variable name by removing the underscore (MyVar for example) or use only lower-case with underscores (my_var for example).
JIRA ticket number

Show Less

0

92

Official Support Articles

Runtime SSH client layout is mangled when the runtime is on Linux

Problem Description Connecting to Runtime with an SSH client such as PuTTY or MobaXterm displays a mangled layout. Root Cause This is ... Show More

Problem Description

Connecting to Runtime with an SSH client such as PuTTY or MobaXterm displays a mangled layout.

Root Cause

This is a client terminal configuration issue rather than ESB Runtime issue.

Solution

With PuTTY SSH client, enable Implicit CR in every LF and Implicit LF in every CR on the PuTTY Configuration > Terminal tab.

Show Less

0

93

Official Support Articles

ActiveMQ issue with UNC path an NFS shared directory

Problem Description In a Windows environment, when configuring ActiveMQ for master/slave setup, the following UNC path to an NFS share drive is s... Show More

Problem Description

In a Windows environment, when configuring ActiveMQ for master/slave setup, the following UNC path to an NFS share drive is set in the ActiveMQ configuration file activemq.xml:

<persistenceAdapter>
  <kahaDB directory="\\server_ip\amqshare\kahadb" />
</persistenceAdapter>

This prevents the ActiveMQ master from starting, with the following error message:

| INFO | Database \\server_ip\amqshare\kahadb\lock is locked by another server. 
This broker is now in slave mode waiting a lock to be acquired |
org.apache.activemq.store.SharedFileLocker | main

Root Cause

NFS file lock does not work on the Java level. The Lock operation is not supported for NFS share on Windows using UNC with Microsoft NFS client.

Solution

NFS mount the share drive, and map it to the P drive, for example, and use P:\...\kahadb in activemq.xml:

<persistenceAdapter>
  <kahaDB directory="P:\kahadb" />
</persistenceAdapter>

Show Less

0

234

Official Support Articles

cSplitter support for Tokenizer and TokenizeXML languages

Question How do you split a large file, line by line, using cSplitter? Solution Currently cSplitter support for Tokenizer and TokenizeXML... Show More

Question

How do you split a large file, line by line, using cSplitter?

Solution

Currently cSplitter support for Tokenizer and TokenizeXML languages is not available.

Workaround

Double-click the cSplitter component to display the Basic settings view in the Component tab.
Select None from the Language drop-down list, then enter body(String.class).tokenize("\n") in the Expression text box.

Show Less

0

77

Official Support Articles

cSplitter with XPath is not working/splitting as expected

Problem Description cSplitter with XPath language is not splitting as expected when consuming the correct XML input with the correct XPath expres... Show More

Problem Description

cSplitter with XPath language is not splitting as expected when consuming the correct XML input with the correct XPath expression.

In this case, the input XML looks like this:

<elmt1 xmlns="http://xyz.company.com">
  <msg>
    ....
  </msg>
  <msg>
    ....
  </msg>
  ...
</elmt1>

The XPath expression is:

/elmt1/msg

Root Cause

The input XML has namespace definitions.

Solution

Configure the cSplitter to Add Namespaces declarations by performing the following steps:

Select the Add Namespaces check box.
Click the plus (+) sign to add Namespaces.
1. Enter the name of your namespace in the Prefix column, (you can choose any name, it doesn't have to match the one in the input XML), for example, "v".
2. Enter the URL for your namespace in the URL column, for example, "http://xyz.company.com".
Then change the XPath Expression to include the namespace Prefix you added above. In this example, the expression becomes, "/v:elmt1/v:msg".

Show Less

0

75

Official Support Articles

Jobs do not inherit the environment variables of the user profile starting JobSe...

Talend Version 6.4.1 Summary When you start JobServer, the Job does not inherit the environment variables of your user profile.Additional Versions ... Show More

Talend Version	6.4.1
Summary	When you start JobServer, the Job does not inherit the environment variables of your user profile.
Additional Versions
Product	Talend Data Integration
Component	JobServer
Problem Description	After starting JobServer using the start_rs.sh script file, the output of the environment from a terminal matches with the output of the environment generated by the Job. This was also crosschecked with the /proc/PID/environ file used by JobServer.
Problem root cause	Different sets of environment variables were noticed when JobServer was started using the systemd option.
Solution or Workaround	For the environment variables to work correctly when JobServer is started using systemctl, you have to use either an Environment or an EnvironmentFile directive. Solution 1 [Unit] Requires=coreos-metadata.service After=coreos-metadata.service [Service] EnvironmentFile=/run/metadata/coreos Solution 2 [Service] # Client Env Vars Environment=ETCD_CA_FILE=/path/to/CA.pem More information, see Using environment variables in systemd units
JIRA ticket number

Show Less

0

137

Official Support Articles

Undelivered messages are not sent to the ActiveMQ Dead Letter Queue (DLQ)

Talend Version All versions Summary Undelivered messages are not sent to the ActiveMQ Dead Letter Queue.Additional Versions ProductESBComponentA... Show More

Talend Version	All versions
Summary	Undelivered messages are not sent to the ActiveMQ Dead Letter Queue.
Additional Versions
Product	ESB
Component	ActiveMQ
Problem Description	A JMS message fails in the route exchange flow, and consequently, the JMS transaction is rolled back, which triggers the ActiveMQ redelivery policy. When message redelivery attempts exceed maximumRedeliveries (six times by default), the pending message should be sent to the Dead Letter Queue, but this doesn't happen.
Problem root cause	The original undelivered message is non-persistent, but by default, only undelivered persistent messages are sent to DLQ.
Solution or Workaround	Send persistent messages to ensure that they end up in DLQ if they can't be delivered. See the Apache ActiveMQ Message Redelivery and DLQ Handling page for more information.
JIRA ticket number

Show Less

0

224

Official Support Articles

How to configure the cJMS component with the exclusive consumer feature

Talend Version 6.3.1 Summary Using the ActiveMQ exclusive consumer feature with Talend route.Additional Versions6.2.1ProductTalend ESBCompo... Show More

Talend Version	6.3.1
Summary	Using the ActiveMQ exclusive consumer feature with Talend route.
Additional Versions	6.2.1
Product	Talend ESB
Component
Problem Description
Problem root cause
Solution or Workaround	To enable the exclusive consumer feature, you would need to add the destination.consumer.exclusive=true parameter to the queue property of cJMS. However, this parameter is not currently supported with cMQConnectionFactory + cJMS. Instead, you need to use the generic cMessagingEndpoint component, with the following URI as an example: activemq:queue:SampleQ1?destination.consumer.exclusive=true Where activemq bean is defined in the spring config file as follows: <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent"> <property name="brokerURL" value="tcp://localhost:61616"/> <property name="userName" value="tadmin"/> <property name="password" value="tadmin"/> </bean> Adjust the values for brokerURL, userName, and password as appropriate.
JIRA ticket number

Show Less

0

116

Official Support Articles

TAC locator configuration value in a locator clustered environment

Question In a replicated Service Locator setup with three (or more) locator servers running, which locator URL should be specify on the Configura... Show More

Question

In a replicated Service Locator setup with three (or more) locator servers running, which locator URL should be specify on the Configuration page in TAC?

Answer

The locator configuration in TAC should specify all locator server URLs delimited by a comma.

For example: hostA:2181,hostB:2181,hostC:2181

Show Less

0

70

Official Support Articles

Official Support Articles

Recent Documents

Question

Answer

Environment Setup

Launching a standard Azure Kubernetes Service (AKS)

Using Azure CLI or Cloud shell

Using the Azure portal

Installing Istio with Helm on AKS

Install Helm client

Using Cloud Shell (AKS)

Using shell script (Linux)

From Chocolatey (Windows)

Customizing the Istio installation with Helm

Launching Istio monitoring dashboards for AKS

Publish Talend microservices to Azure Container Registry

Creating an Azure Container Registry (ACR)

Publishing microservices from Talend Studio to ACR

Authentication with Azure Container Registry from AKS using a service principal

Obtaining a service principal

Obtaining the ACR resource ID

Granting role-based access to ACR from AKS

Create a Kubernetes secret with service principal

Using Secret in the Kubernetes resource files

Conclusion

Best practice

Problem Description

Solution

Overview of Best Practice

Installation

Configuration

Add custom context data to every monitoring event

Example: monitoring in Elasticsearch

Camel Event

Camel Trace

Problem Description

Solution

Problem Description

Root Cause

Solution

Problem Description

Root Cause

Solution

Question

Answer

Problem Description

Root Cause

Solution

Problem Description

Root Cause

Solution

Question

Solution

Workaround

Problem Description

Root Cause

Solution

Solution 1

Solution 2

Question

Answer