Managing Talend microservices with Istio service mesh on Amazon Elastic Kubernetes Service

[TalendSolutionExpert]

Content:

• Prerequisites
• Environment Setup
• Launching an Elastic Kubernetes Service (EKS)
• Install AWS CLI
• Install eksctl and the aws-iam-authenticator
• Install eksctl and aws-iam-authenticator (Windows)
• Install eksctl and aws-iam-authenticator (Linux)
• Create cluster
• Installing Istio with Helm on EKS
• Installing Helm client
• Using shell script (Linux )
• From Chocolatey (Windows)
• Customizing the Istio installation with Helm
• Launching Istio monitoring dashboards for AWS
• Publish Talend microservices to Amazon Elastic Container Registry (ECR)
• Creating repositories in ECR
• Publishing microservices from Talend Studio to ECR
• Authentication with Amazon Elastic Container Registry from EKS
• Conclusion

The Talend Community Knowledge Base (KB) article, Managing Talend microservices with Istio service mesh on Kubernetes, shows you how to connect, secure, control, and observe Talend microservices leveraging Istio.

This article describes the installation and platform-specific steps for managing Talend microservices using Istio service mesh on Amazon Elastic Kubernetes Service (EKS).

For more information on installing Talend microservices using Istio on other cloud providers, see the following Talend Community KB articles:

• Managing Talend microservices with Istio service mesh on Google Kubernetes Engine

• Managing Talend microservices with Istio service mesh on Azure Kubernetes Service

Prerequisites

• Familiarity Amazon Web Services (AWS)
• Talend 7.2 installed
• Administrator level access to create EKS cluster and privileges to launch AWS
• Internet access
• Download the EKS_yaml_files.zip file

Environment Setup

Launching an Elastic Kubernetes Service (EKS)

Amazon EKS cluster can be easily created in a single step using eksctl CLI commands and a YAML file with the cluster configuration.

For more information on creating an EKS cluster using AWS Management Console, see the Amazon EKS, Getting Started with the AWS Management Console guide.

Install AWS CLI

Following the instructions in the Amazon EKS, Getting Started with eksctl documentation, complete the following sections:

• Install the Latest AWS CLI

• Configure Your AWS CLI Credentials

• Install and Configure kubectl for Amazon EKS

Install eksctl and the aws-iam-authenticator

Install eksctl and aws-iam-authenticator (Windows)

Log in to the command prompt as administrator.

#install eksctl and aws-iam-authenticator
chocolatey install -y eksctl aws-iam-authenticator

# verify eksctl version
eksctl version

Install eksctl and aws-iam-authenticator (Linux)

1. Login to the bash shell.

2. Download and extract the latest release of eksctl by using the following command:

   curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/latest_release/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp

3. Move the extracted binary to /usr/local/bin and verify the version.

   sudo mv /tmp/eksctl /usr/local/bin
   
   # verify eksctl version
   eksctl version

Create cluster

Download the create_eks_cluster.yaml file (attached to this article) to a directory and execute the following command:

eksctl create cluster -f create_eks_cluster.yaml

Installing Istio with Helm on EKS

Helm is a package manager for Kubernetes. Helm Charts help you define, install, and upgrade even the most complex Kubernetes application.

Istio provides customizable helm templates for installation on a Kubernetes cluster. For more information on other installation options, see the Istio Installation Guides.

Installing Helm client

Install Helm client on Linux / Windows OS, using one of the following options.

Using shell script (Linux )

Helm client can be installed on Linux OS using shell commands.

Connect to the shell, then install Helm client:

curl -LO https://git.io/get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh

From Chocolatey (Windows)

Helm package can be installed on Windows using Chocolatey.

1. Launch a command prompt, then execute the following command:

   choco

   Note: If you get an error "'choco' is not recognized as an internal or external command, operable program or batch file", follow the Install Chocolatey on Windows instructions.

2. Install helm using the choco install command:

   #Install helm client
   choco install kubernetes-helm

Customizing the Istio installation with Helm

From the bash shell or command prompt:

1. Create an istio_install directory.

   # create directory istio_install
   mkdir istio_install (Windows / Linux)
   
   # move into the istio_install
   cd istio_install

2. Download an Istio release, then execute the steps in the Customizable Install with Helm guide.

   # Download the installation file using the curl command
   curl -Ls -O https://github.com/istio/istio/releases/download/1.3.4/istio-1.3.4-linux.tar.gz  (Linux)
   curl -Ls -O https://github.com/istio/istio/releases/download/1.3.4/istio-1.3.4-win.zip (Windows)
   
   # gunzip and untar the file
   gunzip istio-1.3.4-linux.tar.gz 
   tar -xvf istio-1.3.4-linux.tar
   
   # move into the directory istio-1.3.4
   cd istio-1.3.4
   
   # Configure the PATH environment variable
   export PATH=$PWD/bin:$PATH (Linux)
   set PATH=%CD%/bin;%PATH% (Windows)
   
   # Initialize helm
   helm init
   
   #Add Istio to the helm repository
   helm repo add istio.io https://storage.googleapis.com/istio-release/releases/1.3.4/charts/
   
   #Create namespace istio-system
   kubectl create namespace istio-system
   
   #Install all the istio CRDs
   helm template install/kubernetes/helm/istio-init --name istio-init --namespace istio-system | kubectl apply -f -

3. Install Istio using one of the Installation Configuration Profiles.

   Note: This article uses the demo_auth profile.

   # Execute the helm template with the predefined settings in the demo_auth profile
   
   helm template install/kubernetes/helm/istio --name istio --namespace istio-system \
   --values install/kubernetes/helm/istio/values-istio-demo-auth.yaml | kubectl apply -f -

4. Execute the following command, then verify that the status of all the Istio components in the demo-auth configuration profile is Running / Completed.

   kubectl get pods -n istio-system

   

5. Verify an External IP is assigned to the istio-ingressgateway service.

   kubectl get svc -n istio-system

   

   Note:

   • Ingress traffic to application pods or microservices is only made possible using the External / Public IP assigned to the istio-ingressgateway service.

   • An external Load-balancer is launched on EKS automatically, and its IP is assigned to the istio-ingressgateway service.

Launching Istio monitoring dashboards for AWS

With Istio demo-auth profile add-ons for dashboards such as Prometheus, Grafana, Kiali, and Jaeger are enabled by default.

• Prometheus is a web-based graphical user interface for querying the Istio metric values.

• Grafana is a web-based graphical user interface that provides a global view of the mesh along with services and their workloads.

• Kiali is a web-based graphical user interface to view service graphs of the mesh and Istio configuration objects. Different graph types such as App, Versioned App, Workload, and Service are available to view the services in the mesh.

Open a Windows command prompt in Administrator mode, then execute the following commands and launch the dashboards using local port forwarding, as shown below:

# Execute the below port-forward command and access Prometheus using local port forwarding
kubectl port-forward svc/prometheus 9090:9090 -n istio-system

# Execute the below port-forward command and access Grafana using local port forwarding
kubectl port-forward svc/grafana 3000:3000 -n istio-system

# Execute the below port-forward command and access Kiali using local port forwarding
kubectl port-forward svc/kiali 20001:20001 -n istio-system

# Execute the below port-forward command and access Jaeger using local port forwarding
kubectl port-forward svc/jaeger 16686:16686 -n istio-system

Publish Talend microservices to Amazon Elastic Container Registry (ECR)

Creating repositories in ECR

1. Log in to the AWS management console, search for the service ECR, then click Create repository.

   

2. Create two repositories in ECR, as shown in the following example:

   

Publishing microservices from Talend Studio to ECR

1. In Talend Studio, fill in the following properties, publish the Orders microservice to ECR.

   1. Docker host: select Remote, then enter your Docker host information

   2. Image name: enter your repository name

   3. Registry: enter your Amazon container Registry URI

   4. Username: enter your access key ID

   5. Password: enter your secret access key

   Click Finish.

   

2. Repeat Step 1 and publish the Customers microservice. Make a note of the URLs of the microservice images published to ECR.

Authentication with Amazon Elastic Container Registry from EKS

Kubernetes has native support for the Amazon Elastic Container Registry (ECR) when nodes are AWS EC2 instances all pods in a cluster have read access to images in a container registry.

Conclusion

This article explains how to launch an Amazon Elastic Kubernetes Cluster using eksctl, install Istio manually using Helm, and how to publish Talend microservices on Amazon Elastic Container Registry.