Design and Development

Utilizing the logging and monitoring capabilities offered by Talend.
View full article
Use the --logging.config parameter.
View full article
Configure Databricks and ADLS Gen2 in Talend.
View full article
Installation and platform-specific steps to manage Talend microservices using Istio on AKS.
View full article
Installation and platform-specific steps to manage Talend microservices using Istio on EKS.
View full article
Installation and platform-specific steps to manage Talend microservices using Istio on GKE.
View full article
Using the tDataEncrypt and tDataDecrypt components to encrypt and decrypt data.
View full article
Overview The Talend Community Knowledge Base article, Containerization and orchestration of Talend microservices with Docker and Kubernetes shows you how to: Build a demo Customer service as a Talend microservice container image Generate Kubernetes resources for the Customer service using the f8-m-p plugin Deploy the Customer service on a Kubernetes cluster using the f8-m-p Invoke the Customer service using a ClusterIP However, in Production environments, it is more likely you'll need to: Expose a few microservices to external endpoints using an external load balancer Route traffic dynamically to several internal endpoints running in a distributed cluster Handle faults, timeouts, and perform retries Roll incremental changes or bug fixes without service outage Deploy multiple versions and route incoming traffic based on a match criteria Observe logs, metrics, and health of the services This article provides another approach to deploying Talend microservices to a distributed Kubernetes cluster using the kubectl utility, and leveraging Istio to manage some of the production-grade requirements for microservices.     Prerequisites Talend 7.2 or above Internet access Access to a Docker repository Install a MySQL database Set up Kubernetes cluster and install Istio (see Environment) Download the sources for the project available in the Microservices-JobDesigns.zip file (attached to this article) Download the demo Kubernetes resource ZIP file (attached to the article) for one of the following cloud platforms: Demo Kubernetes resource files for AKS Demo Kubernetes resource files for GKE Demo Kubernetes resource files for EKS   Environment Before you can understand some of the Istio features, you need to set up a Kubernetes cluster and install Istio, by following the platform-specific instructions using one of the following Talend Community Knowledge Base (KB) articles: Follow the Launching a standard Azure Kubernetes Service (AKS) and Installing Istio with Helm on AKS sections in the Managing Talend microservices with Istio service mesh on Azure Kubernetes Service (AKS) article. Follow the Launch a standard Google Kubernetes Engine (GKE) and Installing Istio with Helm on GKE sections in the Managing Talend microservices with Istio service mesh on Google Kubernetes Engine (GKE) article. Follow the Launching an Elastic Kubernetes Service (EKS) and Installing Istio with Helm on EKS sections in the Managing Talend microservices with Istio service mesh on Amazon Elastic Kubernetes Service (EKS) article.   Use case: Order processing Typically, a request passes through different stages during the order processing lifecycle, such as Create, Get, and Amend.   The following diagram showcases a few of those stages, the sequence of steps, and the services involved in a CreateOrder and GetOrder workflow.   In this scenario, you're building a company's retail car purchasing microservice where: The CreateOrder workflow is responsible for processing all the new orders placed by customers using different sales channels. The GetOrder workflow retrieves the details and status of a previously placed order.   Creating a mock order This section shows you how to create a mock order (referred to as a mock order because it's not a real order) with a sample MySQL database that contains information about classic car models purchased by customers. Launch an RDS MySQL instance. Import the mysql-sampledatabase-classicmodels.sql file (attached to this article) to a folder. Login as root and connect to the MySQL shell: sql connect INSTANCE_NAME --user=root --quiet Create the classicmodels MySQL database by using the source command: source file_location/mysql-sampledatabase-classicmodels.sql   After the classicmodels database is created, observe the relationship between the customers, orders, and products entities.   Implementing the GetOrder The GetOrder workflow retrieves all the orders made by a customer and displays the status of all the orders. This section shows you how to implement the GetOrder services as a Talend microservices.   Microservice Orders Launch Talend Studio, then import the Jobs in the Microservices-JobDesigns.zip (attached to this article) file. Open the Orders microservice.   Click the Job tab, then click Deployment and notice that the Build Type is set to Microservice for ESB. Observe the Job design: Orders service is a REST data service Fetches all the order details from the database using customerNumber Uses three different context environments namely STUDIO, QA, and PROD Run the Job in Studio. By default, the service listens on port 8090.   Microservice Customers Open the Customers microservice.   Click the Job tab, then click Deployment and notice that the Build Type is set to Microservice for ESB. Observe the Job design: Customers service is a REST data service Fetches all the customer details by customerNumber Invokes the Order service and all the orders created by the customer Uses three different context environments namely STUDIO, QA, and PROD In the context variables, observe the endpoint of the REST client, for example, the ORDER_SERVICE_CLIENT_URL parameter The host and port of the ORDER_SERVICE_CLIENT_URL changes for every environment In the PROD context the host is set with a logical host name, for example, order-service Run the Job in Studio. By default, the service listens on port 8090 and fails if the port is already in use.   Build the GetOrder services In the Studio, right-click the Customers microservice, select Build Job, and save the build to a folder demo.   Follow the process in Step 1, and build a Job for Orders. Launch the Windows Command Prompt and move them into the demo folder. Follow the steps in the next section and run both the microservices.   Running the GetOrder service as standalone microservice with QA context Run the Orders service: Java -jar Orders_0.1.jar --server.port=8065 --spring.config.location=classpath:config/contexts/ --spring.config.name=QA Run the Customers service: Java -jar Customers_0.1.jar --server.port=8066 --spring.config.location=classpath:config/contexts/ --spring.config.name=QA Invoke the Customers service using Talend API Tester or in a browser, enter the following URL and click the Send button: http://localhost:8066/services/customer/103   Notice that the GetOrder service returns all the orders created by a customer upon call.   Use case: Order processing with Istio and Kubernetes Istio is an open-source framework that provides a service mesh for microservices and can be installed on Kubernetes.   Why Istio with GetOrder Services? Istio can provide many additional capabilities to the Talend microservices by following a few best practices and without any additional changes to the Job designed in Talend Studio.   Adding Istio capabilities for GetOrder services With Istio sidecar injection, a sidecar proxy Envoy is deployed in front of the GetOrder services (customer-service, order-service) and provides Istio Envoy capabilities. For more information on Envoy and its features, read the Istio Envoy documentation.   Service Discovery Service discovery explains how a service register and deregister its endpoint when deployed on a distributed cluster and how the clients/services can dynamically locate the endpoint URLs. Istio Pilot has a built-in Kubernetes adapter and watches the Kubernetes API server for pod registrations and deregistrations.   Note: Istio service discovery leverages the discovery features provided by platforms like Kubernetes. The diagram above shows how Istio: Adds Envoy proxy in front of the Talend microservice application pods Service discovery works for locating the services internally with in the mesh For example, the customer-service Pods in namespace talend-istio (1) dynamically discovers the endpoint URL of the order-service (4) by making a name lookup (order -service or order-service.talend-istio) using the Istio Kubernetes adapter (2) and the Envoy proxy (3).   For more information, read Discovering services in the Kubernetes using DNS documentation.   Traffic management   The advantage of Istio Traffic management: It allows developers and deployment teams to easily configure routing rules and control the flow of traffic and API calls between services. Operation teams can apply new rules or modifications without stopping the services. Istio Pilot watches for changes in routing rules and dynamically propagates new changes to the Envoy proxy. This article shows you how to configure the routing rules on endpoints of customer-service, order-service, and manage: Service discovery and routing Recovery failures Canary deployments   Gateways The Gateway manages external Ingress traffic to our service mesh. It supports multiple protocols and can be configured to allow traffic on specific ports.   Virtual services A virtual service allows to configure an ordered list of routing rules and decides control how the Envoy proxy should route the traffic.   The following diagram explains how: External traffic is allowed to the service mesh using the Ingress Gateway Define routing rules in a virtual service Apply rules and impose match conditions on the request URI Route traffic to one or more destinations Refer to this VirtualService configuration settings in the Service_Discovery.yaml file.   Destination rules After enforcing the routing rules on the incoming traffic with virtual services, additional policies can be applied using destination rules. For example, the incoming traffic to order-service can be routed to different versions (v1, v2) of the same service using subsets by specifying match conditions and weights.   Refer to this VirtualService and DestinationRule configuration entries in the CanaryDeployments.yaml file.   Service Entry ServiceEntry enables adding additional entries into the Istio internal service registry so that auto-discovered services in the mesh can access/route to these manually specified services.   In the GetOrder use case, the Talend microservice residing inside the service mesh need access to an external RDS MySQL instance.   Refer to this ServiceEntry configuration in the Service_Discovery.yaml file. apiVersion: networking.istio.io/v1alpha3 kind: ServiceEntry metadata: name: mysql-external spec: hosts: - istio-customer-db.some_id.eu-central-1.rds.amazonaws.com ports: - name: tcp number: 3306 protocol: tcp location: MESH_EXTERNAL resolution: DNS   Logical architecture with Istio and Talend microservices The microservices in the GetOrder use case are deployed to Kubernetes using Istio sidecar injection.   The following diagram depicts a high-level logical architecture of the complete setup.   Publishing GetOrder microservices to a container registry To publish the GetOrder microservices to a container registry, using the KB article you selected in the Environment section of this article (Azure Kubernetes Service (AKS), Google Kubernetes Engine (GKE), or Amazon Elastic Kubernetes Service (EKS)), execute the platform-specific instructions in one of the following container registry sections: Publish Talend microservices to Azure Container Registry Publish Talend microservices to Google Container Registry Publish Talend microservices to Amazon Elastic Container Registry (ECR)   Deploying GetOrder microservices with Istio on Kubernetes Deploying the GetOrder services on a Kubernetes cluster requires the following resources: Kubernetes resource files (YAML) Cloud / SDK shell Kubernetes command-line tool (kubectl) These are available in the demo Kubernetes resource file you download in the Prerequisites section of this article.   Set Talend context environment The GetOrder services used in this article use Talend context environment and context variables, for example, the Prod context environment and context variables are used in this article for microservices deployments on a Kubernetes cluster.   Unzip the demo Kubernetes resource file you downloaded in the Prerequisites section of this article. Open the resources files for one of the Kubernetes clusters. Search for the API type Deployment. Observe the value of the spring.config.location property. spring.config.location is set with the classpath location of Talend context environment properties file, for example, PROD.properties command: ["/bin/sh", "/maven/Customers/Customers_run.sh"] args: ["--spring.config.location=classpath:config/contexts/PROD.properties"]   Enable Istio sidecar injection at the namespace level Connect to the cloud shell, create a namespace called talend-istio: kubectl create ns talend-istio Enable Istio sidecars injection on the new namespace: kubectl label namespace talend-istio istio-injection=enabled   Deploy the GetOrder services using a Kubernetes resource file Using the kubectl utility, execute the apply command along with the resource file. kubectl apply -n talend-istio -f FILE_NAME   From the cloud / SDK shell, observe that the Status of the pods in the talend-istio namespace is Running. kubectl get pods -n talend-istio   Observe the External-IP associated with the istio-ingressgateway. kubectl -n istio-system get service istio-ingressgateway Note: The IP address assigned to istio-ingressgateway is the same as the IP address of the external LoadBalancer.   Undeploy the GetOrder services using a Kubernetes resource file Using the kubectl utility, execute the delete command along with the resource file. kubectl delete -n talend-istio -f FILE_NAME   From the cloud / SDK shell, observe that the pods in the talend-istio namespace were deleted. kubectl get pods -n talend-istio Demo   The demo uses Istio Dashboards for Monitoring, execute the platform-specific instructions, and open the dashboards for Kiali, Grafana, and Jaeger: Launching Istio monitoring dashboards for GKE Launching Istio monitoring dashboards for AKS Launching Istio monitoring dashboards for AWS   Service discovery and routing Using the kubectl utility, execute the apply command along with the resource file. kubectl apply -n talend-istio -f Service_Discovery.yaml Before executing the apply command download and modify the cloud-specific template files, and make the following changes: Search for the API type Deployment, replace the value in the image property with your registry location URL: image: REPLACE_WITH_YOUR_RESPOSITORY_URL/k8s_microservice/customer/microservices/orders:0.2.0 If you publish the images to Azure Container Registry, then Use Kubernetes Secrets in the resource files in the API of type Deployment, for more detailed steps follow the section Authentication with Azure Container Registry from AKS using Service principal. Search for the API type ServiceEntry, replace the hosts and number property values with the URL and port number of an external database. hosts: - istio-customer-db.some_id.eu-central-1.rds.amazonaws.com ports: - name: tcp number: 3306 From the cloud / SDK shell, observe that the Status of the pods in the talend-istio namespace is Running. kubectl get pods -n talend-istio From your local machine (Windows / Linux), connect to Kiali, observe that there are two applications in the talend-istio namespace, and there is no traffic.   Invoke the Customers service endpoint using Talend API Tester, enter the following URL, click the Send button, and observe the response. http://REPLACE_WITH_YOUR_INGRESS_EXTERNAL_IP/customer/103 Create more load on the Customers service, invoke the Customers service endpoint by clicking the Send button 4 to 5 times. Notice that the dashboard detects some traffic.   In the Dashboard, from the menu on the left, click Graph, then select the Workload graph. In the Workload graph, observe the last 5 minutes of traffic. Notice that the services are arranged in a sequence following their order of invocation, for example, istio-ingressgateway > customer-service proxy > customer-service application > istio-customer-db > order-service proxy > then order-service application. Observe that the HTTP traffic is green, indicating there are no failures.   Click the Namespace down arrow next talend-istio, and add the istio-system namespace. Review the Istio service graph, and notice that the istio-telemetry service is added to the graph istio-telemetry gathers metrics data from the sidecars customer-service and order-service   From your local machine (Windows / Linux), open the Google Cloud SDK shell, then using local port forwarding, connect to Grafana. In the Istio Workload Dashboard, observe the request/response duration and size.   From your local machine (Windows / Linux), connect to Jaeger. Find Traces for customer-service, observe the dependency between the services and latency.   Note: Applications must manually propagate appropriate b3 HTTP headers for correlating spans into a single trace. For more information, read Trace context propagation and Zipkin b3-propagation Recovery failures This section helps you understand how Istio handles recovery failures such as faults, timeouts, and retries by enhancing the configuration from the Service discovery and routing section of this article.   In this scenario, assume the order-service runs CPU intensive tasks and not accepting new requests. To understand this scenario, configure the Virtual service as below: Configure Istio to create a delay of two secs before routing the request to order-service Customer-service timeouts due to the latency induced by order-service On timeout, customer-service performs a retry Using the kubectl utility, execute the delete command and clean the previous deployments: kubectl delete -n talend-istio -f Service_Discovery.yaml Using the kubectl utility, execute the apply command along with the resource file: kubectl apply -n talend-istio -f RecoveryFailures.yaml Note: Before executing the apply command, modify the YAML file for environment-specific changes as follows: Search for the API type Deployment, replace the value in the image property with your registry location URL. image: eu.gcr.io/customer-success-account_id/k8s_microservice/customer/microservices/orders:0.2.0 Search for the API type ServiceEntry, replace the hosts, and number property values with the URL and port number of an external database. hosts: - istio-customer-db.some_id.eu-central-1.rds.amazonaws.com ports: - name: tcp number: 3306 From the cloud / SDK shell, observe that the Status of the pods in the talend-istio namespace is Running. kubectl get pods -n talend-istio Invoke the Customer service endpoint using Talend API Tester, enter the following URL, and observe the response. http://REPLACE_WITH_YOUR_INGRESS_EXTERNAL_IP/customer/103 Create more load on the Customer service by clicking the customer-service endpoints 4 to 5 times. Notice that the dashboard detects some traffic. From your local machine (Windows / Linux), open the Google Cloud SDK shell, then using local port forwarding, connect to Kiali. Open the Versioned app graph, observe the customer-service is Red and 10% of the HTTP traffic fails due to timeouts induced on order-service.   Canary deployments Using the kubectl utility, execute the delete command and clean the previous deployments. kubectl delete -n talend-istio -f RecoveryFailures.yaml Using the kubectl utility, execute the apply command along with the resource file. kubectl apply -n talend-istio -f CanaryDeployments.yaml Note: Before executing the apply command, modify the YAML file for environment-specific changes as follows: Search for the API type Deployment, replace the value in the image property with your registry location URL. image: eu.gcr.io/customer-success-account_id/k8s_microservice/customer/microservices/orders:0.2.0 Search for the API type ServiceEntry, replace the hosts, and number property values with the URL and port number of an external database. hosts: - istio-customer-db.some_id.eu-central-1.rds.amazonaws.com ports: - name: tcp number: 3306 From the cloud / SDK shell, observe that the Status of the pods in the talend-istio namespace is Running. kubectl get pods -n talend-istio Invoke the Customer service endpoint using Talend API Tester, enter the following URL, and observe the response. http://REPLACE_WITH_YOUR_INGRESS_EXTERNAL_IP/customer/103 Create more load on the Customer service by clicking the customer-service endpoints 4 to 5 times. Notice that the dashboard detects some traffic. From your local machine (Windows / Linux), open the Google Cloud SDK shell, then using local port forwarding connect to Kiali.   From your local machine (Windows / Linux), open the Google Cloud SDK shell, then using local port forwarding, connect to Grafana. In the Istio Workload Dashboard, observe the traffic was split into two versions of the order-service.   Conclusion This article explains how Talend integrates with distributed platforms like Kubernetes and how you can leverage Istio for managing traffic, service discovery, monitoring, and tracing services in a service mesh. Services are designed, developed, and published with the help of Talend Studio.
View full article
Using a third-party artifact repository makes the Talend Job deployment process easier in Talend Cloud.
View full article
Learn how to set up a zero install CI.
View full article
Top Contributors