Introduction to Talend Microservices with Kubernetes

Overview

You may have heard of Docker or microservice architecture, but may not be familiar with them. Microservice architecture has become the standard for next gen applications, and Docker and containers are now commonly used for running microservice-based applications.

 

As you move towards a microservice architecture, creating one manually is very easy, but what about production deployment and high availability? There are many orchestrators that allow you to deploy multi-node applications such as Docker Swarm, Mesos, and Kubernetes.

 

This article will show you how to use Kubernetes, but if you would like to read more about microservice architecture in general, or some of the tools specifically, you may find these resources helpful:

Sources for the project are available in the attached Zip file.

 

Building a Talend Microservice

In this tutorial, you will see how to build a simple microservice with Talend ESB (route). To do it, you need to design a Talend Route based on Apache Camel that will simulate a customer service with static data. Usually, when you develop this type of service, your service will be connected to a database or some sort of storage. But this demo keeps it simple.

demojob.png

 

  1. Create this service with two operations:

    • Get all customers: retrieve all customers
    • Get customer by Id: retrieve one customer for a given identifier

    demoREST.png

     

  2. Generate the customers list using a bean in the component cBeanRegister (customers).

    cBeancustomers.png

     

  3. To Export the route as a microservice, during the build, for build type select ESB Microservice (Spring Boot).

    Note: Do not check Add maven script as it only generates sources and pom.xml files.

    buildType.png

     

  4. You now have a Talend Microservice ready to run. You can test it by unzipping it and running it like this:

    $ DemoRESTRoute/DemoRESTRoute_run.sh --server.port=8040
            ______     __              __                 
           /_  __/__  / /__ _____  __ / /                 
            / / / _ `/ /  -_) _ \\/ _  /                  
           /_/ \\_,_/_/\\__/_//_/\\_,_/                 
            (Talend ESB 6.3.1)
    
    Hit '<ctrl-c>' to stop and quit the running.
  5. Use a browser to test it: http://localhost:8040/services/customers.

 

Building Images

You can use Kubernetes to deploy Docker containers, but before you can deploy them, you need to create your Docker images.

 

In this example, you need two images:

  • Java:8: build an image containing only a JDK 1.8. It will your base image for future microservices.
  • customer-service:latest: this is your Talend microservice.

 

Building the Java base image

  1. Use the Dockerfile below to create your base image:

    FROM ubuntu:latest
    
    LABEL maintainer "user <user@company.com>"
    
    # Environment variables (build purpose)
    ENV UPDATE_VERSION=8u131 
    ENV JAVA_VERSION=1.8.0_131 
    ENV BUILD=b11
    ENV JAVA_HOME=/opt/java
    
    # Change Word directory to opt
    WORKDIR /opt/
    
    # Java installation
    RUN apt-get update && \
        apt-get -y install wget && \
        wget --quiet --no-check-certificate -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/${UPDATE_VERSION}-${BUILD}/d54c1d3a095b4ff2b6607d096fa80163/jdk-${UPDATE_VERSION}-linux-x64.tar.gz -O /tmp/java.tar.gz && \
        tar xzf /tmp/java.tar.gz -C /opt && \
        mv jdk${JAVA_VERSION}/ java/ && \
        update-alternatives --install /usr/bin/java java /opt/java/bin/java 1 && \
        update-alternatives --install /usr/bin/javac javac /opt/java/bin/javac 1 && \
        update-alternatives --install /usr/bin/javaws javaws /opt/java/bin/javaws 1 && \
        rm -Rf /tmp/java.tar.gz && \
        apt-get clean
  2. To build the image, use the docker build command:

    $ cd docker/java/
    $ docker build -t user/java:8 .
    
    $ docker images
    REPOSITORY     TAG    IMAGE ID      CREATED             SIZE
    user/java      8      51325f0bdca3  43 seconds ago      540 MB
    ubuntu         latest 7b9b13f7b9c0  12 days ago         118 MB

 

Building the REST Service image

Prerequisite: Export the Zip file in the source folder: docker/rest-service/source

  1. Use the Dockerfile below to create your microservice image:

    FROM user/java:8
    
    LABEL maintainer "user <user@company.com>"
    
    # Change Word directory to opt
    WORKDIR /opt/rest-service
    
    COPY ./source ./
    
    EXPOSE 8040
    
    ENTRYPOINT ["DemoRESTRoute/DemoRESTRoute_run.sh","--server.port=8040"]

    Using --server.port=8040 allows you to fix a port in the container. This port will be mapped to a local port using the -p option in the docker run command.

  2. To build the image, use the docker build command:

    $ cd docker/rest-service
    $ docker build -t user/customer-service:latest .
    
    $ docker images
    REPOSITORY              TAG    IMAGE ID     CREATED            SIZE
    user/customer-service   latest 6ccedc100de5 About a minute ago 542 MB
    user/java               8      51325f0bdca3 7 minutes ago      540 MB
    ubuntu                  latest 7b9b13f7b9c0 12 days ago        118 MB
  3. To test the image locally, run a container with the docker run command:

    $ docker run -d --name customer-service -p 8040:8040 user/customer-service:latest
    $ docker ps
    CONTAINER ID    IMAGE                            COMMAND                  
    d8a8ce1e2332    user/customer-service:latest "DemoRESTRoute/Dem..."
    CREATED         STATUS         PORTS                   NAMES
    22 seconds ago  Up 22 seconds  0.0.0.0:8040->8040/tcp  customer-service
  4. Use a browser to test it: http://localhost:8040/services/customers

    localhost.png

     

  5. Access your microservice container logs with the docker logs command:

    $ docker logs customer-service 
            ______     __              __                 
           /_  __/__  / /__ _____  __ / /                 
            / / / _ `/ /  -_) _ \\/ _  /                  
           /_/ \\_,_/_/\\__/_//_/\\_,_/                 
            (Talend ESB 6.3.1)
    
    Hit '<ctrl-c>' to stop and quit the running.
    
    [WARN ]: cxfrs_provider.cLog_1 - Exchange[ExchangePattern: InOut, BodyType: beans.Customers, Body: beans.Customers@6231e438]
  6. You can stop the container and remove it with the following command:

    $ docker stop customer-service && docker rm -v customer-service

 

Creating a Docker repository

For this section of the tutorial, you need a Docker Hub login. You will use a private repository on Docker Hub, as it will allow a demonstration of how to use secrets on Kubernetes.

 

You could also use a private registry, but you would have to adjust some parameters.

 

Creating a private repository

  1. Select Create Repository.

    dockerHub.png

     

  2. Name the repository customer-service, select private in the Visibility list box, then click Create.

    createRepo.png

     

    The new repository is now available to you.

    newRepo.png

     

  3. To push your image to the new repository, log in with the docker login command:

    $ docker login
    
    Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
    Username (user): 
    Password: 
    Login Succeeded
  4. Use the docker push command to send the image to the Docker Hub.

    $ docker push user/customer-service:latest
    The push refers to a repository [docker.io/user/customer-service]
    5f1dd919a45f: Pushed 
    ef1e9d4703a2: Pushed 
    fd6d079e8e61: Pushed 
    fd12f029b813: Pushed 
    d8b353eb3025: Mounted from library/ubuntu 
    f2e85bc0b7b1: Mounted from library/ubuntu 
    fc9e1e5e38f7: Mounted from library/ubuntu 
    fe9a3f9c4559: Mounted from library/ubuntu 
    6a8bf8c8edbd: Mounted from library/ubuntu 
    latest: digest: sha256:6a5f426b388f89b376cbb84a67ce5ac7714528586951b637d83ee16c86ebbc91 size: 2201

 

Using Kubernetes with MiniKube

There are many ways to use Kubernetes. The three main platforms are:

For this tutorial, you will use Minikube because it makes Kubernetes easier to test.

  1. To install Minikube, follow the instructions here: https://github.com/kubernetes/minikube.
  2. Once installed, start Minikube:

    $ minikube start
    Starting local Kubernetes cluster...
    Starting VM...
    SSH-ing files into VM...
    Setting up certs...
    Starting cluster components...
    Connecting to cluster...
    Setting up kubeconfig...
    Kubectl is now configured to use the cluster.
  3. Start the Minikube dashboard:

    minicubeDashboard.png

 

Deployment structure and concepts

You will deploy the following structure:

diagram.png

 

To simplify the idea of an ingress, picture it as a Load Balancer for your applications. By default, Pods and Services have IP addresses that are only routable inside your cluster. Ingress provides the routing feature between service and external client.

For more information, see: https://kubernetes.io/docs/concepts/services-networking/ingress/

 

Each pod gets an IP address inside the cluster, and by default they are dynamic, so if a pod fails and restarts it gets a new IP address. To ensure that your application will still be available, a service provides the kind of abstraction that you need to access pods in these conditions.

For more information, see: https://kubernetes.io/docs/concepts/services-networking/service/

 

A deployment will allow you to deploy several pod “replicas” that will run at any time. In this example, it will provide the ability to deploy three replicas.

For more information, see: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/

 

A pod is a group of one or more containers. To simplify the idea, a pod is our application that can be made of one or more containers. In our case, the pod will contain only one container (our customer service).

For more information, see: https://kubernetes.io/docs/concepts/workloads/pods/pod/

 

Creating a Secret

To deploy your application, you need to retrieve the Docker image from your private repository. To enable Kubernetes to deploy your application, you need to give it access to your private repository on Docker Hub.

 

For this, you need to create a secret that will contain your authentication.

For more information, see: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

 

  1. Create a secret for a Docker registry by running the following command:

    $ kubectl create secret docker-registry registry-secret --docker-username=user --docker-password=XXXX@@@@ --docker-email=user@company.com
  2. Display the result using the kubectl command, or by checking the Minikube dashboard:

    $ kubectl get secret registry-secret --output=yaml
    apiVersion: v1
    data:
      .dockercfg: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@==
    kind: Secret
    metadata:
      creationTimestamp: 2017-06-20T11:07:37Z
      name: registry-secret
      namespace: default
      resourceVersion: "32223"
      selfLink: /api/v1/namespaces/default/secrets/registry-secret
      uid: ab0f99ba-55a8-11e7-a031-080027a84992
    type: kubernetes.io/dockercfg

    registrySecret.png

     

 

Creating a Deployment

The sections in the kubernetes/deployment.deployment.yaml file allow you to configure your deployment:

  • Replicas field: Defines that your applications will be deployed in three pods containing your application.
  • Containers section: Defines which Docker images and configuration will be used to run a container.
  • ImagePullSecrets section: Defines which secret will be used to pull an image from your private repository.
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: customer-srv-deployment
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: customer-service
    spec:
      containers:
      - name: customer-service
        image: user/customer-service
        ports:
        - containerPort: 8040
      imagePullSecrets:
        - name: registry-secret
  1. To create your deployment, run the following commands:

    $ cd kubernetes/deployment/
    $ kubectl create -f deployment.yaml
    deployment "customer-srv-deployment" created
  2. To list deployments in your cluster, run the following command:

    $ kubectl get deployment
    NAME                    DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
    customer-srv-deployment 3       3       3          0         16s

    As you can see, deployment has three replicas.

  3. To see the pods' status, run the following command:

    $ kubectl get pod
    NAME                                     READY   STATUS  RESTARTS AGE
    customer-srv-deployment-2126652496-m53h0 1/1     Running 0        19s
    customer-srv-deployment-2126652496-mgxbj 1/1     Running 0        19s
    customer-srv-deployment-2126652496-zqlb2 1/1     Running 0        19s

    As you can see, there are three pods running.

    deployments.png

     

 

Creating a Service

The content of kubernetes/service/service.yaml allows you to expose your application (deployment) through a service. By default, services use the ClusterIP option.

ClusterIP exposes the service on a cluster-internal IP. Choosing this value makes the service reachable only from within the cluster. This is the default ServiceType.

 

This is why this example will use an ingress later.

kind: Service
apiVersion: v1
metadata:
  name: customer-srv-service
spec:
  selector:
    app: customer-service
  ports:
    - name: http
      protocol: TCP
      port: 8040
      targetPort: 8040
  1. To create your service, run the following command:

    $ cd kubernetes/service/
    $ kubectl create -f service.yaml
    service "customer-srv-service" created
  2. To list the services in your cluster, run the following command:

    $ kubectl get services
    NAME                   CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
    customer-srv-service   10.0.0.175   <none>        8040/TCP   5s
    kubernetes             10.0.0.1     <none>        443/TCP    4d
    

 

Creating an Ingress

An ingress is similar to an entrypoint/load balancer for external requests. Before installing an ingress, you need to set up an ingress controller. An ingress controller is a daemon deployed as a pod that will be used by an ingress. In this case, you will use a NGINX Controller.

 

Helm installation and ingress controller set up

To make set up easier, you will use Helm. Helm is a package manager for Kubernetes, similar to YUM and APT on Linux systems, which "hides" the complexity of installing the NGINX Ingress Controller. Helm is composed of two parts:

  • Helm Client (helm)
  • Tiller Server (tiller): Running inside your Kubernetes cluster

For more information about Helm, see What is Helm?

 

Installing Helm

To install Helm, follow the instructions here: https://github.com/kubernetes/helm.

 

Setting up the Tiller Server
  1. Once Helm is installed, set up the server part (Tiller) by running the helm init command:

    $ helm init
    $HELM_HOME has been configured at /home/user/.helm.
    
    Tiller (the helm server side component) has been installed into your Kubernetes Cluster.
    Happy Helming!
  2. You can then search for your ingress controller:

    $ helm search ingress
    NAME                 VERSION DESCRIPTION                                       
    stable/nginx-ingress 0.3.2   An nginx Ingress controller that uses ConfigMap...
    stable/nginx-lego    0.2.1   Chart for nginx-ingress-controller and kube-lego  
    stable/traefik       1.2.1-a A Traefik based Kubernetes ingress controller w…

    For this example, you should select the stable/nginx-ingress.

  3. To install a package using Helm, run the following command:

    $ helm install stable/nginx-ingress
    NAME:   bailing-meerkat
    LAST DEPLOYED: Tue Jun 20 13:28:59 2017
    NAMESPACE: default
    STATUS: DEPLOYED
    
    RESOURCES:
    ==> v1/ConfigMap
    NAME                                      DATA  AGE
    bailing-meerkat-nginx-ingress-controller  1     0s
    
    ==> v1/Service
    NAME                                           CLUSTER-IP  EXTERNAL-IP  PORT(S)                     AGE
    bailing-meerkat-nginx-ingress-controller       10.0.0.140  <pending>    80:32004/TCP,443:30853/TCP  0s
    bailing-meerkat-nginx-ingress-default-backend  10.0.0.133  <none>       80/TCP                      0s
    
    
    ==> v1beta1/Deployment
    NAME                                       DESIRED CURRENT UP-TO-DATE AVAILABLE  AGE
    bailing-meerkat-nginx-ingress-controller       1       1       1          0      0s
    bailing-meerkat-nginx-ingress-default-backend  1       1       1          0      0s
    
    …
    

Now you need to install your ingress.

 

Installing an ingress

To install an ingress, use the file kubernetes/ingress/ingress.yaml. As you can see:

  • Annotations will state that you will use a nginx ingress class, which refers to your ingress controller.
  • In the rules section, you define:

    • a hostname: talenddemo.net
    • a root path for your service
    • the backend, defined by the service you want to call and the port
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
      annotations:
        kubernetes.io/ingress.class: nginx
      name: customer-srv-ingress
    spec:
      rules:
      - host: talenddemo.net
        http:
          paths:
          - path: /
            backend:
              serviceName: customer-srv-service
              servicePort: 8040
  1. To install the ingress, run this command:

    $ cd kubernetes/ingress/
    $ kubectl create -f ingress.yaml
    ingress "customer-srv-ingress" created
  2. To list your ingress, run this command:

    $ kubectl get ingress
    NAME                   HOSTS            ADDRESS          PORTS     AGE
    customer-srv-ingress   talenddemo.net   xxx.xxx.xx.xxx   80        1d
  3. Because you specified a hostname, your application will be bound to it, so you need to update your hosts file using the IP address of Minikube:

    $ vim /etc/hosts
    #Kubernetes
    xxx.xxx.xx.xxx talenddemo.net
  4. Test your application by calling the following URLs:

 

Appendix

Creating a package using a Helm Chart

In this tutorial, you installed the items below one by one:

  • Deployment (pods)
  • Service
  • Ingress

In this case, all these items are considered to be part of your application, and you can just use the deployment. However, instead of creating all these elements one by one, you can also create a “package” using Helm.

 

To do so, you will define a Helm chart. For more information about Helm charts, see Developing Charts. Here is the basic structure of a chart:

.
└── customer-srv
    ├── Chart.yaml
    ├── templates
    │   ├── deployment.yaml
    │   ├── ingress.yaml
    │   └── service.yaml
    └── values.yaml

 

customer-srv your package
Chart.yaml a YAML file containing information about the chart
templates a directory of templates that, when combined with values, will generate valid Kubernetes manifest files
values.yaml the default configuration values for this chart

 

In your application, these files will be configured similar to the following examples:

  • Chart.yaml: this is a description of your chart

    apiVersion: v1
    description: A Helm chart for Kubernetes
    name: customer-srv
    version: 0.1.0
    keywords:
     - talend
     - helm
     - microservice
     - demo
     - kubernetes
    home: https://github.com/user/kubernetes-demo.git
    maintainers:
     - name: "User"
       email: "user@company.com"
  • values.yaml: defined pairs of keys/values that will be used as variables in your template files

    replicaCount: 3
    labels:
      app: customer-service
    image:
      repository: user/customer-service
      tag: latest
    service:
      name: nginx
      externalPort: 8040
      internalPort: 8040
    ingress:
      host: talenddemo.net
      annotations:
        kubernetes.io/ingress.class: nginx
    registry:
      secret: registry-secret
  • templates: Helm chart templates are written in the Go template language. For example:

          {{ .Values.replicaCount }}: get the value from values.yaml and the field replicaCount

          {{ .Chart.Name }}: get value from Chart.yaml

     

    • deployment.yaml:

      apiVersion: apps/v1beta1
      kind: Deployment
      metadata:
        name: {{ .Chart.Name }}-deployment
      spec:
        replicas: {{ .Values.replicaCount }}
        template:
          metadata:
            labels:
              app: {{ .Values.labels.app }}
          spec:
            containers:
            - name: {{ .Values.labels.app }}
              image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
              ports:
              - containerPort: {{ .Values.service.internalPort }}
            imagePullSecrets:
              - name: {{ .Values.registry.secret }}
    • service.yaml

      kind: Service
      apiVersion: v1
      metadata:
        name: {{ .Chart.Name }}-service
      spec:
        selector:
          app: {{ .Values.labels.app }}
        ports:
          - name: {{ .Values.service.name }}
            protocol: TCP
            port: {{ .Values.service.externalPort }}
            targetPort: {{ .Values.service.internalPort }}
    • ingress.yaml

      apiVersion: extensions/v1beta1
      kind: Ingress
      metadata:
        annotations:
          {{- range $key, $value := .Values.ingress.annotations }}
            {{ $key }}: {{ $value | quote }}
          {{- end }}
        name: {{ .Chart.Name }}-ingress
      spec:
        rules:
        - host: {{ .Values.ingress.host }}
          http:
            paths:
            - path: /
              backend:
                serviceName: {{ .Chart.Name }}-service
                servicePort: {{ .Values.service.externalPort }}

 

Deploying a Helm chart

Prerequisites:

  • Secret must be created
  • Ingress controller must be set up
  1. To deploy a Helm chart, run the following command, being careful to confirm the IP address talenddemo.net:

    $ cd helm/
    $ helm install customer-srv/
    NAME:   filled-shark
    LAST DEPLOYED: Fri Jun 23 12:10:06 2017
    NAMESPACE: default
    STATUS: DEPLOYED
    
    RESOURCES:
    ==> v1beta1/Deployment
    NAME                     DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
    customer-srv-deployment  3        3        3           0          0s
    
    ==> v1beta1/Ingress
    NAME                  HOSTS           ADDRESS         PORTS  AGE
    customer-srv-ingress  talenddemo.net  xxx.xxx.xx.xxx  80     0s
    
    ==> v1/Service
    NAME                  CLUSTER-IP  EXTERNAL-IP  PORT(S)   AGE
    customer-srv-service  10.0.0.52   <none>       8040/TCP  0s
  2. Wait until your chart is fully deployed, then test your service by calling this URL: http://talenddemo.net/services/customers.
Version history
Revision #:
16 of 16
Last update:
‎09-29-2017 07:37 PM
Updated by:
 
Labels (1)
Contributors