Introduction to Talend deployment with Ansible

Why you should use an automation tool

 

Before learning more about Ansible, you need to understand what an automation engine/platform is. Imagine performing the same task, for example, deploying an application, many times. You can do it in two ways:

  • Manually, by doing the install by hand over and over, but this is subject to error (no one is perfect)
  • Automatically, by creating a script or using a tool

This is where automation tools are useful. The purpose of these applications is to manage your infrastructure programmatically, not server by server but as a stack based on your needs.

 

If you already know about automation engines and Infrastructure as Code, then you have probably heard about Puppet, Chef, SaltStack, and Ansible.

 

This is not an article about “why you should use Ansible”. It will not make a comparison or give any recommendations about which tool to use, there are several blogs out there to tell you that. The only recommendation is to read about them, and decide which one best fits your needs.

 

What is Ansible?

Ansible is an automation platform, owned by Red Hat and developed in Python (which is required for the installation). It is designed as a client-only architecture, as opposed to Chef and Puppet, which are designed as a client/server architecture. Ansible uses SSH to connect to remote servers.

 

Common use cases are:

 

Environment

In this example, you work on an installation with:

  • A Talend Administration Center (TAC) server
  • Three JobServer agents (one per server)

You learn how to deploy Talend 6.4.1 TAC and JobServers with Ansible, and how to deploy your infrastructure with Ansible Playbooks.

 

The architecture for this example is shown in the diagram below:

 

env.png

 

Ansible installation

 

Prerequisites

This example requires five servers:

  • Ansible server: acts as a source and Ansible client/server
  • Talend Administration Server (TAC): you will deploy Java on this server
  • Three JobServer servers: each server hosts one JobServer

This example also requires you to perform these tasks:

  • Ensure that every server can be recognized by its name and not its IP address
  • Modify the IP addresses based on your architecture
  • Perform the following commands on each server
#!/bin/bash
 
# Edit host file to resolve names
sudo su - -c "echo '192.168.199.2   ansible.example.com ansible' >> /etc/hosts"
sudo su - -c "echo '192.168.199.3   tac.example.com tac' >> /etc/hosts"
sudo su - -c "echo '192.168.199.4   jobserver1.example.com  jobserver1' >> /etc/hosts"
sudo su - -c "echo '192.168.199.5   jobserver2.example.com  jobserver2' >> /etc/hosts"
sudo su - -c "echo '192.168.199.6   jobserver3.example.com  jobserver3' >> /etc/hosts"

 

Remember that Ansible uses SSH to manage infrastructure; therefore, you must ensure that each server has a user with sudo rights. In this case, name the user ansible and use the password ansiblepwd.

#!/bin/bash
 
# Add ansible user with sudo rights (with no password)
sudo useradd ansible -m -s /bin/bash
sudo su - -c 'echo ansible:ansiblepwd | sudo chpasswd'
sudo su - -c 'echo "ansible ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers'
 
#Comment
sudo sed -i '/PasswordAuthentication no/s/^/#/' /etc/ssh/sshd_config
#Uncomment
sudo sed -i '/^#.*PasswordAuthentication yes/s/^#//' /etc/ssh/sshd_config
 
#Restart ssh service
sudo systemctl restart sshd

Note: If you are using Amazon EC2 instances, which do not allow password authentication, you must update the ssh_config file as shown above.

 

Installation

Installing Ansible is very simple. This example uses Red Hat 7 as a base server for Ansible.

 

Install Ansible on a server called ansible.example.com:

#!/bin/bash
 
# Centos/Red Hat
sudo yum install –y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install -y epel-release
sudo yum install -y ansible

For more information on installation, see the Ansible Installation Guide.

 

Verification

  1. Ensure that the correct version of Ansible is installed:

    # Check Ansible version
    ansible --version
    ansible 2.3.1.0
      config file = /etc/ansible/ansible.cfg
      configured module search path = Default w/o overrides
      python version = 2.7.5 (default, May 3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)]

    This example requires that you configure ssh passwordless. This is not the only way to use authentication with Ansible, but in this case it is the easiest.

  2. From the Ansible server, run these commands to generate an SSH key and copy it to the distant servers.

    #!/bin/bash
     
    # Add ssh key
    sudo su - ansible -c 'echo -e "\n" | ssh-keygen -t rsa -N ""'
    sudo su - ansible -c 'sshpass -p "ansiblepwd" ssh-copy-id -o StrictHostKeyChecking=no ansible@tac.example.com'
    sudo su - ansible -c 'sshpass -p "ansiblepwd" ssh-copy-id -o StrictHostKeyChecking=no ansible@jobserver1.example.com'
    sudo su - ansible -c 'sshpass -p "ansiblepwd" ssh-copy-id -o StrictHostKeyChecking=no ansible@jobserver2.example.com'
    sudo su - ansible -c 'sshpass -p "ansiblepwd" ssh-copy-id -o StrictHostKeyChecking=no ansible@jobserver3.example.com'
  3. Check all connections by running these commands:

    sudo su ansible
    ssh tac.example.com
    ssh jobserver1.example.com
    ssh jobserver2.example.com
    ssh jobserver3.example.com

 

Configuration

 

Inventory

An inventory is a list of servers grouped by type. Categorize your hosts by group, as follows:

  1. Edit the default inventory file:

    sudo vi /etc/ansible/hosts
  2. Add this section to the end of the file:

    [java]
    tac.example.com
    jobserver[1:3].example.com
    
    [tac]
    tac.example.com
    
    [jobserver]
    jobserver[1:3].example.com

     

    In this case, you have three groups:

    • Java: where the Java Development Kit (JDK) is deployed
    • TAC: where Talend Administration Center (TAC) is deployed
    • JobServer: where the JobServer application is deployed

    As you can see, all of the servers are in the Java group, this is because each Talend application needs Java to run.

     

    Note: Be careful, this is a quick example. Using an inventory like this is a shortcut, not a best practice. Talend recommends using roles, as you can assign roles to a group of servers.

  3. Now you can test if all of the servers are accessible:

    [ansible@ip-172-31-30-19 ec2-user]$ ansible all -m ping
    jobserver3.example.com | SUCCESS => {
        "changed": false,
        "ping": "pong"
    }
    jobserver2.example.com | SUCCESS => {
        "changed": false,
        "ping": "pong"
    }
    jobserver1.example.com | SUCCESS => {
        "changed": false,
        "ping": "pong"
    }
    tac.example.com | SUCCESS => {
        "changed": false,
        "ping": "pong"
    }

 

For more information on inventory, see Ansible Documentation, Working with Inventory.

 

Playbooks

Playbooks are Ansible's configuration, deployment, and orchestration language. Consider a playbook to be a recipe of what you want to do for your deployment.

 

For more information Playbooks, see Ansible Documentation, Working With Playbooks

 

Java

First, define the Java playbook. The sections are discussed below.

 

---
- name: install Java JDK 8
  hosts: java
  vars:
    archive_path: /home/ansible/sources
  become_method: sudo
  become_user: root
 
  tasks:
  - name: create archive directory
    file:
      path: "{{ archive_path }}/java"
      state: directory
      recurse: yes
      owner: ansible
      group: ansible
    become: yes
  - name: Push JDK file to target server
    copy:
      src: "{{ archive_path }}/java/jdk-8u144-linux-x64.rpm"
      dest: "{{ archive_path }}/java/jdk-8u144-linux-x64.rpm"
      owner: ansible
      group: ansible
      mode: 0644
  - name: install JDK rpm from a local file
    yum:
      name: "{{ archive_path }}/java/jdk-8u144-linux-x64.rpm"
      state: present
    become: yes

 

Describe and configure the playbook

The first part of the playbook is used to describe and configure the playbook.

---
- name: install Java JDK 8
  hosts: java
  vars:
    archive_path: /home/ansible/sources
  become_method: sudo
  become_user: root

 

  • name: playbook name
  • hosts: define the hosts from the inventory where you want to perform your tasks
  • vars: variables used to configure actions
  • become_method and become_user: define which user you want to be for some actions, and how to impersonate that user

 

Describe the tasks to perform

The second part of the playbook describes the tasks you want to perform.

tasks:
  - name: create archive directory
    file:
      path: "{{ archive_path }}/java"
      state: directory
      recurse: yes
      owner: ansible
      group: ansible
    become: yes
  - name: Push JDK file to target server
    copy:
      src: "{{ archive_path }}/java/jdk-8u144-linux-x64.rpm"
      dest: "{{ archive_path }}/java/jdk-8u144-linux-x64.rpm"
      owner: ansible
      group: ansible
      mode: 0644
  - name: install JDK rpm from a local file
    yum:
      name: "{{ archive_path }}/java/jdk-8u144-linux-x64.rpm"
      state: present
    become: yes

 

Each task is defined by a name and a module (file, copy, yum). A module represents a program or an action to perform, with its parameters.

 

For the list of all modules, see Ansible documentation, All modules.

 

Notice that some tasks contain become: yes, which is linked to become_user and become_method you saw in the first part of the playbook. It means that you want to perform this task as root.

 

JobServers

To install all JobServers at once, use this playbook in which you define a list of steps to install the JobServers:

  1. Creating the installation directory on the remote servers.
  2. Creating an archive directory to store files pushed from the master/server.
  3. Pushing files from the Ansible server to the JobServer servers.
  4. Unarchiving the jobserver.tar.gz (custom distribution) file.
  5. Configuring and pushing the service file to the targets.
  6. Changing the configuration file of the JobServer.
---
- name: install and start Talend Jobserver
  hosts: jobserver
  vars:
    command_server_port: 8000
    file_server_port: 8001
    monitoring_port: 8888
    root_path: ./TalendJobServersFiles
    install_path: /opt/talend/6.4.1
    archive_path: /home/ansible/sources
    jobserver_file: jobserver.tar.gz
    run_as_user: ansible
  become_method: sudo
  become_user: root
 
  tasks:
 
  - stat: path="{{ install_path }}/jobserver"
    register: jobserver_folder
 
  - name: create installation directory
    file:
      path: "{{ install_path }}"
      state: directory
      recurse: yes
      owner: ansible
      group: ansible
    become: yes
    when: not jobserver_folder.stat.exists
 
  - name: create archive directory
    file:
      path: "{{ archive_path }}/jobserver"
      state: directory
      recurse: yes
      owner: ansible
      group: ansible
    become: yes
 
  - name: copy jobserver files
    copy:
      src: "{{ archive_path }}/jobserver/{{ jobserver_file }}"
      dest: "{{ archive_path }}/jobserver/{{ jobserver_file }}"
      mode: 0644
    become: yes
 
  - name: unarchive jobserver file
    unarchive:
      src: "{{ archive_path }}/jobserver/{{ jobserver_file }}"
      dest: "{{ install_path }}"
      remote_src: yes
      owner: ansible
      group: ansible
    become: yes
    when: not jobserver_folder.stat.exists
 
  - name: install jobserver systemd unit file
    template:
      src: "{{ archive_path }}/jobserver/service/talend-jobserver.service"
      dest: /etc/systemd/system/talend-jobserver.service
      mode: 0644
    become: yes
 
  - name: copy configuration files to jobserver
    template:
      src: "{{ archive_path }}/jobserver/configuration/TalendJobServer.properties"
      dest: "{{ install_path }}/jobserver/conf/"
      mode: 0644
    become: yes
    notify: "restart talend-jobserver"
 
  handlers:
    - name: restart talend-jobserver
      service: name=talend-jobserver state=restarted
      listen: "restart talend-jobserver"
      become: yes

 

The playbook for JobServer includes two new elements:

  • State validation: checks if the JobServer installation folder already exists.

    - stat: path="{{ install_path }}/jobserver"
        register: jobserver_folder

    This condition is used later on tasks with the keyword when.

    when: not jobserver_folder.stat.exists

     

  • Handlers: can be triggered by tasks that detect a change.

    In this case, you restart the talend-jobserver service.

    handlers:
        - name: restart talend-jobserver
          service: name=talend-jobserver state=restarted
          listen: "restart talend-jobserver"
          become: yes

    To call this handler, add one task:

    notify: "restart talend-jobserver"

    It will notify the handler to execute.

     

Another interesting thing is the use of the module template. The template module uses Jinja2, which is a Python template engine to configure files, like your service.

[Unit]
Description=Talend JobServer Service
After=network.target
[Service]
WorkingDirectory={{ install_path }}/jobserver
ExecStart=/bin/bash start_rs.sh
ExecStop=/bin/bash stop_rs.sh
Type=simple
User={{ run_as_user }}
[Install]
WantedBy=default.target

In this example, {{ install_path }} will be replaced by your variable from the playbook.

 

Talend Administration Center

The playbook used to install Talend Administration Center is specific and includes the following steps:

  1. Creating an installation directory on the remote servers.
  2. Creating an archive directory to store files pushed from the master server.
  3. Copying the installation files.
  4. Copying the option file for the unattended installation.
  5. Running the installation with the shell module.
  6. Copying the configuration file.

 

In this case, you use the Talend Installer with a dist file. Normally you would do a manual installation, which will give you flexibility in your installation, and also will be lighter than pushing the whole installer to the remote server. But as an example, it is interesting to see the unattended installation.

 

---
- name: install and start Talend Administration Center
  hosts: tac
  vars:
    tac_port: 8080
    tac_user: security@admin.com
    tac_passwd: admin
    install_path: /opt/talend/6.4.1
    archive_path: /home/ansible/sources
    license_path: /home/ansible/sources/tac/license_BDRT
  become_method: sudo
  become_user: root
 
  tasks:
  - stat: path="{{ install_path }}/tac"
    register: tac_folder
 
  - name: create installation directory
    file:
      path: "{{install_path}}"
      state: directory
      recurse: yes
      owner: ansible
      group: ansible
    become: yes
 
  - name: create archive directory
    file:
      path: "{{ archive_path }}/tac"
      state: directory
      recurse: yes
      owner: ansible
      group: ansible
    become: yes
 
  - name: copy tac dist
    copy:
      src: "{{ archive_path }}/tac/{{ item }}"
      dest: "{{ archive_path }}/tac/{{ item }}"
      mode: 0744
    become: yes
    with_items:
     - dist
     - Talend-Installer-20170623_1246-V6.4.1-linux64-installer.run
     - license_BDRT
 
  - name: Create unattended optionfile
    template:
      src: "{{ archive_path }}/tac/configuration/tac_setup.txt"
      dest: "{{install_path}}"
      mode: 0644
    become: yes
 
  - stat: path="{{ install_path }}/tac"
    register: tac_folder
 
  - name: Run TAC Installation
    shell: "{{ archive_path }}/tac/Talend-Installer-20170623_1246-V6.4.1-linux64-installer.run --optionfile {{install_path}}/tac_setup.txt"
    become: yes
    when: not tac_folder.stat.exists
 
  - name: copy configuration files to tac
    template:
      src: "{{ archive_path }}/tac/configuration/configuration.properties"
      dest: "{{ install_path }}/tac/apache-tomcat/webapps/org.talend.administrator/WEB-INF/classes/"
      mode: 0644
    become: yes
    notify: "restart talend-tac"
 
  handlers:
    - name: restart talend-tac
      service: name=talend-tac-6.4.1 state=restarted
      listen: "restart talend--tac"
      become: yes

As you can see, you do not have to create a service, because the installation did it for you.

 

Execution

To run a playbook, use the following command:

ansible-playbook my-playbook.yml

 

To run a pre-check, use the following command:

ansible-playbook my-playbook.yml –check

 

This is an example of the execution for the Java playbook.

$ ansible-playbook playbooks/java/java-playbook.yml
 
PLAY [install Java JDK 8] ***********************************************************************************
 
TASK [Gathering Facts] ***********************************************************************************
ok: [tac.example.com]
ok: [jobserver2.example.com]
ok: [jobserver3.example.com]
ok: [jobserver1.example.com]
 
TASK [create archive directory] ***********************************************************************************
changed: [jobserver1.example.com]
changed: [tac.example.com]
changed: [jobserver2.example.com]
changed: [jobserver3.example.com]
 
TASK [Push JDK file to target server] ***********************************************************************************
changed: [jobserver3.example.com]
changed: [jobserver2.example.com]
changed: [jobserver1.example.com]
changed: [tac.example.com]
 
TASK [install JDK rpm from a local file] ***********************************************************************************
changed: [jobserver1.example.com]
changed: [jobserver2.example.com]
changed: [tac.example.com]
changed: [jobserver3.example.com]
 
PLAY RECAP ***********************************************************************************
changed=3    unreachable=0    failed=0
jobserver2.example.com     : ok=4    changed=3    unreachable=0    failed=0
jobserver3.example.com     : ok=4    changed=3    unreachable=0    failed=0
tac.example.com            : ok=4    changed=3    unreachable=0    failed=0
 

 

If you run it again without making any changes, it will skip these steps.

Version history
Revision #:
13 of 13
Last update:
‎09-29-2018 12:17 AM
Updated by:
 
Labels (2)