AWS - Using OpsWorks to manage Talend ESB configuration files

Overview

AWS OpsWorks for Chef Automate provides a fully-managed Chef server. Chef is a very popular tool in the DevOps area that lets you manage your configuration and automate your infrastructure set up and deployment.

 

As you may know, Talend ESB supports dynamic configuration. This means that any change in its configuration files is detected and reloaded dynamically by the Talend ESB Runtime. When deploying a fleet of Talend ESB Runtime instances on Amazon Web Services, it can be interesting to use Chef to have a centralized and flexible management of Talend ESB Runtime configuration files.

 

This article is a step-by-step guide on how to use AWS OpsWorks for Chef Automate to seamlessly manage Talend ESB Runtime configuration files on AWS. After following this tutorial, you will gain better insight on:

  • How to install and use Chef and AWS OpsWorks
  • How to create a basic cookbook for managing Talend ESB configuration files
  • How to deploy an EC2 instance with Talend ESB and the chef-client agent
  • The benefits of using these technologies

 

Architecture

The figure below comes from the Chef official website. It describes the main modules of Chef and the interactions between them.

chef_arch.png

 

In this tutorial, you will:

The diagram below shows the target architecture.

target_arch.png

 

  1. The administrator uses the Chef Development Kit to write a Chef cookbook to manage Talend ESB Runtime configuration files.
  2. The administrator uses Knife to upload the Chef cookbook to AWS Opworks.
  3. The polling Chef client retrieves the latest cookbook updates from the AWS OpsWorks Chef server.
  4. The Chef client automatically updates the configuration Talend ESB instance using the latest cookbook information.

 

Environment

Talend ESB 6.3.1

Talend Data Services Platform 6.3.1 and later

Chef client 12.16.42

 

Assumptions

  1. Amazon Web Services (AWS):

    • You should be familiar with the AWS platform, since this article will not provide a deep dive into details regarding administration and management of AWS services. Read the Amazon Web Services (AWS) - Getting Started on the Talend Help Center for information on all the AWS functionalities that Talend provides.
    • You should have full administrator access to the main AWS services described in the Prerequisites section below.
  2. Talend:

  3. Chef:

    • No need to be an expert in Chef. You just need to be a bit familiar with Chef concepts.
    • Start by reading this page first: Learn the Chef Basics.

 

Prerequisites

  1. A valid AWS account with full administrator access to following services:

  2. Talend ESB Runtime (Commercial Edition)—https://www.talend.com/products/application-integration/

 

Create an OpsWorks Chef server

To work through the steps in this tutorial, you must connect to the AWS Console and choose a region.

Note: Considerations when choosing an AWS region include latency, compliance requirements, and availability of a specific service. This example will use the Ireland region.

  1. Open the OpsWorks console.

  2. Click Create Chef Automate Server.

    create_chef.png

     

  3. On the Step 1: Set the name, region, and type screen, use the following configuration for your Chef server:

    • Chef Automate server name = Talend-Chef-Automate
    • Chef Automate server region = EU (Ireland)
    • EC2 Instance type = t2.medium

      create_chef_step1.png

     

  4. Click Next.
  5. On the Step 2: Select an SSH Key screen, select Use an existing EC2 key pair, then select your existing EC2 key pair or generate a new one.

    create_chef_step2.png

     

  6. Click Next.
  7. On the Step 3: Configure advanced settings screen, keep it simple with the values below:

    • VPC: choose your default VPC
    • Subnet: choose a default subnet
    • Associate Public IP Address: Yes
    • Security group = Generate a new one
    • Service role: leave with default value if any or generate a new one
    • Instance profile: leave with default value if any, or generate a new one
    • System Maintenance: leave all fields with default value
    • Enable Automated backup: No

      create_chef_step3.png

       

  8. Click Next.
  9. Review the configuration.

    create_chef_review.png

     

  10. Click Launch to create the Chef Automate server.

    The creation process will take about 20 minutes to create an Elastic IP, launch an EC2 instance, then install the Chef Automate server.

    create_chef_is_creating.png

  11. While waiting for the Chef Automate server to be created, download the sign-in credentials and the starter kit to your local laptop.
    1. Create a directory called opsworks on your laptop. This folder will be the base directory of the Chef DK installation. The path used in this tutorial is C:\talend\opsworks.
    2. Click Download Credentials, then save the file in your opsworks directory.
    3. Click Download Starter Kit then save the file in the same directory.
  12. Once the Chef Automate server has been successfully created, connect to the interface:

    1. In the OpsWorks console, click Open Chef Automate dashboard.

      This will open a new tab. You can ignore an SSL warning if one appears: it is due to the self-signed certificate being used by Chef server.

    2. Log in to the Chef Automate server using the credentials provided in the credentials file you downloaded.

      chef_signin.png

       

    3. The Chef Automate dashboard appears when you successfully log in:

      chef_dashboard.png

       

    Congratulations! You have successfully created your Chef server using AWS OpsWorks. The next step is to install the Chef Development Kit on your local laptop.

 

Install the Chef Development Kit on your laptop

The Chef Development Kit is easy to install. First, download Chef Installer, then execute the installer wizard.

 

The Chef Development Kit is available for several operating systems, but this tutorial was built using the Chef Development Kit on Windows 10. If you are using Windows 10, refer to the steps below that guide you through the installation and configuration of Chef DK. If you are using another operating system, see Install the Chef DK.

  1. Download the Windows 10 installer from the Chef Downloads page (https://downloads.chef.io/chefdk).

    win10inst.png

     

  2. Double-click the installer to launch the wizard. From the initial Setup Wizard screen, click Next.

    chef_inst.png

     

  3. Read and accept the license terms. Click Next.

    chef_eula.png

     

  4. Click Next to accept the default install settings (this tutorial was written using the defaults), or you customize the settings to your needs. Click Next.

    chef_custom.png

     

  5. Click Install to start the installation.

    chef_readyinst.png

     

  6. Click Finish to exit the Setup Wizard.

    chef_finished.png

     

  7. By default, the Chef DK is installed in C:\opscode\chefdk. Open Windows Explorer to check that folder, or look in the correct folder if you changed the install location:

    chef_installDir.png

     

Congratulations! You have successfully installed the Chef DK on your laptop. The next step is to configure it.

 

Configure the Chef DK

Configuration of the Chef DK is straightforward. If you are using Windows 10, as in this article, follow the steps below to configure the Chef DK and set up the Chef repository.

  1. Launch Windows PowerShell as Administrator.

  2. Execute the following command:

    chef shell-init powershell | Invoke-Expression

    powershell.png

     

    Note: This tutorial does not show installing GIT as the source version control for the Chef DK. However, Talend highly recommends that you use a version control system such as GIT to store and version the Chef recipes that you develop.

  3. Open the folder where you downloaded the Starter Kit (in this tutorial, C:\talend\opsworks), then unzip the starter_kit.zip file into the same directory:

    opsworks.png

     

  4. Open the uncompressed starter_kit folder. You should see a folder having the same name as your OpsWorks Chef Server with an appended suffix, for example talend-chef-automate-x8tbhjq2xiulhqur (the suffix may be different for your installation).

    starterKit.png

     

  5. Open the folder, which will be your Chef repo directory. It should contain several folders and files related to Chef:

    talend-chef.png

     

  6. To verify that the Chef repo is configured properly, execute the following command from your Windows PowerShell (adapt it as necessary to match your own Chef repo folder):

    cd C:\talend\opsworks\starter_kit\talend-chef-automate-x8tbhjq2xiulhqur

    powershell_cd.png

     

  7. Use the knife command below to check that your Chef DK can connect to your remote OpsWorks Chef server based on your Chef repo configuration:

    knife client list

    If successful, it should display the following message:

    default-validator

Congratulations! You have successfully configured the Chef DK on your laptop. The next step is to create a cookbook for Talend ESB.

 

Create a cookbook for Talend ESB

This section shows how to use knife to create a cookbook for Talend ESB.

 

This cookbook will contain configuration files for Talend runtimes, and will be used by the Chef agent on each EC2 instance to manage Talend ESB configuration files as described in the Architecture.

 

For simplicity, this tutorial is limited to the management of a single configuration file called org.ops4j.pax.web.cfg. This file defines the web configuration of Talend ESB, and thus contains configuration information concerning HTTP and SSL transport, keystore, and so on.

  1. Use your previous Windows PowerShell session, or launch a new session as Administrator.

  2. Create a new cookbook using the following knife command:

    knife cookbook create talend-esb

    powershell_esb.png

     

  3. Using Windows Explorer, locate the cookbooks folder under the Chef repo directory. In this tutorial, the default is C:\talend\opsworks\starter_kit\talend-chef-automate-x8tbhjq2xiulhqur\cookbooks.

  4. In the cookbooks folder, check that a folder called talend-esb was created (as a result of the knife command).

    talend-esb.png

     

  5. Check the talend-esb folder. It should contain several folders and files.

    talend-esb2.png

     

  6. For simplicity, the original org.ops4j.pax.web.cfg configuration file will be managed as a Chef template resource that will be used by the recipe. If you are not familiar with Chef template resources, see the documentation at https://docs.chef.io/resource_template.html.

  7. Open the talend-esb\templates\default directory, then create a file called org.ops4j.pax.web.cfg.erb (note the .erb extension).
  8. Use a text editor to update the file with the content below. The HTTP port is being set at 9045 instead of the default 8040.

    ###
    # #%L
    # TESB :: Assembly
    # %%
    # Copyright (C) 2011 - 2012 Talend Inc.
    # %%
    # Licensed under the Apache License, Version 2.0 (the "License");
    # you may not use this file except in compliance with the License.
    # You may obtain a copy of the License at
    #
    #      http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.
    # #L%
    ###
    #
    # Default port for the OSGI HTTP Service
    #
    org.osgi.service.http.port=9045
    org.osgi.service.http.port.secure=9001
    org.osgi.service.http.secure.enabled=true
    org.ops4j.pax.web.ssl.keystore=${karaf.base}/etc/keystores/keystore.jks
    org.ops4j.pax.web.ssl.password=password
    org.ops4j.pax.web.ssl.keypassword=password
    #org.ops4j.pax.web.config.file=${karaf.base}/etc/jetty.xml
  9. To create the recipe that will use the template, open the directory ...\cookbooks\talend-esb\recipes.
  10. Open the file default.rb with your favorite editor, then add the following lines to the file:

    template '/home/ec2-user/talend/Talend-ESB-V6.3.1/container/etc/org.ops4j.pax.web.cfg' do
        content 'org.ops4j.pax.web.cfg.erb'
    end

    default_rb.png

     

  11. Save the file.

Congratulations! You have just created a recipe telling Chef to manage the ESB configuration file org.ops4j.pax.web.cfg by using the template file org.ops4j.pax.web.cfg.erb.

 

Upload the cookbook

Now that you have created the cookbook, upload it to the OpsWorks Chef server using knife.

  1. In the root directory of your Chef repo, execute the following command:

    knife upload .

    knife_upload.png

     

  2. To verify that the cookbook has been successfully uploaded, execute the following command in your Windows PowerShell session:

    knife cookbook list

    The resulting output should contain talend-esb.

    knife_list.png

     

    Note: Whenever you make any changes to the cookbook in your Chef repo, you must upload it again to commit the changes to your OpsWorks Chef server.

Congratulations! The cookbook talend-esb is now uploaded to your OpsWorks Chef server. You can now proceed to install Talend ESB on EC2.

Create an IAM role to use as your instance profile

  1. To create an IAM role that will be used by EC2 instances to interact with the OpsWorks Chef server, go to the IAM Console, then click Policies.

  2. Create a new policy with these parameters:

    • Policy name: Talend_EC2_opsworks_policy

    • Policy document:

      { "Version": "2012-10-17",
      "Statement": [ { "Action": [ "opsworks-cm:AssociateNode", "opsworks-cm:DescribeNodeAssociationStatus" ],
      "Effect": "Allow",
      "Resource": [ "*" ] } ] }
  3. Click Validate Policy.
  4. Click Create Policy.
  5. Go to the IAM Console, then click Roles.
  6. Create a new role: role name = Talend_EC2_opsworks_role
  7. Select AWS Service Roles > Amazon EC2
  8. Attach policy: select the filter Customer Managed, then choose the policy Talend_EC2_opsworks_policy
  9. Click Next Step.
  10. Review.
  11. Click Create Role.

Congratulations! You have created the role with OpsWorks permissions that will be attached to the EC2 instance. Now launch an EC2 instance and install Talend ESB onto it.

 

Launch an EC2 instance

Launch a new EC2 instance with the following information:

  1. Choose AMI: select an Amazon Linux AMI.
  2. Choose the Instance Type: t2.medium is a good fit.
  3. Configure Instance: Number of Instances = 1.
  4. Network: choose your default VPC.
  5. Subnet: choose your default subnet. For simplicity, this tutorial is limited to one instance in one subnet.
  6. Auto-assign Public IP: Select Enable. You will use the public IP to SSH into the instance for further configuration.
  7. IAM Role: select Talend_EC2_opsworks_role. Leave all other fields as default.
  8. Add Storage = 32 GB.
  9. Add Tags: add a name tag with value Talend ESB, and any other tags that are relevant for you.
  10. Configure Security Group:

    1. Select Create a new security group.
    2. Security group name: Talend - ESB Security Group.
    3. Rules: Add the rules as shown in the following screenshot:

      rules.png

       

      • The SSH rule on port 22 allows SSH sessions on the instance for management purposes. AWS best practices recommend that you only open this port to your IP or corporate directory range of IPs.
      • The Custom TCP rules on ports 9045 and 8040 allow connections from any host. AWS best practice is to use a load-balancer to manage traffic coming from the internet, and thus allow only the load-balancer security group on these ports.
    4. Click Review and Launch: review your instance configuration, then click Launch.
    5. Select your existing key pair, or create a new one. For more information about AWS Key pairs, refer to AWS documentation at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html.
    6. Click Launch Instances.

Once the EC2 instance has started, you can install Talend ESB.

 

Install Talend ESB

The procedure to properly install Talend ESB on a Linux machine is fully described in the Talend ESB 6.3.1 - Installation Guide for Linux (EN), available from the Talend Help Center.

 

For the cookbook to work correctly, install Talend ESB in the folder /home/ec2-user/talend/Talend-ESB-V6.3.1, or adapt the recipe to match your installation path.

linux-esb.png

 

After you have successfully installed and started Talend ESB, use your favorite web browser to check the access to the Talend ESB Services page on port 8040. It should display either a list of running services, or most probably a message saying "No services have been found" since this is a newly-installed Talend ESB runtime. In both cases, this proves Talend ESB is fully operational and is listening on default HTTP port 8040.

 

Install and configure the chef-client on the EC2 instance

At this point, Talend ESB has been installed on the EC2 instance and has been configured to automatically start when the server boots. Now you can install the chef-client on the Talend ESB instance. You will configure the chef-client to periodically poll the Chef server for new changes in the cookbook to apply to Talend ESB configuration files.

 

Create Instances by using an Unattended Association Script: the steps below are adapted from the AWS Documentation at http://docs.aws.amazon.com/en_en/opsworks/latest/userguide/opscm-unattend-assoc.html. The version below and in the attached Zip file have been adapted to this tutorial.

  1. Download the attached Zip file and uncompress the script install_chef_client.sh, or copy the text below and create the file locally.
  2. Open the script with your favorite text editor.
  3. Update the following parameters with your own values:

    1. NODE_NAME: Update the IP address as appropriate.
    2. REGION: Use the correct region code. Stay with the same region as before; for this tutorial, it's eu-west-1 for Ireland.
    3. CHEF_SERVER_NAME
    4. CHEF_SERVER_ENDPOINT
    #!/bin/bash
    # Required settings
    NODE_NAME="$(curl --silent --show-error --retry 3 http://169.254.169.254/latest/meta-data/instance-id)" # This uses the EC2 instance ID as the node name
    REGION="eu-west-1" # Valid values are us-east-1, us-west-2, or eu-west-1
    CHEF_SERVER_NAME="Talend-Chef-Automate" # The name of your Chef server
    CHEF_SERVER_ENDPOINT="talend-chef-automate-lyjdv5vs5etq2fbh.eu-west-1.opsworks-cm.io" # Provide the FQDN or endpoint; it's the string after 'https://'
    # Optional settings
    CHEF_ORGANIZATION="default"    # Leave as "default"; do not change. AWS OpsWorks for Chef Automate always creates the organization "default"
    NODE_ENVIRONMENT=""            # e.g. development, staging, onebox ...
    CHEF_CLIENT_VERSION="12.16.42" # latest if empty
    # Recommended: upload the chef-client cookbook from the chef supermarket  https://supermarket.chef.io/cookbooks/chef-client
    # Use this to apply sensible default settings for your chef-client configuration like logrotate, and running as a service.
    # You can add more cookbooks in the run list, based on your needs
    RUN_LIST="talend-esb" # e.g. "recipe[chef-client],recipe[apache2]"
    # ---------------------------
    set -e -o pipefail
    AWS_CLI_TMP_FOLDER=$(mktemp --directory "/tmp/awscli_XXXX")
    CHEF_CA_PATH="/etc/chef/opsworks-cm-ca-2016-root.pem"
    install_aws_cli() {
      # see: http://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-bundle-other-os
      cd "$AWS_CLI_TMP_FOLDER"
      curl --retry 3 -L -o "awscli-bundle.zip" "https://s3.amazonaws.com/opsworks-cm-us-east-1-prod-default-assets/misc/awscli-bundle-opsworks-cm.zip"
      unzip "awscli-bundle.zip"
      ./awscli-bundle/install -i "$PWD"
    }
    aws_cli() {
      "${AWS_CLI_TMP_FOLDER}/bin/aws" opsworks-cm --region "${REGION}" --output text "$@" --server-name "${CHEF_SERVER_NAME}"
    }
    associate_node() {
      client_key="/etc/chef/client.pem"
      mkdir /etc/chef
      ( umask 077; openssl genrsa -out "${client_key}" 2048 )
      aws_cli associate-node \
        --node-name "${NODE_NAME}" \
        --engine-attributes \
        "Name=CHEF_ORGANIZATION,Value=${CHEF_ORGANIZATION}" \
        "Name=CHEF_NODE_PUBLIC_KEY,Value='$(openssl rsa -in "${client_key}" -pubout)'"
    }
    write_chef_config() {
      (
        echo "chef_server_url   'https://${CHEF_SERVER_ENDPOINT}/organizations/${CHEF_ORGANIZATION}'"
        echo "node_name         '${NODE_NAME}'"
        echo "ssl_ca_file       '${CHEF_CA_PATH}'"
      ) >> /etc/chef/client.rb
    }
    install_chef_client() {
      # see: https://docs.chef.io/install_omnibus.html
      curl --silent --show-error --retry 3 --location https://omnitruck.chef.io/install.sh | bash -s -- -v "${CHEF_CLIENT_VERSION}"
    }
    install_trusted_certs() {
      curl --silent --show-error --retry 3 --location --output "${CHEF_CA_PATH}" \
        "https://s3.amazonaws.com/opsworks-cm-us-east-1-prod-default-assets/misc/opsworks-cm-ca-2016-root.pem"
    }
    wait_node_associated() {
      aws_cli wait node-associated --node-association-status-token "$1"
    }
    install_aws_cli
    node_association_status_token="$(associate_node)"
    install_chef_client
    write_chef_config
    install_trusted_certs
    wait_node_associated "${node_association_status_token}"
    #if [ -z "${NODE_ENVIRONMENT}" ]; then
     # chef-client -r "${RUN_LIST}"
    #else
     # chef-client -r "${RUN_LIST}" -e "${NODE_ENVIRONMENT}"
    #fi
    ###For Demos: runs the cookbook "talend-esb" every 30 seconds
    #chef-client -i 30 -r "talend-esb"
  4. Save the file.

Read the script carefully to understand the steps it performs:

  1. Associate the current EC2 node with the Chef server.
  2. Install the Chef client.
  3. Write the Chef configuration.
  4. Install the certificates.
  5. Optionally: Uncomment the last line to periodically run the chef-client to poll changes from the Chef server.

For all these procedures to be performed, you need to perform the following two steps:

  1. Upload the script into your EC2 instance using an FTP client such as Filezilla.

    upload_script.png

     

  2. Execute the script with the following command:

    sudo sh install_chef_client.sh

    install_script.png

     

The script will take some time to install the chef-client. After it is successfully installed, you can start the chef-client.

Note: Best practice is to automatically install and execute the chef-client when the host machine starts up. This can be achieved by using the AWS cloud-init capacity for example, described in the AWS documentation here: http://docs.aws.amazon.com/en_en/AWSEC2/latest/UserGuide/user-data.html. To keep things simple, this tutorial shows how how to start it manually.

 

Run the chef-client

In the previous section, you successfully installed the chef-client. Now start it by following these steps.

  1. On your EC2 instance, run the chef-client with this command:

    sudo chef-client -i 30 -r "talend-esb"

    After an initial delay of 30 seconds, this command makes the chef-client poll the Chef server for cookbook updates every 30 seconds.

    chef_poll.png

    Note: This command will be executed in the foreground until you terminate it with a Ctrl-C command. Best practice for running the chef-client is either using crontab to schedule the chef-client execution, or configuring the chef-client as a daemon. For the daemon mode, see Chef documentation at https://docs.chef.io/ctl_chef_client.html.

  2. When you installed the Talend ESB runtime, the default HTTP port of Talend ESB was 8040. After the first execution, the chef-client should have fetched from the Chef Server and applied the configuration with port 9045. This means that the Talend ESB runtime should have been updated, and it should be now listening on port 9045 (thanks to hot dynamic configuration).

  3. To test this, first try the URL with port 8040:

    inaccessible.png

     

    Talend ESB is no longer accessible on port 8040.

  4. Now try on port 9045 as specified in the Chef recipe.

    no_svcs.png

    Talend ESB is now listening on port 9045, as updated in the cookbook.

 

Test the update process

You launched the chef-client in the previous section; as long as it is running, it will poll the Chef server for changes every 30 seconds.

 

To test if changes are automatically applied to the Talend ESB configuration by Chef, update the talend-esb cookbook, upload it, then check the updates.

  1. Open the template file org.ops4j.pax.web.cfg.erb on your local laptop.

  2. Update the HTTP port from 9045 back to 8040 as shown below:

    change_port.png

     

  3. Save and close the file.
  4. Use PowerShell to upload the changes:

    1. Change to the root Chef repo, in this example C:\talend\opsworks\starter_kit\talend-chef-automate-x8tbhjq2xiulhqur.

      powershell_root.png

       

    2. Execute this command to upload changes to the OpsWorks Chef server:

      knife upload .

      knife_upload2.png

       

  5. Since the chef-client on Talend ESB Server side polls the Chef server every 30 seconds for updates, wait 30 seconds then try to access the Talend ESB Services page again, first using the old port 9045 then with port 8040. The page should now be accessible on port 8040.

Congratulations! This proves that your architecture with Chef is fully set up and running, and that concludes this tutorial.

Version history
Revision #:
22 of 22
Last update:
‎09-16-2017 09:09 PM
Updated by:
 
Labels (3)