Agile DevOps - People and Process then Automation!!!

Use Cloud and DevOps to build infrastructure by rapid agile iterative development using collaborative open source tools:
-Terraform or Cloud Formation for easy orchestration.
-chef cookbooks, puppet modules or ansible playbooks to build servers.
-Test Kitchen to deploy and test them.
-Packer for build server images or containers.
-EC2Dream a graphic user interface that provides a 'single pane of glass' to do agile devops primarily on cloud servers.
-Amazon AWS, Azure, Google Compute Engine, IBM, Openstack, Local and Hosted servers.

Networking Multi-Tier Applications

Consider a multi-tier application that consists of
  1. Web First Tier e.g. Apache
  2. Clustered Middle Tier e.g. JBoss cluster
  3. Data Tier e.g. MySQL, Postgres SQL etc

Amazon uses dynamic IP addresses so running multi-tier applcations has some challenges.
Amazon instances have 2 IP address:
  1. A Public Address accessible from the internet.
  2. A Private Address accessible within Amazon that is used when instances communicate with each other.
An instance can have an Elastic IP Address so the address is static however there is a limit of 5 IP addresses available. These can be increase via this form

  Typically the web first tier has a static IP address and the DNS provider maps this static IP address to the domain name of the application.
  To handle the dynamic IP Addresses it is best to use hostnames and build a local hosts file from EC2 meta-data. Amazon hostnames are IP-<private IP Address> in the Rightscale Centos Images.
  The following ruby script updates a local hosts file from EC2 meta-data.

# generate hosts files in the following order
# 1. running instances  tag called name set to name of instance
# 2. running instance security group name
# NOTE:  This script can be run via cron to do regular updating of hosts file
def generate_hosts_file(ec2, server_tag)
   if server_tag == nil
      server_tag = "name"
   hosts_file ="/etc/hosts")"/etc/hosts.bak", "w") {|f| f.write hosts_file }
   i = 0
   host_file_updated = false
   host_lines  =
   hosts_file.each do |line|
      host_lines[i] = line
   server_name = ""
   ec2.describe_instances.each do |r|
      if r[:aws_state] == "running"
         server_name = r[:groups][0][:group_name]
         r[:tags].each do |k,v|
           if k == server_tag
              server_name = v
         private_ip_address = r[:private_ip_address]
         puts "ip address #{private_ip_address} server #{server_name}"
         host_found = false
         host_lines.each_index do |l|
            if host_lines[l] != nil and host_lines[l] != ""
               a = host_lines[l].split(" ")
               if a.length > 0 and a[1] ==  server_name
                 if a[0] != nil and a[0] != "" and a[0] != private_ip_address
                   if a[0].length >= 7
                      a[0] = private_ip_address
                      host_lines[l] = ""
                      host_file_updated = true
                      host_found = true
                      a.each do |d|
                         host_lines[l] = "#{host_lines[l]} #{d}"
                      print_message(" updated host #{host_lines[l]}")
                    if a[0] != nil and a[0] != "" and a[0] == private_ip_address 
                       host_found = true
         if !host_found
            print_message("adding host #{private_ip_address} #{server_name}") 
            host_lines[host_lines.length+1] = "#{private_ip_address} #{server_name}"
            host_file_updated = true
   if host_file_updated"/etc/hosts", "w") do |f|
        host_lines.each do |l|
           if l != nil
           ec2 =, secretaccesskey,:endpoint_url => region)
          generate_hosts_file(ec2, "name")

 To run the script automatically every 5 mins:
        crontab -e
        */5 * * * * ruby /root/cloud_init.rb generate_hosts=Y

Security: There is a good article on Securing n-tier and distributed applications on EC2. I have a similar setup for security as mentioned in this article.

Multi-Casting: Amazon does not support multi-casting (I guess to preserve bandwidth). If your application uses multi-casting (e.g JBoss Cache, monitoring tools) they will need to be reconfigured to use unicasting.


Scott said...

What does the structure of the file /root/settings look like?

Many Thanks

Post a Comment