Agile DevOps - People and Process then Automation!!!

Use Cloud and DevOps to build infrastructure by rapid agile iterative development using collaborative open source tools:
-Terraform or Cloud Formation for easy orchestration.
-chef cookbooks, puppet modules or ansible playbooks to build servers.
-Test Kitchen to deploy and test them.
-Packer for build server images or containers.
-EC2Dream a graphic user interface that provides a 'single pane of glass' to do agile devops primarily on cloud servers.
-Amazon AWS, Azure, Google Compute Engine, IBM, Openstack, Local and Hosted servers.

Networking in EC2

Instances running in EC2 have both a Public and Private DNS. Amazon allocates addresses dynamically when an instance starts.

Access from outside EC2 (eg HTTP and ssh) is via the Public DNS IP address. This can be set to a static address by using an Elastic IP address.

Security Groups provide a system wide firewall to restrict access to the machines.
Second level protection can be provided by a firewall on the instance or configuration of IP addresses in some applications.

Access between machines is via the Private DSN. As IP addresses are dynamic the best way is to create a hosts file and then do configuration via hostname. Then when an IP address changes the host file just needs to be updated. A script can be written to use Amazon EC2 Metadata as a simple dns (see below).

Amazon say that "bandwidth is at least 250Mbps, and gets as high as 1000Mb depending on your instance size and that this should be more than enough for most servers". Bandwidth is higher on the large machine instances compared to the small machine instances.

Use Amazon EC2 Metadata as a simple dns
I use the amazon metadata for creating /etc/hosts and do this on a cron schedule. This does everything I need. Instead of fancy DynDNS tricks or having to run and manage an internal DNS server I just have a ruby script that looks at the metadata ec2 to build /etc/hosts. It's easy. To set it up yourself and try it all you need are 3 easy steps.

Step 1 - Start each of your instances with unique security group that matches what you want their internal hostname to be or with a tag like name set to the hostname.

Step 2 - Make sure you have ruby, rubygems and right_aws (rubygem) installed. The following script will update the /etc/hosts file:

# generate hosts files in the following order
# 1. running instances  tag called name set to name of instance
# 2. running instance security group name
# NOTE:  This script can be run via cron to do regular updating of hosts file
def generate_hosts_file(ec2, server_tag)
   if server_tag == nil
      server_tag = "name"
   hosts_file ="/etc/hosts")"/etc/hosts.bak", "w") {|f| f.write hosts_file }
   i = 0
   host_file_updated = false
   host_lines  =
   hosts_file.each do |line|
      host_lines[i] = line
   server_name = ""
   ec2.describe_instances.each do |r|
      if r[:aws_state] == "running"
         server_name = r[:groups][0][:group_name]
         r[:tags].each do |k,v|
           if k == server_tag
              server_name = v
         private_ip_address = r[:private_ip_address]
         puts "ip address #{private_ip_address} server #{server_name}"
         host_found = false
         host_lines.each_index do |l|
            if host_lines[l] != nil and host_lines[l] != ""
               a = host_lines[l].split(" ")
               if a.length > 0 and a[1] ==  server_name
                 if a[0] != nil and a[0] != "" and a[0] != private_ip_address
                   if a[0].length >= 7
                      a[0] = private_ip_address
                      host_lines[l] = ""
                      host_file_updated = true
                      host_found = true
                      a.each do |d|
                         host_lines[l] = "#{host_lines[l]} #{d}"
                      print_message(" updated host #{host_lines[l]}")
                    if a[0] != nil and a[0] != "" and a[0] == private_ip_address 
                       host_found = true
         if !host_found
            print_message("adding host #{private_ip_address} #{server_name}") 
            host_lines[host_lines.length+1] = "#{private_ip_address} #{server_name}"
            host_file_updated = true
   if host_file_updated"/etc/hosts", "w") do |f|
        host_lines.each do |l|
           if l != nil
           ec2 =, secretaccesskey,:endpoint_url =region)
          generate_hosts_file(ec2, "name")

 Step 3 -To run the script automatically every 5 mins:
        crontab -e
        */5 * * * * ruby /root/cloud_init.rb generate_hosts=Y


Post a Comment