Agile DevOps - architecture, technical practices, and culture

Use Cloud and DevOps to build infrastructure by rapid agile iterative development using collaborative open source tools:
-chef cookbooks, puppet modules or ansible playbooks to build servers.
-Test Kitchen to deploy and test them.
-EC2Dream a graphic user interface that provides a 'single pane of glass' to do agile devops primarily on cloud servers.
-Cloud Formation or Terraform using cfndsl Ruby or Troposphere DSL for easy orchestration.
-Amazon AWS, Azure, Google Compute Engine, IBM, Openstack, Local and Hosted servers.

Networking in EC2


Instances running in EC2 have both a Public and Private DNS. Amazon allocates addresses dynamically when an instance starts.

Access from outside EC2 (eg HTTP and ssh) is via the Public DNS IP address. This can be set to a static address by using an Elastic IP address.

Security Groups provide a system wide firewall to restrict access to the machines.
Second level protection can be provided by a firewall on the instance or configuration of IP addresses in some applications.

Access between machines is via the Private DSN. As IP addresses are dynamic the best way is to create a hosts file and then do configuration via hostname. Then when an IP address changes the host file just needs to be updated. A script can be written to use Amazon EC2 Metadata as a simple dns (see below).


Amazon say that "bandwidth is at least 250Mbps, and gets as high as 1000Mb depending on your instance size and that this should be more than enough for most servers". Bandwidth is higher on the large machine instances compared to the small machine instances.

Use Amazon EC2 Metadata as a simple dns
I use the amazon metadata for creating /etc/hosts and do this on a cron schedule. This does everything I need. Instead of fancy DynDNS tricks or having to run and manage an internal DNS server I just have a ruby script that looks at the metadata ec2 to build /etc/hosts. It's easy. To set it up yourself and try it all you need are 3 easy steps.

Step 1 - Start each of your instances with unique security group that matches what you want their internal hostname to be or with a tag like name set to the hostname.

Step 2 - Make sure you have ruby, rubygems and right_aws (rubygem) installed. The following script will update the /etc/hosts file:

#
# generate hosts files in the following order
# 1. running instances  tag called name set to name of instance
# 2. running instance security group name
#
# NOTE:  This script can be run via cron to do regular updating of hosts file
#
def generate_hosts_file(ec2, server_tag)
   if server_tag == nil
      server_tag = "name"
   end
   hosts_file = File.open("/etc/hosts")
   File.open("/etc/hosts.bak", "w") {|f| f.write hosts_file }
   i = 0
   host_file_updated = false
   host_lines  = Array.new
   hosts_file.each do |line|
      host_lines[i] = line
      i=i+1
   end
   server_name = ""
   ec2.describe_instances.each do |r|
      if r[:aws_state] == "running"
         server_name = r[:groups][0][:group_name]
         r[:tags].each do |k,v|
           if k == server_tag
              server_name = v
           end
         end
         private_ip_address = r[:private_ip_address]
         puts "ip address #{private_ip_address} server #{server_name}"
         host_found = false
         host_lines.each_index do |l|
            if host_lines[l] != nil and host_lines[l] != ""
               a = host_lines[l].split(" ")
               if a.length > 0 and a[1] ==  server_name
                 if a[0] != nil and a[0] != "" and a[0] != private_ip_address
                   if a[0].length >= 7
                      a[0] = private_ip_address
                      host_lines[l] = ""
                      host_file_updated = true
                      host_found = true
                      a.each do |d|
                         host_lines[l] = "#{host_lines[l]} #{d}"
                      end
                      print_message(" updated host #{host_lines[l]}")
                   end
                 else
                    if a[0] != nil and a[0] != "" and a[0] == private_ip_address 
                       host_found = true
                    end
                 end
               end
            end
         end  
         if !host_found
            print_message("adding host #{private_ip_address} #{server_name}") 
            host_lines[host_lines.length+1] = "#{private_ip_address} #{server_name}"
            host_file_updated = true
         end
      end 
   end  
   if host_file_updated
      File.open("/etc/hosts", "w") do |f|
        host_lines.each do |l|
           if l != nil
              f.puts(l)
           end
        end 
      end
   end   
end
           ec2 = RightAws::Ec2.new(accesskey, secretaccesskey,:endpoint_url =region)
          generate_hosts_file(ec2, "name")

 Step 3 -To run the script automatically every 5 mins:
        crontab -e
        */5 * * * * ruby /root/cloud_init.rb generate_hosts=Y


0 comments:

Post a Comment