Agile DevOps - Test-Kitchen and EC2Dream

I have a dream... that we have collaborative open source tools and use Cloud and DevOps to rapidly build infrastructure just like developers currently use tools to do agile development of code.....EC2Dream is a small attempt to develop a gui to do agile devops primarily on cloud servers by using:
-Rapid agile iterative development of chef cookbooks, puppet modules or ansibe playbooks to build servers.
-Test Kitchen to deploy and test them.
-Cloud Formation or OpenStack Heat using cfndsl Ruby DSL for easy orchestration.
-Amazon AWS and compatible clouds Eucalyptus, CloudStack including VPC, Autoscaling, ability to list most entities.
-Google Compute Engine
-Openstack Private Clouds and Rackspace and HP Cloud
-Local and Hosted servers.

Lightweight DevOps

"One of my mantras - focus and simplicity. Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. But it's worth it in the end because once you get there, you can move mountains." - Steve Jobs

Lightweight DevOps is NOT using Chef Server or PuppetMaster to pull running updates regularly on your servers. Instead use:
  • Puppet apply, chef solo/zero or ansible to config servers by pushing changes to servers when required.

  • Containers by creating generic images without the deployed application or build docker containers.

  • Puppet/chef/ansible in development to create containers that deploy applications in live from say S3 and be autoscaled.

  • Puppet/chef/ansible minimially in live environments.

  • Immutable infrastructure where ever possible - replace servers instead of updating them.

  • Only one git repository for all environments.

  • Puppet forge, chef community or ansible galaxy to minimise the amount of config management to write. Contribute if there is no generic cookbook/module/playbook for your software component.

  • Tools like test-kitchen to run verification tests.

It works best with a cloud where the cloud can be used to:
  • Know where your servers are and what their roles are instead of chef server or puppermaster. (eg AWS Console).

  • Orchestrate the servers with a dsl on top (eg cloudformation with cfndsl).

  • Monitor the servers, manage the log files and generate alarms (e.g. AWS CloudWatch).

  • Run your databases as services (e.g. AWS RDS).

Try and keep it simple and build as little as possible yourself as this will make you as efficient as possible.


References
DevOps is Not a Feature!
De-Centralise and Conquer: Masterless Puppet in a Dynamic Environment
Why You Should Be Using Nodeless-Masterless Puppet
Scaling Puppet in EC2
Keep Chef out of your Docker containers
Chef Cookbook Continuous Integration