Agile DevOps - Test-Kitchen and EC2Dream

I have a dream... that we have collaborative open source tools and use Cloud and DevOps to rapidly build infrastructure just like developers use agile development.....EC2Dream is a graphic user interface that provides a 'single pane of glass' to do agile devops primarily on cloud servers by using:
-Rapid agile iterative development of chef cookbooks, puppet modules or ansibe playbooks to build servers.
-Test Kitchen to deploy and test them.
-Cloud Formation or OpenStack Heat using cfndsl Ruby DSL for easy orchestration.
-Amazon AWS, Google Compute Engine, Openstack, Rackspace, HP Cloud, Local and Hosted servers.

Lightweight DevOps

"One of my mantras - focus and simplicity. Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. But it's worth it in the end because once you get there, you can move mountains." - Steve Jobs

Lightweight DevOps is NOT using Chef Server or Puppet Master to pull running updates regularly on all your servers. Instead servers fall into 3 groups:
  • Serverless infrastructure. Cloud services like AWS RDS etc.

  • Custom Images or Containers. Use chef/puppet/ansible in development to build custom images to use in test and live.

  • Config individual servers via Puppet apply, chef solo/zero or ansible by pushing changes to servers using tools like test-kitchen or pulling changes from AWS S3.

Use:
  • Immutable infrastructure where ever possible - replace servers instead of updating them.

  • Only one git repository for all environments.

  • Puppet Forge, Chef Community or Ansible Galaxy to minimise the amount of config management to write. Contribute if there is no generic cookbook/module/playbook for your software component.

  • Tools like test-kitchen to run verification tests.

It works best with a cloud where it can replace some of the functions of Chef Server or Puppet Master:
  • Know where your servers are and what their roles are instead of chef server or puppermaster. (eg AWS Console).

  • Orchestrate the servers with a dsl (eg cloudformation with cfndsl).

  • Monitor the servers, manage the log files and generate alarms (e.g. AWS CloudWatch).

  • Run your databases as services (e.g. AWS RDS).

Try and keep it simple and build as little as possible yourself as this will make you as efficient as possible.


References
DevOps is Not a Feature!
De-Centralise and Conquer: Masterless Puppet in a Dynamic Environment
Why You Should Be Using Nodeless-Masterless Puppet
Testing infrastructure with Test Kitchen
Scaling Puppet in EC2
Keep Chef out of your Docker containers
Chef Cookbook Continuous Integration