EC2Dream - Exploring Visual DevOps

Cloud, DevOps and open source infrastructure are the future. The number of command line DevOps tools is exploding, fast agile iterative development of chef cookbooks and puppet modules is the way to build servers and Test Kitchen is the way to test them. EC2Dream explores using a gui to simplify devops on remote servers. it supports:
-Local and Hosted servers
-Amazon AWS and compatible clouds Eucalyptus, CloudStack including VPC, Autoscaling and the ability to list most entities
-Google Compute Engine
-Openstack Clouds Rackspace and HP Cloud and CloudFoundry Open Platform as a Service

Jenkins Continuous Integration Pipeline

Devops is following the development trend of continuous integration i.e.having an automated system for building (testing, source managing & deploying) your chef cookbooks or puppet modules.

In the following we use Jenkins Continuous Integration System, Test Kitchen and a test chef or puppet repo to implement this.

1. Pre-Requisites - Install Chef or Puppet
Install Chef from getchef or
Install puppet from Docs: Installing Puppet. (On windows don't do a gem install for puppet).

2. To use vagrant with Test Kitchen install:
    Oracle Virtual Box
    Vagrant

3. Install required software for testing
For Both:
    gem install test-kitchen
    gem install kitchen-vagrant
For Chef:
    gem install foodcritic
    gem install chefspec
    gem install fauxhai
    gem install berkshelf
For Puppet:
    gem install puppet-lint
    gem install rspec-puppet
    gem install librarian-puppet
    gem install https://github.com/neillturner/kitchen-puppet/blob/master/kitchen-puppet-0.0.5.gem

4. Install Jenkins
For Windows:
    Jenkins running Test-Kitchen via Vagrant. On Windows
For Mac: either
    Jenkins CI on Mac OS X to install using Jenkins-OSX installer package
    or Running jenkins on a mac, from homebrew, as a daemon to install using homebrew.
For Linux:
    Installing and Configuring Jenkins

5. Install the Git Plugin in Jenkins.
To be able to pull and push code to github or a git server you need to setup access.
For Linux and Mac:
    - The easiest way is to use ssh keys and the way to set these up is described in the links about.
    - The ssh keys by default need to be in ~/.ssh/id_rsa for the user running jenkins.
For Windows:
    - The easiest way is to change the logon user of the Jenkins service from local account to your own user then it can use the same credentials as your git client on your workstation.
    - However this is not as secure as using the Local System account so the alternative is to also use ssh keys. See the section Jenkins, GIT plugin and Windows in the Git Plugin documentation.

6. Clone the git hub repo https://github.com/neillturner/chef-repo or https://github.com/neillturner/puppet_repo to your own github user or git server.

7. On your workstation use git to clone the new repo.

8. Setup in jenkins a free-style software project item called say chef_test:
    a. In source code management
        - set to git and a url of your chef-repo on github or git server.
        - setup credential of your git user and password (or use ssh keys).
        - in the advanced section set name to say chef-repo
        - set branches to build to master
    b. Check Poll SCM and enter a schedule of say every fifteen minutes
        H/15 * * * *
    c. Add a windows batch task or a shell command of
    For Chef:
        foodcritic -f any site-cookbooks
    For Puppet:
        puppet-lint --with-filename modules-mycompany
    d. Add a windows batch task or a shell command of
    For Chef:
        rspec site-cookbooks/*/spec/*_spec.rb
    For Puppet:
         cd rspec_directory && rake spec
    e. Add a windows batch task or a shell command of
        kitchen test default-ubuntu-1204
    f. Add a git publisher to publish to the release branch with a build number tag.
        -check Push Only If Build Succeeds
        -set Tag to push to build#$BUILD_NUMBER
        -check Create new tag
        -set Target remote name to chef-repo or puppet_repo
        -add a branch and set Branch to push to release and set Target remote name to chef-repo or puppet_repo

9. Modify the chef-repo or puppet_repo on your workstation and push to the github or git server.

10. Within 15 minutes the test pipeline should run.

Additional Steps
    - If you have a team developing cookbooks you might want to develop on feature branches and merge them on a successful build. See the section Using Git, Jenkins and pre-build branch merging in the Git Plugin documentation.
    - As the number of servers and cookbooks increase it gets better to put cookbooks in separate repositories and run a build for a cookbook. The cookbook name can be a parameter to the build.
    - Before installing in live or qa or doing the feature merge you may want to do a human code review. Products like gerrit can be used.
    - deployment to live or qa can be via Chef Server or Chef Solo.

References
    Leveling Up on Chef Best Practices
    Automating System Provisioning and Application Deployment With Chef
    Using Jenkins to create git tags branches automatically
    Don't Phunk with My Stable Branch: Jenkins Pre-tested Commits to Stop Breaking Stable Branches
    Git, Feature Branches, and Jenkins – or how I learned to stop worrying about broken builds
    Introducing Gerrit - a scalable code review system