“It is not the strongest of the species that survive, nor the most intelligent, but the one most responsive to change.” — Charles Darwin

“DevOps is a set of practices that seeks to reduce the gap between software development and software operation.”

Test Driven Development with Chef, Puppet and Ansible

"build the right thing and the thing right"

An increased focus on testing, continuous integration and agile development is occuring in software development. This is also influencing configuration tools Chef, Puppet and Ansible. We are seeing influential books like Test-Driven Infrastructure with Chef, 2nd Edition emerge.

The following tools seem to be becoming well established:

Test TypeChefPuppetAnsible
StyleFoodcriticPuppet-lintAnsible-lint
UnitChefspecRspec-PuppetRspec
Integration    Serverspec    Serverspec or Beaker-rspecServerspec or Ansible_spec
Test Harness    Test Kitchen    Test Kitchen or BeakerTest Kitchen

Style and Syntax Testing
Foodcritic, Puppet-lint and Ansible-lint can quickly test the syntax and style of your cookbooks/modules on your own workstation without writing any tests so this is the easiest testing to do.
See explanations of Foodcritic messages or Puppet-lint checks or ansible-lint

Unit Testing
Chefspec and rspec-puppet are good for regression testing and checking that the cookbooks or modules build what they are suppose to be building. Here you have to write the tests in rspec format but you don't need to have a server and run chef/puppet because it simulates this. But there are limitations so not all features of your cookbook/module can be testing.
To learn chefspec see chefspec exploration, chefspec examples, chefspec doc and rubydoc
To learn rspec-puppet see rspec-puppet tutorial and the matches

Integration Testing
Once again Serverspec tests need to be written but they are in a similar format to rspec tests. However integration tests are more time consuming as it requires chef or puppet to be run on a remote server to converge it before running the tests.
Because chef/puppet/ansible converge the node you don't need to write tests to prove that chef/puppet/ansible works. The tests should be equivalent to what system admin does when they log on to the server and check various files, directory and logs to check all is ok.
To learn serverspec see Resource Types and serverspec examples
Beaker-rspec is also in rspec format and see the Beaker DSL with examples.

Test Harness
Test Kitchen is fast becoming the standard way to run integration tests for chef as it automatically creates the servers and runs the tests and logs the results. It has a great Getting Started Tutorial.
See Test Kitchen Getting Started for test kitchen commands.
See demo of test-kitchen 1.0 chef repository
Puppet is supported in Test Kitchen using the kitchen-puppet plugin.
Ansible is supported in Test Kitchen using the kitchen-ansible plugin.

Beaker is the puppet developed equivalent of Test Kitchen. It supports provisioning of servers in vagrant or AWS etc. Unlike Test Kitchen it has its own beaker-rspec language for writing tests similar to serverspec. Because it is ruby code other testing tools can be called from this code.
See Beaker wiki for documentation. Currently there are not many examples of creating beaker DSL tests.

Ansible-spec is an alternative to test-kitchen and serverspec. It has a particular directory structure to follow ansible-sample-tdd. It should be possible to call ansible-spec from test-kitchen if so desired.

Continuous Integration
We are starting to see continuous integration pipelines being set in systems like Jenkins that automate the test process.

Auditing
Many companies run puppet with the --noop option regularly over the server estate to check the configuration are correct. Ansible also support a similar option called dryrun. However often these option canno fully simulate complex modules or playbooks so they are not a guarantee that they will work with run for real.

EC2Dream facilitates both cookbook development and testing development by supplying a sample repository and:
a. A graphical interface to test kitchen
b. The ability to push a repository and tests to a remote machine and run chef or puppet and the test tools.

Chef References
foodcritic docs
chefspec docs
Outside In TDD with Chefspec
Chef cookbooks development by TDD
Day 11 Lazy Sysadmins Guide to Test Driven Chef Cookbooks
Testing Chef Cookbooks. Part 2. Chefspec
Unit Testing Chef Cookbooks
Chef Recipe Code Coverage
Test-Driven Chef Cookbooks with Test Kitchen
Chef Testing at PagerDuty
Chef Cookbook Testing and Continous Integration

Puppet References
BetterSpecs
Rspec Puppet Tutorial
Automated Puppet Testing
Verifying Puppet: Checking Syntax and Writing Automated Tests
Test Driven Development with Puppet
The Next Generation of Puppet Module Testing
RSpec tests for puppet modules
Puppet Testing Part 1
Testing puppet modules
Puppet modules in Jenkins.

0 comments:

Post a Comment