Martin Englund posted an open question to the Puppet mailing list a few days ago asking how people are verifying their systems are built as expected:

When you write code, you always use unit testing & integration testing to verify that the application is working as expected, but why don't we use that when we install a system?

What are you using to verify that your system is correctly configured and behaves the way you want?

He linked to a blog post demonstrating how he was verifying his machines using Cucumber.

Coincidentally, about a week earlier at Devopsdays in Gent, I was talking to Felix Kronlage and Bernd Ahlers from bytemine about doing similar things through testing SSH and mail delivery with cucumber-nagios.

It's pretty cool people are thinking about doing BDD/TDD with infrastructure, and it's even cooler that the tools are at the point where doing this is actually possible.

When doing software testing, your testing tool is normally separate from the language and libraries you're building the software with (but almost always written in the same language). When testing your infrastructure, I think it makes perfect sense to apply this practice.

So to practise Behaviour Driven Infrastructure right now, you can use Cucumber as the testing tool, and Puppet as the programming language.

One advantage of practicising BDD within sysadmin world is that the testing tools aren't closely coupled to the language our systems are built with - i.e. if you hate Puppet you can use Cfengine, and if Cucumber isn't cutting it use PyUnit.

But to something tangible!

Building on Martin's excellent examples, i've pushed out a new version of cucumber-nagios that includes some basic SSH interaction steps, so you can start building behavioural tests for your infrastructure:

Feature: example.org ssh logins
  As a user of example.org
  I need to login remotely

  Scenario: Basic login
    Given I have no public keys set
    Then I can ssh to "example.org" with the following credentials: 
     | username | password    |
     | lindsay  | spoonofdoom |

  Scenario: Login to multiple hosts
    Given I have no public keys set
    Then I can ssh to the following hosts with these credentials: 
     | hostname           | username | password      |
     | example.org        | matthew  | spladeofpain  |
     | mail.example.org   | john     | forkoffury    |
     | web04.example.org  | steve    | sporkofpork   |

  Scenario: Login with a key
    Given I have the following public keys: 
     | keyfile                   |
     | /home/user/.ssh/id_dsa |
    Then I can ssh to the following hosts with these credentials: 
     | hostname         | username |
     | example.org      | matthew  |
     | mail.example.org | mark     |
    
  Scenario: Login with an inline key
    Then I can ssh to the following hosts with these credentials: 
     | hostname         | username | keyfile                   |
     | example.org      | luke     | /home/luke/.ssh/id_dsa |
     | mail.example.org | john     | /home/john/.ssh/id_dsa |

The above example shows there's lots of ways to test the same thing (all depending on what you're trying to achieve), but there is now also suppport for executing shell commands remotely:

  Scenario: Checking /etc/passwd
    When I ssh to "example.org" with the following credentials: 
     | username | password      | keyfile                 |
     | jacob    | spifeofstrife | /home/jacob/.ssh/id_dsa |
    And I run "cat /etc/passwd" 
    Then I should see "jacob" in the output

I don't expect you would do a cat /etc/passwd in a real test, however the step definition is a good example of how to interact with an established SSH connection:

When /^I run "([^\"]*)"$/ do |command|
  @output = @connection.exec!(command)
end

Then /^I should see "([^\"]*)" in the output$/ do |string|
  @output.should =~ /#{string}/
end

You'd use this to write specific tests for checking system behaviour, such as local user logins vs LDAP logins, or the presence of a daemon.

So the resulting process may look something like this:

1. Use cucumber-nagios to write a specification of how you expect your infrastructure to behave.
2. Hook your new cucumber-nagios checks into Nagios.
3. Start writing your manifests/cookbooks.
4. Run your configuration management tool on the node you're configuring.
5. Iterate until your monitoring system is silent.

Not only do you have a functional definition of how your machines work that you can use to build your machines, but if your systems deviate from the expected behaviour at any point in the future, you'll get an alert from your monitoring system.

Maintaining both a configuration management system and a set of integration tests might get annoying after a while, but if you ever decide to migrate to another configuration management system or move your machines into the cloud you'd have a set of tests you could apply immediately.

This could also be useful for moving existing machines into a configuration management system. Write a set of integration tests for your unmanaged machines, run your configuration management system over the existing machines, see if anything is broken.

I'd be interested to hear how this process or similar works for people!

On cucumber-nagios:

And Flapjack:

Yesterday at the excellent Devopsdays in Gent, Belgium, I proposed an open session to flesh out an idea I had a few weeks ago - to use Cucumber as a general scripting language.

Cucumber's Given/When/Then steps are well suited to procedural tasks like shell script, and you would be writing your "scripts" in straightforward language that non-technical users such as managers and clients could understand. Also, as writing a scenario without a Then to close it feels unbalanced, you'd get in the mindset of testing the actions of your "scripts" fairly quickly.

With little more than the hypothesis above, a group of us found a room and started modeling some scenarios. Our focus was on file manipulation, as it was a low hanging fruit and something most scripts do.

We came up with this:

Feature: Copy files around
  
  Scenario: A single file
    Given I am in "/tmp"
    And the file "spoons" exists
    When I copy the file "spoons" to "forks"
    Then the file "forks" should exist
    And the file "forks" should be readable

  Scenario: Multiple files
    Given I am in "/tmp"
    Given the following table of tasty fruit:
      | filename |
      | apples   | 
      | oranges  |
      | bananas  |
      | ananas   |
      | file with lots o spaces |
      | spoons of : doom |
    When I create the directory "/tmp/some_other_dir"
    When I copy the tasty fruit in the table to "/tmp/some_other_dir"
    Then the tasty fruit in the table should exist in "/tmp/some_other_dir"

The first scenario is fairly self explanatory, but the second one is where the interesting stuff starts happening.

In the implementation of the "following table" step, we create an instance variable that persists the list of files between steps. This way, we can reference the "tasty fruit" throughout our other steps:

Given /^the following table of (.+):$/ do |name, table|                          
  @tables = {}                                                                   
  @tables[name] = table.hashes                                                   
end

We use the (.+) regex to capture the name of the table so we can poke at it later on. This design lets you easily use multiple tables throughout your steps that won't conflict with one another:

  Scenario: Multiple files from multiple tables
    Given the following table of tasty fruit:
      | filename |
      | apples   | 
      | oranges  |
    And the following table of baggy baggage:
      | filename |
      | suitcase | 
      | backpack |
    When I copy the baggy baggage in the table to "/tmp/some_other_dir"
    And I copy the tasty fruit in the table to "/tmp/some_other_dir"
    Then the tasty fruit in the table should exist in "/tmp/some_other_dir"
    And the baggy baggage in the table should exist in "/tmp/some_other_dir"

Other steps can reference data in the table by accepting a name and looking it up in the hash of tables:

Then /^the (.+) in the table should exist in "([^\"]*)"$/ do |name, destination| 
  @tables[name].each do |file|                                                   
    File.exists?(File.join(destination, file["filename"])).should be_true        
  end                                                                            
end 

We also looked at handling permission problems:

  Scenario: Do things i'm not allowed to 
    When I create the directory "/usr/bin/wtf"

Here the step will raise an Errno::EACCES exception, and as Cucumber uses a pretty formatter by default, the failed step will appear in red.

Finally we tried copying files with a glob. The initial implementation I banged out was very Unix focused (it used *, which is a very explicit globbing syntax), so we scrapped that idea and wrote our intentions in plain English:

  Scenario: Copy based on a pattern
    Given I am in "/tmp"
    When I create the directory "/tmp/pattern_dir"
    And I copy files beginning with the letters z,y,x to "/tmp/pattern_dir"
    Then they should exist there

The implementation is obvious, and is very understandable (and seemingly powerful) to someone with no knowledge of globbing.

People who have used Cucumber in web development will likely note that the above implementation is an example of tightly coupled steps, which is sometimes regarded as an anti-pattern. I'm of the opinion that this is a lot more painful in a web development context than in a procedural/scripting tool one.

From my recollection of Euruko earlier this year, when Aslak was asked whether he considers it an antipattern, he said it can be ok to use depending on the problem you're trying to solve, so I take that as tacit permission that it is ok this context. :-)

I posted the results of the session to a Gist yesterday, and I have also published a repo with a bundler-ready install process, so people can hack on it more.

After the session I remembered that the feature file doesn't actually have to start with Feature, so it's possible to write standalone scenarios one after another.

When wrapping up, someone in the room pointed out that our implementation actually went one better than being readable by non-technical users - they could probably write the scripts themselves.

This is pretty powerful, and coupled with Cucumber's very cool step generation when running scenarios with undefined steps, makes it very easy to start prototyping a standard library of human readable scripting commands.

There was chatter on the Cucumber mailing list a few weeks ago about providing alternate interfaces for writing and executing Cucumber features, and it could be cool to see a drag-and-drop interface with a library of common tasks that calls out to Cucumber to execute them. You could even build something quite beautiful with HotCocoa.

Anyhow, if you think anything mentioned above is a cool idea, check out the code and start hacking!

I've just released a new version of cucumber-nagios, and this release is quite a milestone!

Big changes in this release include:

• Removal of the ghetto bundler in favour of wycats/carllerche's bundler.

  In previous releases, you'd use a rake task to freeze in dependencies. This produced all sorts of weird problems when new versions of the dependencies were released, it didn't handle gems with C extensions that well, and could be very slow if you ran it multiple times.

  Now that bundler has started maturing, cucumber-nagios has made the switch. It eliminates all the aforementioned issues, and integrates cleanly with RubyGems.

• Renaming of the gem to cucumber-nagios from auxesis-cucumber-nagios, as GitHub have discontinued building gems. The gem is now published on Gemcutter.

• The project generator now prints out helpful instructions when you generate a new project.

• cucumber-nagios projects have built-in steps for benchmarking response times. The following example explains it best:

Feature: slashdot.com
  To keep the geek masses satisfied
  Slashdot must be responsive
    
  Scenario: Visiting a responsive front page
    Given I am benchmarking
    When I go to http://slashdot.org/
    Then the elapsed time should be less than 5 seconds

• A --debug switch can be passed to cucumber-nagios to print out the command line built and executed. This can be useful when writing your features.

• Removal of several unnecessary support files, and cleanups of helpers and Cucumber's World object setup, in line with an updated version of Webrat.

• Refactoring of the Nagios formatter for Cucumber to use Cucumber 0.4.0's formatter interface. For users, this simply means cucumber-nagios now works with Cucumber 0.4.0 (the latest at time of this release).

Although i've done a fair amount of testing, there will invariably be bugs, which can be reported on GitHub.

After a quick migration, i've switched this blog from WordPress to Jekyll.

I've done this for several reasons:

• My blog content is static. Spinning up PHP on every request is overkill.
• I want to write my posts in Markdown.
• Jekyll has awesome syntax highlighting using Pygments.
• I can easily migrate from WordPress to Jekyll.
• Latest WordPress releases segfault Apache on Dapper.

Cool things now i've migrated:

• I can version control my blog.
• My blog content is flat file, so I just edit the content and push. This also means my blog can be easily distributed and backed up.
• Pulling in Flickr photos, Last.fm listening and tweets no longer blocks the page load. I wrote a cute little MooTools class to display the info, and a cron job to fetch it in the background.
• Comments are all preserved, as I switched to Disqus several weeks ago. The WordPress => Disqus import was mind numbingly easy using the Disqus plugin.

If you want minimalism in your blogging engine and full control over its appearance, Jekyll might be worth checking out.

I've spent the last few years trying to find the holy grail of FOSS toolchains for producing and displaying my presentations.

I started with OpenOffice.org Impress (which I have sworn never to use again after it ate several presentations), dallied with Clutter's opt, seriously used KeyJNote, before moving to Mac OS X last year and settling on Apple's Keynote.

While travelling this year without a Mac, i've resumed my search for the perfect toolchain, and I think i've found a setup that works pretty well.

It's uses Inkscape to build the slides, a text file to order slides, KeyJNote to display them, and Rake to tie it all together. Oh, and it's versioned with Bazaar.

Each slide goes on a new line in order.txt:

/home/auxesis/Desktop/devopsdays/slides/blank.png
# http://www.flickr.com/photos/tim_norris/2600844073/
/home/auxesis/Desktop/devopsdays/slides/scalable.png
# http://www.flickr.com/photos/barbour/404053639/
/home/auxesis/Desktop/devopsdays/slides/distributed.png
# http://www.flickr.com/photos/numstead/535460927/
/home/auxesis/Desktop/devopsdays/slides/nagios-plugin-format.png

You can easily add comments between slides to keep track of image sources or write down ideas.

Then there's a Rake task for building the KeyJNote command line and setting up displays:

desc "setup external displays"
task :displays do
  system("xrandr --output VGA --mode 1024x768")
  system("xrandr --output VGA --same-as LVDS")
end
 
desc "perform presentation"
task :perform => :displays do
  options = "--transition Crossfade --transtime 250 -c persistent"
  command = "keyjnote #{options} @#{File.dirname(__FILE__)}/order.txt"
  system(command)
end

Finally, there's a Rake task for building the PNGs from the SVGs created through Inkscape:

desc "build pngs from svgs"
task :build do
  Dir.glob("sources/*.svg").each do |file|
    if modified?(file)
      basename = File.basename(file, '.svg')
      slide = "slides/#{basename}.png"
      system("inkscape -e #{slide} -f #{file}")
    end
  end
end
 
def modified?(filename)
  index_filename = File.join(File.dirname(__FILE__), "cache", "index")
  # read or initialise index
  if File.exists?(index_filename)
    @index = File.open(index_filename, 'r') { |f| Marshal.load(f) }
  else
    @index = {}
  end
 
  # check if modified
  if @index[filename]
    modified = File.mtime(filename) > @index[filename][:mtime]
  else
    modified = true
  end
 
  # update index
  @index[filename] = {:mtime => File.mtime(filename)}
  File.open(index_filename, 'w') { |f| Marshal.dump(@index, f) }
 
  return modified
end

This keeps an index of SVG mtimes, and only rebuilds a slides if you modify the SVG.

Then to view the presentation, it's a simple:

$ rake build ; rake perform

Now that you're just dealing with a bunch of files, you can version control the whole presentation with something like bzr (which handles binary content really well). It's worth setting up an ignore list so all the generated slides don't get versioned:

slides/*
*.cache
cache/*

I've been working on a cool little side project the last week called Visage. It renders graphs of collectd statistics in the browser, making the data interactive.

It's a lot more interactive than the screenshot suggests, so check out an instance of Visage running.

Some background:

collectd is an awesome way to collect statistics from your Unix machines and aggregate the stats in one place (it has a network plugin that makes this a cinch).

So you set up collectd, and you're getting all these great statistics, but you want graphs right? Graphs make the IT manager in all of us smile.

To date there have been two options for viewing graphs of collectd's data: collection.cgi, which comes bundled with collectd on most distros, though sometimes squirreled away weirdly, and the newer collection3.

The problem I have with these interfaces is that they are organised like the RRDs that collectd stores. You basically use the interface to navigate the RRDs, not deduce meaning.

I want to easily see correlations between multiple hosts during a slashdotting. I want to view related stats for a host on a dashboard page. I want to filter out datasets that aren't interesting to me.

What's holding the existing graphing interfaces back is the presentation layer (graphs generated from RRDtool, wrapped in a smattering of Perl) being very tightly coupled with the data layer (the RRDs themselves).

So I set about exposing the RRDs in a more digestible form - JSON.

Once the RRDs are exposed over the web it makes it easy to consume the data and build your own graphing interface as either a thick client, Flash widget, or in the browser. You could also do periodic snapshots and reporting, but I digress.

So once I was able to consume this data, I used the Raphaël JavaScript library to render the graphs, and turned it into a MooTools class for maximum reusability.

So there you have it!

Right now there are a few rough edges (the axis labels keep me up at night), but it's functional. If you give it a go, i'd like to know! You can report any issues you find on GitHub.

The conference season is starting to warm up:

I'll be speaking at the devopsdays 2009 in Ghent, Belgium on 30th/31st of October, on cucumber-nagios and Flapjack. Patrick Debois is doing an awesome job carving out a Velocity-like conference in Europe, so if you're do any sort of operations or sysadmin work it's definitely worth attending.

In January i'll be speaking at linux.conf.au 2010 in Wellington, New Zealand on Flapjack. This year's organisers are putting together a conference so chock-full of awesome your head will be spinning. If you do anything with open source in Australia or New Zealand you can't afford to miss it.

When hacking on small open source projects you end up having to be a jack of all trades - hacker, documentor, community manager. Documentation can be annoying when you just want to hack, but if you want people to your code it's essential.

One problem i've experienced is keeping documentation on the project's website up to date with how the software actually works.

Generally you have two sets of documentation - one in your project's source code repo, and one on your project's website. The website docs tend to cover meta information about the project, and maybe a quick install guide or demo. Your source code docs will cover the nitty-gritty of setting up your app, and maybe how to hack on the code.

More often than not the docs in the project's source will get updated more regularly than on the website, and after a while the website docs may end up diverging from how the software actually works. This can be especially noticeable in installation docs.

I've tried attacking this problem on a new project i'm hacking on.

Lately i've been using nanoc to build a few simple sites. Rather than a full blown application server running in the background a la Rails or Django, nanoc simply compiles the templates and layouts on your site and spits out static HTML.

nanoc makes it easy to write your own helpers to be used during the compile phase, so i've written a simple helper to include documentation from an external source inline with my content:

#lib/external_docs.rb

def generate_docs_from_source(name)
  doc_path = File.join(@config[:code_src], 'doc')
  filename = File.join(doc_path, "#{name.upcase}.md")

  if File.exists?(filename)
    output = `rdiscount #{filename}`
    Haml::Helpers::find_and_preserve(output, ["pre"])
  else
    "<span style='color: red'>Error: #{name.upcase}.md doesn't exist in #{doc_path}</span>"
  end
end

This calls out to rdiscount to generate HTML from a Markdown document. You can use it easily in your templates:

%div#installation
  %div#install.generated
    = generate_docs_from_source('install')

Which will render INSTALL.md in your source code docs/ directory under div#install. This makes it easy to theme the generated docs, courtesy of the .generated selector.

The nanoc command line tool has a simple way to compile a site:

$ nanoc compile

Which is useful when you want to call a compile from other scripts. You can probably see where i'm going with this:

On the main project source repo i've configured a post-receive hook to POST to docs.flapjack-project.com, which is a Sinatra app that triggers a build of the website. Before it does the build, it checks out the latest copy of the project and website sources.

The nice thing about this approach is you can easily interpolate your own static docs with those from the source code:

%div#developing
  %h2 Source code
  %p  
    Flapjack's code is maintained in two repositories on GitHub:
    %ul 
      %li 
        %a{:href => "http://github.com/auxesis/flapjack/tree/master"}> flapjack
        , the core of the monitoring system.
      %li 
        %a{:href => "http://github.com/auxesis/flapjack-admin/tree/master"}> flapjack-admin  
        , the admin interface.
 
  %p  
    Flapjack is open source, released under the 
    = link_to "MIT Licence", "http://en.wikipedia.org/wiki/MIT_License"
    \.  
  
  %div.generated#developing
    = generate_docs_from_source('developing')

The code for those interested:

• Sinatra app, to trigger website builds
• nanoc website source, to compile the site.

Last year I blogged about where to get up-to-date Ruby packages for Ubuntu.

The PPA I suggested hasn't been updated in a while, and there are a few gems that require the latest version of RubyGems to work correctly (Rails, i'm looking at you).

I suggest you remove the PPA and add this to your /etc/apt/sources.list:

deb http://apt.brightbox.net/ hardy main

Sub hardy with intrepid/dapper depending on what release you need.

Then do an apt-get update && apt-get remove rubygems && apt-get install rubygems, and you'll be upgraded to the latest version of RubyGems.

As lots of people are apparently using cucumber-nagios and i'm getting a few bug reports, I thought it'd be worth setting up a proper home for the project.

You can find the project's site over here. Feature requests and bugs can be reported over at the GitHub issues page.

I apologise in advance if i'm a tad unresponsive when replying to new issues or emails about it. I'm backpacking around Europe until December so it's all a matter of finding suitable hacking time.

I pushed out a new release of cucumber-nagios last night. Things of note:

• It's now project focused. Use cucumber-nagios-gen to generate a project. The project provides infrastructure for freezing in dependencies, so you can zip up the project directory and migrate it between machines easily.
• It's released as a gem. You can install it with a plain old gem install cucumber-nagios. This will install everything you need to set up a project.

To get up and running with cucumber-nagios, this is what you need to do:

gem sources -a http://gems.github.com
gem install auxesis-cucumber-nagios
cucumber-nagios-gen project ebay.com.au
cd ebay.com.au 
rake deps

When you generate a project, it also spits out a .bzrignore and .gitignore, so there's no excuse not to be using version control!

All the previous documentation about writing and testing features still applies. I'm planning on adding support for generating features with cucumber-nagios-gen in the next release.

The code and documentation can now be found on GitHub. Launchpad was giving me no love.

James, Michael and I have been working on a small startup called Hardcopy Books for the last 3 months. Here's the elevator pitch:

Hardcopy Books is an online bookstore exclusively for tech books in Australia. How are we different?

1. We only stock tech books.
2. Your orders will be cheaper than through Amazon.
3. We aim to deliver within a week.

We started it because ordering tech books in Australia sucks. If you order from Amazon, the books are cheap, but they can take ages to get here, and shipping is very expensive. Most local book stores don't specialise in tech books, and the ones that do have a very slow turnaround and a horribly tedious ordering process. We aim to change that.

Today we're announcing the Hardcopy Books 2009 Launch Survey.

We're asking you, the local tech community, to fill it out and let us know what sort of books you buy. We're going to be launching with a select group of titles, and the feedback you give us will let us know what books we need to stock up on.

We're also on Twitter, so you can keep up to date with the launch on there.

Interesting thought experiment:

• Cucumber is kick arse way of describing the behaviour of a system.
• Webrat makes interacting with websites blindingly easy.
• Nagios is the industry standard for system/network/application monitoring.

What happens if you combine the three? You get cucumber-nagios.

cucumber-nagios takes the results of a Cucumber run and outputs them in the Nagios plugin format. What does that actually mean?

A sysadmin can describe the behaviour of a system that they manage:

Feature: google.com.au
  It should be up
  And I should be able to search for things

  Scenario: Searching for things
    When I visit "http://www.google.com"
    And I fill in "q" with "wikipedia"
    And I press "Google Search"
    Then I should see "www.wikipedia.org"

Then they can run the feature through cucumber-nagios:

$ cucumber-nagios features/google.com.au/search.feature
Critical: 0, Warning: 0, 4 okay | value=4.000000;;;;

The curious can check out the code on GitHub, and the documentation on the project website.

UPDATE: There have been a few changes to cucumber-nagios since this post. Check out these two posts for more info.

So I don't waste another 30 minutes of my life:

When using the select helper in Merb, make sure you call to_s on whatever you're setting :selected to, i.e.

select :name => "order[status]", :collection => OrderStatus.all, 
       :text_method => :description, :value_method => :id,
       :selected => @order.status.to_s

Otherwise the helper will compare a String (the value) to an Integer (:selected), and you'll never get anything in your select selected!