I've spent the last few years trying to find the holy grail of FOSS toolchains for producing and displaying my presentations.

I started with OpenOffice.org Impress (which I have sworn never to use again after it ate several presentations), dallied with Clutter's opt, seriously used KeyJNote, before moving to Mac OS X last year and settling on Apple's Keynote.

While travelling this year without a Mac, i've resumed my search for the perfect toolchain, and I think i've found a setup that works pretty well.

It's uses Inkscape to build the slides, a text file to order slides, KeyJNote to display them, and Rake to tie it all together. Oh, and it's versioned with Bazaar.

Each slide goes on a new line in order.txt:

/home/auxesis/Desktop/devopsdays/slides/blank.png
# http://www.flickr.com/photos/tim_norris/2600844073/
/home/auxesis/Desktop/devopsdays/slides/scalable.png
# http://www.flickr.com/photos/barbour/404053639/
/home/auxesis/Desktop/devopsdays/slides/distributed.png
# http://www.flickr.com/photos/numstead/535460927/
/home/auxesis/Desktop/devopsdays/slides/nagios-plugin-format.png

You can easily add comments between slides to keep track of image sources or write down ideas.

Then there's a Rake task for building the KeyJNote command line and setting up displays:

desc "setup external displays"
task :displays do
  system("xrandr --output VGA --mode 1024x768")
  system("xrandr --output VGA --same-as LVDS")
end
 
desc "perform presentation"
task :perform => :displays do
  options = "--transition Crossfade --transtime 250 -c persistent"
  command = "keyjnote #{options} @#{File.dirname(__FILE__)}/order.txt"
  system(command)
end

Finally, there's a Rake task for building the PNGs from the SVGs created through Inkscape:

desc "build pngs from svgs"
task :build do
  Dir.glob("sources/*.svg").each do |file|
    if modified?(file)
      basename = File.basename(file, '.svg')
      slide = "slides/#{basename}.png"
      system("inkscape -e #{slide} -f #{file}")
    end
  end
end
 
def modified?(filename)
  index_filename = File.join(File.dirname(__FILE__), "cache", "index")
  # read or initialise index
  if File.exists?(index_filename)
    @index = File.open(index_filename, 'r') { |f| Marshal.load(f) }
  else
    @index = {}
  end
 
  # check if modified
  if @index[filename]
    modified = File.mtime(filename) > @index[filename][:mtime]
  else
    modified = true
  end
 
  # update index
  @index[filename] = {:mtime => File.mtime(filename)}
  File.open(index_filename, 'w') { |f| Marshal.dump(@index, f) }
 
  return modified
end

This keeps an index of SVG mtimes, and only rebuilds a slides if you modify the SVG.

Then to view the presentation, it's a simple:

$ rake build ; rake perform

Now that you're just dealing with a bunch of files, you can version control the whole presentation with something like bzr (which handles binary content really well). It's worth setting up an ignore list so all the generated slides don't get versioned:

slides/*
*.cache
cache/*

I've been working on a cool little side project the last week called Visage. It renders graphs of collectd statistics in the browser, making the data interactive.

It's a lot more interactive than the screenshot suggests, so check out an instance of Visage running.

Some background:

collectd is an awesome way to collect statistics from your Unix machines and aggregate the stats in one place (it has a network plugin that makes this a cinch).

So you set up collectd, and you're getting all these great statistics, but you want graphs right? Graphs make the IT manager in all of us smile.

To date there have been two options for viewing graphs of collectd's data: collection.cgi, which comes bundled with collectd on most distros, though sometimes squirreled away weirdly, and the newer collection3.

The problem I have with these interfaces is that they are organised like the RRDs that collectd stores. You basically use the interface to navigate the RRDs, not deduce meaning.

I want to easily see correlations between multiple hosts during a slashdotting. I want to view related stats for a host on a dashboard page. I want to filter out datasets that aren't interesting to me.

What's holding the existing graphing interfaces back is the presentation layer (graphs generated from RRDtool, wrapped in a smattering of Perl) being very tightly coupled with the data layer (the RRDs themselves).

So I set about exposing the RRDs in a more digestible form - JSON.

Once the RRDs are exposed over the web it makes it easy to consume the data and build your own graphing interface as either a thick client, Flash widget, or in the browser. You could also do periodic snapshots and reporting, but I digress.

So once I was able to consume this data, I used the Raphaël JavaScript library to render the graphs, and turned it into a MooTools class for maximum reusability.

So there you have it!

Right now there are a few rough edges (the axis labels keep me up at night), but it's functional. If you give it a go, i'd like to know! You can report any issues you find on GitHub.

The conference season is starting to warm up:

I'll be speaking at the devopsdays 2009 in Ghent, Belgium on 30th/31st of October, on cucumber-nagios and Flapjack. Patrick Debois is doing an awesome job carving out a Velocity-like conference in Europe, so if you're do any sort of operations or sysadmin work it's definitely worth attending.

In January i'll be speaking at linux.conf.au 2010 in Wellington, New Zealand on Flapjack. This year's organisers are putting together a conference so chock-full of awesome your head will be spinning. If you do anything with open source in Australia or New Zealand you can't afford to miss it.

When hacking on small open source projects you end up having to be a jack of all trades - hacker, documentor, community manager. Documentation can be annoying when you just want to hack, but if you want people to your code it's essential.

One problem i've experienced is keeping documentation on the project's website up to date with how the software actually works.

Generally you have two sets of documentation - one in your project's source code repo, and one on your project's website. The website docs tend to cover meta information about the project, and maybe a quick install guide or demo. Your source code docs will cover the nitty-gritty of setting up your app, and maybe how to hack on the code.

More often than not the docs in the project's source will get updated more regularly than on the website, and after a while the website docs may end up diverging from how the software actually works. This can be especially noticeable in installation docs.

I've tried attacking this problem on a new project i'm hacking on.

Lately i've been using nanoc to build a few simple sites. Rather than a full blown application server running in the background a la Rails or Django, nanoc simply compiles the templates and layouts on your site and spits out static HTML.

nanoc makes it easy to write your own helpers to be used during the compile phase, so i've written a simple helper to include documentation from an external source inline with my content:

#lib/external_docs.rb

def generate_docs_from_source(name)
  doc_path = File.join(@config[:code_src], 'doc')
  filename = File.join(doc_path, "#{name.upcase}.md")

  if File.exists?(filename)
    output = `rdiscount #{filename}`
    Haml::Helpers::find_and_preserve(output, ["pre"])
  else
    "<span style='color: red'>Error: #{name.upcase}.md doesn't exist in #{doc_path}</span>"
  end
end

This calls out to rdiscount to generate HTML from a Markdown document. You can use it easily in your templates:

%div#installation
  %div#install.generated
    = generate_docs_from_source('install')

Which will render INSTALL.md in your source code docs/ directory under div#install. This makes it easy to theme the generated docs, courtesy of the .generated selector.

The nanoc command line tool has a simple way to compile a site:

$ nanoc compile

Which is useful when you want to call a compile from other scripts. You can probably see where i'm going with this:

On the main project source repo i've configured a post-receive hook to POST to docs.flapjack-project.com, which is a Sinatra app that triggers a build of the website. Before it does the build, it checks out the latest copy of the project and website sources.

The nice thing about this approach is you can easily interpolate your own static docs with those from the source code:

%div#developing
  %h2 Source code
  %p  
    Flapjack's code is maintained in two repositories on GitHub:
    %ul 
      %li 
        %a{:href => "http://github.com/auxesis/flapjack/tree/master"}> flapjack
        , the core of the monitoring system.
      %li 
        %a{:href => "http://github.com/auxesis/flapjack-admin/tree/master"}> flapjack-admin  
        , the admin interface.
 
  %p  
    Flapjack is open source, released under the 
    = link_to "MIT Licence", "http://en.wikipedia.org/wiki/MIT_License"
    \.  
  
  %div.generated#developing
    = generate_docs_from_source('developing')

The code for those interested:

• Sinatra app, to trigger website builds
• nanoc website source, to compile the site.

Last year I blogged about where to get up-to-date Ruby packages for Ubuntu.

The PPA I suggested hasn't been updated in a while, and there are a few gems that require the latest version of RubyGems to work correctly (Rails, i'm looking at you).

I suggest you remove the PPA and add this to your /etc/apt/sources.list:

deb http://apt.brightbox.net/ hardy main

Sub hardy with intrepid/dapper depending on what release you need.

Then do an apt-get update && apt-get remove rubygems && apt-get install rubygems, and you'll be upgraded to the latest version of RubyGems.

As lots of people are apparently using cucumber-nagios and i'm getting a few bug reports, I thought it'd be worth setting up a proper home for the project.

You can find the project's site over here. Feature requests and bugs can be reported over at the GitHub issues page.

I apologise in advance if i'm a tad unresponsive when replying to new issues or emails about it. I'm backpacking around Europe until December so it's all a matter of finding suitable hacking time.

I pushed out a new release of cucumber-nagios last night. Things of note:

• It's now project focused. Use cucumber-nagios-gen to generate a project. The project provides infrastructure for freezing in dependencies, so you can zip up the project directory and migrate it between machines easily.
• It's released as a gem. You can install it with a plain old gem install cucumber-nagios. This will install everything you need to set up a project.

To get up and running with cucumber-nagios, this is what you need to do:

gem sources -a http://gems.github.com
gem install auxesis-cucumber-nagios
cucumber-nagios-gen project ebay.com.au
cd ebay.com.au 
rake deps

When you generate a project, it also spits out a .bzrignore and .gitignore, so there's no excuse not to be using version control!

All the previous documentation about writing and testing features still applies. I'm planning on adding support for generating features with cucumber-nagios-gen in the next release.

The code and documentation can now be found on GitHub. Launchpad was giving me no love.

James, Michael and I have been working on a small startup called Hardcopy Books for the last 3 months. Here's the elevator pitch:

Hardcopy Books is an online bookstore exclusively for tech books in Australia. How are we different?

1. We only stock tech books.
2. Your orders will be cheaper than through Amazon.
3. We aim to deliver within a week.

We started it because ordering tech books in Australia sucks. If you order from Amazon, the books are cheap, but they can take ages to get here, and shipping is very expensive. Most local book stores don't specialise in tech books, and the ones that do have a very slow turnaround and a horribly tedious ordering process. We aim to change that.

Today we're announcing the Hardcopy Books 2009 Launch Survey.

We're asking you, the local tech community, to fill it out and let us know what sort of books you buy. We're going to be launching with a select group of titles, and the feedback you give us will let us know what books we need to stock up on.

We're also on Twitter, so you can keep up to date with the launch on there.

Interesting thought experiment:

• Cucumber is kick arse way of describing the behaviour of a system.
• Webrat makes interacting with websites blindingly easy.
• Nagios is the industry standard for system/network/application monitoring.

What happens if you combine the three? You get cucumber-nagios.

cucumber-nagios takes the results of a Cucumber run and outputs them in the Nagios plugin format. What does that actually mean?

A sysadmin can describe the behaviour of a system that they manage:

Feature: google.com.au
  It should be up
  And I should be able to search for things

  Scenario: Searching for things
    When I visit "http://www.google.com"
    And I fill in "q" with "wikipedia"
    And I press "Google Search"
    Then I should see "www.wikipedia.org"

Then they can run the feature through cucumber-nagios:

$ cucumber-nagios features/google.com.au/search.feature
Critical: 0, Warning: 0, 4 okay | value=4.000000;;;;

The curious can check out the code on GitHub, and the documentation on the project website.

UPDATE: There have been a few changes to cucumber-nagios since this post. Check out these two posts for more info.

So I don't waste another 30 minutes of my life:

When using the select helper in Merb, make sure you call to_s on whatever you're setting :selected to, i.e.

select :name => "order[status]", :collection => OrderStatus.all, 
       :text_method => :description, :value_method => :id,
       :selected => @order.status.to_s

Otherwise the helper will compare a String (the value) to an Integer (:selected), and you'll never get anything in your select selected!

It's been interesting watching the flurry of activity in data-storage land over the last few months. CouchDB has been improving in leaps and bounds, and multi dimensioned data stores have been getting a lot more attention in general.

I started using Couch on a project a few weeks ago with the DataMapper adapter, specifically for scalability and search reasons. Migrating from my test SQLite database to Couch was a breeze, however it started getting ugly after a short while.

The main obstacle was that Datamapper's Couch adapter integrated pretty clunkily with DataMapper, particularly:

• You had to mix in DataMapper::CouchResource into your models instead of the bog standard DataMapper::Resource.
• The standard DataMapper finders didn't work, so you had to wrap all your queries in Couch views.

As much as this annoyed me, dm-couchrest-adapter's maintainer is a smart guy, so I knew there was a good reason behind it being that way even if it wasn't immediately apparent.

That good reason presented itself to me a few days ago when I started playing with dm-ferret-adapter to get full text search with Ferret on my models.

I was trying to work out how you do a multi-field search, but as with most things in the Ruby world, the documentation was lacking. The author of the dm-sphinx-adapter fortuitously posted on the mailing list about how his adapter handled the problem, so I went digging around inside dm-ferret-adapter's and dm-is-searchable's internals to work out why it wasn't behaving.

The crux of the problem was that DM was being too smart for its own good and tried to match fields listed in the the :conditions parameter to actual fields in the database, hence passing a big string in a :conditions would explode before it even hit the Ferret adapter.

And thus we've hit a fundamental problem with DataMapper's current implementation: at its core, it's still an ORM for relational databases - adapter authors are always going to be fighting an uphill battle when trying to integrate a non-relational data store.

So back in Couch land, I ended up switching to the CouchRest ORM to talk to Couch. Explaining why he wrote CouchRest as a standalone ORM, Chris Anderson stated...

(I could have written a DataMapper adapter for CouchDB, but much of DataMapper’s code is based around SQL-like problems that CouchDB just doesn’t have.)

... sounds just like the problems I was referring to.

Anyhow, why the title of this post?

Well last year I briefly hacked on some business banking code for Suncorp, and I was introduced to the wonderful world of UniVerse BASIC. I'm guessing that almost none of the readers of this blog have ever heard of UniVerse, but some may have heard of Pick.

Pick was a pre-Unix operating system and rapid application development environment initially released in 1965. Pick's killer feature was the MultiValue database (think hash table), and was specifically targeted at businesses and business analysts.

You're probably thinking "Woo, a hash table - why should I care about this Lindsay? My language already has Hashes/Dictionaries/HashMaps/filing cabinets". Well Pick's hash table implementation was (and still is) pretty kick arse for its time. There's a query language (that's suspiciously similar but slightly different to SQL), and it backs onto an incredibly well tested on-disk data store.

There's also no enforced schema, so it was particularly useful in the accounting world where relational databases with rigidly enforced schema aren't a good fit. Hence, there are a lot of financial applications out there written in Pick or a Pick derivative.

If you've ever worked with any of the EDIFACT data formats, you've indirectly worked with Pick. Those pesky separator/terminators ('+:?) are handled by multivalue databases really well. If you were to represent an EDIFACT segment how UniVerse would process it:

# EDIFACT segment
TVL+240493:1740::2030+JFK+MIA+DL+081+C'

To Python tuples and dictionaries:

( 'TVL', {'240493': ('1740', None, '2030')}, 'JFK', 'MIA', 'DL', '081', 'C')

To Ruby arrays and hashes:

[ 'TVL', {'240493' => ['1740', nil, '2030']},  'JFK', 'MIA', 'DL', '081', 'C']

By now you're thinking "oh god now I have to iterate over a whole bunch of nested data structures", but fortunately Pick's BASIC implementation provided syntax that made this pretty straightforward.

Anyhow, the MultiValue technology behind Pick was licensed to roughly 3 dozen companies during the 70's and 80's, but there's been a lot of consolidation in the Pick market since then, and the main player is now actually IBM. They purchased two implementations, UniVerse and UniData, in the 90's, re-branded them as U2 (sorry, no Bono here), and have been continually developing them ever since.

IBM have written .NET and Java interfaces to U2 data stores, there's integration with RedBack (a web application development framework), and more recently work has gone into PHP, Python, and Ruby bindings.

Pick's usage in the enterprise was and is still phenomenal. Last year at IBM's U2 University in Sydney the U2 product manager quoted a statistic that the U2 team estimate at least 60% of IBM's clients are directly using either UniVerse or UniData. A large majority of these systems are small back office-type setups that were installed decades ago, next-to-nobody touches, but are mission-critical.

So after seeing Tokyo Cabinet do the rounds this week in the Ruby sphere, it's pretty obvious that multi dimensioned data stores are experiencing a bit of resurgence.

Google's success with BigTable has kicked a lot of smart people into gear: CouchDB, HBase, and Tokyo Cabinet are shining examples of awesome work being done in the DBMS sphere.

What I think is going to make a difference this time:

• Implementations are not walled gardens. IBM's U2 products are not open source, and have a significant monetary barrier of entry. It's a problem that the entire Pick marketplace suffers, and why there isn't a lot of young talent in the Pick sphere anymore.
• The multi dimensioned data paradigm maps really well onto existing (and popular!) interchange formats. Take a look at JSON - its take up over the last few years has been impressive to say the least. YAML is another great example. They succeed where rigid data formats don't fit. Also, they're not the new EDIFACT.
• Developers are hitting barriers with RDBMSes. If there's one thing we can learn from the hype-fest that was "Web 2.0", it's that scalability is hard. Multi dimensioned databases aren't a magical elixir for the scalability problems of developers around the world, but they do prompt people to think of alternate ways of storing their data.

It'll be interesting to see whether the industry will start taking up multi dimensioned data stores en mass any time soon.

On Sunday I posted about a Twitter bot (@nswbushfires) I quickly hacked up to post current incident updates to Twitter.

People following the bushfires on Twitter may have noticed that the Rural Fire Service launched an official Twitter bot (@nswrfs) this morning containing information on major fire updates.

I had a brief chat with the Manager of Online Communications from the RFS this afternoon about the datasets available on their site, how data is generated within the RFS, and how that data can best be used.

Basically their bot aggregates major fire updates, which contain information on incidents that may directly affect people or property, and how people should respond. The announcements are a digital form of what gets syndicated to news outlets, and are crafted by the RFS communications team. Generally this information is up-to-the-minute.

On the other hand, my bot aggregates the list of current incidents, which is an extract of an internal RFS system used by people on the ground to track their handling of fires. The data in the current incidents list can potentially be several hours out of date, as it's quite often entered into their internal system after the incident has been handled. That said, it provides a state-wide overview of RFS activity and can be useful for tracking non-critical bushfire activity in your area.

So for people wanting to follow bushfire activity in NSW, I would highly recommend following both bots.

I just scraped together a Twitter bot to post updates from the NSW Rural Fire Service's Current Incidents list.

If you're in NSW, follow @nswbushfires on Twitter.

Almost a week ago today, Julia and I tied the knot.

You can find photos on Flickr, and highlights on Julia's brand new blog.

Gastro has been relaunched at gotgastro.com!

DNS issues should be all fixed, so go nuts and share the site.