Skip to main content

2014 (old posts, page 1)

puppet masterless server automation on a very small scale

The benefits of server automation are well-known; the ability to swiftly build and maintain a group of servers provides a competitive advantage, reducing overhead of setups, minimising likelihood of errors, and generally keeping the bill for headache-tablets low. One brilliant and popular such tool is Puppet, which for a group of servers can be run with a ‘Puppet master’ controlling related servers. But what about automation on a very small scale—say just a couple of boxes? Puppet can also be run in ‘masterless’ or ‘local’ mode, in which approach manifests are applied individually without centralisation. In these notes I highlight the main steps I tend to take when setting up an isolated server to run using this method.

Throughout, I assume a target distribution of Ubuntu Server 12.04 LTS; instructions for other environments should be similar.

bootstrapping the server

As the scale is small, the new server can be bootstrapped by hand. Create the server using your host of choice. If possible, avoid a situation where a root password is sent to you and go straight for an SSH key—but this might not be possible, depending on your host. If you have to take an unsecure route, I recommend following these instructions and establishing a process and manifest that you’re happy with, then destroying the instance, changing the passwords, and building all over again; the build is supposed to be repeatable, after all, and this way you’ll minimise the time that your box is live with security settings which upset you.

Set up your DNS as required.

Update packages and reboot the system, to load any new kernel. I like to do this as the first step so I know I’m working from an up-to-date distribution.

apt-get update && apt-get upgrade -y && reboot

Install up-to-date puppet (not puppetmaster), altering as needed if not using Ubuntu Server 12.04 LTS (‘Precise’).

wget && dpkg -i puppetlabs-release-precise.deb && apt-get update && apt-get install -y puppet

That’s it; everything else is configuring your manifest and applying it on the server.

creating the puppet manifest

For even a single server, I store the Puppet manifest in a private Git repository; that way, I can track changes over time, and also use it as part of the automation process. The basic repository structure can be something as simple as


puppet.conf is the basic configuration file for your Puppet installation; I lift one of these from the installed Puppet, and version that with the rest. modules/ holds any Puppet modules used by your manifest; I tend to use Git Submodules for these. manifests/site.pp is the manifest for the box you are automating. I tend to use a mixture of third-party and private modules, meaning I can easily set up security as required and include SSH keys and such on the box.

# manifests/site.pp

# Firewall

class { 'tp_firewall': }

# Apt

class { 'apt': }
class { 'apt::unattended_upgrades': }

# Users

user { 'root':
  ensure     => 'present',
  password   => 'PASSWORD_HASH_ROOT',

user { 'eguser':
  ensure     => 'present',
  groups     => ['sudo'],
  password   => 'PASSWORD_HASH_EGUSER',
  managehome => true,
  shell      => '/bin/bash',


class { 'tp_ssh': }

tp_ssh::ssh_authorized_key_tp { 'root': }
tp_ssh::ssh_authorized_key_tp { 'eguser': }

tp_firewall is a private module which locks down ports using Puppet Firewall. apt is set up with unattended upgrades, so security patches get applied automatically (this might not be best for you if you want to vet things more carefully). user sets up accounts using the Puppet user type. PASSWORD_HASH_* are set using openssl passwd -1 locally; I change the root password on the first run. tp_ssh is a private module which applies my custom /etc/ssh/sshd_config locking down the SSH installation, and places SSH keys using the Puppet ssh_authorized_key type.

applying the puppet manifest

All that remains is to apply the Puppet manifest itself. For this, use your deployment tool of choice, deploying the manifest with any submodules, and triggering the Puppet run after-deploy. I usually deploy straight to /etc/puppet/, and log to both stdout and a file.

sudo bash -c "puppet apply --logdest /dev/stdout /etc/puppet/**/*.pp | tee -a /var/log/puppet/"

Obviously, this is merely a basic automation. But for situations where you find yourself running a couple of isolated servers on a very small scale, something along these lines should enable you to gain many benefits whilst keeping things simple and cost-effective.

tiredpixel ☮

tunefl 1.4.4 LilyPond Ruby released

tunefl: LilyPond mini-score engraving and sharing service for musicians.

I am pleased to announce tunefl 1.4.4—a release upgrading small things. As a general celebration and overhaul of the tunefl project, I also today announce upgraded tunefl servers. :)

tunefl now has 5 GitHub stars, 1 GitHub fork, and has engraved 2694 valid mini-scores, which are easily shareable on your social network of choice.

As ever, I render grateful thanks to the LilyPond project contributors (, whose continued hard and excellent work makes such a simple, yet useful, service as tunefl possible.


  • start of support for Ruby 2.1.0; end of support for other versions (please wave if this upsets you :) )
  • upgrade of gems
  • timeout long-running compile jobs

tiredpixel ☮

ryespy 1.0.0 Ruby gem released

ryespy: Redis Sidekiq/Resque IMAP, FTP, Amazon S3, Google Cloud Storage, Rackspace Cloud Files listener.

I’m pleased to announce ryespy 1.0.0—the first major release, adding support for Amazon S3, Google Cloud Storage, Rackspace Cloud Files, fixing various things, and freezing the Redis key structure. (Thank you, Fog, for making this support so straightforward. :) )


  • first major release; Redis key structure frozen
  • Redis key structure backwards-incompatible with 0.x.x (sorry! :( )
  • start of support for Ruby 2.1.0
  • end of support for Ruby 1.9.2
  • new Amazon S3 listener (--listener amzn-s3)
  • new Google Cloud Storage listener (--listener goog-cs)
  • new Rackspace Cloud Files listener (--listener rax-cf)
  • change of --verbose mode to --debug mode
  • broader error-catching, in case weird things happen when --eternal
  • missing FTP listener (--listener ftp) --ftp-port fix
  • dynamic requiring of listeners (some have their own dependencies)
  • comprehensive README with lots of examples
  • major refactoring and improvement of code throughout
  • a plethora of tests; most of the core is now covered

tiredpixel ☮

chocolate muffin

i ponder the chocolate muffin’s demise
it feels a little sad :(
but it strikes me that
it is being converted into energy and fulfilling its fate
much like people?
(the allegorical muffin ↦ the allegorical muffin’s energy field)
(*dedicated to T.)

o song, from ancient wells eternal sprung

o song, from ancient wells eternal sprung
through wandering time a coloured thread
the her, the him, the echoed chord
be ever-present, ever-sung
as we do journeys slowly make

and may there listeners always be
to soundless resonance of form
the shade of light, the breath of day
a half-dreamed thought within a dream
forget us not who wearied be

and when at last our shape we lose
now woven back within the core
defined again but as the whole
forgive us for the troubled notes
we cast within the sacred scores

curses! conditional ruby gem installation within a gemspec

Update 2014-03-26: curses 1.0.1, containing ruby/curses#4, removes the need for this workaround. Hurrah!

From Ruby 2.1.0 onwards, curses has been removed from the Ruby standard library and extracted to the curses Ruby gem. For sidekiq-spy, the Ruby gem providing Sidekiq monitoring in the console, that necessitated some enthusiastic dancing using conditional Ruby gem installation within a gemspec, to maintain support for Ruby 1.9.3 and Ruby 2.0.0.

To use curses in Ruby 1.9.3 and Ruby 2.0.0, all that is needed is require ‘curses’. In Ruby 2.1.0, specifying the dependency gem ‘curses’ lets you carry on unharmed. But the curses gem does not appear to work properly in Ruby 1.9.3. Thus, the Ruby gem can be installed conditionally, perhaps for Ruby 2.1.0 onwards. This same approach could be used to install different gems or versions based on which way the wind is blowing, or the Ruby version used. In a non-gem project, this could be accomplished using

# Gemfile

gem 'curses', :platform => :ruby_21

or, because the :ruby_21 platform does not appear to exist in Ruby 1.9.3, maybe something like

# Gemfile

gem 'curses' if RUBY_VERSION >= '2.1'

For a gem, however, this would only be evaluated at build-time, not install-time. The Wikibook Ruby Programming has an excellent chapter entitled RubyGems. This describes an approach where the gem is installed within the umbrella of native extensions building. I found that not all of this appeared to be necessary for my use-case, but the approach seems to work well.

First, register the extension in the gemspec:

# sidekiq-spy.gemspec

spec.extensions << 'ext/mkrf_conf.rb'

Then, create the extension file itself. In my case, I wanted to install curses only for Ruby 2.1.0 or later, but this can easily be adapted.

# ext/mkrf_conf.rb

require 'rubygems/dependency_installer'

di =

  if RUBY_VERSION >= '2.1'
    puts "Installing curses because Ruby #{RUBY_VERSION}"

    di.install "curses", "~> 1.0"
    puts "Not installing curses because Ruby #{RUBY_VERSION}"
rescue => e
  warn "#{$0}: #{e}"


puts "Writing fake Rakefile"

# Write fake Rakefile for rake since Makefile isn't used, 'Rakefile'), 'w') do |f|
  f.write("task :default" + $/)

The puts messages won’t usually show up during installation, but are useful for gem install using the -V flag. This should be all that’s necessary for handling the dependency at install-time, in this case installing the curses gem for Ruby 2.1.0 or later and not otherwise.

Next, ensure that the gem is loaded before being required. I found a variety of information online saying to use a variant of gem ‘curses’ whilst rescuing Gem::LoadError, but this did not appear to be necessary in my case. Merely requiring as before seems to be working fine (spaghetti crossed).

However, this still doesn’t handle the dependency for development-time, without which, curses isn’t available with Ruby 2.1.0. For this, a conditional development-dependency declaration ensures that the curses gem is available, whilst having no effect on install-time (at least, with the usual bundle install --without development test). Alternatively, the Gemfile could be used similarly. As mentioned, I did not have success with using the :platform => ruby_21 approach for Ruby 1.9.3.

# sidekiq-spy.gemspec

if RUBY_VERSION >= '2.1'
  spec.add_development_dependency "curses", "~> 1.0"

Of course, it’s very possible that there are better ways of doing all this. If you know of one, I would be very interested to hear it. :)

tiredpixel ☮