When they KVM modules are loaded, they try and create a device called /dev/kvm. This fails pretty miserably since /dev/kvm is already a directory due to LVM shenanigans.

Not all is lost though if you’ve done a lot of setup like I had. You can rename VGs. Simply boot from your Ubuntu install CD, choose rescue mode and then jump into a shell. First you deactivate the LVs using

vgchange -a n

then you can rename the VG using

vgrename kvm kvmvg

Not sure whether I should file this problem as a bug. It is a bit of a weird situation.

johnf@zoot:~/dev/vquence/metrics/trunk$ sudo rm -rf /usr/lib
^C

The command only ran for a few seconds so the damage wasn’t to bad, but what did I lose?

The locate command came to my rescue. locate runs out of cron, usually once a day, and creates a database with a list of every file on your machine. You can then use it to search for files. So to work out what was missing I did the following.

# Get the list of files before we removed them
locate --regexp '.' > /tmp/before_rm
 
# update the locate database
sudo updatedb
 
# Get the list of current files on the system
locate --regexp '.' > /tmp/after_rm
 
# Create a list of what's missing
diff -u /tmp/before_rm /tmp/after_rm > /tmp/diff_rm
grep '^-' /tmp/diff_rm | sed -e 's/^-//' > /tmp/missing_rm
 
# Ask the dpkg system what packages those files belong to
for i in `cat /tmp/missing_rm`
do
    dpkg -S $i;
done | awk '{print $1}' | sed -e 's/:$//;s/,//g' > /tmp/packages
 
# Reinstall those packages
sudo aptitude reinstall `cat /tmp/packages`

After this process it is probably worth running the step from updatedb again to work out what is still missing.

For the record I lost 102 files and had to reinstall 97 packages.

Now back to real work!

Basically I wanted to create the equivalent of an Ubuntu PPA service running on our own servers so we could place it behind our firewall. This post is basically the process I followed to integrate rebuilld and reprepro to replicate a PPA setup.

So first up install reprepro

aptitude install reprepro

next we need do create a reprepro repository

mkdir -p /srv/reprepro/{conf,incoming,incomingtmp}

Now we need to tell reprepro which distributions we care about. Create /srv/reprepro/conf/distributions with the following contents

Suite: hardy
Version: 8.04
Codename: hardy
Architectures: i386 amd64 source
Components: main
Description: Local Hardy
SignWith: repository@inodes.org
DebIndices: Packages Release . .gz .bz2
DscIndices: Sources Release .gz .bz2
Tracking: all includechanges keepsources
Log: logfile
  --changes /srv/reprepro/bin/build_sources

Suite: intrepid
Version: 8.10
Codename: intrepid
Architectures: i386 amd64 source
Components: main
Description: Local Intrepid
SignWith: repository@inodes.org
DebIndices: Packages Release . .gz .bz2
DscIndices: Sources Release .gz .bz2
Tracking: all includechanges keepsources
Log: logfile
  --changes /srv/reprepro/bin/build_sources

Suite: jaunty
Version: 9.04
Codename: jaunty
Architectures: i386 amd64 source
Components: main
Description: Local Jaunty
SignWith: repository@inodes.org
DebIndices: Packages Release . .gz .bz2
DscIndices: Sources Release .gz .bz2
Tracking: all includechanges keepsources
Log: logfile
  --changes /srv/reprepro/bin/build_sources

I also like to create reprepro options file to setup some defaults, edit /srv/reprepro/conf/options

verbose
verbose
verbose
verbose
verbose

Next we need to setup an incoming queue so that we can use dput to get the source packages into reprepro,
vi /srv/reprepro/conf/incoming

Name: incoming
IncomingDir: incoming
Allow: hardy intrepid jaunty
Cleanup: on_deny on_error
Tempdir: incomingtmp

The repository is now ready to go. So now we can setup apache. Edit /etc/apache/sites-enabled/pppa

<virtualhost *:80>
    ServerName packages.inodes.org
    DocumentRoot /srv/reprepro
</virtualhost>

and we should also configure our sources.list to use these repositories, edit /etc/apt/sources.list

# Sources for rebuildd
deb-src http://packages.inodes.org hardy main
deb-src http://packages.inodes.org intrepid main
deb-src http://packages.inodes.org jaunty main

Next we want to setup our dput.cf to make the magic happen to get the source packages into the archive, edit ~/.dput.cf

[DEFAULT]
default_host_main = notspecified

[local]
fqdn = localhost
method = local
incoming = /srv/reprepro/incoming
allow_unsigned_uploads = 0
run_dinstall = 0
post_upload_command = reprepro -V -b /srv/reprepro processincoming incoming

So now we can do the following

apt-get source squid3
cd squid3*
dch -i # increment version number
dpkg-buildpackage -sa -S
cd ..
dput local *changes
aptitude update
apt-get source squid3

So when you run dput, first it copies the source package files to /srv/reprepro/incoming and then it gets reprepro to process it’s incoming queue. This means that the source package is now sitting in the repository.
So the second apt-get source should have downloaded the source package from our local repository which is exactly what rebuildd will do before it tries to build it.

Next step is to setup rebuildd so that it builds the binary packages and installs them into the repository.

aptitude install rebuildd

Setup so it runs out of init.d and the releases we care about, edit /etc/default/rebuildd

START_REBUILDD=1
START_REBUILDD_HTTPD=1
DISTS="hardy intrepid jaunty"

Now when a source package is uploaded into the repository we want to kick off rebuildd to build the package. We can do this through the reprepro log hooks. You’ll notice in the conf/distributions above the following lines.

Log: logfile
  --changes /srv/reprepro/bin/build_sources

This script will be run any time a .changes file is added to the repository. Create /srv/reprepro/bin/build_sources

#!/bin/bash
 
action=$1
release=$2
package=$3
version=$4
changes_file=$5
 
# Only care about packages being added
if [ "$action" != "accepted" ]
then
	exit 0
fi
 
# Only care about source packages
echo $changes_file | grep -q _source.changes
if [ $? = 1 ]
then
	exit 0
fi
 
# Kick off the job
echo "$package $version 1 $release"  | sudo rebuildd-job add

This script basically checks the right type of package is being added. Then it calls rebuildd-job to ask for that specific package and version to be built for that Ubuntu release.

Now the first thing that rebuildd does is download the source for the package. However we need to update the sources first since our server doesn’t know there are new files in the repository yet. So edit /etc/rebuildd/rebuilddrv an change

apt-get -q --download-only -t ${d} source ${p}=${v}

to

source_cmd = /srv/reprepro/bin/get_sources ${d} ${p} ${v}

and create /srv/reprepro/bin/get_sources with

#!/bin/bash
 
d=$1
p=$2
v=$3
 
sudo aptitude update >/dev/null
apt-get -q --download-only -t ${d} source ${p}=${v}

By this stage we have rebuildd building packages but we need to make sure they get re-injected back into the repository. We can do this with a post script. Edit /etc/rebuildd/rebuilddrc

post_build_cmd = /srv/reprepro/bin/upload_binaries ${d} ${p} ${v} ${a}

and create /srv/reprepro/bin/upload_binaries

#!/bin/bash
 
d=$1
p=$2
v=$3
a=$4
 
su -l -c "reprepro -V -b /srv/reprepro include ${d} /var/cache/pbuilder/result/${p}_${v}_${a}.changes" johnf

Now the su is in there because rebuildd needs to be able to access the GPG passphrase to sign the repository with. So rather than have a passphrase-less key we make sure that gpg-agent is running by adding the following to your .profile.

if test -f $HOME/.gpg-agent-info &&    kill -0 `cut -d: -f 2 $HOME/.gpg-agent-info` 2>/dev/null; then
	GPG_AGENT_INFO=`cat $HOME/.gpg-agent-info`
	export GPG_AGENT_INFO
else
	eval `gpg-agent --daemon`
	echo $GPG_AGENT_INFO >$HOME/.gpg-agent-info
fi
 
GPG_TTY=`tty`
export GPG_TTY

So that’s it you now have your own personal PPA. Just in case you had fallen asleep. Here is a little script I wrote so you can auto build the source packages for each release you care about in one go.

#!/bin/bash
 
set -e
 
RELEASES="hardy intrepid jaunty"
 
if [ ! -f debian/changelog ]
then
	echo "This isn't a debian repo"
	exit 1
fi
 
# Check for changes
if [ `bzr st | wc -l` != "0" ]
then
	echo "You have uncommitted changes!"
	exit 1
fi
 
if [ -d ../tmpbuild ]
then
	echo "The tmpbuild dir exists"
	exit 1
fi
 
bzr export ../tmpbuild
cp debian/changelog ../tmpbuild.changelog
cd ../tmpbuild
 
PACKAGE=`head -1 debian/changelog | awk '{print $1}'`
VERSION=`head -1 debian/changelog | awk '{print $2}' | sed -r -e 's/^\(//;s/\)$//'`
 
for release in $RELEASES
do
 
	sed -r -e "1s/\) [^;]+; /~${release}) ${release}; /" ../tmpbuild.changelog > debian/changelog 
	head -1 debian/changelog
	dpkg-buildpackage -S -sa
	dput local ../${PACKAGE}_${VERSION}~${release}_source.changes
done
 
cd ..
rm -rf tmpbuild

So the above documentation is a bit of a brain dump on what I’ve been working on for the past 2 days and I’m sure I’ve left some bits out. So please give me any feedback you have in the comments.

After hours of searching I finally decided on the USB SATA Drive Dock .

Again a big thank you to the LA council and to Steve Walsh for organising the gift certificate.

This is what we call the Melbourne Cup Network Effect

It seems it really is the race that stops the nation. This is a graph of Bulletproof’s outbound web traffic for today. That’s a 37% drop in outbound traffic just after 3pm.

Make sure you take note of my l33t gimp skills!

While running around fixing up all our servers, most of them in one felll swoop thanks to puppet, I realised two of our servers were still running Edgy. I figured it was high time I moved them to Hardy.

Everything went fairly smoothly with some minor hicups, except for SMTP-AUTH for exim. We use an ldap backed SMTP-AUTH and this just wouldn’t work after the upgrade. The following error was appearing in the logs.

ldap_search failed: -7, Bad search filter

This lead to hours upon hours of google searches, staring at debug messages and even at one stage resorting to using GDB. Eventually after staring at debug messages harder it twigged when I saw the following.

perform_ldap_search: ldapdn URL = "ldap:///ou=people,o=vquence?dn?sub?(uid=moo) "

Notice the space just before the closing double quote. It seems that the new openldap libraries don’t like errant spaces in your search filter.

Now to remember what I was doing yesterday morning before this whole derailment began.

Note: Before anyone comments I will completely deny that during these upgrades I did anything as silly as rm -rf `dpkg -L random-font-package`, no matter what twitter says.

passwd -l root

In gutsy the above would simply lock the account by placing an ! in front of the passwd in your /etc/shadow file.

In hardy it now also sets the account as expired. Meaning you can’t ssh to it even if you have SSH keys in place.

Time to go and rebuild my EC2 AMI.

Update: To get the old behavour back you can do the following

passwd -l root
usermod -e "" root

To ensure a minimum setup time on a new server I wanted the setup to be as simple as

• echo ‘DAEMON_OPTS=”-w 120 –fqdn newserver.vquence.com –server puppetmaster.vquence.com” > /etc/default/puppet
• aptitude install puppet

This means that the puppet client will use newserver.vquence.com as the common name in the SSL certificate it creates for itself. On the puppet master the SSL cert name is then used to pick a node rather than the hostname reported by facter.

This means that I don’t need to worry about setting up /etc/hostname, even better /etc/hostname can be managed by puppet.

You can control this functionality on the puppet master by using the node_name option. From the docs

    # How the puppetmaster determines the client's identity
    # and sets the 'hostname' fact for use in the manifest, in particular
    # for determining which 'node' statement applies to the client.
    # Possible values are 'cert' (use the subject's CN in the client's
    # certificate) and 'facter' (use the hostname that the client
    # reported in its facts)
    # The default value is 'cert'.
    # node_name = cert

The problem was that the ‘hostname’ fact wasn’t being set. It looks like there was a regression in SVN#1673 when some refactoring was performed.

I’ve filed bug #1133 and you can clone my git repository.

I haven’t included any tests in the patch as I’m not sure how to. The master.rb test already tests this functionality but doesn’t test that the facts object has actually been changed. I think a test on getconfig is probably required but I’m not sure how you would access the facts after calling it.

Update: This patch is now in puppet as of 0.24.3.

This works without any problems when using the java based command line tools. However I was getting the following error when using the EC2 Ruby GEM.

/usr/lib/ruby/1.8/net/protocol.rb:133:in `sysread': Connection reset by peer (Errno::ECONNRESET)
	from /usr/lib/ruby/1.8/net/protocol.rb:133:in `rbuf_fill'
	from /usr/lib/ruby/1.8/timeout.rb:56:in `timeout'
	from /usr/lib/ruby/1.8/timeout.rb:76:in `timeout'
	from /usr/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill'
	from /usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
	from /usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
	from /usr/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'
	from /usr/lib/ruby/1.8/net/http.rb:2009:in `read_new'
	 ... 6 levels...
	from ./lib/ec2helpers.rb:43:in `start_instance'
	from ./ec2-puppet:107
	from ./ec2-puppet:89:in `each_pair'
	from ./ec2-puppet:89

Doing some tcpdumping indicated that after receiving the request Amazon waits for a while and then sends a TCP RESET. Not very nice at all. My next step was to use ngrep to compare the output from the command line tools and the ruby gem. This got nowhere fast since the command line tools use the SOAP API while the ruby gem uses the Query API.

What I did notice however is that while the command line tools performed a POST the ruby library performed a GET. At this stage I decided to test how much data I could send. So I started trying different user data sizes. The offending amount was around 7.8k, suspiciously close to exactly 8k.

The HTTP/1.1 spec doesn’t place an actual limit on the length but leaves it up to the server.

The HTTP protocol does not place any a priori limit on the length of
a URI. Servers MUST be able to handle the URI of any resource they
serve, and SHOULD be able to handle URIs of unbounded length if they
provide GET-based forms that could generate such URIs. A server
SHOULD return 414 (Request-URI Too Long) status if a URI is longer
than the server can handle (see section 10.4.15).

Note: Servers ought to be cautious about depending on URI lengths
above 255 bytes, because some older client or proxy
implementations might not properly support these lengths.

Apache for example limits this by default to 8190 bytes including the method and the protocol. You can change this using the LimitRequestLine directive.

I created a patch to modify the EC2 Gem to use a POST instead of a GET which has no such limitations. You can find the git tree for it at http://inodes.org/~johnf/git/amazon-ec2

------------     ---------     ------------
| Internet |---->| Squid |---->| Mongrels |
------------     ---------     ------------

(Who needs Inkscape when you have ASCII art)

This infrastructure is hosted in the US and up until recently squid hadn’t been doing much of anything except really sitting there.

Now a few months ago when we signed a contract with an Australian customer we decided we needed to place a squid cache in Australia which would actually cache content. For two reasons, firstly the US is a long way away and the 300ms latency is really noticeable and secondly because some of our pages involving graphs have long statistical calculations which can take minutes to render. (OK its really because no one has had a chance to optimise them yet but lets pretend that’s not the case). So we changed the above setup for the Australian customers to look like the following.

------------     ------------     ------------     ------------
| Internet |---->| Squid AU |---->| Squid US |---->| Mongrels |
------------     ------------     ------------     ------------

We hand out urls like http://www.client.b2b.vquence.com/widget to Australian customers and the rails backend is smart enough to make sure all the URLs look similar (I’ll blog about how I did that another time).

Without much time to look into thing properly I did some really nasty things on the AU squid cache to make sure it cached the pages.

refresh_pattern /client/graph  1440    0%    1440    ignore-no-cache ignore-reload
refresh_pattern /client/static 1440    0%    1440    ignore-no-cache ignore-reload
refresh_pattern /client/video  1440    0%    1440    ignore-no-cache ignore-reload

This is evil, breaks a whole heap of RFCs but it did the trick and got us out of a bind quickly.

A few weeks ago I moved the production site to Rails 2.0, I noticed around this time that the caching had stopped working. The client was no longer using our services as their campaign had finished so it wasn’t an urgent concern.

It seems that Rails 2.0 goes one step further to ensure that caches don’t cache content and instead of just sending

Cache-Control: no-cache

it now sends

Cache-Control: private, max-age=0, must-revalidate

I tried adding ignore-private, since if you’re breaking some aspects of the RFC you may as well break a couple more, but squid still refused to cache the content. After struggling with this for a bit I decided that the universe was trying to tell me I should actually do things properly.

So with squid set back to its defaults I went exploring how to accomplish this. Google wasn’t all that helpful at first since most Rails caching articles talk about caching to static files as most sites don’t implement reverse proxying for caching. It turns out however its fairly simple. In the appropriate actions in your controllers simply do the following.

class VideoController < ApplicationController

    def vquence
        # Lots of code here

        expires_in 8.hours, :private => false
        render :template => "videos/vquence"
    end

end

This will send the following header and cache the page for 8 hours.

Cache-Control: max-age=28800

Now everything is much faster!!