LCA2011

We are now well and truly into the run up to linux,conf.au 2011. The LCA Call for Papers, Miniconfs, Posters and Tutorials (that is quite a mouthful!), has been open since the 13th of July. The CFP closes at midnight tonight, so it’s still not too late to get a proposal in.

Everyone should spend the next 5 minutes thinking of the one person or topic they would love to be able to listen to at LCA next year. Now, go and email that person and convince them to submit something. But hurry, you don’t have long.

The Paper Review Committee will be performing an online review of all the papers over the next 2 weeks. They will then meet in Sydney for a final one day review to decide what makes it into the conference. I’ve been involved in the process for the last few years, and I can tell you that it is not an easy process. The quality of the submissions we receive for LCA each year is extremely high and it is a very difficult task to whittle down 200-300 submissions into the 90 or so proposals we have space for.

LCA2012 Bid process

Submissions for LCA2012 close on the 15th of August, which is tomorrow night! So far we have had an expression of interest from Ballarat, and the odd rumour that other cities also have some teams thinking about it.

Once the bids come in, the council will take time to review them, and then we will begin visiting each team so that they can pitch their bid to us in person why they should earn the honour of hosting the next LCA.

This year we have changed the process slightly and asked all the teams to post their submission publicly. I’m looking forward to reading the proposals and having a healthy community discussion about which city should host LCA.

Software Freedom Day

Software Freedom Day is just around the corner, being held on the 18th of September. SFD is a worldwide celebration of FOSS and also serves to educate the general public about the benefits of FOSS.

According to the SFD website, it looks like we have about 6 teams registered in Australia. Noticeably missing are most of our capital cities. Please bring up SFD at your next LUG meeting or on your LUGs mailing list and try to organise an event in your area.

This year Linux Australia will be assisting SFD teams by providing schwag from past LCAs to give away at events. You should see an email to the list with more details about this shortly.

Also don’t forget that when you register your team on the official SFD website, Software Freedom International will send out SFD schwag for you to use on the day.

Australian Treasury Department, SBR and Auskey (Update)

As I mentioned last month, I’ve been doing some work in my capacity as President as well as my day job in regards to creating an Open Source project around the Australian Treasury’s Standard Business Reporting (SBR) project.

We recently held a meeting with some representatives from the Department of Treasury, where we were able to discuss our plans and what is required to make SBR and Auskey available for the Open Source community. SBR have shown a keen interest in the project and have been quite helpful in making resources and people available to help us with the project.

SBR have also recently announced that they will be supporting Linux on the AusKey website. This has not been possible up till this point as a browser plug-in is required to be able to interact with Auskey. SBR hopes to have a solution released by the end of the year and will be initially supporting Ubuntu. This means that Australian businesses using Open Source Software will soon be able to submit their BASs online again.

There has been the odd murmur that Linux Australia is not doing a good enough
job of communicating with the community, and I would have to agree. While we
are sending out meeting minutes every fortnight, I think we are lacking a more
direct communication as to what the Council and Linux Australia are up to. To
that end, I would like to initiate a monthly President’s report to try and get
the word out as to what we are doing.

Changes to the Council

First of all, on behalf of the Council and the rest of the community, I would
like to send a large Thank You to James Turnbull for all the work he did during
the first half of the year in his role as president. I would especially like to
draw attention to the work that James put into the Linux Australia Membership
Survey, results of which we plan to release in the next month. James will be
sorely missed, and we wish him all the best in his future endeavours in
Portland.

I’d also like to welcome Joshua Hesketh to the Council. Josh is already doing a
wonderful job as treasurer, as well as our liaison with the LCA2011 team.

Australian Treasury Department, SBR and Auskey

As many of you may be aware, the Australian Treasury has just released a new
project called Standard Business Reporting (SBR). This project aims to
standardise reporting to government, with an aim to becoming a centralised
point where business can submit forms to government. In essence, it is an API
which will allow standard government documents, like a BAS or employment
declaration, to be filed electronically. At the moment the ATO, ASIC and
various Offices of State Revenue are involved in the project. However, there is
a large amount of interest from other departments, like Medicare and
Centerlink. Hand-in-hand with this project is another sub-project called
AusKey, which is an all-of-government PKI system that is already beginning to
replace the existing ECI system used at the ATO to authenticate BAS filing.

A few months ago, I was contacted by Ron Skeoch from Muli Management. Muli have
been involved in the Open Source community for a number of years, and support a
piece of accounting software targeted at the construction industry. Muli need
to have their software support the SBR system, and they were interested in my
assistance; firstly helping them write the software to interface with the SBR,
but secondly in assisting them create this as a fully fledged open source
project that other projects could then use. At this stage, I put my Linux
Australia hat on, and indicated that we would like to work together with Muli
to help make that happen.

While this process is still at an early stage, we have already submitted a
document to Treasury outlining the requirements for the Open Source community
to be able to interact with SBR. We also pointed out the current issues with
AusKey in relation to being able to file a BAS. The response from Treasury has
been very promising, and they are quite eager to work with Linux Australia and
Muli to try and aid the Open Source community in any way they can; including
potentially even placing the reference clients under an appropriate license, so
that we can make use of them.

*For purposes of transparency I would like to point out a potential
conflict of interest here. Muli Management is a customer of my business and has
engaged me to among other things write the code and help create the open source
project.*

LCA2011

Preparations for linux.conf.au 2011 in Brisbane are well under way. Some
members of the Council, along with past LCA organisers and the new LCA team,
met for Ghosts in April in Brisbane. This was an extremely valuable experience
where past organisers were able to pass on some wisdom, and the current team
was able to pass on some of the ideas they have in store for us next year. The
meeting was held at the venue itself, where we were able to take a short tour
of where the conference will be held as well as some of the surrounding areas.
I have a lot of confidence that Shaun and his team are going to put together an
excellent conference. The Call for Papers should open shortly, so now is the
time to start thinking about the presentation you want to give at the next LCA.

LCA2012 Bid process

We recently announced our request for formal submissions for hosting
linux.conf.au 2012. So far we have an official expression of Interest from
Ballarat, and I have heard the odd rumour of goings on in Sydney and Canberra.
Submissions close on August 15th, just over a month away. That is still plenty
of time to put in a bid for the conference. If you think you might have it in
you, but need some co-conspirators, then please feel free to send the Council a
quick email. We may know of people in your area who are in the same position
and can help put you in touch with each other.

Media Sub-Committee

One area in which we have been lacking recently is getting our message about
things we care about out effectively to the media. This is in relation to
events we are holding, announcements about linux.conf.au and opinions on
relevant issues. The idea of a media sub-committee was originally raised at the
Face to Face meeting in February although it is not a new idea. There was a
press team once upon a time; the mailing list even still exists! I’ve asked
James Purser to put together a team and a framework for it to work in, so that
not too great a burden is placed on any one member. If you are interested in
helping out with media related activities, whether on twitter or with media
organisations directly, please get in touch with James.

Linux Australia Membership Survey

As mentioned above, we recently ran a survey of Linux Australia Members. The
survey was aimed at the Australian FOSS community and our aim was to gather
information to aid us in making decisions about what Linux Australia is, and
the directions that it should take as an organisation. We had an excellent
response with 528 submissions, including three people claiming to be Linus
Torvalds. The Council is working at the moment on collating all of the results.
Our plan is to release all of the anonymised raw data to the community in the
next month. It is our hope that the community will help us in spending some
time to analyse the data and tell us what they think it means. In due course,
the Council will present some analyses of its own.

Events

We recently had two very successful events which were supported by Linux
Australia. The first was PyCon AU 2010, this is the first time that this event
has been run in Australia and was possible due to the hard work of Tim Ansell,
Neil Davenport and Richard Jones. I hear that the event was a tremendous
success, and sold out before close of registrations. A few attendees I’ve
talked to were very excited and can’t wait for next years conference. The
conference is running on a model of the same team running it twice in a row in
the same city and a formal request for bids to host PyCon AU 2012-2013 in the
next few months.

The other event was the Sydney Education Expo. The Linux Australia stand at
this event was organised by Patrick Elliott-Brennan who did a wonderful job in
preparing everything required for the stand at the expo. Sridhar Dhanapalan
also assisted in his role as Technical Manager at OLPC Australia, who shared
the stand with us and provided some sponsorship.

That’s all for this month. It feels like we’ve been fairly busy. Hopefully I’ll
have just as much to write about next month. See you then!

I’ll be giving a presentation entitiled Commit early, Deploy often. I’ll be talking about using package management to empower developers to deploy applications locally just as they would in production. This also means sysadmins can deploy using the exact same environment.

There are still a few tickets left, so if you are in Sydney this weekend and are either a developer or a sysadmin then make sure you come along.

Disclaimer: I’m also sponsoring the event.

I remembered listening to a Ruby5 podcast which mentioned a great little gem called Delorean.

Delorean easily allows you to mock time in your tests. In no time I had hooked it up to cucumber.

In features/support/delorean.rb:

require 'delorean'  
 
# Make sure we fix the time up after each scenario
After do
  Delorean.back_to_the_present
end

and then in features/step_definitions/delorean_steps.rb

Given /^I time travel to (.+)$/ do |period|
  Delorean.time_travel_to(period)
end

this lets me create steps like

  Scenario: Link attributes are correct for yesterday
    Given I time travel to 2010-02-01 05:00
    When I GET the videos keywords feeds page
    Then I should see "start_time=2010-02-01T00:00:00"

Some other examples you can use with Delorean are

• 2 weeks ago
• tomorrow
• next tuesday 5pm

You can find more examples in the Chronic gem documentation which Delorean uses to achieve this functionality.

When they KVM modules are loaded, they try and create a device called /dev/kvm. This fails pretty miserably since /dev/kvm is already a directory due to LVM shenanigans.

Not all is lost though if you’ve done a lot of setup like I had. You can rename VGs. Simply boot from your Ubuntu install CD, choose rescue mode and then jump into a shell. First you deactivate the LVs using

vgchange -a n

then you can rename the VG using

vgrename kvm kvmvg

Not sure whether I should file this problem as a bug. It is a bit of a weird situation.

CD-ROM is in ISO 9660 format
System id: LINUX
Volume id: Ubuntu-Server 9.10 i386
Volume set id:
Publisher id:
Data preparer id:
Application id: GENISOIMAGE ISO 9660/HFS FILESYSTEM CREATOR
   (C) 1993 E.YOUNGDALE (C) 1997-2006 J.PEARSON/J.SCHILLING
   (C) 2006-2007 CDRKIT TEAM
Copyright File id:
Abstract File id:
Bibliographic File id:
Volume set size is: 1
Volume set sequence number is: 1
Logical block size is: 2048
Volume size is: 327972
El Torito VD version 1 found, boot catalog is in sector 1804
Joliet with UCS level 3 found
Rock Ridge signatures version 1 found
Eltorito validation header:
    Hid 1
    Arch 0 (x86)
    ID ''
    Key 55 AA
    Eltorito defaultboot header:
        Bootid 88 (bootable)
        Boot media 0 (No Emulation Boot)
        Load segment 0
        Sys type 0
        Nsect 4
        Bootoff 704 1796

/.disk
/README.diskdefines
<snip>File system listing</snip>

Looks like less runs isoinfo -f -R -J -i ubuntu-9.10-server-i386.iso. So I did have the tools I needed installed, I just didn’t know it yet .

I’m running for the position of Treasurer, but you don’t need to vote for me since I’m running unopposed.

I’m running on a common platform with a group of other like minded individuals. You can find the details of the platform here.

The main reason I’m running is I believe that Linux Australia can achieve so much more than it does today. Linux Australia should not simply be a conduit for linux.conf.au.

I want to help turn Linux Australia into an organisation that is relevant to all of us. It should be an organisation that not only fosters and supports the community but also represents the community.

We should offer supportive services to our members, spread the FOSS message through the community as well as actively lobby government for the things we believe in.

Most importantly it is essential that we all become involved. The community is nothing without people to move it forwards. So I would encourage you to vote for

President James Turnbull
Vice President Lindsay Holmwood
Secretary Peter Lieverdink
Treasurer John Ferlito

Ordinary Committee Members
Alice Boxhall
Elspeth Thorne

Once you have finished voting, go and join the mailing lists and get involved.

We do this by specifying two databases in config/database.yml.

development:
  database: vqmetrics_devel
  < <: *login_dev_local
 
vqdata_development: &VQDATA_TEST
  database: vqdata_devel
  <<: *login_dev_local

So by default the vqmetrics_devel database will be used. When we need to specify a model where we need to connect to the vqdata_devel database we use

class Video < ActiveRecord::Base
  establish_connection "vqdata_#{RAILS_ENV}"
end

and for migrations that need to connect to this database we do the following.

class InitialSetup < ActiveRecord::Migration
  def self.connection
    Video.connection
  end
end

This setup works really well. However recently I moved this application to using Cucumber for testing. Tests worked fine the first time they are run but not the second time.

I discovered that the transaction on the second database where not being rolled back as they should be. Cucumber only sets up the first database for roll back by using

ActiveRecord::Base.connection

where it should be rolling them all back by looping through

ActiveRecord::Base.connection_handler.connection_pools.values.map {|pool| pool.connection}

I've filed a bug at lighthouseapp.

johnf@zoot:~/dev/vquence/metrics/trunk$ sudo rm -rf /usr/lib
^C

The command only ran for a few seconds so the damage wasn’t to bad, but what did I lose?

The locate command came to my rescue. locate runs out of cron, usually once a day, and creates a database with a list of every file on your machine. You can then use it to search for files. So to work out what was missing I did the following.

# Get the list of files before we removed them
locate --regexp '.' > /tmp/before_rm
 
# update the locate database
sudo updatedb
 
# Get the list of current files on the system
locate --regexp '.' > /tmp/after_rm
 
# Create a list of what's missing
diff -u /tmp/before_rm /tmp/after_rm > /tmp/diff_rm
grep '^-' /tmp/diff_rm | sed -e 's/^-//' > /tmp/missing_rm
 
# Ask the dpkg system what packages those files belong to
for i in `cat /tmp/missing_rm`
do
    dpkg -S $i;
done | awk '{print $1}' | sed -e 's/:$//;s/,//g' > /tmp/packages
 
# Reinstall those packages
sudo aptitude reinstall `cat /tmp/packages`

After this process it is probably worth running the step from updatedb again to work out what is still missing.

For the record I lost 102 files and had to reinstall 97 packages.

Now back to real work!

Basically I wanted to create the equivalent of an Ubuntu PPA service running on our own servers so we could place it behind our firewall. This post is basically the process I followed to integrate rebuilld and reprepro to replicate a PPA setup.

So first up install reprepro

aptitude install reprepro

next we need do create a reprepro repository

mkdir -p /srv/reprepro/{conf,incoming,incomingtmp}

Now we need to tell reprepro which distributions we care about. Create /srv/reprepro/conf/distributions with the following contents

Suite: hardy
Version: 8.04
Codename: hardy
Architectures: i386 amd64 source
Components: main
Description: Local Hardy
SignWith: repository@inodes.org
DebIndices: Packages Release . .gz .bz2
DscIndices: Sources Release .gz .bz2
Tracking: all includechanges keepsources
Log: logfile
  --changes /srv/reprepro/bin/build_sources

Suite: intrepid
Version: 8.10
Codename: intrepid
Architectures: i386 amd64 source
Components: main
Description: Local Intrepid
SignWith: repository@inodes.org
DebIndices: Packages Release . .gz .bz2
DscIndices: Sources Release .gz .bz2
Tracking: all includechanges keepsources
Log: logfile
  --changes /srv/reprepro/bin/build_sources

Suite: jaunty
Version: 9.04
Codename: jaunty
Architectures: i386 amd64 source
Components: main
Description: Local Jaunty
SignWith: repository@inodes.org
DebIndices: Packages Release . .gz .bz2
DscIndices: Sources Release .gz .bz2
Tracking: all includechanges keepsources
Log: logfile
  --changes /srv/reprepro/bin/build_sources

I also like to create reprepro options file to setup some defaults, edit /srv/reprepro/conf/options

verbose
verbose
verbose
verbose
verbose

Next we need to setup an incoming queue so that we can use dput to get the source packages into reprepro,
vi /srv/reprepro/conf/incoming

Name: incoming
IncomingDir: incoming
Allow: hardy intrepid jaunty
Cleanup: on_deny on_error
Tempdir: incomingtmp

The repository is now ready to go. So now we can setup apache. Edit /etc/apache/sites-enabled/pppa

<virtualhost *:80>
    ServerName packages.inodes.org
    DocumentRoot /srv/reprepro
</virtualhost>

and we should also configure our sources.list to use these repositories, edit /etc/apt/sources.list

# Sources for rebuildd
deb-src http://packages.inodes.org hardy main
deb-src http://packages.inodes.org intrepid main
deb-src http://packages.inodes.org jaunty main

Next we want to setup our dput.cf to make the magic happen to get the source packages into the archive, edit ~/.dput.cf

[DEFAULT]
default_host_main = notspecified

[local]
fqdn = localhost
method = local
incoming = /srv/reprepro/incoming
allow_unsigned_uploads = 0
run_dinstall = 0
post_upload_command = reprepro -V -b /srv/reprepro processincoming incoming

So now we can do the following

apt-get source squid3
cd squid3*
dch -i # increment version number
dpkg-buildpackage -sa -S
cd ..
dput local *changes
aptitude update
apt-get source squid3

So when you run dput, first it copies the source package files to /srv/reprepro/incoming and then it gets reprepro to process it’s incoming queue. This means that the source package is now sitting in the repository.
So the second apt-get source should have downloaded the source package from our local repository which is exactly what rebuildd will do before it tries to build it.

Next step is to setup rebuildd so that it builds the binary packages and installs them into the repository.

aptitude install rebuildd

Setup so it runs out of init.d and the releases we care about, edit /etc/default/rebuildd

START_REBUILDD=1
START_REBUILDD_HTTPD=1
DISTS="hardy intrepid jaunty"

Now when a source package is uploaded into the repository we want to kick off rebuildd to build the package. We can do this through the reprepro log hooks. You’ll notice in the conf/distributions above the following lines.

Log: logfile
  --changes /srv/reprepro/bin/build_sources

This script will be run any time a .changes file is added to the repository. Create /srv/reprepro/bin/build_sources

#!/bin/bash
 
action=$1
release=$2
package=$3
version=$4
changes_file=$5
 
# Only care about packages being added
if [ "$action" != "accepted" ]
then
	exit 0
fi
 
# Only care about source packages
echo $changes_file | grep -q _source.changes
if [ $? = 1 ]
then
	exit 0
fi
 
# Kick off the job
echo "$package $version 1 $release"  | sudo rebuildd-job add

This script basically checks the right type of package is being added. Then it calls rebuildd-job to ask for that specific package and version to be built for that Ubuntu release.

Now the first thing that rebuildd does is download the source for the package. However we need to update the sources first since our server doesn’t know there are new files in the repository yet. So edit /etc/rebuildd/rebuilddrv an change

apt-get -q --download-only -t ${d} source ${p}=${v}

to

source_cmd = /srv/reprepro/bin/get_sources ${d} ${p} ${v}

and create /srv/reprepro/bin/get_sources with

#!/bin/bash
 
d=$1
p=$2
v=$3
 
sudo aptitude update >/dev/null
apt-get -q --download-only -t ${d} source ${p}=${v}

By this stage we have rebuildd building packages but we need to make sure they get re-injected back into the repository. We can do this with a post script. Edit /etc/rebuildd/rebuilddrc

post_build_cmd = /srv/reprepro/bin/upload_binaries ${d} ${p} ${v} ${a}

and create /srv/reprepro/bin/upload_binaries

#!/bin/bash
 
d=$1
p=$2
v=$3
a=$4
 
su -l -c "reprepro -V -b /srv/reprepro include ${d} /var/cache/pbuilder/result/${p}_${v}_${a}.changes" johnf

Now the su is in there because rebuildd needs to be able to access the GPG passphrase to sign the repository with. So rather than have a passphrase-less key we make sure that gpg-agent is running by adding the following to your .profile.

if test -f $HOME/.gpg-agent-info &&    kill -0 `cut -d: -f 2 $HOME/.gpg-agent-info` 2>/dev/null; then
	GPG_AGENT_INFO=`cat $HOME/.gpg-agent-info`
	export GPG_AGENT_INFO
else
	eval `gpg-agent --daemon`
	echo $GPG_AGENT_INFO >$HOME/.gpg-agent-info
fi
 
GPG_TTY=`tty`
export GPG_TTY

So that’s it you now have your own personal PPA. Just in case you had fallen asleep. Here is a little script I wrote so you can auto build the source packages for each release you care about in one go.

#!/bin/bash
 
set -e
 
RELEASES="hardy intrepid jaunty"
 
if [ ! -f debian/changelog ]
then
	echo "This isn't a debian repo"
	exit 1
fi
 
# Check for changes
if [ `bzr st | wc -l` != "0" ]
then
	echo "You have uncommitted changes!"
	exit 1
fi
 
if [ -d ../tmpbuild ]
then
	echo "The tmpbuild dir exists"
	exit 1
fi
 
bzr export ../tmpbuild
cp debian/changelog ../tmpbuild.changelog
cd ../tmpbuild
 
PACKAGE=`head -1 debian/changelog | awk '{print $1}'`
VERSION=`head -1 debian/changelog | awk '{print $2}' | sed -r -e 's/^\(//;s/\)$//'`
 
for release in $RELEASES
do
 
	sed -r -e "1s/\) [^;]+; /~${release}) ${release}; /" ../tmpbuild.changelog > debian/changelog 
	head -1 debian/changelog
	dpkg-buildpackage -S -sa
	dput local ../${PACKAGE}_${VERSION}~${release}_source.changes
done
 
cd ..
rm -rf tmpbuild

So the above documentation is a bit of a brain dump on what I’ve been working on for the past 2 days and I’m sure I’ve left some bits out. So please give me any feedback you have in the comments.