Inodes

Fractional CTO Consulting

Building a Private PPA on Ubuntu

· · 13 Comments

One of the things I love about the Ubuntu project and launchpad is the Personal Package Archive. PPAs make it so simple and easy to backport packages. The only problem with PPAs is that they are public. I had a need to be able to host some private internal packages as well as squid with SSL support, which you can’t distribute in binary form due to licensing restrictions.

Basically I wanted to create the equivalent of an Ubuntu PPA service running on our own servers so we could place it behind our firewall. This post is basically the process I followed to integrate rebuilld and reprepro to replicate a PPA setup.

So first up install reprepro

Bash
aptitude install reprepro

next we need do create a reprepro repository

Bash
mkdir -p /srv/reprepro/{conf,incoming,incomingtmp}

Now we need to tell reprepro which distributions we care about. Create /srv/reprepro/conf/distributions with the following contents

YAML
Suite: hardy
Version: 8.04
Codename: hardy
Architectures: i386 amd64 source
Components: main
Description: Local Hardy
SignWith: repository@inodes.org
DebIndices: Packages Release . .gz .bz2
DscIndices: Sources Release .gz .bz2
Tracking: all includechanges keepsources
Log: logfile
  --changes /srv/reprepro/bin/build_sources

Suite: intrepid
Version: 8.10
Codename: intrepid
Architectures: i386 amd64 source
Components: main
Description: Local Intrepid
SignWith: repository@inodes.org
DebIndices: Packages Release . .gz .bz2
DscIndices: Sources Release .gz .bz2
Tracking: all includechanges keepsources
Log: logfile
  --changes /srv/reprepro/bin/build_sources

Suite: jaunty
Version: 9.04
Codename: jaunty
Architectures: i386 amd64 source
Components: main
Description: Local Jaunty
SignWith: repository@inodes.org
DebIndices: Packages Release . .gz .bz2
DscIndices: Sources Release .gz .bz2
Tracking: all includechanges keepsources
Log: logfile
  --changes /srv/reprepro/bin/build_sources

I also like to create reprepro options file to setup some defaults, edit /srv/reprepro/conf/options

Bash
verbose
verbose
verbose
verbose
verbose

Next we need to setup an incoming queue so that we can use dput to get the source packages into reprepro,
vi /srv/reprepro/conf/incoming

YAML
Name: incoming
IncomingDir: incoming
Allow: hardy intrepid jaunty
Cleanup: on_deny on_error
Tempdir: incomingtmp

The repository is now ready to go. So now we can setup apache. Edit /etc/apache/sites-enabled/pppa

Apache
ServerName packages.inodes.org
DocumentRoot /srv/reprepro

and we should also configure our sources.list to use these repositories, edit /etc/apt/sources.list

Bash
# Sources for rebuildd
deb-src http://packages.inodes.org hardy main
deb-src http://packages.inodes.org intrepid main
deb-src http://packages.inodes.org jaunty main

Next we want to setup our dput.cf to make the magic happen to get the source packages into the archive, edit ~/.dput.cf

INI
[DEFAULT]
default_host_main = notspecified

[local]
fqdn = localhost
method = local
incoming = /srv/reprepro/incoming
allow_unsigned_uploads = 0
run_dinstall = 0
post_upload_command = reprepro -V -b /srv/reprepro processincoming incoming

So now we can do the following

Bash
apt-get source squid3
cd squid3*
dch -i # increment version number
dpkg-buildpackage -sa -S
cd ..
dput local *changes
aptitude update
apt-get source squid3

So when you run dput, first it copies the source package files to /srv/reprepro/incoming and then it gets reprepro to process it’s incoming queue. This means that the source package is now sitting in the repository.
So the second apt-get source should have downloaded the source package from our local repository which is exactly what rebuildd will do before it tries to build it.

Next step is to setup rebuildd so that it builds the binary packages and installs them into the repository.

Bash
aptitude install rebuildd

Setup so it runs out of init.d and the releases we care about, edit /etc/default/rebuildd

Bash
START_REBUILDD=1
START_REBUILDD_HTTPD=1
DISTS="hardy intrepid jaunty"

Now when a source package is uploaded into the repository we want to kick off rebuildd to build the package. We can do this through the reprepro log hooks. You’ll notice in the conf/distributions above the following lines.

YAML
Log: logfile
  --changes /srv/reprepro/bin/build_sources

This script will be run any time a .changes file is added to the repository. Create /srv/reprepro/bin/build_sources

Bash
#!/bin/bash

action=$1
release=$2
package=$3
version=$4
changes_file=$5

# Only care about packages being added
if [ "$action" != "accepted" ]
then
	exit 0
fi

# Only care about source packages
echo $changes_file | grep -q _source.changes
if [ $? = 1 ]
then
	exit 0
fi

# Kick off the job
echo "$package $version 1 $release"  | sudo rebuildd-job add

This script basically checks the right type of package is being added. Then it calls rebuildd-job to ask for that specific package and version to be built for that Ubuntu release.

Now the first thing that rebuildd does is download the source for the package. However we need to update the sources first since our server doesn’t know there are new files in the repository yet. So edit /etc/rebuildd/rebuilddrv an change

Bash
apt-get -q --download-only -t ${d} source ${p}=${v}

to

Bash
source_cmd = /srv/reprepro/bin/get_sources ${d} ${p} ${v}

and create /srv/reprepro/bin/get_sources with

Bash
#!/bin/bash

d=$1
p=$2
v=$3

sudo aptitude update >/dev/null
apt-get -q --download-only -t ${d} source ${p}=${v}

By this stage we have rebuildd building packages but we need to make sure they get re-injected back into the repository. We can do this with a post script. Edit /etc/rebuildd/rebuilddrc

Bash
post_build_cmd = /srv/reprepro/bin/upload_binaries ${d} ${p} ${v} ${a}

and create /srv/reprepro/bin/upload_binaries

Bash
#!/bin/bash

d=$1
p=$2
v=$3
a=$4

su -l -c "reprepro -V -b /srv/reprepro include ${d} /var/cache/pbuilder/result/${p}_${v}_${a}.changes" johnf

Now the su is in there because rebuildd needs to be able to access the GPG passphrase to sign the repository with. So rather than have a passphrase-less key we make sure that gpg-agent is running by adding the following to your .profile.

Bash
if test -f $HOME/.gpg-agent-info &&    kill -0 `cut -d: -f 2 $HOME/.gpg-agent-info` 2>/dev/null; then
	GPG_AGENT_INFO=`cat $HOME/.gpg-agent-info`
	export GPG_AGENT_INFO
else
	eval `gpg-agent --daemon`
	echo $GPG_AGENT_INFO >$HOME/.gpg-agent-info
fi

GPG_TTY=`tty`
export GPG_TTY

So that’s it you now have your own personal PPA. Just in case you had fallen asleep. Here is a little script I wrote so you can auto build the source packages for each release you care about in one go.

Bash
#!/bin/bash

set -e

RELEASES="hardy intrepid jaunty"

if [ ! -f debian/changelog ]
then
	echo "This isn't a debian repo"
	exit 1
fi

# Check for changes
if [ `bzr st | wc -l` != "0" ]
then
	echo "You have uncommitted changes!"
	exit 1
fi

if [ -d ../tmpbuild ]
then
	echo "The tmpbuild dir exists"
	exit 1
fi

bzr export ../tmpbuild
cp debian/changelog ../tmpbuild.changelog
cd ../tmpbuild

PACKAGE=`head -1 debian/changelog | awk '{print $1}'`
VERSION=`head -1 debian/changelog | awk '{print $2}' | sed -r -e 's/^(//;s/)$//'`

for release in $RELEASES
do
	
	sed -r -e "1s/) [^;]+; /~${release}) ${release}; /" ../tmpbuild.changelog > debian/changelog 
	head -1 debian/changelog
	dpkg-buildpackage -S -sa
	dput local ../${PACKAGE}_${VERSION}~${release}_source.changes
done

cd ..
rm -rf tmpbuild

So the above documentation is a bit of a brain dump on what I’ve been working on for the past 2 days and I’m sure I’ve left some bits out. So please give me any feedback you have in the comments.

Reader Interactions

Comments

    • That’s one thing I forgot to mention. You can get it to satisfy everything from the PPA.

      Basically you need to add the PPA to your sources.list so that it is used to grab dependencies from. You also need a pre-depend hook to perform an aptitude update.

      Edit /etc/pbuilder/pbuilderrc

       HOOKDIR="/etc/pbuilder/hooks"

      then

      mkdir /etc/pbuilder/hooks
echo -e "#!/bin/shn/usr/bin/aptitude updaten" > /etc/pbuilder/hooks/D70aptitude_update
      Reply

  4. Thanks a lot for the howto! I was looking forward to building hardened packages with hardening-wrapper on a local mirror…

    Reply

  5. That’s one thing I forgot to mention. You can get it to satisfy everything from the PPA.

    Basically you need to add the PPA to your sources.list so that it is used to grab dependencies from. You also need a pre-depend hook to perform an aptitude update.

    Edit /etc/pbuilder/pbuilderrc

     HOOKDIR="/etc/pbuilder/hooks"

    then

    mkdir /etc/pbuilder/hooks
echo -e "#!/bin/shn/usr/bin/aptitude updaten" > /etc/pbuilder/hooks/D70aptitude_update
    Reply

  6. For getting my debs builds in again on a easy way, i did hack the upload_binaries script a bit.

    #!/bin/bash

    d=$1 p=$2 v=$3 a=$4

    basedir=/srv/reprepro
    resultdir=/var/cache/pbuilder/result

    echo “upload_binaries” “${p}_${v}_${a}”

    # Search now for packages builded by these sources
    for i in ${resultdir}/*_${v}_*.deb; do
    su -l -c “reprepro -V -b ${basedir} includedeb ${d} ${i}” frans
    done

    Reply

  7. Hi , thank you for the great manual .

    but I run into one problem .

    for example I have a source package which I have built on my system – i have .dsc file _source.changes , and so on .

    now I upload with dupload or dput this _source.changes file and company to the repository server .

    after I’ve uploaded stuff to incoming on the repository server I run
    reprepro processincoming incoming

    and my .dsc and other files are added to the repository

    but when the build process spawned by Log : is finished , and rebuildd tries to put the new package to the repository the checksum on the new package is different and I get something like following :

    File “pool/main/libp/libpam-abl/libpam-abl_0.4.1-1.dsc” is already registered with different checksums!
    md5 expected: 79c0aaadf3f1deb26722d959db052cd5, got: b59bc17aa5cc072c8f2a091dbb745949
    sha1 expected: 0e2c1c73ecc8625ae5a0459a7bf77bc51e745294, got: 1c22b4a0664a1c16ea0e28c62a9f3e660582177b
    sha256 expected: 3edbfb50a5d228c717a8244ddcda58d067a76ed98c01f3afd5e89f8365c08274, got: 6aa915dd021165cead23bd1389b9af3797f3efd53cd6bba50f82b2d8b05768e8
    size expected: 1848, got: 964
    There have been errors!

    So the here is more general problem – because there is only one pool , it is not possible to have the same package-version in different distributions like sid and lenny .

    Anyway thank you a lot for sharing such a nice manual .

    best regards ,
    Alex

    Reply

  8. I use key with a passphrase and gpg-agent ,

    so using script mentioned by comment #7 Frans van Berckel

    I would need to input passphrase at some point , but using next constructions one can avoid passwordless gpg key and reuse gpg-agent with su

    su -l -c “bash -i -c “/usr/bin/reprepro -V -b ${basedir} includedeb ${d} ${i}”” ftpmaster

    with bash -i -c GPG_AGENT_INFO variable will be setup correctly from .bashrc

    here is the part of .bashrc responsible for gpg-agent

    pgrep -u $USER gpg-agent || gpg-agent –daemon –enable-ssh-support –write-env-file “${HOME}/.gpg-agent-info” –default-cache-ttl 600

    if [ -f “${HOME}/.gpg-agent-info” ]; then
    . “${HOME}/.gpg-agent-info”
    export GPG_AGENT_INFO
    export SSH_AUTH_SOCK
    export SSH_AGENT_PID
    fi
    GPG_TTY=$(tty)
    export GPG_TTY

    Reply

  9. Hi

    This looks really useful but I am struggling 🙁 I have followed all the steps, but when I dput a changes file the source package is accepted into the local repo but rebuildd never builds the binary. I can see rebuildd idling but the build never kicks in? Any ideas?

    Many thanks!

    Reply

  10. Ok I’ve made some progress. The /srv/reprepro/bin/build_sources script needs to be executable. Now rebuildd picks up the package but it still wont build. In the log file I see:

    ERROR Couldn’t find dist/arch in the config file for hello-debhelper_2.2-2ubuntu1 on precise/amd64, don’t adding it

    Note: I am playing with the hello-debhelper package instead of squid.

    Any help will be greatly appreciated!

    Reply

  12. I’m seeing the same problem Mono does :

    ERROR Couldn’t find dist/arch in the config file for on saucy/amd64, not adding it

    Any pointers or so would be very much appreciated.

    Also, thanks for the guide. The rest of it works like a charm!

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Hit the ground running with a minimalist look. Learn More