Linux Administration: Creating an official Debian mirror with apt-mirror, debmirror or ftpsync

In this article I'll demonstrate 3 ways of creating a local Debian mirror using the apt-mirror, debmirror and the ftpsync tools. I'll be mirroring Debian Wheezy, which will download about 79GB of packages. If your environment consists of many server builds each day, having a local Debian mirror will save you time and bandwidth.

I'll be using a Debian Wheezy server for this purpose.

First install the apt-mirror package and apache:

The apt-mirror config file is /etc/apt/mirror.list and should contain the following:

File: gistfile1.sh ------------------ [root@mirror:~]# cat /etc/apt/mirror.list deb http://ftp.us.debian.org/debian wheezy main contrib non-free deb-src http://ftp.us.debian.org/debian wheezy main contrib non-free clean http://ftp.us.debian.org/debian

You can put whatever mirror is closer to you and add or remove different repository sections. In this example I am interested in mirroring the main, contrib and the non-free sections.

Now that the config is there just run:

File: gistfile1.sh ------------------ [root@mirror:~]# apt-mirror /etc/apt/mirror.list Downloading 22 index files using 20 threads... Begin time: Wed Nov  6 15:53:51 2013 [20]... [19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... End time: Wed Nov  6 15:53:53 2013 Proceed indexes: [SP] 79.7 GiB will be downloaded into archive. Downloading 89109 archive files using 20 threads... Begin time: Wed Nov  6 15:54:04 2013 [20]...[19]... [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... End time: Wed Nov  6 16:38:47 2013 0.0 bytes in 0 files and 0 directories can be freed. Run /var/spool/apt-mirror/var/clean.sh for this purpose. Running the Post Mirror script ... (/var/spool/apt-mirror/var/postmirror.sh) Post Mirror script has completed. See above output for any possible errors.

This will have the packages downloaded to

Let's expose the repo over HTTP:

And use it:

File: gistfile1.sh ------------------ [root@mirror:~]# cat /etc/apt/sources.list deb http://YOUR_MIRROR_IP/debian/ wheezy main contrib non-free deb-src http://YOUR_MIRROR_IP/debian/ wheezy main contrib non-free [root@mirror:~]# apt-get update Get:1 http://YOUR_MIRROR_IP wheezy Release.gpg [1,672 B] Get:2 http://YOUR_MIRROR_IP wheezy Release [168 kB] Get:3 http://YOUR_MIRROR_IP wheezy/main Sources [5,959 kB] Get:4 http://YOUR_MIRROR_IP wheezy/contrib Sources [47.8 kB] Get:5 http://YOUR_MIRROR_IP wheezy/non-free Sources [93.4 kB] Get:6 http://YOUR_MIRROR_IP wheezy/main amd64 Packages [5,848 kB] Get:7 http://YOUR_MIRROR_IP wheezy/contrib amd64 Packages [42.0 kB] Get:8 http://YOUR_MIRROR_IP wheezy/non-free amd64 Packages [80.8 kB] Ign http://YOUR_MIRROR_IP wheezy/contrib Translation-en Ign http://YOUR_MIRROR_IP wheezy/main Translation-en Ign http://YOUR_MIRROR_IP wheezy/non-free Translation-en Fetched 12.2 MB in 5s (2,344 kB/s) Reading package lists... Done [root@mirror:~]# apt-get install iptraf Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: iptraf 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 175 kB of archives. After this operation, 609 kB of additional disk space will be used. Get:1 http://YOUR_MIRROR_IP/debian/ wheezy/main iptraf amd64 3.0.0-8.1 [175 kB] Fetched 175 kB in 0s (5,723 kB/s) Selecting previously unselected package iptraf. (Reading database ... 27685 files and directories currently installed.) Unpacking iptraf (from .../iptraf_3.0.0-8.1_amd64.deb) ... Processing triggers for man-db ... Setting up iptraf (3.0.0-8.1) ...

In some cases you might need to have a stable section in your repo. In this case you can just create a symlink to Wheezy:

To keep the repository in sync with the upstream Debian you can create a cronjob that pulls new packages at midnight:

Alternatively you can mirror any repo e.g. Percona using debmirror with this one liner:

File: gistfile1.sh ------------------ [root@mirror:~]# debmirror -a amd64 -v --ignore-release-gpg --no-source -s main -h repo.percona.com -d wheezy -r /apt --diff=none --progress -e rsync /var/www/mirror/percona

The recommended way of mirroring Debian repository is by using the ftpsync mirroring tool [1].

Let's install it and configure it to mirror wheezy and squeesze:

File: gistfile1.sh ------------------ [root@mirror:~]# adduser --disabled-password archvsync [root@mirror:~]# mkdir -p /srv/mirrors/debian [root@mirror:~]# chown -R archvsync:archvsync /srv/mirrors [root@mirror:~]# su - archvsync [archvsync@mirror:~]# wget http://ftp-master.debian.org/ftpsync.tar.gz [archvsync@mirror:~]# tar zxfv ftpsync.tar.gz [archvsync@mirror:~]# mv distrib/bin/ . [archvsync@mirror:~]# mv distrib/etc/ . [archvsync@mirror:~]# mkdir log [archvsync@mirror:~]# cp etc/ftpsync.conf.sample etc/ftpsync.conf

The config file should at a minimum look like this (if you only want to mirror i386 and amd64 architectures):

File: gistfile1.sh ------------------ [archvsync@mirror:~]# cat etc/ftpsync.conf | grep -vi "^#" | sed '/^$/d' MIRRORNAME=`hostname -f` TO="/srv/mirrors/debian" RSYNC_PATH="debian" RSYNC_HOST=ftp.us.debian.org LOGDIR="${BASEDIR}/log" EXCLUDE="--exclude=jessie* --exclude=oldstable* --exclude=sid* --exclude=testing* --exclude=unstable* --exclude=experimental* --exclude=Debian* --exclude=stable*" ARCH_EXCLUDE="alpha arm armel armhf hppa hurd-i386 ia64 kfreebsd-amd64 kfreebsd-i386 m68k mipsel mips powerpc s390 s390x sh sparc source"

With all that in place time to rsync the files:

Or if you rather run the two rsync stages one at a time, execute:

This really just runs these two rsync commands one after the other:

File: gistfile1.sh ------------------ rsync --bwlimit=0 -prltvHSB8192 --timeout 3600 --stats --filter=protect_Archive-Update-in-Progress-ftpsync --filter=protect_project/trace/ftpsync --filter=protect_Archive-Update-Required-ftpsync --exclude=Packages* --exclude=Sources* --exclude=Release* --exclude=InRelease --exclude=i18n/* --exclude=ls-lR* --exclude=jessie* --exclude=oldstable* --exclude=sid* --exclude=testing* --exclude=unstable* --exclude=experimental* --exclude=Debian* --exclude=stable* --exclude=.~tmp~/ ftp.us.debian.org::debian /srv/mirrors/debian rsync --bwlimit=0 -prltvHSB8192 --timeout 3600 --stats --filter=protect_Archive-Update-in-Progress-heroku-test --filter=protect_project/trace/heroku-test --filter=protect_Archive-Update-Required-heroku-test --max-delete=40000 --delay-updates --delete --delete-excluded --delete-delay --exclude=.~tmp~/ ftp.us.debian.org::debian/dists/wheezy /srv/mirrors/debian

To mirror the OpenVZ repo at download.openvz.org first let's find what rsync shares are supported:

After some trial and error the openvz-download is the share that works. The config file should look like so:

File: gistfile1.sh ------------------ [archvsync@mirror:~]# cat etc/ftpsync-OPENVZ.conf | grep -vi "^#" | sed '/^$/d' MIRRORNAME=`hostname -f` TO="/srv/mirrors/openvz" RSYNC_PATH="openvz-download/debian" RSYNC_HOST=download.openvz.org LOGDIR="${BASEDIR}/log" LOG="${LOGDIR}/${NAME}.log"

To rsync the repo run:
Notice how the config file has the OPENVZ name in it and how we specify which config to run by passing the sync:archive:OPENVZ parameters to the ftpsync script.

Resources:

[1]. http://ftp-master.debian.org/ftpsync.tar.gz

Pages

Creating an official Debian mirror with apt-mirror, debmirror or ftpsync