Creating a mirror

From ParabolaWiki
Jump to: navigation, search
Note: See Mirroring on demand instead for an alternative in which packages are fetch only when they are requested by clients of the mirror (basically caching), useful for multiple clients on internal networks.

1 mirror-snapshot

It creates hard-linked snapshots like http://alfplayer.com/.

http://evc.link/mirror-snapshot/mirror-snapshot.tar.gz contains all files.

These files are needed.

http://evc.link/mirror-snapshot/mirror-snapshot

http://evc.link/mirror-snapshot/mirror-snapshot-config

1.1 mirror-snapshot-sequence

mirror-snapshot-sequence (untested) is also included in the tarball. It runs mirror-snapshot and serves the same purpose but it fetches from Arch and ArchARM servers before syncing from a Parabola server. It depends on mirror-rename.

http://evc.link/mirror-snapshot/mirror-snapshot-sequence

http://evc.link/mirror-snapshot/mirror-rename

2 Bash script

Creating a mirror can be done with the script given below.

You should adjust the target and tmp directory before using the script. You can change the source_url (currently rsync://repo.parabola.nu:875/repos/) and the lastupdate_url to use another source.

To synchronise, just execute the script as a user with writing permissions on the target and tmp directory. You should run this script regularly to keep your mirror up to date by configuring a cron job to synchronise e.g. once a day (once an hour maximum, please sync on a random minute). Running this script as a cron job will only cause it to sync when the ./lastupdate file is changed.

Finally, you can set up a webserver such as Apache or nginx to serve files from target directory to the public. If you have enough resources to be a public mirror, please post your mirror details in the dev-mailinglist.

#!/bin/bash

# Directory where the repo is stored locally. Example: /srv/repo
target="/srv/repo"

# Directory where files are downloaded to before being moved in place.
# This should be on the same filesystem as $target, but not a subdirectory of $target.
# Example: /srv/tmp
tmp="/srv/tmp"

# Lockfile path
lock="/var/lock/syncrepo.lck"

# If you want to limit the bandwidth used by rsync set this.
# Use 0 to disable the limit.
# The default unit is KiB (see man rsync /--bwlimit for more)
bwlimit=0

# The source URL of the mirror you want to sync from.
source_url='rsync://repo.parabola.nu:875/repos/'

# An HTTP(S) URL pointing to the 'lastupdate' file on your chosen mirror.
lastupdate_url='https://repo.parabola.nu/lastupdate'

#### END CONFIG

[ ! -d "${target}" ] && mkdir -p "${target}"
[ ! -d "${tmp}" ] && mkdir -p "${tmp}"

exec 9>"${lock}"
/usr/bin/flock -n 9 || exit

rsync_cmd() {
        local -a cmd=(/usr/bin/rsync -rtlH --safe-links --delete-after ${VERBOSE} "--timeout=600" "--contimeout=60" -p \
                --delay-updates --no-motd "--temp-dir=${tmp}")

        if /bin/stty &>/dev/null; then
                cmd+=(-h -v --progress)
        else
                cmd+=(--quiet)
        fi

        if ((bwlimit>0)); then
                cmd+=("--bwlimit=$bwlimit")
        fi

        "${cmd[@]}" "$@"
}


# if we are called without a tty (cronjob) only run when there are changes
if ! /usr/bin/tty -s && [[ -f "$target/lastupdate" ]] && /usr/bin/diff -b <(/usr/bin/curl -Ls "$lastupdate_url") "$target/lastupdate" >/dev/null; then
        # keep lastsync file in sync for statistics
        rsync_cmd "$source_url/lastsync" "$target/lastsync"
        exit 0
fi

rsync_cmd \
        --exclude='*.links.tar.gz*' \
        "${source_url}" \
        "${target}"