Mirror Master

Lee McLoughlin

and

Zoë Leech

22 May 1998 lmjm@icparc.ic.ac.uk zl@icparc.ic.ac.uk

Warning
Introduction
Description
Flags
Configuration File
Examples
See Also
Author

Warning

mm is not for the faint-hearted

Introduction

mm [flags] [configuration file]

Description

mm is a Perl program designed to manage the running of multiple mirrors. It will keep as many mirrors running in parallel as it can, up to a user defined maximum, automatically starting another when one finishes. It will also handle locking so that only one mirror of a package can occur at a time and allows configuration of the minimum time between attempts to mirror a package.

mm presumes that all the package details are stored in a directory called packages. All the packages for a given site must be in a file whose name is the same as that of the site. So in packages/sunsite.org.uk will be all the package details for the host sunsite.org.uk.

Flags

-t	Ignore the minimal retry timers.
-o site:package	Only mirror the given site and package.
-debug	Enable debugging. If this argument is given more than once the level of debugging will increase. Currently the maximum possible level is four.
-s	Enable status debugging. This will print out the status of any subprocesses as they are spawned or as they exit.

Any unrecognized arguments are passed as arguments to mirror for example:
mm -d mmin
will pass mirror the -d flag (so mirror will run in debug mode).

Configuration File

The configuration file is parsed as a series of statements. Blank lines and lines beginning with a hash are ignored. There are are two kinds of statements: control statements and site:package details.

Control statements are of the form:

keyword=value

Whitespace around the equals sign is ignored.

Statements are obeyed in sequence as they are reached.

Here is a list of the keywords and their values and any defaults:

home Directory that mm chdirs to before doing any work. The default is the current directory (i.e. .)

max Maximum number of mirrors that can be running in parallel at any one time. The default is 6.

mirror This is the way to call mirror.
The default is:
exec ./mirror \$args -p'\$package' packages/\$site > logs/\$ site:\$pkg 2>&1
$args, $package and $site are replaced with the appropriate values from the package file. Any $pkg is replaced with a version of $package with all characters that cause problems in filenames converted to underscore.

cmd Run the given command. This can occur multiple times. It can be used to delete temporary files or send mail.

cmdin Run the given command and use its output as mm input. This can occur multiple times. For example
cmdin=./pkgs_to_mmin packages/*

skip Skip the given site:package when it occurs.

In addition to control statements there are also package details. Each package details statement is of the form:

site:package min-restart-last-ok min-restart-last-notok mirror-args

The site and package fields are matched against those in the files in the packages directory. A mirror for this site:package will only be run if the last mirror run was successful and was more than min-restart-last-ok hours ago. If the last mirror of this was unsuccessful then min-restart-last-notok will be used to decide if the mirror will be run. The information about the last mirror run is stored in a file called mm.status in the directory defined by home

Examples

Here is a mm configuration file to manage the mirroring of two packages.

# Configuration file for mm

max=1

cmd=/usr/bin/rm -f /public/mirror/tmp

home=/public/mirror

ftp.cl.cam.ac.uk:JIPS-nrs 20 20
hcrl.open.ac.uk:hcrl-psion 20 20

However maintaining this file where there are a large number of mirrors can be a problem. To overcome this the program pkgs_to_mmin, was
designed to take a list of package files as input and turn these into the necessary input for mm

pkgs_to_mmin [-y min-restart-last-ok] [-n min-restart-last-notok] package-files-to-use

In this mm configuration file pkgs_to_mmin is used to manage all the packages in the /public/mirror/packages directory.