Whole document tree

Debian Developer's Reference
Chapter 5 - The Debian Archive

5.1 Overview

The Debian GNU/Linux distribution consists of a lot of Debian packages (.deb's, currently around 6800) and a few additional files (documentation, installation disk images, etc.).

Here is an example directory tree of a complete Debian archive:

     dists/stable/main/
     dists/stable/main/binary-all/
     dists/stable/main/binary-all/admin/
     dists/stable/main/binary-all/base/
     dists/stable/main/binary-all/comm/
     dists/stable/main/binary-all/devel/
          ...
     dists/stable/main/binary-i386/
     dists/stable/main/binary-i386/admin/
     dists/stable/main/binary-i386/base/
          ...
     dists/stable/main/binary-m68k/
     dists/stable/main/binary-m68k/admin/
     dists/stable/main/binary-m68k/base/
          ...
     dists/stable/main/source/
     dists/stable/main/source/admin/
     dists/stable/main/source/base/
          ...
     dists/stable/main/disks-i386/
     dists/stable/main/disks-m68k/
          ...
     
     dists/stable/contrib/
     dists/stable/contrib/binary-all/
     dists/stable/contrib/binary-i386/
     dists/stable/contrib/binary-m68k/
          ...
     dists/stable/contrib/source/
     
     dists/stable/non-free/
     dists/stable/non-free/binary-all/
     dists/stable/non-free/binary-i386/
     dists/stable/non-free/binary-m68k/
          ...
     dists/stable/non-free/source/
     
     dists/testing/
     dists/testing/main/
          ...
     dists/testing/contrib/
          ...
     dists/testing/non-free/
          ...
     
     dists/unstable
     dists/unstable/main/
          ...
     dists/unstable/contrib/
          ...
     dists/unstable/non-free/
          ...
     
     pool/
     pool/a/
     pool/a/apt/
          ...
     pool/b/
     pool/b/bash/
          ...
     pool/liba/
     pool/liba/libalias-perl/
          ...
     pool/m/
     pool/m/mailx/
          ...

As you can see, the top-level directory contains two directories, dists/ and pool/. The latter is a ``pool'' in which the packages actually are, and which is handled by the archive maintenance database and the accompanying programs. The former contains the distributions, stable, testing and unstable. Each of those distribution directories is divided in equivalent subdirectories purpose of which is equal, so we will only explain how it looks in stable. The Packages and Sources files in the distribution subdirectories can reference files in the pool/ directory.

dists/stable contains three directories, namely main, contrib, and non-free.

In each of the areas, there is a directory with the source packages (source), a directory for each supported architecture (binary-i386, binary-m68k, etc.), and a directory for architecture independent packages (binary-all).

The main area contains additional directories which holds the disk images and some essential pieces of documentation required for installing the Debian distribution on a specific architecture (disks-i386, disks-m68k, etc.).

The binary-* and source directories are divided further into subsections.

5.2 Sections

The main section of the Debian archive is what makes up the official Debian GNU/Linux distribution. The main section is official because it fully complies with all our guidelines. The other two sections do not, to different degrees; as such, they are not officially part of Debian GNU/Linux.

Every package in the main section must fully comply with the Debian Free Software Guidelines (DFSG) and with all other policy requirements as described in the Debian Policy Manual. The DFSG is our definition of ``free software.'' Check out the Debian Policy Manual for details.

Packages in the contrib section have to comply with the DFSG, but may fail other requirements. For instance, they may depend on non-free packages.

Packages which do not apply to the DFSG are placed in the non-free section. These packages are not considered as part of the Debian distribution, though we support their use, and we provide infrastructure (such as our bug-tracking system and mailing lists) for non-free software packages.

The Debian Policy Manual contains a more exact definition of the three sections. The above discussion is just an introduction.

The separation of the three sections at the top-level of the archive is important for all people who want to distribute Debian, either via FTP servers on the Internet or on CD-ROMs: by distributing only the main and contrib sections, one can avoid any legal risks. Some packages in the non-free section do not allow commercial distribution, for example.

On the other hand, a CD-ROM vendor could easily check the individual package licenses of the packages in non-free and include as many on the CD-ROMs as he's allowed to. (Since this varies greatly from vendor to vendor, this job can't be done by the Debian developers.)

5.3 Architectures

In the first days, the Linux kernel was only available for the Intel i386 (or greater) platforms, and so was Debian. But when Linux became more and more popular, the kernel was ported to other architectures, too.

The Linux 2.0 kernel supports Intel x86, DEC Alpha, SPARC, Motorola 680x0 (like Atari, Amiga and Macintoshes), MIPS, and PowerPC. The Linux 2.2 kernel supports even more architectures, including ARM and UltraSPARC. Since Linux supports these platforms, Debian decided that it should, too. Therefore, Debian has ports underway; in fact, we also have ports underway to non-Linux kernel. Aside from i386 (our name for Intel x86), there is m68k, alpha, powerpc, sparc, hurd-i386, and arm, as of this writing.

Debian GNU/Linux 1.3 is only available as i386. Debian 2.0 shipped for i386 and m68k architectures. Debian 2.1 ships for the i386, m68k, alpha, and sparc architectures. Debian 2.2 adds support for the powerpc and arm architectures.

Information for developers or uses about the specific ports are available at the Debian Ports web pages.

5.4 Subsections

The sections main, contrib, and non-free are split into subsections to simplify the installation process and the maintainance of the archive. Subsections are not formally defined, except perhaps the `base' subsection. Subsections simply exist to simplify the organization and browsing of available packages. Please check the current Debian distribution to see which sections are available.

Note however that with the introduction of package pools (see the top-level pool/ directory), the subsections in the form of subdirectories will eventually cease to exist. They will be kept in the packages' `Section' header fields, though.

5.5 Packages

There are two types of Debian packages, namely source and binary packages.

Source packages consist of either two or three files: a .dsc file, and either a .tar.gz file or both an .orig.tar.gz and a .diff.gz file.

If a package is developed specially for Debian and is not distributed outside of Debian, there is just one .tar.gz file which contains the sources of the program. If a package is distributed elsewhere too, the .orig.tar.gz file stores the so-called upstream source code, that is the source code that's distributed from the upstream maintainer (often the author of the software). In this case, the .diff.gz contains the changes made by the Debian maintainer.

The .dsc lists all the files in the source package together with checksums (md5sums) and some additional info about the package (maintainer, version, etc.).

5.6 Distribution directories

The directory system described in the previous chapter is itself contained within distribution directories. Each distribution is actually contained in the pool directory in the top-level of the Debian archive itself.

To summarize, the Debian archive has a root directory within an FTP server. For instance, at the mirror site, ftp.us.debian.org, the Debian archive itself is contained in /debian, which is a common location (another is /pub/debian).

A distribution is comprised of Debian source and binary packages, and the respective Sources and Packages index files, containing the header information from all those packages. The former are kept in the pool/ directory, while the latter are kept in the dists/ directory of the archive (because of backwards compatibility).

5.6.1 Stable, testing, and unstable

There are always distributions called stable (residing in dists/stable), one called testing (residing in dists/testing), and one called unstable (residing in dists/unstable). This reflects the development process of the Debian project.

Active development is done in the unstable distribution (that's why this distribution is sometimes called the development distribution). Every Debian developer can update his or her packages in this distribution at any time. Thus, the contents of this distribution change from day-to-day. Since no special effort is done to make sure everything in this distribution is working properly, it is sometimes ``unstable.''

Packages get copied from unstable to testing if they satisfy certain criteria. To get into testing distribution, a package needs to be in the archive for two weeks and not have any release critical bugs. After that period, it will propagate into testing as soon as anything it depends on is also added. This process is automatic. You can see some notes on this system as well as update_excuses (describing which packages are valid candidates, which are not, and why not) at http://ftp-master.debian.org/testing/.

After a period of development, once the release manager deems fit, the testing distribution is frozen, meaning that the policies which control how packages move from unstable to testing are tightened. Packages which are too buggy are removed. No changes are allowed into testing except for bug fixes. After some time has elapsed, depending on progress, the testing distribution goes into a `deep freeze', when no changes are made to it except those needed for the installation system. This is called a ``test cycle'', and it can last up to two weeks. There can be several test cycles, until the distribution is prepared for release, as decided by the release manager. At the end of the last test cycle, the testing distribution is renamed to stable, overriding the old stable distribution, which is removed at that time (although it can be found at archive.debian.org).

This development cycle is based on the assumption that the unstable distribution becomes stable after passing a period of being in testing. Even once a distribution is considered stable, a few bugs inevitably remain — that's why the stable distribution is updated every now and then. However, these updates are tested very carefully and have to be introduced into the archive individually to reduce the risk of introducing new bugs. You can find proposed additions to stable in the proposed-updates directory. Those packages in proposed-updates that pass muster are periodically moved as a batch into the stable distribution and the revision level of the stable distribution is incremented (e.g., `1.3' becomes `1.3r1', `2.0r2' becomes `2.0r3', and so forth).

Note that development under unstable continues during the ``freeze'' period, since the unstable distribution remains in place in parallel with testing.

5.6.2 Experimental

The experimental distribution is a specialty distribution. It is not a full distribution in the same sense as `stable' and `unstable' are. Instead, it is meant to be a temporary staging area for highly experimental software where there's a good chance that the software could break your system, or software that's just too unstable even for the unstable distribution (but there is a reason to package it nevertheless). Users who download and install packages from experimental are expected to have been duly warned. In short, all bets are off for the experimental distribution.

If there is a chance that the software could do grave damage to a system, it is likely to be better to put it into experimental. For instance, an experimental compressed file system should probably go into experimental.

Whenever there is a new upstream version of a package that introduces new features but breaks a lot of old ones, it should either not be uploaded, or be uploaded to experimental. A new, beta, version of some software which uses completely different configuration can go into experimental, at the maintainer's discretion. If you are working on an incompatible or complex upgrade situation, you can also use experimental as a staging area, so that testers can get early access.

Some experimental software can still go into unstable, with a few warnings in the description, but that isn't recommended because packages from unstable are expected to propagate to testing and thus to stable.

New software which isn't likely to damage your system can go directly into unstable.

An alternative to experimental is to use your personal web space on people.debian.org (klecker.debian.org).

5.7 Release code names

Every released Debian distribution has a code name: Debian 1.1 is called `buzz'; Debian 1.2, `rex'; Debian 1.3, `bo'; Debian 2.0, `hamm'; Debian 2.1, `slink'; Debian 2.2, `potato'; and Debian 3.0, `woody'. There is also a ``pseudo-distribution'', called `sid', which is the current `unstable' distribution; since packages are moved from `unstable' to `testing' as they approach stability, `sid' itself is never released. As well as the usual contents of a Debian distribution, `sid' contains packages for architectures which are not yet officially supported or released by Debian. These architectures are planned to be integrated into the mainstream distribution at some future date.

Since Debian has an open development model (i.e., everyone can participate and follow the development) even the `unstable' and `testing' distributions are distributed to the Internet through the Debian FTP and HTTP server network. Thus, if we had called the directory which contains the release candidate version `testing', then we would have to rename it to `stable' when the version is released, which would cause all FTP mirrors to re-retrieve the whole distribution (which is quite large).

On the other hand, if we called the distribution directories Debian-x.y from the beginning, people would think that Debian release x.y is available. (This happened in the past, where a CD-ROM vendor built a Debian 1.0 CD-ROM based on a pre-1.0 development version. That's the reason why the first official Debian release was 1.1, and not 1.0.)

Thus, the names of the distribution directories in the archive are determined by their code names and not their release status (e.g., `slink'). These names stay the same during the development period and after the release; symbolic links, which can be changed easily, indicate the currently released stable distribution. That's why the real distribution directories use the code names, while symbolic links for stable, testing, and unstable point to the appropriate release directories.

Debian Developer's Reference