JIGDO-FILESection: User Commands (1)
Updated: 28 June 2002
Return to Main Contents
NAMEjigdo-file - Prepare files for Jigsaw Download (distribution of huge files, e.g. CD images).
[ --image=cdrom.iso ] [ --jigdo=cdrom.jigdo ] [ --template=cdrom.template ] [ --force ] [ MORE OPTIONS ] [ FILES ... | --files-from=f ]
Jigsaw Download, or short jigdo, is a scheme developed primarily to make it easy to distribute huge filesystem images (e.g. CD (ISO9660) or DVD (UDF) images) over the internet, but it could also be used for other data which is awkward to handle due to its size, like audio/video files or large software packages.
jigdo tries to ensure that the large file (always called image from now on) is downloaded in small parts which can be stored on different servers. People who want to download the image do so by telling the jigdo(1) (NOT IMPLEMENTED YET) download tool to process one `.jigdo' file; using it, jigdo downloads the parts and reassembles the image. jigdo-file is used to prepare the files for download.
What makes jigdo special is that the parts that are used to reconstruct the image can have any size and content - they only need to be contained in a contiguous region anywhere in the image.
For example, if you wish to distribute an ISO9660 image which contains a snapshot of an FTP server, you can instruct jigdo-file to prepare the download data in such a way that when people use jigdo to download the image, jigdo actually fetches the individual files from the FTP server and assembles them into an exact copy of your image - during the download! (If the image is not a filesystem dump, you can use split(1) to create the small parts that the image will be reassembled from.)
You are completely free to choose where the individual parts of the image are stored: They may be in entirely different directories on different servers (e.g. because of storage/bandwidth constraints), but this is invisible to the people downloading your image. The information about available servers only needs to be added to the `.jigdo' file by you before distributing it.
Many options are specific to a particular COMMAND; the ones below are general or used by several commands. Further options are listed below with the individual commands. All options are silently ignored if they are not applicable to the current command. For any BYTES parameters to options, you can append one of the letters `k', `M' or `G' to the amount you specify, to indicate kilobytes, megabytes or gigabytes.
The command name is the first non-option argument passed to jigdo-file. Most commands have short abbreviations as well as long names. The short command names should not be used in scripts - there may be incompatible changes to them in the future!
Reads image and FILES, creates `.jigdo' and `.template'. This is the main functionality of jigdo-file.
It is possible to specify both --image=- and --files-from=-. In this case, first the list of files is read from standard input until an empty line is encountered. Everything following it is assumed to be the image data. This can be useful if you use mkisofs(1) or similar programs that can output the complete image on their standard output, because there is no need to store the image on disc temporarily.
If a FILES argument contains the characters `//' (Unix) or `\.\' (Windows), this has special meaning. In the final jigdo file that users will download, each of the parts is referenced in the `[Parts]' section with a URI of the form `Label:some/filename'. (See `FORMAT OF .JIGDO FILES' below for a detailed description.) The `[Servers]' section gives a mapping of labels to servers on the internet, with lines like `Label=http://myserver.org/jigdofiles/'. Using this information, jigdo will create the final download URI for the part, `http://myserver.org/jigdofiles/some/filename'. Specifying `//' (or `\.\') in a file or directory name serves to `cut off' the names at the right directory level. For example, if the Unix path of one of your FILES is `/path/some/filename', you can tell jigdo-file to cut off after the `/path' by passing it the argument `/path//some/filename', or `/path//' if you want the whole directory scanned. The path names need not be absolute; `somedirectory//' is also possible.
Reads `.template' and FILES, creates image (or `imagename.tmp'). Provides a rudimentary way of reassembling images - jigdo is usually better suited for this task. However, in contrast to jigdo, no `.jigdo' file is required.
If the image is to be written to a file (and not to standard output), it is possible to create the image in several steps, with several invocations of `jigdo-file make-image', as follows: You first invoke jigdo-file, specifying as many files as are available at this time. The program scans the files, and those that are contained in the image are copied to a temporary file, whose name is formed by appending `.tmp' to the image filename.
For all further files which could be parts of the image, you repeat this process. As soon as all parts are present, the temporary file will be truncated slightly (to delete some administrative data that jigdo-file appends at the end) and renamed to the final image name. The possibility of reassembling the image in several steps is especially useful for gathering files from removable media, e.g. several older CDs.
Scripts using make-image can detect whether image creation is complete by checking the exit status: 0 signals successful creation, whereas 1 means that more files need to be supplied. Other errors result in an exit status of 2 (`recoverable', e.g. file not found) or 3 (non-recoverable, e.g. write error).
Reads `.jigdo', `.template' and (if present) `imagename.tmp', outputs a list of URIs still needed to completely reassemble the image.
Together with the make-image command, this provides most of the functionality of jigdo on the command line.
For each part that is not yet present in the temporary image file, the file checksum is looked up in the `[Parts]' section of the jigdo file. Any label in the corresponding entry is then expanded according to the label definitions in the `[Servers]' section and printed on standard output. jigdo allows you to specify several alternative locations for each label in this section, but print-missing will only output the first one for each missing part.
If the checksum cannot be found in the `[Parts]' section (this Should Not Happen unless you deleted that section), a lookup is instead made for `MD5Sum:<checksum>', just like with jigdo. (Thus, if you want to get rid of the `[Parts]' section, you can do so if you rename each part to its own checksum.)
Just like print-missing, this command outputs a list of URIs still needed to completely reassemble the image. However, all alternative download locations are printed instead of just one. In the output, the URIs for a file are separated from other files' URIs with blank lines. The --uri option has the same effect as for print-missing.
Reads image (presumably generated with make-image) and `.template', checks for correct checksum of image.
The template data does not only contain checksums of the individual parts, but also of the image as a whole. make-image already performs a number of internal checks, but if you like, you can additionally check the image with this command.
Reads all the FILES and prints out MD5 checksums of their contents. This command is quite similar to md5sum(1), except that the checksum is output in the Base64-like encoding which is also used elsewhere by jigdo-file.
The FILES arguments are processed in the same way as with the other commands, which means that recursion automatically takes place for any arguments that are directories, and that symbolic links are not listed except when the file(s) they point to are not reachable directly.
In the checksum list printed on standard output, only the part of the filename following any `//' (or `\.\' on Windows) is printed. Any --cache will be used for querying files' MD5 checksums and/or writing the checksums of scanned files.
Reads a `.template' file and outputs low-level information about the image and all parts contained in it, including offset, length and checksum.
You can also use this command with temporary image files (by specifying something like --template=imagename.tmp) - in that case, the output also distinguishes between parts that have been written to the image and parts that haven't.
The exact output format may change incompatibly between different jigdo releases. The following different types of lines can be output. `have-file' only occurs for `.tmp' files, indicating a file that has already been successfully written to the temporary file:
in-template offset-in-image length need-file offset-in-image length file-md5sum filestart-rsyncsum have-file offset-in-image length file-md5sum filestart-rsyncsum image-info image-length image-md5sum rsyncsum-size
Jigsaw Download was created with the format of ISO9660 CD images in mind - however, the following also applies to many other filesystem formats, as well as to `tar' archives and uncompressed `zip' archives. A CD image contains both information for organizing the filesystem (header with disc name etc., ISO9660 directory data, data of extensions such as Joliet or RockRidge, zero padding) and the files contained on the CD. An important property that jigdo relies on is that each file is stored in one contiguous section of the image; it is not split into two or more parts.
When jigdo-file is given a number of files that might be contained in an image, it detects whether any of the files are present using a `rolling checksum' inspired by the one used by rsync(1). The resulting data is written to the `.template' file: If a section of the image could not be matched (e.g. it was directory information), the data is compressed and written directly to the template. However, if a matching file was found, its data is omitted from the template. Instead, only a reference (an MD5 checksum of the file) is inserted in the template.
Note that the template data only contains binary data, it does not contain any filenames or URIs, since it cannot be easily edited in case any of these values need to be changed. All that information is stored in the `.jigdo' file, a text file to which you can add URLs for your server(s). The jigdo file provides a mapping for each MD5 checksum to one or more alternative download locations for the corresponding part.
Apart from the mapping of MD5 sums to URIs, the jigdo file also contains an URI pointing to a download location for the template file. This way, the jigdo download tool only needs to be given one URI (that of the `.jigdo' file) to be able to download and reassemble the complete image.
FORMAT OF .JIGDO FILES
The overall format of `.jigdo' files follows that of `.ini' files, as also used by the Gnome and KDE projects for some data. The file is organized into sections, each of which is preceded by a line reading `[Sectionname]'. Within each section, lines have the form `Label=Value'. Such lines are also called `entries' below.
Comments are introduced with the `#' character and extend to the end of the line. Whitespace is ignored at line start and end as well as to the left and right of section names and the `=' in entries. Furthermore, the jigdo utilities split up the text of the entry value (i.e. the part after the `=') into whitespace-separated words, much like the Unix shell. Single '' and double "" quotes can be used to prevent that e.g. URIs containing whitespace are split apart. Similarly, characters with special meaning (the characters '"#\ and space/tab) must be quoted with \ to appear in the value. As with the shell, there is a difference between '' and "": Within '', the characters "#\ and whitespace lose their special meaning, whereas within "", only the characters '# and whitespace lose their special meaning - in other words, backslash escapes still work inside "", but not ''.
[Jigdo] Version=1.0 Generator=jigdo-file/0.5.3 Info=long description
Information about the version of the jigdo file format used, and the program that generated it. The `Info' label can contain a description of the contents of the jigdo file.
[Image] Filename="filename for saving on user's disc" Template="URI where to fetch template file" Template-MD5Sum=OQ8riqT1BuyzsrT9964A7g Selected=yes|no ShortInfo=single-line description, for displaying menu of images Info=long description
The `Selected' label indicates whether the image is selected for download by default when jigdo is run for this file. (UNIMPLEMENTED)
Instead of an URI (which can be absolute or relative to the URL of the jigdo file), the value for the `Template' entry can also be a string of the form `Label:pathname' (UNIMPLEMENTED), as described below.
The `Template-MD5Sum' entry is added by jigdo-file and specifies the MD5 checksum of the generated `.template' file. It is used by jigdo to detect cases where the downloaded template data is corrupted or belongs to a different image.
[Parts] xJNkjrq8NYMraeGavUpllw=LabelA:part0 GoTResP2EC6Lb_2wTsqOoQ=LabelA:part1 kyfebwu6clbYqqWUdFIyaw=LabelB:some/path/part2 -J9UAimo0Bqg9c0oOXI1mQ=http://some.where.com/part3
All lines in the section, which provides the mapping from MD5 checksums to URIs, have the same format: On the left side of the `=' the checksum (encoded with a Base64-like encoding) is given, and on the right a string corresponding to the part with this checksum; either a complete URI or a string of the form `Label:pathname', which is expanded into one or more URIs by looking up the definition(s) for the Label in the `[Servers]' section.
In case a particular MD5 checksum cannot be found in any `[Parts]' section by jigdo, the program will perform a lookup for `MD5Sum:<checksum>', e.g. for `MD5Sum:xJNkjrq8NYMraeGavUpllw' if you deleted the line for `part0' above.
A checksum appearing multiple times in this section indicates alternative download locations for the part.
There may be any number of `[Parts]' sections in the file; they are all considered when looking up MD5 checksums.
jigdo-file always puts the `[Parts]' section at the end of the file, and it even rearranges any file specified with --merge to have only one such section at the end. This is done to allow jigdo to display the list of images contained in a jigdo file while the rest of that file is still being downloaded.
[Servers] LabelA=http://myserver.org/ LabelA=ftp://mirror.myserver.org/ LabelB=LabelC:subdirectory/ LabelC=http://some.where.com/jigdo/
All lines in the section, which provides the mapping from server labels to server locations, have the same format: On the left side of the `=' the label name is given, and on the right the value to expand the label name to.
A label name appearing multiple times in this section indicates alternative download locations for the parts that use the label in the `[Parts]' section. This notation makes it very easy to add mirrors to the jigdo file.
As shown by the example above, the label values may themselves reference other labels. In this case, the entry `LabelB:some/path/part2' in the `[Parts]' section will expand to `http://some.where.com/jigdo/subdirectory/some/path/part2'. Loops in the label definitions result in undefined behaviour and must be avoided.
[Comment] Any text, except that lines must not begin with `['.
Any file specified with the --cache option is used to store information about the FILES presented to jigdo-file. When querying the cache, a file is considered unchanged (and the cached data is used) only if filename, file size and last modification time (mtime) match exactly. For the filename match, not the entire file name is used, but only the part following any `//', so that any changes to the part before the `//' will not invalidate the cache.
Old cache entries are removed from the cache if they have not been read from or written to for the amount of time specified with --cache-expiry. Entries are not immediately removed from the cache if the file they refer to no longer exists - this makes it possible to cache information about files on removable media.
Cache expiry only takes place after jigdo-file has done its main work - if any old entries are accessed before expiry takes place, they will be kept. For example, if the program is run using the default expiry time of 30 days, but accesses a cache file with entries generated 2 months ago, then entries in that cache will be considered, and only those cache entries that were not needed during the program run will be expired.
Due to a peculiarity of the underlying database library (libdb3), cache files never shrink, they only grow. If a large number of entries was expired from your cache file and you want it to shrink, you can either just delete it (of course then everything will have to be regenerated) or use the utilities accompanying libdb3 to dump and restore the database, with a command like `db3_dump old-cache.db | db3_load new-cache.db'. For Debian, these programs are supplied in the package `libdb3-util'.
If a different --md5-block-size is specified, the entire file needs to be re-read to update its cache entry. If a different --min-length is specified, only the first `md5-block-size' bytes of the file need to be re-read.
PREPARING YOUR CD IMAGE FOR DISTRIBUTION
You have created a CD image `image.iso' from some of the files stored in the directory `/home/ftp' on your harddisc, which is also available online as `ftp://mysite.org'. As you don't want to waste space by effectively hosting the same data twice (once as files on the FTP server, once inside the image), and you are fed up with users' downloads aborting after 200MB and their restarting the download dozens of times, you decide to use jigdo. How do you prepare the image for download?
In fact, only one command is necessary:
Note that nothing prevents you from doing the same for an FTP server that isn't administrated by you - in that case, you only need to host the `.jigdo' and `.template' files on your own server/homepage.
PREPARING AN ARBITRARY LARGE FILE FOR DISTRIBUTION
We assume that you have a large file that is not a filesystem, e.g. `movie.mpeg'. Because of space problems, you want to distribute the data on two servers.
In this case, the parts of the image need to be generated artificially with the split command. For example, to create chunks of 4MB each, use `split -b 4m movie.mpeg part'. Copy the resulting files `partXX' into two directories `1' and `2' that you create, according to how you want the files distributed between the servers. Next, create the jigdo and template files with `jigdo-file make-template --image=movie.mpeg 1// 2//'. You will need to edit the `.jigdo' file and provide the right URIs for the two servers that you are going to upload the `partXX' files to.
CUSTOMIZED VERSIONS OF IMAGES
Because it is possible to assign a different URI for each part of an image if necessary, jigdo is very flexible. Only one example is the possibility of customized versions of images: Suppose that someone is distributing a CD image, and that you want to make a few small changes to it and redistribute your own version. You download the `official.iso' CD image with jigdo (passing it the URL of `official.jigdo'), write it to CD-R, make your changes (say, adding files from the `myfiles' directory on your harddisc) and produce your own version, `myversion.iso'. Next, you instruct jigdo-file to create the jigdo and template files for your modified image, using the command
Now you can upload the `.jigdo' file, the `.template' file and also the files in `myfiles' to `http://my.homepage.net/'. Thus, for people to download your modified image, you do not need to upload the complete image contents to your web space, but only the changes you made!
COMBINING MANY JIGDO-MANAGED IMAGES INTO ONE
It is also no problem to combine data from several sources that use jigdo. For example, if of five different and unrelated servers each one distributes a different CD image via jigdo, you can create a customized DVD image that contains the data from all these CDs. When people use jigdo to download your image, the individual files on the DVD are fetched from the same sources as the original CDs.
For certain contents of one of the input files, most notably a sequence of zero bytes longer than --min-length at the start of the file, jigdo-file make-template may fail to find the file in the image. Unfortunately, this restriction cannot be avoided because the program could become very slow otherwise. jigdo-file may also fail to find the file that follows an all-zeroes file in the image. To avoid this, exclude the all-zeroes file from FILES - this means that its data will end up in the template data, but an all-zeroes file can be compressed very well.
In fact, not only all-zeroes files trigger this behaviour, but also any files which contain at their start a long sequence of short identical strings. For example, both a file containing only `a' characters and one containing `abcabcabcabc...' are problematic.
This document was created by man2html, using the manual pages.
Time: 19:09:49 GMT, December 08, 2021