view release on metacpan or  search on metacpan

docs/Architecture.md  view on Meta::CPAN

# File::Unpack2 Architecture

This document explains how the unpacker works and, more importantly, *why* it is built the way it is. It is
meant to be read start to finish by someone new to the project — most likely someone working their way into
[Cavil](https://github.com/openSUSE/cavil) — before they open the source. It talks about concepts, not
functions or line numbers.

## Why this exists

Cavil reviews the licensing of software by scanning source code for the text of known licenses. Before it can
scan anything, it has to get the text *out* of whatever a package happens to be: a tarball, an RPM, a zip
inside a tarball inside an RPM, a PDF, an office document. `File::Unpack2` is the component that does that. Its
one job is to take an arbitrary input and expose as much readable payload as possible as ordinary files on
disk, so the rest of Cavil can read them.

That single goal — *expose everything readable* — explains most of the design decisions below. Unpacking goes
as deep as it can rather than stopping at the first layer; it identifies files by content so it is not fooled
by names; and when one file cannot be unpacked it is logged and skipped rather than aborting the whole run,
because a distribution has thousands of files and one bad apple must not sink the batch.

## Identify by content, not by name

The first thing done to any file is to determine its mime type, and this is deliberately *not* based on the
file's suffix. Source packages are full of files whose names lie — a `.bin` that is really a zip, a `.gz` with
no extension at all, a shell script with an archive glued onto the end. Trusting the suffix would miss real
payload and waste effort on the wrong helper.

Detection is layered because no single engine is enough. The primary source is `libmagic` (via
`File::LibMagic`), the same engine as the `file` command, which returns a mime type, a charset and a
free-text description. The freedesktop `shared-mime-info` database (via `File::MimeInfo::Magic`) is consulted
where libmagic is weak. And a thin layer of the module's own logic handles cases that carry no reliable magic
of their own — most importantly raw LZMA, which is almost content-free at the start of the stream. The
human-readable description is cross-checked against the mime type, so a file that libmagic mislabels can be
caught and corrected before the wrong helper runs. The two magic libraries are loaded lazily; only mime
detection needs them, and the module degrades gracefully if one is missing.

## Dispatch to mime helpers

Once a file's type is known, unpacking it is delegated to a **mime helper** for that type. A helper is just
"the thing that knows how to open this kind of file". Common formats are covered by **built-in helpers**:
entries baked into the module that wrap the standard command line tools — `tar`, `xz`/`lzcat`, `unzip`,
`rpm2cpio` piped into `cpio`, `7z`, `unrar`, `cabextract`, `ar` for `.deb`, `pdftotext`/`pdfimages` for PDFs,
and so on. Each entry pairs a mime-type pattern with a suffix hint and the command (with redirections and
pipelines) to run.

The built-ins cover the archive and compression formats a distribution is made of. For anything else, the
**extension mechanism** lets a consumer teach File::Unpack2 a new format without changing the module.

### Extending: two ways to add a helper

- **Programmatically**, with the `mime_helper` method: register a command for a mime type, using `%(src)s`,
  `%(destfile)s` and friends as placeholders that are filled in at call time. This is how a consumer adds a
  format it cares about; Cavil, for instance, registers a `zstd` helper this way at startup.
- **From a directory**, with the `mime_helper_dir` method (or the `FILE_UNPACK2_HELPER_DIR` environment
  variable, or the `helper_dir` constructor argument): every executable in the directory is registered as a
  helper, named after the mime type it handles. Nothing is scanned unless a directory is explicitly configured.

The naming convention ties a helper to its type. A helper's name (or the `mime_helper` pattern) is the mime
type with `/` written as `=` (filesystems dislike `/` in names), and an `x-` or `ANY+` prefix after the `=` is
treated as implicit — so `application=x-debian-package` handles `application/x-debian-package`. When several
registered helpers could match, the most specific wins: an exact name beats one with wildcards, a wildcard
*after* the `=` beats one before it, and a more-recently-added helper takes precedence. This lets a consumer
override a built-in simply by registering a helper of the same name.

### Writing a mime helper

A directory helper is an ordinary executable — a few lines of shell is typical. It is invoked with its working
directory already set to a fresh, empty output directory, and it is handed six arguments:

```
$1  source path        the file to unpack (absolute)
$2  suggested name     a destination name the helper may use
$3  destination dir    the output directory (also the current working directory)
$4  mime type          as detected
$5  description         libmagic's human-readable description
$6  config dir          a directory holding a JSON dump of the unpacker's config
```

Its job is to place the unpacked contents into the current directory, using *relative* paths only. It signals
success with exit status zero; a non-zero exit is recorded as an error against that one file, and unpacking
continues elsewhere. A helper that determines the file is already as unpacked as it can be may symlink the
suggested name to the source to say "take it as is", which stops recursion into it. Because the contract is
this small, and because it is the same whether the helper is built in, registered by `mime_helper`, or found
in a directory, adding format support is cheap. Both registration styles are demonstrated end to end in
`t/12-mime-helper.t`.

## Recursion: peeling every layer

Unpacking is recursive. After a helper runs, every file it produced is itself fed back through the same
identify-and-dispatch process, so nested archives are peeled apart layer by layer until nothing left looks