File-Unpack2

 view release on metacpan or  search on metacpan

MANIFEST  view on Meta::CPAN

t/data/lxknf09SCc0.bin
t/data/monotone.info
t/data/pdftex-a.txt
t/data/recursive-3level-removal-test.tar.gz
t/data/recursive-3level-removal-test.tar.xz
t/data/ruhyphal.tex
t/data/test.mht
t/data/test2.tga
t/data/Times-Roman-snippet.afm
t/data/wzbc-2009-06-28-17-00.m3u
t/data/xterm-snippet.desktop
t/pod-coverage.t
t/pod.t
META.yml                                 Module YAML meta-data (added by MakeMaker)
META.json                                Module JSON meta-data (added by MakeMaker)

README.md  view on Meta::CPAN

```
perl Makefile.PL
make
make test
sudo make install
```

The Perl prerequisites are pulled in by `Makefile.PL`. Unpacking itself shells out to the usual command line
tools (`tar`, `xz`, `unzip`, `rpm2cpio`, `cpio`, `7z`, `unrar`, `cabextract`, `pdftotext`, …); install the ones
you need for the formats you care about. `libmagic` and the freedesktop `shared-mime-info` database power
mime-type detection.

## Synopsis

As a library:

```perl
use File::Unpack2;

# Recursively unpack an archive into a destination directory, capturing a JSON

docs/Architecture.md  view on Meta::CPAN

## Identify by content, not by name

The first thing done to any file is to determine its mime type, and this is deliberately *not* based on the
file's suffix. Source packages are full of files whose names lie — a `.bin` that is really a zip, a `.gz` with
no extension at all, a shell script with an archive glued onto the end. Trusting the suffix would miss real
payload and waste effort on the wrong helper.

Detection is layered because no single engine is enough. The primary source is `libmagic` (via
`File::LibMagic`), the same engine as the `file` command, which returns a mime type, a charset and a
free-text description. The freedesktop `shared-mime-info` database (via `File::MimeInfo::Magic`) is consulted
where libmagic is weak. And a thin layer of the module's own logic handles cases that carry no reliable magic
of their own — most importantly raw LZMA, which is almost content-free at the start of the stream. The
human-readable description is cross-checked against the mime type, so a file that libmagic mislabels can be
caught and corrected before the wrong helper runs. The two magic libraries are loaded lazily; only mime
detection needs them, and the module degrades gracefully if one is missing.

## Dispatch to mime helpers

Once a file's type is known, unpacking it is delegated to a **mime helper** for that type. A helper is just
"the thing that knows how to open this kind of file". Common formats are covered by **built-in helpers**:

lib/File/Unpack2.pm  view on Meta::CPAN

$u->mime(buf => "#!/bin ...", file => "what-was-read")

$u->mime(fd => \*STDIN, file => "what-was-opened")

Determines the MIME type (and optionally additional information) of a file.
The file can be specified by filename, by a provided buffer or an opened file descriptor.
For the latter two cases, specifying a filename is optional, and used only for diagnostics.

C<mime> uses libmagic by Christos Zoulas exposed via File::LibMagic and also uses
the shared-mime-info database from freedesktop.org exposed via
File::MimeInfo::Magic, if available.  Either one is sufficient, but having both
is better. LibMagic sometimes says 'text/x-pascal', although we have a F<.desktop>
file, or says 'text/plain', but has contradicting details in its description.

C<File::MimeInfo::Magic::magic> is consulted where the libmagic output is dubious. E.g. when 
the desciption says something interesting like 'Debian binary package (format 2.0)' but the 
mimetype says 'application/octet-stream'. The combination of both libraries gives us 
excellent reliability in the critical field of MIME type recognition.

This implementation also features multi-level MIME type recognition for efficient unpacking.
When e.g. unpacking a large bzipped tar archive, this saves us from creating a
huge temporary tar-file which C<unpack> would extract in a second step.  The multi-level recognition

lib/File/Unpack2.pm  view on Meta::CPAN

  if ($mime1 =~ m{^application/xml})
    {
      # This is horrible from a greedy text cruncher perspective:
      # although xml is a plain text syntax, it is reported by flm to be 
      # outside text/*
      $r[0] = "text/x-application-xml";
    }

  if ($mime1 =~ m{^text/x-(?:pascal|fortran)$})
    {
      # xterm.desktop
      # ['text/x-pascal; charset=utf-8','UTF-8 Unicode Pascal program text']
      # 'application/x-desktop'
      #
      # Times-Roman.afm
      # ['text/x-fortran; charset=us-ascii','ASCII font metrics']
      # 'application/x-font-afm'
      #
      # debian/rules
      # ['text/x-pascal; charset=us-ascii','a /usr/bin/make -f  script text']
      # 'text/x-makefile'
      if ($mime2 ||= eval { open my $fd,'<',\$in{buf}; File::MimeInfo::Magic::magic($fd); })
        {

lib/File/Unpack2.pm  view on Meta::CPAN

  $r[3] = [ $mime1, $mime2 ] if $mime1 ne $r[0] or ($mime2 and $mime2 ne $mime1);

  return \@r;
}

=head1 MIME TYPE DETECTION

File::Unpack2 identifies files by content, not by name. Detection is layered:
L<File::LibMagic> (the same C<libmagic> engine as F</usr/bin/file>) is the primary
source of mime type, charset and a human description; L<File::MimeInfo::Magic>
(the freedesktop.org shared-mime-info database) fills the gaps where libmagic is
weak; and a little extra logic on top recognises compression that carries no
usable magic of its own, most notably raw LZMA. The C<description> string is
cross-checked against the mime type to catch mislabellings before a helper is
chosen. Both magic modules are loaded lazily and only L</mime> requires them.

=head1 SEE ALSO

=over 2

=item *

t/02-mime.t  view on Meta::CPAN

  ## these two are from SUSE:Factory:Head/qpdf%5.1.0%r23/qpdf-5.1.0/qpdf/qtest/qpdf/
  'bad34.pdf' => 
  	[ 'application/pdf', 'us-ascii', 'PDF document, version 1.3' ],
  'good10.pdf' => 
  	[ 'application/pdf', 'us-ascii', 'PDF document, version 1.3' ],

  ## 0.22 used to say application/x-lzma, but true binary data. Not even compressed.
  'lxknf09SCc0.bin' => 
  	[ 'application/octet-stream', qr{^(binary|unknown|)$} ], 

  ## actually 'application/x-desktop' or 'text/x-desktop'
  'Desktop.directory' => 
  	[ 'text/plain', 'utf-8', 'UTF-8 Unicode text' ],

  ## text/plain seen on 12.1, was text/x-desktop before
  'xterm-snippet.desktop' => 
  	[ qr{^text/(plain|x\-desktop)$}, 'utf-8', 
	 'UTF-8 Unicode Pascal program text', ['text/x-pascal','application/x-desktop']],

  'IPA-snippet.pfa' => 
  	[ 'text/x-font-type1', qr{^(us-ascii|)$}, 
	  'PostScript Type 1 font text (OmegaSerifIPA 001.000)', 
	  [ 'text/plain', 'application/x-font-type1' ] ],

  'Times-Roman-snippet.afm' => 
  	[ qr{^(application|text)/x-font-sunos-news$}, 
	  'us-ascii','ASCII font metrics',['text/x-fortran','application/x-font-sunos-news']], 

t/data/Desktop.directory  view on Meta::CPAN

[Desktop Entry]
BgImage=
Encoding=UTF-8
Icon=user-desktop
Name=Desktop
Name[af]=Werkskerm
Name[ar]=سطح المكتب
Name[az]=Masa Üstü
Name[be]=Працоўны стол
Name[bg]=Работен плот
Name[bn]=ডেস্কটপ
Name[br]=Gorretaol
Name[bs]=Radna površina
Name[ca]=Escriptori