view release on metacpan or  search on metacpan

lib/Lingua/Identify.pm  view on Meta::CPAN

read and their content merged and then parsed for language
identification.

=cut

sub langof_file {
  my %config = ();
  if (ref($_[0]) eq 'HASH') {%config = (%config, %{+shift})}

=head3 OPTIONS

I<langof_file> accepts all the options I<langof> does, so refer to
those first (up in this document).

  $language = langof_file(\%config, $path);

I<langof_file> currently only reads the first 10,000 bytes of each
file.

You can force an input encoding with C<< { encoding => 'ISO-8859-1' } >> 
in the configuration hash.

=cut

  # select max-size
  my $maxsize = defined $config{'max-size'} ? $config{'max-size'}
                                            : $default_maxsize;

  my @files = @_;
  my $text  = '';

  for my $file (@files) {
      #-r and -e or next;
      if (exists($config{encoding})) {
          open(FILE, "<:encoding($config{encoding})", $file) or next;
      } else {
          open(FILE, "<:utf8", $file) or next;
      }
      local $/ = \$maxsize;
      $text .= <FILE>;
      close(FILE);
  }

  return langof(\%config,$text);
}

=head2 confidence

After getting the results into an array, its first element is the most probable
language. That doesn't mean it is very probable or not.

You can find more about the likeliness of the results to be accurate by
computing its confidence level.

  use Lingua::Identify qw/:language_identification/;
  my @results = langof($text);
  my $confidence_level = confidence(@results);
  # $confidence_level now holds a value between 0.5 and 1; the higher that
  # value, the more accurate the results seem to be

The formula used is pretty simple: p1 / (p1 + p2) , where p1 is the
probability of the most likely language and p2 is the probability of
the language which came in second. A couple of examples to illustrate
this:

English 50% Portuguese 10% ...

confidence level: 50 / (50 + 10) = 0.83

Another example:

Spanish 30% Portuguese 10% ...

confidence level: 30 / (25 + 30) = 0.55

French 10% German 5% ...

confidence level: 10 / (10 + 5) = 0.67

As you can see, the first example is probably the most accurate one.
Are there any doubts? The English language has five times the
probability of the second language.

The second example is a bit more tricky. 55% confidence. The
confidence level is always above 50%, for obvious reasons. 55% doesn't
make anyone confident in the results, and one shouldn't be, with
results such as these.

Notice the third example. The confidence level goes up to 67%, but the
probability of French is of mere 10%. So what? It's twice as much as
the second language. The low probability may well be caused by a great
number of languages in play.

=cut

sub confidence {
    defined $_[1] and $_[1] or return 0;
    defined $_[3] and $_[3] or return 1;
    $_[1] / ($_[1] + $_[3]);
}

=head2 get_all_methods

Returns a list comprised of all the available methods for language
identification.

=cut

sub get_all_methods {
  qw/smallwords
     prefixes1 prefixes2 prefixes3 prefixes4
     suffixes1 suffixes2 suffixes3 suffixes4
     ngrams1 ngrams2 ngrams3 ngrams4/
}

=head1 LANGUAGE IDENTIFICATION IN GENERAL

Language identification is based in patterns.

In order to identify the language a given text is written in, we repeat a given
process for each active language (see section LANGUAGES MANIPULATION); in that