view release on metacpan or  search on metacpan

Encode.pm  view on Meta::CPAN

and to convert it back:

  from_to($octets, "cp1250", "iso-8859-1");

Because the conversion happens in place, the data to be
converted cannot be a string constant: it must be a scalar variable.

C<from_to()> returns the length of the converted string in octets on success,
and C<undef> on error.

B<CAVEAT>: The following operations may look the same, but are not:

  from_to($data, "iso-8859-1", "UTF-8"); #1
  $data = decode("iso-8859-1", $data);  #2

Both #1 and #2 make $data consist of a completely valid UTF-8 string,
but only #2 turns the UTF8 flag on.  #1 is equivalent to:

  $data = encode("UTF-8", decode("iso-8859-1", $data));

See L</"The UTF8 flag"> below.

Also note that:

  from_to($octets, $from, $to, $check);

is equivalent to:

  $octets = encode($to, decode($from, $octets), $check);

Yes, it does I<not> respect the $check during decoding.  It is
deliberately done that way.  If you need minute control, use C<decode>
followed by C<encode> as follows:

  $octets = encode($to, decode($from, $octets, $check_from), $check_to);

=head3 encode_utf8

  $octets = encode_utf8($string);

B<WARNING>: L<This function can produce invalid UTF-8!|/UTF-8 vs. utf8 vs. UTF8>
Do not use it for data exchange.
Unless you want Perl's older "lax" mode, prefer
C<$octets = encode("UTF-8", $string)>.

Equivalent to C<$octets = encode("utf8", $string)>.  The characters in
$string are encoded in Perl's internal format, and the result is returned
as a sequence of octets.  Because all possible characters in Perl have a
(loose, not strict) utf8 representation, this function cannot fail.

=head3 decode_utf8

  $string = decode_utf8($octets [, CHECK]);

B<WARNING>: L<This function accepts invalid UTF-8!|/UTF-8 vs. utf8 vs. UTF8>
Do not use it for data exchange.
Unless you want Perl's older "lax" mode, prefer
C<$string = decode("UTF-8", $octets [, CHECK])>.

Equivalent to C<$string = decode("utf8", $octets [, CHECK])>.
The sequence of octets represented by $octets is decoded
from (loose, not strict) utf8 into a sequence of logical characters.
Because not all sequences of octets are valid not strict utf8,
it is quite possible for this function to fail.
For CHECK, see L</"Handling Malformed Data">.

B<CAVEAT>: the input I<$octets> might be modified in-place depending on
what is set in CHECK. See L</LEAVE_SRC> if you want your inputs to be
left unchanged.

=head2 Listing available encodings

  use Encode;
  @list = Encode->encodings();

Returns a list of canonical names of available encodings that have already
been loaded.  To get a list of all available encodings including those that
have not yet been loaded, say:

  @all_encodings = Encode->encodings(":all");

Or you can give the name of a specific module:

  @with_jp = Encode->encodings("Encode::JP");

When "C<::>" is not in the name, "C<Encode::>" is assumed.

  @ebcdic = Encode->encodings("EBCDIC");

To find out in detail which encodings are supported by this package,
see L<Encode::Supported>.

=head2 Defining Aliases

To add a new alias to a given encoding, use:

  use Encode;
  use Encode::Alias;
  define_alias(NEWNAME => ENCODING);

After that, I<NEWNAME> can be used as an alias for I<ENCODING>.
I<ENCODING> may be either the name of an encoding or an
I<encoding object>.

Before you do that, first make sure the alias is nonexistent using
C<resolve_alias()>, which returns the canonical name thereof.
For example:

  Encode::resolve_alias("latin1") eq "iso-8859-1" # true
  Encode::resolve_alias("iso-8859-12")   # false; nonexistent
  Encode::resolve_alias($name) eq $name  # true if $name is canonical

C<resolve_alias()> does not need C<use Encode::Alias>; it can be
imported via C<use Encode qw(resolve_alias)>.

See L<Encode::Alias> for details.

=head2 Finding IANA Character Set Registry names

The canonical name of a given encoding does not necessarily agree with
IANA Character Set Registry, commonly seen as C<< Content-Type:

Encode.pm  view on Meta::CPAN

When you decode, the Unicode REPLACEMENT CHARACTER, code point U+FFFD, is
used.  If the data is supposed to be UTF-8, an optional lexical warning of
warning category C<"utf8"> is given.

=head3 FB_CROAK

  CHECK = Encode::FB_CROAK ( == 1)

If I<CHECK> is 1, methods immediately die with an error
message.  Therefore, when I<CHECK> is 1, you should trap
exceptions with C<eval{}>, unless you really want to let it C<die>.

=head3 FB_QUIET

  CHECK = Encode::FB_QUIET

If I<CHECK> is set to C<Encode::FB_QUIET>, encoding and decoding immediately
return the portion of the data that has been processed so far when an
error occurs. The data argument is overwritten with everything
after that point; that is, the unprocessed portion of the data.  This is
handy when you have to call C<decode> repeatedly in the case where your
source data may contain partial multi-byte character sequences,
(that is, you are reading with a fixed-width buffer). Here's some sample
code to do exactly that:

    my($buffer, $string) = ("", "");
    while (read($fh, $buffer, 256, length($buffer))) {
        $string .= decode($encoding, $buffer, Encode::FB_QUIET);
        # $buffer now contains the unprocessed partial character
    }

=head3 FB_WARN

  CHECK = Encode::FB_WARN

This is the same as C<FB_QUIET> above, except that instead of being silent
on errors, it issues a warning.  This is handy for when you are debugging.

B<CAVEAT>: All warnings from Encode module are reported, independently of
L<pragma warnings|warnings> settings. If you want to follow settings of
lexical warnings configured by L<pragma warnings|warnings> then append
also check value C<ENCODE::ONLY_PRAGMA_WARNINGS>. This value is available
since Encode version 2.99.

=head3 FB_PERLQQ FB_HTMLCREF FB_XMLCREF

=over 2

=item perlqq mode (I<CHECK> = Encode::FB_PERLQQ)

=item HTML charref mode (I<CHECK> = Encode::FB_HTMLCREF)

=item XML charref mode (I<CHECK> = Encode::FB_XMLCREF)

=back

For encodings that are implemented by the C<Encode::XS> module, C<CHECK> C<==>
C<Encode::FB_PERLQQ> puts C<encode> and C<decode> into C<perlqq> fallback mode.

When you decode, C<\xI<HH>> is inserted for a malformed character, where
I<HH> is the hex representation of the octet that could not be decoded to
utf8.  When you encode, C<\x{I<HHHH>}> will be inserted, where I<HHHH> is
the Unicode code point (in any number of hex digits) of the character that
cannot be found in the character repertoire of the encoding.

The HTML/XML character reference modes are about the same. In place of
C<\x{I<HHHH>}>, HTML uses C<&#I<NNN>;> where I<NNN> is a decimal number, and
XML uses C<&#xI<HHHH>;> where I<HHHH> is the hexadecimal number.

In C<Encode> 2.10 or later, C<LEAVE_SRC> is also implied.

=head3 The bitmask

These modes are all actually set via a bitmask.  Here is how the C<FB_I<XXX>>
constants are laid out.  You can import the C<FB_I<XXX>> constants via
C<use Encode qw(:fallbacks)>, and you can import the generic bitmask
constants via C<use Encode qw(:fallback_all)>.

                     FB_DEFAULT FB_CROAK FB_QUIET FB_WARN  FB_PERLQQ
 DIE_ON_ERR    0x0001             X
 WARN_ON_ERR   0x0002                               X
 RETURN_ON_ERR 0x0004                      X        X
 LEAVE_SRC     0x0008                                        X
 PERLQQ        0x0100                                        X
 HTMLCREF      0x0200
 XMLCREF       0x0400

=head3 LEAVE_SRC

  Encode::LEAVE_SRC

If the C<Encode::LEAVE_SRC> bit is I<not> set but I<CHECK> is set, then the
source string to encode() or decode() will be overwritten in place.
If you're not interested in this, then bitwise-OR it with the bitmask.

=head2 coderef for CHECK

As of C<Encode> 2.12, C<CHECK> can also be a code reference which takes the
ordinal value of the unmapped character as an argument and returns
octets that represent the fallback character.  For instance:

  $ascii = encode("ascii", $utf8, sub{ sprintf "<U+%04X>", shift });

Acts like C<FB_PERLQQ> but U+I<XXXX> is used instead of C<\x{I<XXXX>}>.

Fallback for C<decode> must return decoded string (sequence of characters)
and takes a list of ordinal values as its arguments. So for
example if you wish to decode octets as UTF-8, and use ISO-8859-15 as
a fallback for bytes that are not valid UTF-8, you could write

    $str = decode 'UTF-8', $octets, sub {
        my $tmp = join '', map chr, @_;
        return decode 'ISO-8859-15', $tmp;
    };

=head1 Defining Encodings

To define a new encoding, use:

    use Encode qw(define_encoding);
    define_encoding($object, CANONICAL_NAME [, alias...]);

I<CANONICAL_NAME> will be associated with I<$object>.  The object
should provide the interface described in L<Encode::Encoding>.
If more than two arguments are provided, additional
arguments are considered aliases for I<$object>.

See L<Encode::Encoding> for details.

=head1 The UTF8 flag

Before the introduction of Unicode support in Perl, the C<eq> operator
just compared the strings represented by two scalars. Beginning with
Perl 5.8, C<eq> compares two strings with simultaneous consideration of
I<the UTF8 flag>. To explain why we made it so, I quote from page 402 of
I<Programming Perl, 3rd ed.>

=over 2

=item Goal #1:

Old byte-oriented programs should not spontaneously break on the old
byte-oriented data they used to work on.

=item Goal #2:

Old byte-oriented programs should magically start working on the new
character-oriented data when appropriate.

=item Goal #3:

Programs should run just as fast in the new character-oriented mode
as in the old byte-oriented mode.

=item Goal #4:

Perl should remain one language, rather than forking into a
byte-oriented Perl and a character-oriented Perl.

=back

When I<Programming Perl, 3rd ed.> was written, not even Perl 5.6.0 had been
born yet, many features documented in the book remained unimplemented for a
long time.  Perl 5.8 corrected much of this, and the introduction of the
UTF8 flag is one of them.  You can think of there being two fundamentally
different kinds of strings and string-operations in Perl: one a