view release on metacpan or  search on metacpan

lib/AI/MicroStructure/RemoteList.pm  view on Meta::CPAN

            ref $src
            ? ( POST => $src->[0],
                [ content_type => 'application/x-www-form-urlencoded' ],
                $src->[1]
                )
            : ( GET => $src )
        );

        my $res = $ua->request( $request );
        if ( ! $res->is_success() ) {
            carp "Failed to get content at $src (" . $res->status_line();
            return;
        }

        # extract, cleanup and return the data
        # if decoding the content fails, we just deal with the raw content
        push @items =>
            $class->extract( $res->decoded_content() || $res->content(),
               $category || () );

    }

    # return unique items
    my %seen;
    return grep { !$seen{$_}++ } @items;
}

#
# transformation subroutines
#
sub tr_nonword {
    my $str = shift;
    $str =~ tr/a-zA-Z0-9_/_/c;
    $str;
}

sub tr_accent {
    my $str = shift;
    $str =~ tr{ÀÁÂÃÄÅÇÈÉÊËÌÍÎÏÑÒÓÔÕÖØÙÚÛÜÝàáâãäåçèéêëìíîïñòóôõöøùúûüýÿ}
              {AAAAAACEEEEIIIINOOOOOOUUUUYaaaaaaceeeeiiiinoooooouuuuyy};
    return $str;
}

my %utf2asc = (
    "æ"        => 'ae',
    "Æ"        => 'AE',
    "\xc5\xa0" => 'S',
    "\x{0160}" => 'S',
    # for pokemons
    "\x{0101}"     => 'a',
    "\x{012b}"     => 'i',
    "\x{014d}"     => 'o',
    "\x{016b}"     => 'u',
    "\xe2\x99\x80" => 'female',
    "\xe2\x99\x82" => 'male',
    "\x{2640}"     => 'female',
    "\x{2642}"     => 'male',
);
my $utf_re = qr/(@{[join( '|', sort keys %utf2asc )]})/;

sub tr_utf8_basic {
    my $str = shift;
    $str =~ s/$utf_re/$utf2asc{$1}/go;
    return $str;
}

1;

__END__

=head1 NAME

AI::MicroStructure::RemoteList - Retrieval of a remote source for a structure

=head1 SYNOPSIS

    package AI::MicroStructure::contributors;
    use strict;
    use AI::MicroStructure::List;
    our @ISA = qw( AI::MicroStructure::List );

    # data regarding the remote source
    our %Remote = (
        source =>
            'http://search.cpan.org/dist/AI-MicroStructure/CONTRIBUTORS',
        extract => sub {
            my $content = shift;
            my @items   =
                map { AI::MicroStructure::RemoteList::tr_nonword($_) }
                map { AI::MicroStructure::RemoteList::tr_accent($_) }
                $content =~ /^\* (.*?)\s*$/gm;
            return @items;
        },
    );

    __PACKAGE__->init();

    1;

    # and the usual documentation and list definition

=head1 DESCRIPTION

This base class adds the capability to fetch a fresh list of items from a
remote source to any structure that requires it.

To be able to fetch remote items, an C<AI::MicroStructure> structure must
define the package hash variable C<%Remote> with the appropriate keys.

The keys are:

=over 4

=item C<source>

The URL where the data is available. The content will be passed to the
C<extract> subroutine.

Because of the various way the data can be made available on the web
and can be used in L<AI::MicroStructure>, this scheme has evolved to
support several cases:

lib/AI/MicroStructure/RemoteList.pm  view on Meta::CPAN

(C<AI::MicroStructure::List> and C<AI::MicroStructure::Locale> are
subclasses of C<AI::MicroStructure::RemoteList>.

=head1 METHODS

As an ancestor, this class adds the following methods to an
C<AI::MicroStructure> structure:

=over 4

=item remote_list()

Returns the list of items available at the remote source, or an empty
list in case of error.

=item has_remotelist()

Return a boolean indicating if the C<source> key is defined (and therefore
if the structure actually has a remote list).

=item source()

Return the data structure containing the source URLs. This can be quite
different depending on the class: a single scalar (URL), an array
reference (list of URLs) or a hash reference (each value being either
a scalar or an array reference) for structures that are subclasses of
C<AI::MicroStructure::MultiList>.

=item sources( [ $category ] )

Return the list of source URL. The C<$category> parameter can be used
to select the sources for a sub-category of the structure (in the case of
C<AI::MicroStructure::MultiList>).

C<$category> can be an array reference containing a list of categories.

=item extract( $content )

Return a list of items from the C<$content> string. C<$content> is
expected to be the content available at the URL given by C<source()>.

=back

=head1 TRANSFORMATION SUBROUTINES

The C<AI::MicroStructure::RemoteList> class also provides a few helper
subroutines that simplify the normalisation of items:

=over 4

=item tr_nonword( $str )

Return a copy of C<$str> with all non-word characters turned into
underscores (C<_>).

=item tr_accent( $str )

Return a copy of C<$str> will all iso-8859-1 accented characters turned
into basic ASCII characters.

=item tr_utf8_basic( $str )

Return a copy of C<$str> with some of the utf-8 accented characters turned
into basic ASCII characters. This is very crude, but I didn't to bother
and depend on the proper module to do that.

=back

=head1 AUTHOR

'santex'  << <santex@cpan.org> >>.

=head1 SEE ALSO

L<AI::MicroStructure>, L<AI::MicroStructure::List>,
L<AI::MicroStructure::Locale>.

=head1 COPYRIGHT

Copyright 2009-2016 Hagen Geissler, All Rights Reserved.

=head1 LICENSE
This program is free software; you can redistribute it and/or modify it
under the same terms as Perl itself.

=cut