view release on metacpan or  search on metacpan

lib/AnyEvent/Beanstalk/Worker.pm  view on Meta::CPAN

B<AnyEvent::Beanstalk::Worker> implements the following attributes.

=head2 beanstalk

This is a handle to the internal B<AnyEvent::Beanstalk> object.

=head2 job_count

This returns the number of outstanding jobs this worker is handling.

=head2 handled_jobs

This returns the number of jobs this worker has reserved and begun
work on.

=head2 concurrency

Sets or gets the number of jobs this worker can handle at the same
time.

=head1 SIGNALS

B<AnyEvent::Beanstalk::Worker> receives the following signals:

=head2 INT

A C<INT> signal will cause the worker to invoke its B<stop> method,
which will process any outstanding events before shutting down.

=head2 TERM

A C<TERM> signal is handled in the same way as C<INT>.

=head2 USR2

A C<USR2> signal will bump the log level of the worker up until it
reaches I<trace>; after trace it wraps around and starts again at
I<critical>. See L<AnyEvent::Log> for available log levels.

=head1 LOGGING

B<AnyEvent::Beanstalk::Worker> implements logging via
B<AnyEvent::Log>; it probably doesn't do this as well as it could and
more work needs to be done here.

=head1 EXAMPLES

The F<eg> directory has several working examples of using this module,
including one that shows how to subclass it.

=head1 SUPPLEMENTAL

This section contains additional information not directly needed to
use this module, but may be useful for those unfamiliar with any of
the underlying technologies.

=head2 Caveat

This module represents the current results of an ongoing experiment
involving queues (beanstalk, AnyEvent::Beanstalk), non-blocking and
asynchronous events (AnyEvent), and state machines as means of a
simpler to understand method of event-driven programming.

=head2 Introduction to beanstalk

B<beanstalkd> is a small, fast work queue written in C. When you need
to do lots of jobs (work units--call them what you will), such as
sending an email, fetching and parsing a web page, image processing,
etc.), a I<producer> (a small worker that creates jobs) adds jobs to
the queue. One or more I<consumer> workers come along and ask for jobs
from the queue, and then work on them. When the consumer worker is
done, it deletes the job from the queue and asks for another job.

=head2 Introduction to AnyEvent

B<AnyEvent> is an elegantly designed, generic interface to a variety
of event loops.

=head2 Introduction to state machines

The idea behind state machines is you have a "machine" (or program
modeling a machine) with a set of I<states> and a set of events that
when triggered alter the state of the machine. For example, we could
model a web crawler as a state machine. Our states will be I<get url>,
I<fetch>, I<parse>, and I<add url>, and our events will be I<got url>,
I<fetched>, I<parsed>, and I<added>.

                +---------+
                | get url |
                +-/-----^-+
      (got url)  /       \
                /         \ (added)
         +-----v-+     +---\-----+
         | fetch |     | add url |
         +-----\-+     +-^-------+
      (fetched) \       /
                 \     / (parsed)
                +-v---/-+
                | parse |
                +-------+

In the I<get url> state, we take a URL from a list of URLs (perhaps we
seed it with one URL), then we emit the I<got url> event. This causes
our machine to move to the I<fetch> state. In the I<fetch> state, we
make an HTTP C<GET> request on that URL and then emit the I<fetched>
event, which moves our machine to the I<parse> state where we parse
the incoming web page. Then we add any URLs we find into the queue and
start over.

If we use our B<WebWorker> class above, the result might look like
this:

    #!/usr/bin/env perl
    use strict;
    use warnings;
    use feature 'say';

    use WebWorker;

    my $w = WebWorker->new
      ( concurrency     => 1,

lib/AnyEvent/Beanstalk/Worker.pm  view on Meta::CPAN

            return $self->finish(delete => $job->id);
        }

        say STDERR "parsing " . $job->data;
        eval {
            $tx->res->dom->at("html body")->find('a[href]')
              ->each(sub { $self->emit(add_url => shift->{href}) });
        };

        return $self->finish(delete => $job->id);
    });

    $w->on(add_url => sub {
        my ($self, $url) = @_;

        return unless $url =~ /^http/;

        $self->beanstalk
          ->put({ priority => 100,
                  ttr      => 15,
                  delay    => 1,
                  data     => $url },
                sub { say STDERR "URL $url added" });
    });

    $w->start;

    AnyEvent->condvar->recv;

We've just written a simple (and impolite--should read F<robots.txt>)
web crawler.

See F<eg/web-state.pl> and F<eg/web-state-add.pl> for this example.

=head2 Introduction to event loops

I couldn't find any gentle introductions into event loops; I was going
to write one myself but realized it would probably turn into a
book. Additionally, I'm not qualified to write said book. With that
disclaimer, here is a brief, "close enough" introduction to event
loops which may help some people get an approximate mental model, good
enough to begin event programming.

An event loop can be as simple as this:

    my @events = ();
    my %watchers = ();

    while (1) {
        my $event = pop @events;
        handle($event);
    }

    sub handle {
        my $event = shift;

        $_->($event) for @{$watchers{$event->{type}}};
    }

The C<@events> list (or queue, since events are read as a FIFO) might
be populated asynchronously from system events, such as receiving
signals, network data, disk I/O, timers, or other sources. The
C<handle()> subroutine checks the C<%watchers> hash to see if there
are any watchers or handlers for this event and calls those
subroutines as needed. Some of these subroutines may add more events
to the event queue. Then the loop starts again.

Most of the time you never see the event loop--you just start it. For
example, most of the time when I'm programming with B<EV>, this is all
I ever see of it:

    EV::run;

B<EV> receives all kinds of events from the system, but you can tell
it about more events. Then you register event I<handlers> to fire off
when a particular kind of event is received.

=head1 SEE ALSO

B<beanstalkd>, by Keith Rarick: L<http://kr.github.io/beanstalkd/>

B<AnyEvent::Beanstalk>, by Graham Barr: L<AnyEvent::Beanstalk>

B<AnyEvent>, by Marc Lehmann: L<http://anyevent.schmorp.de>

=head1 AUTHOR

Scott Wiersdorf, E<lt>scott@perlcode.orgE<gt>

=head1 COPYRIGHT AND LICENSE

Copyright (C) 2014 by Scott Wiersdorf

This library is free software; you can redistribute it and/or modify
it under the same terms as Perl itself, either Perl version 5.16.1 or,
at your option, any later version of Perl 5 you may have available.

=cut