view release on metacpan or  search on metacpan

devdata/http_advent.perldancer.org_2018_16  view on Meta::CPAN

<p>With that in place, it was easy for our <code>queue_job()</code> method to throw an error if the developer tried to add a job to an invalid queue:</p>
<pre class="prettyprint">sub queue_job( $self, $args ) {
    my $job_name = $args-&gt;{ name     } or die "queue_job(): must define job name!";
    my $guid     = $args-&gt;{ guid     } or die "queue_job(): must have GUID to process!";
    my $title    = $args-&gt;{ title    } // $job_name;
    my $queue    = $args-&gt;{ queue    } // 'default';
    my $job_args = $args-&gt;{ job_args };

    die "queue_job(): Invalid job queue '$queue' specified" 
        if $self-&gt;has_invalid_queues( $queue );

    my %notes = ( title =&gt; $title, guid  =&gt; $guid );

    return $self-&gt;runner-&gt;enqueue( $job_name =&gt; $job_args =&gt; 
        { notes =&gt; \%notes, queue =&gt; $queue });
}</pre>

<h2><a name="creating_jobs"></a>Creating Jobs</h2>

<p>In our base model class (Moose-based), we would create an attribute for our job runner:</p>
<pre class="prettyprint">has 'job_runner' =&gt; (
    is      =&gt; 'ro',
    isa     =&gt; 'MyJob::JobQueue',
    lazy    =&gt; 1,
    default =&gt; sub( $self ) { return MyJob::JobQueue-&gt;new-&gt;runner; },
);</pre>

<p>And in the models themselves, creating a new queueable task was as easy as:</p>
<pre class="prettyprint">$self-&gt;runner-&gt;add_task( InstantXML =&gt; 
    sub( $job, $request_path, $guid, $company_db, $force, $die_on_error = 0 ) {
        $job-&gt;note( 
            request_path =&gt; $request_path,
            feed_id      =&gt; 2098,
            group        =&gt; $company_db,
        );
        MyJob::Models::FooBar-&gt;new( request_path =&gt; 
          $request_path )-&gt;generate_xml({
            pdf_guid     =&gt; $guid,
            group        =&gt; $company_db,
            force        =&gt; $force,
            die_on_error =&gt; $die_on_error,
        });
});</pre>

<h2><a name="running_jobs"></a>Running Jobs</h2>

<p>Starting a job from Dancer was super easy:</p>
<pre class="prettyprint">use Dancer2;
use MyJob::JobQueue;

sub job_queue {
    return MyJob::JobQueue-&gt;new;
}

get '/my/api/route/:guid/:group/:force' =&gt; sub {
    my $guid  = route_parameters-&gt;get( 'guid' );
    my $group = route_parameters-&gt;get( 'group' );
    my $force = route_parameters-&gt;get( 'force' );

    debug "GENERATING XML ONLY FOR $guid";
    job_queue-&gt;queue_job({
        name     =&gt; "InstantXML",
        guid     =&gt; $guid,
        title    =&gt; "Instant XML Generator",
        queue    =&gt; 'InstantXML',
        job_args =&gt; [ $self-&gt;request_path, $guid, $group, $force ],
    }); 
}</pre>

<h2><a name="creating_and_configuring_the_job_queue_worker"></a>Creating and Configuring the Job Queue Worker</h2>

<p>We wanted to easily configure our Minions for all hosts and environments in one spot. Since we use a lot of YAML in Dancer, specifying the Minion configuration in YAML made a lot of sense
to us:</p>
<pre class="prettyprint"># What port does the dashboard listen on?
dashboard_port: 4000

# Add the rest later.
dashboards:
    UNKNOWN: http://localhost:3000/
    DEV: http://my.development.host.tld:8001/

# Hosts that have no entry assume the default configuration
default:
    max_children: 4
    queues:
        - default

# Host-specific settings
jcrome-precision-3510:
    max_children: 8
    queues:
        - default
        - InstantXML
        - PayrollXML
        - ChangeRequest</pre>

<p>Our job queue workers look like:</p>
<pre class="prettyprint">#!/usr/bin/env perl

use MyJob::Base;
use MyJob::JobQueue;
use MyJob::Log4p;
use MyJob::Util::Logger;
use MyJob::Util::SysTools qw(get_hostname);

my $config     = MyJob::Config-&gt;new-&gt;config;
my $hostconfig = get_hostconfig();
my $minion     = MyJob::JobQueue-&gt;new;
my $worker     = $minion-&gt;runner-&gt;worker;

my $log_eng = MyJob::Log4p-&gt;new({ logger_name =&gt; "Minion" });
my $logger  = MyJob::Util::Logger-&gt;new-&gt;logger($log_eng);</pre>

<p>The above is mostly typical boilerplate for us. Read our configuration, and create a logger the worker can use.</p>
<p>Next, when a job is dequeued, we want to log that the worker picked up a job (needed for auditing purposes) and we alter the process name so if a process hangs, we know what that process
was attempting to run. If an unchecked exception occurs in a job, the worker will catch it and log it for us:</p>
<pre class="prettyprint">$worker-&gt;on( dequeue =&gt; sub( $worker, $job ) {
    my $id    = $job-&gt;id;
    my $notes = $job-&gt;info-&gt;{ notes };
    my $title = $notes-&gt;{ title };