view release on metacpan or  search on metacpan

devdata/http_advent.perldancer.org_2018_16  view on Meta::CPAN

<li><a name="item_it_s_fragile"></a><b>it's fragile</b>
</li>
<li><a name="item_it_doesn_t_scale"></a><b>it doesn't scale</b>
</li>
<li><a name="item_it_s_too_easy_to_screw_something_up_as_a_developer"></a><b>it's too easy to screw something up as a developer</b>
</li>
</ul>
<p>In 2019, we have to ramp up to take on a significantly larger workload. The current solution simply will not handle the amount of work we anticipate needing to handle. Enter <a href="https://metacpan.org/pod/Minion">Minion</a>.</p>
<p><b>Note:</b> The techniques used in this article work equally well with Dancer or Dancer2.</p>
<h2><a name="why_minion"></a>Why Minion?</h2>

<p>We looked at several alternatives to Minion, including <a href="https://beanstalkd.github.io/">beanstalkd</a> and <a href="http://www.celeryproject.org/">celeryd</a>. Using either one of these meant involving our already over-taxed
infrastructure team, however; using Minion allowed us to use expertise that my team already has without having to burden someone else with assisting us. From a development standpoint, using a product that
was developed in Perl gave us the quickest time to implementation.</p>
<p>Scaling our existing setup was near impossible. It's not only difficult to get a handle on what resources are consumed by processes we've forked, but it was impossible to run the jobs on more than one server. 
Starting over with Minion also gave us a much needed opportunity to clean up some code in sore need of refactoring. With a minimal amount of work, we were able to clean up our XML rendering code and make it work
from Minion. This cleanup allowed us to more easily get information as to how much memory and CPU was consumed by an XML rendering job. This information is vital for us in planning future capacity.</p>
<h2><a name="accessing_minion"></a>Accessing Minion</h2>

<p>Since we are a Dancer shop, and not Mojolicious, a lot of things you'd get from Mojolicious for working with Minion isn't as available to us. Given we are also sharing some Minion-based
code with our business models, we had to build some of our own plumbing around Minion:</p>
<pre class="prettyprint">package MyJob::JobQueue;

use Moose;
use Minion;

use MyJob::Models::FooBar;
with 'MyJob::Roles::ConfigReader';

has 'runner' =&gt; (
    is      =&gt; 'ro',
    isa     =&gt; 'Minion',
    lazy    =&gt; 1,
    default =&gt; sub( $self ) {
        $ENV{ MOJO_PUBSUB_EXPERIMENTAL } = 1;
        Minion-&gt;new( mysql =&gt; 
          MyJob::DBConnectionManager-&gt;new-&gt;get_connection_uri({ 
            db_type =&gt; 'feeds', 
            io_type =&gt; 'rw',
        }));
    },
);</pre>

<p>We wrapped a simple Moose class around Minion to make it easy to add to any class or Dancer application with the extra functionality we wanted.</p>
<p>We ran into an issue at one point where jobs weren't running since we added them to a queue that no worker was configured to handle. To prevent this from happening to us again,
we added code to prevent us from adding code to a queue that didn't exist:</p>
<pre class="prettyprint">my @QUEUE_TYPES = qw( default InstantXML PayrollXML ChangeRequest );

sub has_invalid_queues( $self, @queues ) {
    return 1 if $self-&gt;get_invalid_queues( @queues );
    return 0;
}

sub get_invalid_queues( $self, @queues ) {
    my %queue_map;
    @queue_map{ @QUEUE_TYPES } = (); 
    my @invalid_queues = grep !exists $queue_map{ $_ }, @queues;
    return @invalid_queues;
}</pre>

<p>With that in place, it was easy for our <code>queue_job()</code> method to throw an error if the developer tried to add a job to an invalid queue:</p>
<pre class="prettyprint">sub queue_job( $self, $args ) {
    my $job_name = $args-&gt;{ name     } or die "queue_job(): must define job name!";
    my $guid     = $args-&gt;{ guid     } or die "queue_job(): must have GUID to process!";
    my $title    = $args-&gt;{ title    } // $job_name;
    my $queue    = $args-&gt;{ queue    } // 'default';
    my $job_args = $args-&gt;{ job_args };

    die "queue_job(): Invalid job queue '$queue' specified" 
        if $self-&gt;has_invalid_queues( $queue );

    my %notes = ( title =&gt; $title, guid  =&gt; $guid );

    return $self-&gt;runner-&gt;enqueue( $job_name =&gt; $job_args =&gt; 
        { notes =&gt; \%notes, queue =&gt; $queue });
}</pre>

<h2><a name="creating_jobs"></a>Creating Jobs</h2>

<p>In our base model class (Moose-based), we would create an attribute for our job runner:</p>
<pre class="prettyprint">has 'job_runner' =&gt; (
    is      =&gt; 'ro',
    isa     =&gt; 'MyJob::JobQueue',
    lazy    =&gt; 1,
    default =&gt; sub( $self ) { return MyJob::JobQueue-&gt;new-&gt;runner; },
);</pre>

<p>And in the models themselves, creating a new queueable task was as easy as:</p>
<pre class="prettyprint">$self-&gt;runner-&gt;add_task( InstantXML =&gt; 
    sub( $job, $request_path, $guid, $company_db, $force, $die_on_error = 0 ) {
        $job-&gt;note( 
            request_path =&gt; $request_path,
            feed_id      =&gt; 2098,
            group        =&gt; $company_db,
        );
        MyJob::Models::FooBar-&gt;new( request_path =&gt; 
          $request_path )-&gt;generate_xml({
            pdf_guid     =&gt; $guid,
            group        =&gt; $company_db,
            force        =&gt; $force,
            die_on_error =&gt; $die_on_error,
        });
});</pre>

<h2><a name="running_jobs"></a>Running Jobs</h2>

<p>Starting a job from Dancer was super easy:</p>
<pre class="prettyprint">use Dancer2;
use MyJob::JobQueue;

sub job_queue {
    return MyJob::JobQueue-&gt;new;
}

get '/my/api/route/:guid/:group/:force' =&gt; sub {
    my $guid  = route_parameters-&gt;get( 'guid' );
    my $group = route_parameters-&gt;get( 'group' );
    my $force = route_parameters-&gt;get( 'force' );

    debug "GENERATING XML ONLY FOR $guid";
    job_queue-&gt;queue_job({