Acme-CPANModulesBundle-Import-PerlDancerAdvent-2018

 view release on metacpan or  search on metacpan

devdata/http_advent.perldancer.org_2018_14  view on Meta::CPAN

</li>
<li><a name="item_Benefits_management_software"></a><b>Benefits management software</b>
</li>
<li><a name="item_thegamecrafter_com__and_their_family_of_sites_applications_"></a><b>thegamecrafter.com (and their family of sites/applications)</b>
</li>
<li><a name="item_Unattended_installation_portal"></a><b>Unattended installation portal</b>
</li>
</ul>
<h2><a name="wrapping_it_all_up"></a>Wrapping it all up</h2>

<p>We received a lot of constructive feedback from this process, and while we might
not do a survey every year, from time to time, this will be a good barometer
of where things are and where we need to go.</p>
<p>Some of the feedback we received has already been acted upon (websockets, doc
improvements), and you can look forward to us addressing your other feedback
as we are able. Right now, we are working on a better manual that has more 
information about the things you'd most like to see. Stay tuned!</p>
<h2><a name="author"></a>Author</h2>

<p>This article has been written by Jason Crome (CromeDome) for the Perl Dancer 
Advent Calendar 2018.</p>

devdata/http_advent.perldancer.org_2018_16  view on Meta::CPAN

</li>
</ul>
</div>


<div id="content">
<div class="pod-document"><h1><a name="using_minion_in_dancer_apps"></a>Using Minion in Dancer Apps</h1>

<p>At <code>$work</code>, we have built an API with Dancer that generates PDF documents and XML files. This API is a critical component of an insurance enrollment system: PDFs are generated to deliver to the client in a web browser 
immediately, and the XML is delivered to the carrier as soon as it becomes available. Since the XML often takes a significant amount of time to generate, the job is generated in the background so as not to tie up the 
application server for an extended amount of time. When this was done, a homegrown process management system was developed, and works by <code>fork()</code>ing a process, tracking its pid, and hoping we can later successfully
reap the completed process.</p>
<p>There have been several problems with this approach:</p>
<ul>
<li><a name="item_it_s_fragile"></a><b>it's fragile</b>
</li>
<li><a name="item_it_doesn_t_scale"></a><b>it doesn't scale</b>
</li>
<li><a name="item_it_s_too_easy_to_screw_something_up_as_a_developer"></a><b>it's too easy to screw something up as a developer</b>
</li>
</ul>
<p>In 2019, we have to ramp up to take on a significantly larger workload. The current solution simply will not handle the amount of work we anticipate needing to handle. Enter <a href="https://metacpan.org/pod/Minion">Minion</a>.</p>
<p><b>Note:</b> The techniques used in this article work equally well with Dancer or Dancer2.</p>
<h2><a name="why_minion"></a>Why Minion?</h2>

<p>We looked at several alternatives to Minion, including <a href="https://beanstalkd.github.io/">beanstalkd</a> and <a href="http://www.celeryproject.org/">celeryd</a>. Using either one of these meant involving our already over-taxed
infrastructure team, however; using Minion allowed us to use expertise that my team already has without having to burden someone else with assisting us. From a development standpoint, using a product that
was developed in Perl gave us the quickest time to implementation.</p>
<p>Scaling our existing setup was near impossible. It's not only difficult to get a handle on what resources are consumed by processes we've forked, but it was impossible to run the jobs on more than one server. 
Starting over with Minion also gave us a much needed opportunity to clean up some code in sore need of refactoring. With a minimal amount of work, we were able to clean up our XML rendering code and make it work
from Minion. This cleanup allowed us to more easily get information as to how much memory and CPU was consumed by an XML rendering job. This information is vital for us in planning future capacity.</p>
<h2><a name="accessing_minion"></a>Accessing Minion</h2>

<p>Since we are a Dancer shop, and not Mojolicious, a lot of things you'd get from Mojolicious for working with Minion isn't as available to us. Given we are also sharing some Minion-based
code with our business models, we had to build some of our own plumbing around Minion:</p>
<pre class="prettyprint">package MyJob::JobQueue;

use Moose;
use Minion;

devdata/http_advent.perldancer.org_2018_16  view on Meta::CPAN

sub get_invalid_queues( $self, @queues ) {
    my %queue_map;
    @queue_map{ @QUEUE_TYPES } = (); 
    my @invalid_queues = grep !exists $queue_map{ $_ }, @queues;
    return @invalid_queues;
}</pre>

<p>With that in place, it was easy for our <code>queue_job()</code> method to throw an error if the developer tried to add a job to an invalid queue:</p>
<pre class="prettyprint">sub queue_job( $self, $args ) {
    my $job_name = $args-&gt;{ name     } or die "queue_job(): must define job name!";
    my $guid     = $args-&gt;{ guid     } or die "queue_job(): must have GUID to process!";
    my $title    = $args-&gt;{ title    } // $job_name;
    my $queue    = $args-&gt;{ queue    } // 'default';
    my $job_args = $args-&gt;{ job_args };

    die "queue_job(): Invalid job queue '$queue' specified" 
        if $self-&gt;has_invalid_queues( $queue );

    my %notes = ( title =&gt; $title, guid  =&gt; $guid );

    return $self-&gt;runner-&gt;enqueue( $job_name =&gt; $job_args =&gt; 

devdata/http_advent.perldancer.org_2018_16  view on Meta::CPAN

my $config     = MyJob::Config->new->config;
my $hostconfig = get_hostconfig();
my $minion     = MyJob::JobQueue->new;
my $worker     = $minion->runner->worker;

my $log_eng = MyJob::Log4p->new({ logger_name => "Minion" });
my $logger  = MyJob::Util::Logger-&gt;new-&gt;logger($log_eng);</pre>

<p>The above is mostly typical boilerplate for us. Read our configuration, and create a logger the worker can use.</p>
<p>Next, when a job is dequeued, we want to log that the worker picked up a job (needed for auditing purposes) and we alter the process name so if a process hangs, we know what that process
was attempting to run. If an unchecked exception occurs in a job, the worker will catch it and log it for us:</p>
<pre class="prettyprint">$worker-&gt;on( dequeue =&gt; sub( $worker, $job ) {
    my $id    = $job-&gt;id;
    my $notes = $job-&gt;info-&gt;{ notes };
    my $title = $notes-&gt;{ title };
    my $guid  = $notes-&gt;{ guid };

    $job-&gt;on( spawn =&gt; sub( $job, $pid ) {  
        $0 = "$title $guid";
        $logger-&gt;info( 
            "$title: Created child process $pid for job $id by parent $$ - $guid");
    });
        
    $job-&gt;on( failed =&gt; sub( $job, $error ) {
        chomp $error;
        $logger-&gt;error( $error );
    });
});</pre>

<p>To help us for future capacity planning, we want our workers to tell us if they are running at peak capacity, so log when this event occurs:</p>
<pre class="prettyprint">$worker-&gt;on( busy =&gt; sub( $worker ) {

devdata/http_advent.perldancer.org_2018_18  view on Meta::CPAN

  # At this point, we can run the original scaffolding
  $self->SUPER::execute( $opt, $args );

  # Now we finished generating, but we can contineu customizing what we have
}

1;</pre>

<p>Writing your own generation on top of the existing generation allows
you to manage the input (including additional validation) and the
output, giving you full control over the scaffolding process.</p>
<p>Some examples on which customizations you might want to perform:</p>
<ul>
<li><a name="item_Add_additional_default_imported_classes"></a><b>Add additional default imported classes</b>
</li>
<li><a name="item_Change_the_output_directory_name"></a><b>Change the output directory name</b>
</li>
<li><a name="item_Update_a_database_that_we_have_a_new_application"></a><b>Update a database that we have a new application</b>
</li>
<li><a name="item_Update_your_team_with_an_email_or_IRC_Slack_message"></a><b>Update your team with an email or IRC/Slack message</b>
</li>