Acme-CPANModulesBundle-Import-MojoliciousAdvent-2018
view release on metacpan or search on metacpan
devdata/https_mojolicious.io_blog_2018_12_08_authenticating-with-ldap_ view on Meta::CPAN
my $sth = $dbh->prepare($statement);
$sth->execute($username) or return;
my ($encoded) = $sth->fetchrow_array();
$sth->finish();
# WAIT! where did $self come from
return $self->scrypt_verify($password, $encoded);
}
</code></pre>
<p>Oh, dear. The above crashes because of a design decision made early on in the writing process.
I invoked <code>check_credentials</code> as a plain sub, not the method of an object.
Using a Plugin depends on having the controller available, so the following changes are necessary.</p>
<pre><code>sub on_user_login {
my $self = shift;
...
if ($self->check_credentials($username, $password)) {
...
}
devdata/https_mojolicious.io_blog_2018_12_08_authenticating-with-ldap_ view on Meta::CPAN
<pre><code>$r->post('/login')->name('do_login')->to('Secure#on_user_login');
</code></pre>
<p>which I named <em>do_login</em>. Using
<a href="https://mojolicious.io/blog/2017/12/03/day-3-using-named-routes/">named routes</a>
gives you the flexibility of changing URLs without much hassle.</p>
<h2>Where next?</h2>
<p>I go through the whole process of authenticating and maintaining sessions at my
<a href="https://github.com/duffee/Mojolicious_session_example">Mojolicious session tutorial</a>
which will be getting some updates in the New Year to reflect what I've learned.
Contributions welcome!</p>
</section>
<small><p><a href="https://www.flickr.com/photos/maxlfly/1834725880/">Image</a> by <a href="https://www.flickr.com/people/85105517@N00">Alex Grech</a> <a href="https://creativecommons.org/licenses/by/2.0" title="Creative Commons Attribu...
</small>
<p class="tags">
<span>Tagged in </span>:
devdata/https_mojolicious.io_blog_2018_12_10_minion-stands-alone_ view on Meta::CPAN
<p>Now, if I never run a web daemon, this will never be a website. So, can
this be called a web application if nobody ever uses a web browser to
access it? 🤔</p>
<h2>Add Minion</h2>
<p>To use Minion, add the Minion plugin to the app. For this example, I'll
use <a href="https://metacpan.org/pod/Minion::Backend::SQLite">the SQLite Minion
backend</a> so that
I don't need a separate database process running, but Minion can work
across multiple machines if you have a database that accepts remote
connections.</p>
<pre><code>plugin Minion => {
SQLite => 'sqlite:' . app->home->child('minion.db'),
};
</code></pre>
<p>With the Minion plugin loaded, my application gains some new features:</p>
devdata/https_mojolicious.io_blog_2018_12_10_minion-stands-alone_ view on Meta::CPAN
<h2>Running a Worker</h2>
<p>I've enqueued jobs, but nothing's happening, and nothing will happen
until I run a worker using <a href="https://mojolicious.org/perldoc/Minion/Command/minion/worker">the <code>minion worker</code>
command</a>:</p>
<pre><code>$ perl myapp.pl minion worker
</code></pre>
<p>Once the worker starts up, it will immediately begin processing the jobs
I told it to run.</p>
<p>And that's it! I'm using Minion without a Mojolicious
web application. View the source of the <a href="minion.pl">Minion app</a> and the
<a href="enqueue.pl">enqueue.pl script</a>.</p>
</section>
<small><p>Original artwork by Doug Bell, released under CC-BY-SA 4.0. It includes
a screenshot of the Mojolicious.org website (fair use), the Mojolicious
logo (CC-BY-SA 4.0), and the Minion logo (CC-BY-SA 4.0)</p>
devdata/https_mojolicious.io_blog_2018_12_12_dancer-and-minion_ view on Meta::CPAN
<div class="post-thumb">
<!-- theme suggests 1300x500 -->
<img alt="Stylistic photograph of Disney-style minion toys" src="/blog/2018/12/12/dancer-and-minion/banner.jpg">
</div>
<div class="post-content">
<section id="section-1">
<p>At <code>$work</code>, we have built an API with <a href="https://metacpan.org/pod/Dancer">Dancer</a> that generates PDF documents and XML files. This API is a critical component of an insurance enrollment system: PDFs are genera...
immediately, and the XML is delivered to the carrier as soon as it becomes available. Since the XML often takes a significant amount of time to generate, the job is generated in the background so as not to tie up the
application server for an extended amount of time. When this was done, a homegrown process management system was developed, and works by <code>fork()</code>ing a process, tracking its pid, and hoping we can later successfully
reap the completed process.</p>
<p>There have been several problems with this approach:
- it's fragile
- it doesn't scale
- it's too easy to screw something up as a developer</p>
<p>In 2019, we have to ramp up to take on a significantly larger workload. The current solution simply will not handle the amount of work we anticipate needing to handle. Enter <a href="https://metacpan.org/pod/Minion">Minion</a>.</p>
<p><em>Note:</em> The techniques used in this article work equally well with Dancer or <a href="https://metacpan.org/pod/Dancer2">Dancer2</a>.</p>
</section>
<section id="section-2">
<h2>Why Minion?</h2>
<p>We looked at several alternatives to Minion, including <a href="https://beanstalkd.github.io/">beanstalkd</a> and <a href="http://www.celeryproject.org/">celeryd</a>. Using either one of these meant involving our already over-taxed
infrastructure team, however; using Minion allowed us to use expertise that my team already has without having to burden someone else with assisting us. From a development standpoint, using a product that
was developed in Perl gave us the quickest time to implementation.</p>
<p>Scaling our existing setup was near impossible. It's not only difficult to get a handle on what resources are consumed by processes we've forked, but it was impossible to run the jobs on more than one server.
Starting over with Minion also gave us a much needed opportunity to clean up some code in sore need of refactoring. With a minimal amount of work, we were able to clean up our XML rendering code and make it work
from Minion. This cleanup allowed us to more easily get information as to how much memory and CPU was consumed by an XML rendering job. This information is vital for us in planning future capacity.</p>
<h2>Accessing Minion</h2>
<p>Since we are a Dancer shop, and not Mojolicious, a lot of convenience you'd get from Mojolicious for working with Minion isn't as available to us. Given we are also sharing some Minion-based
code with our business models, we had to build some of our own plumbing around Minion:</p>
<pre><code>package MyJob::JobQueue;
devdata/https_mojolicious.io_blog_2018_12_12_dancer-and-minion_ view on Meta::CPAN
@queue_map{ @QUEUE_TYPES } = ();
my @invalid_queues = grep !exists $queue_map{ $_ }, @queues;
return @invalid_queues;
}
</code></pre>
<p>With that in place, it was easy for our <code>queue_job()</code> method to throw an error if the developer tried to add a job to an invalid queue:</p>
<pre><code>sub queue_job( $self, $args ) {
my $job_name = $args->{ name } or die "queue_job(): must define job name!";
my $guid = $args->{ guid } or die "queue_job(): must have GUID to process!";
my $title = $args->{ title } // $job_name;
my $queue = $args->{ queue } // 'default';
my $job_args = $args->{ job_args };
die "queue_job(): Invalid job queue '$queue' specified" if $self->has_invalid_queues( $queue );
my %notes = ( title => $title, guid => $guid );
return $self->runner->enqueue( $job_name => $job_args => { notes => \%notes, queue => $queue });
}
devdata/https_mojolicious.io_blog_2018_12_12_dancer-and-minion_ view on Meta::CPAN
my $hostconfig = get_hostconfig();
my $minion = MyJob::JobQueue->new;
my $worker = $minion->runner->worker;
my $log_eng = MyJob::Log4p->new({ logger_name => "Minion" });
my $logger = MyJob::Util::Logger->new->logger($log_eng);
</code></pre>
<p>The above is mostly typical boilerplate for us. Read our configuration, and create a logger the worker can use.</p>
<p>Next, when a job is dequeued, we want to log that the worker picked up a job (needed for auditing purposes) and we alter the process name so if a process hangs, we know what that process
was attempting to run. If an unchecked exception occurs in a job, the worker will catch it and log it for us:</p>
<pre><code>$worker->on( dequeue => sub( $worker, $job ) {
my $id = $job->id;
my $notes = $job->info->{ notes };
my $title = $notes->{ title };
my $guid = $notes->{ guid };
$job->on( spawn => sub( $job, $pid ) {
$0 = "$title $guid";
$logger->info( "$title: Created child process $pid for job $id by parent $$ - $guid");
});
$job->on( failed => sub( $job, $error ) {
chomp $error;
$logger->error( $error );
});
});
</code></pre>
<p>To help us for future capacity planning, we want our workers to tell us if they are running at peak capacity, so log when this event occurs:</p>
devdata/https_mojolicious.io_blog_2018_12_14_a-practical-example-of-mojo-dom_ view on Meta::CPAN
<div class="post-content">
<section id="section-1">
<p>With recent versions of Mojolicious, <a href="https://mojolicious.org/perldoc/Mojo/DOM">Mojo::DOM</a> gained a lot of power that I have been excited to try, but haven't had the time for. Recently, I had a problem at my real ...
</section>
<section id="section-2">
<h2>The task - simple, but tedious.</h2>
<p>3D models and drawings are fantastic tools, but in reality things are not so perfect. Construction tolerances being what they are, our company relies a lot on <a href="https://www.youtube.com/watch?v=H-uNzEmt5sw">laser scanning</a>, where we go o...
<p>The problem is when our 3D modeling software (<a href="https://www.tekla.com/products/tekla-structures">Tekla Structures</a>) processes the point clouds, it changes the file names of each one from something human readable, such as <code>Pipe Rack ...
<p><img alt="Point clouds add information that is not available in the 3D model" src="pointcloud1.jpg">
<em>Point clouds provide critical information that is either too difficult or too costly to model directly</em></p>
<p>Fortunately, Tekla uses a lot of standard file formats that anyone can edit – including using XML to describe each point cloud it has processed. Of course, I could just hand edit them to change the names, but that would have to be done for e...
<p>Conveniently, the XML file contains both the hash name and the original file name – so I knew I could use Mojo::DOM to parse the XML and rename the point clouds to be human readable. This simple task is the perfect example of how Mojolicious ...
<pre><code>#!/usr/bin/perl
use Mojo::Base -strict;
use Mojo::Util qw(getopt);
use Mojo::File;
use Mojo::DOM;
getopt 'p|path=s' => \my $path;
devdata/https_mojolicious.io_blog_2018_12_14_a-practical-example-of-mojo-dom_ view on Meta::CPAN
<p><a href="https://mojolicious.org/perldoc/Mojo/File">Mojo::File</a> makes reading <code>pointclouds.xml</code> so I can parse it with Mojo::DOM simple:</p>
<pre><code>my $file = Mojo::File->new($path, 'pointclouds.xml');
my $dom = Mojo::DOM->new($file->slurp);
</code></pre>
<p>In the <a href="https://cgi-lib.berkeley.edu/2.18/cgi-lib.pl.txt">bad old days</a>, I probably hand wrote 15 lines of (horrible) code every time I wanted to read a file.</p>
<h2>Mojo::DOM - what can't it do?</h2>
<p>And of course, <a href="https://mojolicious.org/perldoc/Mojo/DOM">Mojo::DOM</a> makes finding the right values in the XML easy - it also handles HTML and CSS selectors. Basically, I just iterate through the contents of <code>PointCloudData</code>...
<pre><code>for my $e ($dom->find('PointCloudData')->each) {
$e->{Folder} = rename_files($e) and $e->{Hash} = '' if $e->{Hash};
}
</code></pre>
<p>I only run <code>rename_files</code> if <code>Hash</code> is populated, and if it is, I empty it so I don't try to rename them again. <code>rename_files</code> is about the only Perl code I had to write myself - almost everything else was cop...
<p>A substitution stores the desired file & folder name in <code>$newname</code> (non-destructive modifier <code>/r</code> allows me to work on a copy of <code>$e</code> without changing the original). Then I simply rename the point cloud folder...
devdata/https_mojolicious.io_blog_2018_12_16_browser-diet_ view on Meta::CPAN
<!-- theme suggests 1300x500 -->
<img alt="Too many nuts. I'm gonna need to slim down" src="/blog/2018/12/16/browser-diet/squirrel.jpg">
</div>
<div class="post-content">
<section id="section-1">
<p>You've just read
<a href="https://browserdiet.com">How to lose Weight in the Browser</a>
and you want to know to slim down your Mojo app.
Part of that process is preventing the browser from requesting files
that hardly change.
I spent a well-caffeinated afternoon trying to do that with
Mojolicious.
I've been 'round the houses, and <em>spoiler alert</em> I didn't find
the answer until the very end, kind of like your favourite Christmas
animated special with a small woodland creature narrating
"The Gruffalo's HTTP header".</p>
<h1>A Children's Story</h1>
devdata/https_mojolicious.io_blog_2018_12_21_a-little-christmas-template-cooking_ view on Meta::CPAN
</div>
<div class="post-thumb">
<!-- theme suggests 1300x500 -->
<img alt="Peanut Butter Kisses cookies" src="/blog/2018/12/21/a-little-christmas-template-cooking/banner.jpg">
</div>
<div class="post-content">
<section id="section-1">
<p>The Advent Calendar has shown you many great ways to use Mojolicious, and since you already have Mojo installed you can use it for things besides web processing. Today's recipe uses The templating rendering engine for somethi...
</section>
<section id="section-2">
<p>First, process some string templates. Here's an example lifted from the <a href="https://mojolicious.org/perldoc/Mojo/Template">Mojo::Template</a>, using the <a href="https://www.effectiveperlprogramming.com/2016/12/strip-lea...
<pre><code>use Mojo::Template;
my $mt = Mojo::Template->new;
say $mt->render(<<~'EOF');
% use Time::Piece;
<div>
% my $now = localtime;
Time: <%= $now->hms %>
devdata/https_mojolicious.io_blog_2018_12_21_a-little-christmas-template-cooking_ view on Meta::CPAN
foreach my $file ( @ARGV ) {
Mojo::File
->new( $file =~ s/\.ep\z//r )
->spurt(
$mt->render_file( $file, { time => $now->hms } )
);
}
</code></pre>
<p>Now that you know everything about Mojo templates, you can process a directory full of them to start a new project:</p>
<pre><code>use v5.14;
use File::Find qw(find);
use Time::Piece;
use Mojo::File;
use Mojo::Template;
my $mt = Mojo::Template->new->vars(1);
devdata/https_mojolicious.io_blog_2018_12_23_mojolicious-and-angular_ view on Meta::CPAN
Sachin@01:07 PM[~/workspace/project/mojo_angular]$
</code></pre>
<p>Now that the Mojolicious full-app is created, start <a href="https://mojolicious.org/perldoc/Mojo/Server/Hypnotoad">hypnotoad</a>, a production web server.</p>
<pre><code>Sachin@01:07 PM[~/workspace/project/mojo_angular]$ cd mojo_angular_app/
Sachin@01:08 PM[~/workspace/project/mojo_angular/mojo_angular_app]$ hypnotoad -f script/mojo_angular_app
[Sat Dec 15 13:08:52 2018] [info] Listening at "http://*:8080"
Server available at http://127.0.0.1:8080
[Sat Dec 15 13:08:52 2018] [info] Manager 38209 started
[Sat Dec 15 13:08:52 2018] [info] Creating process id file "/Users/Sachin/workspace/project/mojo_angular/mojo_angular_app/script/hypnotoad.pid"
[Sat Dec 15 13:08:52 2018] [info] Worker 38210 started
[Sat Dec 15 13:08:52 2018] [info] Worker 38211 started
[Sat Dec 15 13:08:52 2018] [info] Worker 38213 started
[Sat Dec 15 13:08:52 2018] [info] Worker 38212 started
</code></pre>
<p>Let's open browser and have a look at our mojo app.</p>
<p><img alt="basic Mojolicious full app" src="mojo_app.png"></p>
devdata/https_mojolicious.io_blog_2018_12_23_mojolicious-and-angular_ view on Meta::CPAN
</code></pre>
<p>Of course if this were a proper project, you could generate it to build in the public directory, or even build at start time using something like <a href="https://metacpan.org/pod/Mojolicious::Plugin::AssetPack">Mojolicious::Plugin::AssetPack</a>....
<p>Finally, let's run the Mojo (hypnotoad) server to see if the Angular page is served as it was from the Angular server.</p>
<pre><code>Sachin@02:48 PM[~/workspace/project/mojo_angular/mojo_angular_app]$ hypnotoad -f script/mojo_angular_app
[Sat Dec 15 14:49:03 2018] [info] Listening at "http://*:8080"
Server available at http://127.0.0.1:8080
[Sat Dec 15 14:49:03 2018] [info] Manager 40633 started
[Sat Dec 15 14:49:03 2018] [info] Creating process id file "/Users/Sachin/workspace/project/mojo_angular/mojo_angular_app/script/hypnotoad.pid"
[Sat Dec 15 14:49:03 2018] [info] Worker 40634 started
[Sat Dec 15 14:49:03 2018] [info] Worker 40635 started
[Sat Dec 15 14:49:03 2018] [info] Worker 40637 started
[Sat Dec 15 14:49:03 2018] [info] Worker 40636 started
</code></pre>
<p><img alt="mojolicious serving angular SPA" src="mojo_serving_angular.png"></p>
<p>Congratulations! we have served angular app with Mojolicious. Please note that the URL will be <code>http://localhost:8080/NgDemo/</code>.</p>
devdata/https_mojolicious.io_blog_2018_12_24_async-await-the-mojo-way_ view on Meta::CPAN
<div class="post-content">
<section id="section-1">
<h2>The Problems with Thinking in Asynchronous Code</h2>
<p>I've thought a lot about asynchronous code.
Learning it, writing it, teaching it.
Async is hard.</p>
<blockquote>
<p>Learning async programming is about being completely confused and overcomplicating everything and eventually having an 'aha' moment and then being utterly frustrated you don't have a way to teach other people without them needing to ...
<cite><a href="https://shadow.cat/blog/matt-s-trout/">Matt S. Trout</a></cite></p>
</blockquote>
<p>While Matt is right, I've thought a lot about that quote and I think I've come up with an underlying problem.
This may sound trite and it may seem obvious, but the problem is that writing asynchronous code is just fundamentally different than writing blocking code.</p>
<p>We've always known how to make one instruction follow another, that's easy, it happens on the next line of code.
Line one executes and then line two.
If line two needs something from line one it will be there.</p>
devdata/https_mojolicious.io_blog_2018_12_24_async-await-the-mojo-way_ view on Meta::CPAN
return trim $tx->res->dom->at('title')->text;
});
$promise->then(sub ($title) {
say $title;
})->wait;
</code></pre>
<p>This is important if say you had a function that was intended to return a promise that resolved to a title.
Perhaps you might have a function called <code>get_title_p</code> that needs to be called from elsewhere in your project.
Rather than relying on the promise that the user-agent returned, you can now post-process and return the title rather than the HTTP response.</p>
<pre><code>sub get_title_p ($url) {
my $promise = $ua->get_p($url)->then(sub ($tx) {
return trim $tx->res->dom->at('title')->text;
});
return $promise;
}
get_title_p($url)->then(sub ($title) {
say $title;
devdata/https_mojolicious.io_blog_2018_12_24_async-await-the-mojo-way_ view on Meta::CPAN
Everything in it is applicable even as this article takes it one step further below ...</em></p>
<h2>Async/Await</h2>
<p>What we really wish we could tell the Perl interpreter to do is</p>
<ul>
<li>suspend execution until this promise resolves or is rejected</li>
<li>then move on to handle tasks</li>
<li>when it eventually does resolve or reject</li>
<li>then resume processing right here or throw an exception</li>
</ul>
<p>It is a big ask, but if you could say that, you'd basically get linearity back.
Promises give us the control we'd need for such a mechanism, but until now we in Perl-land lack the ability to suspend and resume the interpreter.
Indeed, some languages already have this mechanism and the result is called the Async/Await pattern.
With a little added magic, howver, we can do just that.</p>
<p>That was a lot of introduction, but now I'm finally ready to introduce <a href="https://metacpan.org/pod/Mojo::AsyncAwait">Mojo::AsyncAwait</a>!</p>
<pre><code>use Mojo::AsyncAwait;
( run in 0.309 second using v1.01-cache-2.11-cpan-8d75d55dd25 )