map results from the CPAN

Acme-CPANModulesBundle-Import-MojoliciousAdvent-2017

<h2>Let&#39;s Build a Weather App</h2>

<p>Rather than give several small examples I&#39;m going to change it up this time and give one big example.
It will be ok if you don&#39;t understand every line, I&#39;m skipping a few pedagogical steps to make this example.</p>

<p>Why don&#39;t we build a weather caching app?
I&#39;ve put the entire thing on github at <a href="https://github.com/jberger/MyWeatherApp">https://github.com/jberger/MyWeatherApp</a>.
I&#39;m going to copy portions of the code into this article for clarity, but consider that that site is probably the most up to date.</p>

<p>Of course, we&#39;ll need some data.
I&#39;ve chosen to use <a href="https://openweathermap.org/">https://openweathermap.org/</a>
To do run this you&#39;ll need to sign up for a <a href="http://home.openweathermap.org/users/sign_up">free account</a>.
It only needs an email address, I tried but I really couldn&#39;t find a totally open access weather API.
From there you can get an API key, which will need to go in a configuration file.
Once you have it create a configuration file called <code>myweatherapp.conf</code> and fill it in like so:</p>

<pre><code>{
  appid =&gt; &#39;XXXXXXXXX&#39;,
}
</code></pre>

<h3>The Script</h3>

devdata/https_mojolicious.io_blog_2017_12_06_day-6-adding-your-own-commands view on Meta::CPAN

use Mojo::URL;
use Mojo::UserAgent;

has appid  =&gt; sub { Carp::croak &#39;appid is required&#39; };
has sqlite =&gt; sub { Carp::croak &#39;sqlite is required&#39; };
has ua     =&gt; sub { Mojo::UserAgent-&gt;new };
has &#39;units&#39;;

sub fetch {
  my ($self, $search) = @_;
  my $url = Mojo::URL-&gt;new(&#39;http://api.openweathermap.org/data/2.5/weather&#39;);
  $url-&gt;query(
    q =&gt; $search,
    APPID =&gt; $self-&gt;appid,
    units =&gt; $self-&gt;units || &#39;metric&#39;,
  );
  return $self-&gt;ua-&gt;get($url)-&gt;result-&gt;json;
}

sub insert {
  my ($self, $search, $result) = @_;

devdata/https_mojolicious.io_blog_2017_12_10_day-10-give-the-customer-what-they-want view on Meta::CPAN

</code></pre>

<p>In Mojolicious the overall <a href="http://mojolicious.org/perldoc/Mojolicious/Renderer#default_format">default format</a> is html, but of course can be changed.</p>

<pre><code>app-&gt;renderer-&gt;default_format(&#39;json&#39;);
</code></pre>

<p>There are also mechanisms to limit the format detection, but those are beyond the scope of this article.
The links above contain more.</p>

<p>Note also that the mappings between file extensions and MIME types obviously are in play here.
If you have special format types you can add them to the <a href="http://mojolicious.org/perldoc/Mojolicious#types">types</a> object on the application.</p>

<h2>Responding to the Format Request</h2>

<p>There are two methods which help render what the client wants: <a href="http://mojolicious.org/perldoc/Mojolicious/Controller#respond_to"><code>respond_to</code></a> and <a href="http://mojolicious.org/perldoc/Mojolicious/Plugin/DefaultHelpers#acc...

<p>The former, <code>respond_to</code>, is much more high level and should be your go-to choice.
It takes key value pairs where the keys are the file types that should be handled (in extension format).
The values are either stash values that should be used when rendering or else a callback to be invoked.</p>

devdata/https_mojolicious.io_blog_2017_12_10_day-10-give-the-customer-what-they-want view on Meta::CPAN


@@ reindeer.xml.ep
&lt;?xml version=&quot;1.0&quot;?&gt;
&lt;Reindeer&gt;
  &lt;Name&gt;&lt;%= $reindeer-&gt;{name} =%&gt;&lt;/Name&gt;
  &lt;Description&gt;&lt;%= $reindeer-&gt;{description} =%&gt;&lt;/Description&gt;
&lt;/Reindeer&gt;

</code></pre>

<p>In it you can see each file type maps to a set of stash values.
The test cases use a variety of ways to request the different response types.</p>

<p>N.B. I like showing test cases as examples because it not only shows the code, it shows how to test it and what the expected responses should be.</p>

<h2>A More Advanced Case</h2>

<p>To demonstrate how powerful this mechanism is, let me show you some code that I wrote for a previous job.
That company was friendly to Open Source and so it lives on CPAN as <a href="https://metacpan.org/pod/Mojolicious::Plugin::ReplyTable">Mojolicious::Plugin::ReplyTable</a>.
I won&#39;t copy and paste the whole module, you can see it on <a href="https://github.com/jberger/Mojolicious-Plugin-ReplyTable/blob/master/lib/Mojolicious/Plugin/ReplyTable.pm">Github</a>.</p>

devdata/https_mojolicious.io_blog_2017_12_14_day-14-you-promised-to-call view on Meta::CPAN

    my ($tx) = @_;
    $tx-&gt;result;
  })-&gt;then(sub {
    my ($result) = @_;
    say &quot;$url: &quot;, $result-&gt;dom-&gt;at(&#39;title&#39;)-&gt;text;
    fetchpages(); # returns a promise, but of a whole new page to process
  });
}

# Process two requests at a time
my @promises = map fetchpages(), 1 .. 2;
Mojo::Promise-&gt;all(@promises)-&gt;wait if @promises;
</code></pre>

<p>Another option for dealing with a number of concurrent activities, if you just want the first one that completes, is <a href="http://mojolicious.org/perldoc/Mojo/Promise#race"><code>race</code></a>.</p>

<h2>What if something doesn&#39;t work?</h2>

<p>In the above, we assumed that everything worked. What if it doesn&#39;t? Promises as a standard offer two other methods: <code>catch</code>, and <code>finally</code>.</p>

<p><code>catch</code> is given a code-ref, which will be called when a Promise is &quot;rejected&quot;. When things work as above, each Promise is &quot;resolved&quot;. That means the value it was resolved with gets passed to the next <code>then</cod...

devdata/https_mojolicious.io_blog_2017_12_14_day-14-you-promised-to-call view on Meta::CPAN

usage &quot;$outdir: $!&quot; if ! -d $outdir;
usage &quot;$suffixesfile: $!&quot; if ! -f $suffixesfile;

my $outpath = path($outdir);
my @suffixes = getsuffixes($suffixesfile, $outpath);
my $ua = Mojo::UserAgent-&gt;new;

sub getsuffixes {
  my ($suffixesfile, $outpath) = @_;
  open my $fh, &#39;&lt;&#39;, $suffixesfile or die $!;
  grep { !-f $outpath-&gt;child($_); } map { chomp; $_ } &lt;$fh&gt;;
}
</code></pre>

<p>We also want a procedure to handle results that are ready, to store them in a file if successful:</p>

<pre><code>sub handle_result {
  my ($outpath, $tx, $s) = @_;
  if ($tx-&gt;res-&gt;is_success) {
    print &quot;got $s\n&quot;;
    $outpath-&gt;child($s)-&gt;spurt($tx-&gt;res-&gt;body);
  } else {
    print &quot;error $s\n&quot;;
  }
}
</code></pre>

<p>And now, the Promise part:</p>

<pre><code>my @promises = map makepromise($urlbase, $ua, \@suffixes, $outpath), (1..$MAXREQ);
Mojo::Promise-&gt;all(@promises)-&gt;wait if @promises;

sub makepromise {
  my ($urlbase, $ua, $suffixes, $outpath) = @_;
  my $s = shift @$suffixes;
  return if !defined $s;
  my $url = $urlbase . $s;
  print &quot;getting $url\n&quot;;
  $ua-&gt;get_p($url)-&gt;then(sub {
    my ($tx) = @_;

devdata/https_mojolicious.io_blog_2017_12_18_day-18-the-wishlist-model view on Meta::CPAN

<p>Most modern web applications adhere to a pattern called <a href="https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller">Model View Controller</a> or MVC.
Much has been written about MVC, more than could be conveyed here.
Quickly though, the view is how data is displayed, the template.
The model is the database, both read and write access, and how to manipulated it.
This is usually thought of as the &quot;business logic&quot;.
Finally the controller is supposed to be the minimal amount of logic that can be used to connect the two and process web requests.</p>

<p>In Perl almost all relational database access is via <a href="https://metacpan.org/pod/DBI">DBI</a> whether directly or indirectly.
However this isn&#39;t a model as such, a model really needs to have defined data layout (schema) and manipulation.</p>

<p>Many Perl users turn to <a href="https://metacpan.org/pod/DBIx::Class">DBIx::Class</a>, an <a href="https://en.wikipedia.org/wiki/Object-relational_mapping">Object-Relational Mapper</a> (ORM), and this is a great choice.
It hides most of the SQL and creates classes for the tables.
However some developers like staying a little closer to the SQL.</p>

<h2>Mojo-Flavored DBI</h2>

<p>The Mojolicious community has several modules that live partway between DBI and DBIx::Class.
I lovingly call them &quot;Mojo-Flavored DBI&quot; collectively.
The first of these was <a href="https://metacpan.org/pod/Mojo::Pg">Mojo::Pg</a> for <a href="https://www.postgresql.org/">PostgreSQL</a>.
Quickly, copycat modules were made, <a href="https://metacpan.org/pod/Mojo::mysql">Mojo::mysql</a> for <a href="https://www.mysql.com/">MySQL</a> and <a href="https://metacpan.org/pod/Mojo::SQLite">Mojo::SQLite</a> for the embedded database <a href="...

devdata/https_mojolicious.io_blog_2017_12_18_day-18-the-wishlist-model view on Meta::CPAN

  <span class="hljs-keyword">my</span> <span class="hljs-type">$self</span> = <span class="hljs-function">shift</span>;
  <span class="hljs-keyword">return</span> <span class="hljs-type">$self</span><span class="hljs-type">
</span>    -&gt;sqlite
    -&gt;db
    -&gt;<span class="hljs-function">select</span>(
      &#39;<span class="hljs-string">users</span>&#39; =&gt; [&#39;<span class="hljs-string">name</span>&#39;],
      <span class="hljs-function">undef</span>,
      {-asc =&gt; &#39;<span class="hljs-string">name</span>&#39;},
    )
    -&gt;arrays
    -&gt;<span class="hljs-function">map</span>(<span class="hljs-keyword">sub</span>{ <span class="hljs-variable">$_</span>-&gt;[0] });
}

<span class="hljs-keyword">sub </span><span class="hljs-function">add_item</span> {
  <span class="hljs-keyword">my</span> (<span class="hljs-type">$self</span>, <span class="hljs-type">$user</span>, <span class="hljs-type">$item</span>) = <span class="hljs-type">@_</span>;
  <span class="hljs-type">$item</span>-&gt;{user_id} = <span class="hljs-type">$user</span>-&gt;{id};
  <span class="hljs-keyword">return</span> <span class="hljs-type">$self</span><span class="hljs-type">
</span>    -&gt;sqlite
    -&gt;db
    -&gt;insert(&#39;<span class="hljs-string">items</span>&#39; =&gt; <span class="hljs-type">$item</span>)
    -&gt;last_insert_id;

devdata/https_mojolicious.io_blog_2017_12_21_day-21-virtually-a-lumberjack view on Meta::CPAN

  return Mojo::Asset::File-&gt;new(path =&gt; $path)-&gt;cleanup(0)-&gt;add_chunk(
    data_section(__PACKAGE__, &#39;pgm_header&#39;),
  );
};

sub stdin_read {
  my ($self, $stream, $bytes) = @_;

  foreach my $packet (split /\n/, $bytes) {
    my $chunk = join &#39; &#39;, (
      map { sprintf &#39;%03i&#39;, hex($_) } split /:/, $packet
    );
    $self-&gt;add_chunk($chunk . &quot;\n&quot;);
  }
}

__DATA__

@@ pgm_header
P2
00000000 00000000

devdata/https_mojolicious.io_blog_2017_12_21_day-21-virtually-a-lumberjack view on Meta::CPAN

to the next one.
The text is what is output to the user.
We also use it to get the name of the file to create.</p>

<p>This could be formatted in a lot more of a structured way (e.g. YAML), but
this works fine for our purpose!</p>

<p>Let&#39;s make the instructions get loaded in, skipping any empty lines:</p>

<pre><code>has instructions =&gt; sub {[
  map {[split / /, $_, 2]}
  grep { length $_ }
  split /\n/, data_section(__PACKAGE__, &#39;instructions&#39;)
]};
</code></pre>

<p>And add a method which progresses us through the script:</p>

<pre><code>sub instruction_show {
  my $self = shift;

devdata/https_mojolicious.io_blog_2017_12_23_day-23-one-liners-for-fun-and-profit view on Meta::CPAN

<p>A simple use could be to fetch the title from a webpage.</p>

<pre><code>perl -Mojo -E &#39;print g(&quot;mojolicious.org&quot;)-&gt;dom-&gt;at(&quot;title&quot;)-&gt;text&#39;
</code></pre>

<p>This simple request is something you could have done with the <a href="http://mojolicious.org/perldoc/Mojolicious/Command/get">get</a> command,</p>

<pre><code>mojo get mojolicious.org title text
</code></pre>

<p>however the one liner form can let you do much more, like complex mapping.
For example, to get the text and link from each link on a site as a data structure.</p>

<pre><code>perl -Mojo -E &#39;print r({ g(&quot;mojolicious.io/blog&quot;)-&gt;dom-&gt;find(&quot;a&quot;)-&gt;map(sub{ $_-&gt;text =&gt; $_-&gt;{href} })-&gt;each })&#39;
</code></pre>

<p>You can also use Perl&#39;s looping constructs.
Perhaps you have a file full of sites for which you want to get some data.</p>

<pre><code>$ cat sites
mojolicious.org
mojolicious.io
mojocasts.com
</code></pre>

devdata/https_mojolicious.io_blog_2017_12_23_day-23-one-liners-for-fun-and-profit view on Meta::CPAN

<pre><code>perl -Mojo -nlE &#39;say b(g($_)-&gt;dom-&gt;at(&quot;title&quot;)-&gt;text)-&gt;trim&#39; sites
</code></pre>

<p>We can slurp the <code>sites</code> file itself, split the lines and ouput it as JSON.</p>

<pre><code>perl -Mojo -E &#39;print j({sites =&gt; b(f(shift)-&gt;slurp)-&gt;split(&quot;\n&quot;)})&#39; sites
</code></pre>

<p>We can even <code>trim</code> and <code>sort</code> while we do it</p>

<pre><code>perl -Mojo -E &#39;print j({sites =&gt; b(f(shift)-&gt;slurp)-&gt;split(&quot;\n&quot;)-&gt;map(&quot;trim&quot;)-&gt;sort})&#39; sites
</code></pre>

<h3>The Lite App</h3>

<p>If all of that weren&#39;t enough for you, remember that your one-liner is also a Lite app.
This means that you get all of the Lite app keywords.
I like to show the Mojolicious error page, featuring the glorious Fail Raptor, to the uninitiated.
While I could find the image in the repo, just making a &quot;fail app&quot; is so much easier!</p>

<pre><code>perl -Mojo -E &#39;get &quot;/&quot; =&gt; sub { die }; app-&gt;start&#39; daemon -m production

devscripts/update view on Meta::CPAN

    "24" => "https://mojolicious.io/blog/2017/12/24/day-24-release-and-wrap-up",
);

gen_curried_sub(
    'App::CreateAcmeCPANModulesImportModules::create_acme_cpanmodules_import_modules',
    {
        modules => [
            {
                name => '2017',
                url => "https://mojolicious.io/page/advent/2017/",
                extract_urls => [map {$daily_urls{$_}} "01".."24"],
                summary => 'Modules mentioned in Mojolicious Advent Calendar 2017',
                add_modules => [map {@$_} values %add_modules],
            },
            (map {
                +{
                    name => "2017_12_$_",
                    url => $daily_urls{$_},
                    summary => "Modules mentioned in Mojolicious Advent Calendar 2017 (day $_)",
                    add_modules => [@{ $add_modules{$_} // [] }],
                },
            } "01".."24"),
        ],
        typos => \%typos,
        ignore_empty => 1,

t/00-compile.t view on Meta::CPAN

use IO::Handle;

open my $stdin, '<', File::Spec->devnull or die "can't open devnull: $!";

my @warnings;
for my $lib (@module_files)
{
    # see L<perlfaq8/How can I capture STDERR from an external command?>
    my $stderr = IO::Handle->new;

    diag('Running: ', join(', ', map { my $str = $_; $str =~ s/'/\\'/g; q{'} . $str . q{'} }
            $^X, @switches, '-e', "require q[$lib]"))
        if $ENV{PERL_COMPILE_TEST_DEBUG};

    my $pid = open3($stdin, '>&STDERR', $stderr, $^X, @switches, '-e', "require q[$lib]");
    binmode $stderr, ':crlf' if $^O eq 'MSWin32';
    my @_warnings = <$stderr>;
    waitpid($pid, 0);
    is($?, 0, "$lib loaded ok");

    shift @_warnings if @_warnings and $_warnings[0] =~ /^Using .*\bblib/

( run in 2.231 seconds using v1.01-cache-2.11-cpan-49f99fa48dc )