Fsdb
view release on metacpan or search on metacpan
README.html view on Meta::CPAN
<?xml version="1.0" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Fsdb - a flat-text database for shell scripting</title>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<link rev="made" href="mailto:root@localhost" />
</head>
<body>
<h1 id="NAME">NAME</h1>
<p>Fsdb - a flat-text database for shell scripting</p>
<h1 id="SYNOPSIS">SYNOPSIS</h1>
<p>Fsdb, the flatfile streaming database is package of commands for manipulating flat-ASCII databases from shell scripts. Fsdb is useful to process medium amounts of data (with very little data you'd do it by hand, with megabytes you might want a...
<p>Fsdb is very good at doing things like:</p>
<ul>
<li><p>extracting measurements from experimental output</p>
</li>
<li><p>examining data to address different hypotheses</p>
</li>
<li><p>joining data from different experiments</p>
</li>
<li><p>eliminating/detecting outliers</p>
</li>
<li><p>computing statistics on data (mean, confidence intervals, correlations, histograms)</p>
</li>
<li><p>reformatting data for graphing programs</p>
</li>
</ul>
<p>Fsdb is built around the idea of a flat text file as a database. Fsdb files (by convention, with the extension <i>.fsdb</i>), have a header documenting the schema (what the columns mean), and then each line represents a database record (or row).</...
<p>For example:</p>
<pre><code>#fsdb experiment duration
ufs_mab_sys 37.2
ufs_mab_sys 37.3
ufs_rcp_real 264.5
ufs_rcp_real 277.9</code></pre>
<p>Is a simple file with four experiments (the rows), each with a description, size parameter, and run time in the first, second, and third columns.</p>
<p>Rather than hand-code scripts to do each special case, Fsdb provides higher-level functions. Although it's often easy throw together a custom script to do any single task, I believe that there are several advantages to using Fsdb:</p>
<ul>
<li><p>these programs provide a higher level interface than plain Perl, so</p>
<dl>
<dt id="pod">**</dt>
<dd>
<p>Fewer lines of simpler code:</p>
<pre><code>dbrow '_experiment eq "ufs_mab_sys"' | dbcolstats duration</code></pre>
<p>Picks out just one type of experiment and computes statistics on it, rather than:</p>
<pre><code>while (<>) { split; $sum+=$F[1]; $ss+=$F[1]**2; $n++; }
$mean = $sum / $n; $std_dev = ...</code></pre>
<p>in dozens of places.</p>
</dd>
README.html view on Meta::CPAN
<dd>
<p><a>dbmapreduce</a> now generates plausible output when given no rows of input.</p>
</dd>
<dt id="ENHANCEMENT10">ENHANCEMENT</dt>
<dd>
<p><a>dbroweval</a> the warnings option was backwards; now corrected. As a result, warnings in user code now default off (like in fsdb-1.x).</p>
</dd>
<dt id="BUG-FIX9">BUG FIX</dt>
<dd>
<p><a>dbcolpercentile</a> now defaults to assuming the target column is numeric. The new option <code>-N</code> allows selection of a non-numeric target.</p>
</dd>
<dt id="BUG-FIX10">BUG FIX</dt>
<dd>
<p><a>dbcolscorrelate</a> now includes <code>--sample</code> and <code>--nosample</code> options to compute the sample or full population correlation coefficients. Thanks to Xue Cai for finding this bug.</p>
</dd>
</dl>
<h2 id="Oct-08">2.11, 14-Oct-08</h2>
<p>Still in beta, but picking up some bug fixes.</p>
<dl>
<dt id="ENHANCEMENT11">ENHANCEMENT</dt>
<dd>
<p><a>html_table_to_db</a> is now more aggressive about filling in empty cells with the official empty value, rather than leaving them blank or as whitespace.</p>
</dd>
<dt id="ENHANCEMENT12">ENHANCEMENT</dt>
<dd>
<p><a>dbpipeline</a> now catches failures during pipeline element setup and exits reasonably gracefully.</p>
</dd>
<dt id="BUG-FIX11">BUG FIX</dt>
<dd>
<p><a>dbsubprocess</a> now reaps child processes, thus avoiding running out of processes when used a lot.</p>
</dd>
</dl>
<h2 id="Oct-081">2.12, 16-Oct-08</h2>
<p>Finally, a full (non-beta) 2.x release!</p>
<dl>
<dt id="INCOMPATIBLE-CHANGE6">INCOMPATIBLE CHANGE</dt>
<dd>
<p>Jdb has been renamed Fsdb, the flatfile-streaming database. This change affects all internal Perl APIs, but no shell command-level APIs. While Jdb served well for more than ten years, it is easily confused with the Java debugger (even though Jdb w...
<p>If you just used the shell commands, this change should not affect you. If you used the Perl-level libraries directly in your code, you should be able to rename "Jdb" to "Fsdb" to move to 2.12.</p>
<p>The jdb-announce list not yet been renamed, but it will be shortly.</p>
<p>With this release I've accomplished everything I wanted to in fsdb-2.x. I therefore expect to return to boring, bugfix releases.</p>
</dd>
</dl>
<h2 id="Oct-082">2.13, 30-Oct-08</h2>
<dl>
<dt id="BUG-FIX12">BUG FIX</dt>
<dd>
<p><a>dbrowaccumulate</a> now treats non-numeric data as zero by default.</p>
</dd>
<dt id="BUG-FIX13">BUG FIX</dt>
<dd>
<p>Fixed a perl-5.10ism in <a>dbmapreduce</a> that breaks that program under 5.8. Thanks to Martin Lukac for reporting the bug.</p>
</dd>
</dl>
<h2 id="Nov-08">2.14, 26-Nov-08</h2>
<dl>
<dt id="BUG-FIX14">BUG FIX</dt>
<dd>
<p>Improved documentation for <a>dbmapreduce</a>'s <code>-f</code> option.</p>
</dd>
<dt id="ENHANCEMENT13">ENHANCEMENT</dt>
<dd>
<p><a>dbcolmovingstats</a> how computes a moving standard deviation in addition to a moving mean.</p>
</dd>
</dl>
<h2 id="Apr-09">2.15, 13-Apr-09</h2>
<dl>
<dt id="BUG-FIX15">BUG FIX</dt>
<dd>
<p>Fix a <i>make install</i> bug reported by Shalindra Fernando.</p>
</dd>
</dl>
<h2 id="Apr-091">2.16, 14-Apr-09</h2>
( run in 2.920 seconds using v1.01-cache-2.11-cpan-39bf76dae61 )