view release on metacpan or  search on metacpan

docs/intro.html  view on Meta::CPAN

    <h2><a name="philosophy">Design Philosophy</a></h2>

    <h3>Store only the interesting data</h3>

    <p>The rsyslog daemon running on the central log server is doing a
    very good job of storing the complete set of system log messages
    for each machine into files. This is a lightweight and simple
    approach which guarantees a high degree of reliability. From
    examining the data it was clear that most of it is of little
    interest to us on a daily basis. For example, on a busy SSH server
    the ssh daemon authentication data that we actually wish to
    analyse only constitutes 10% of the entire daily system logs. With
    this in mind we decided to develop a data importing pipeline which
    could process the log messages stored in the files on a regular
    basis (hourly, daily, weekly, as necessary) and filter out the
    data of interest for storage in a separate database. The aim was
    that the design should not prevent the addition, at some later
    date, of a facility to import log messages &quot;live&quot; from
    the incoming stream of messages but that this would not be
    necessary.</p>