LTSV-LINQ

 view release on metacpan or  search on metacpan

README  view on Meta::CPAN

TARGET USE CASES

  - Querying LTSV access logs (Apache, nginx, HTTP::Handy, etc.)
  - Filtering and transforming arrays of hashrefs
  - Data analysis and aggregation without external dependencies
  - Educational use: learning LINQ concepts in Perl

LIMITATIONS

  - All data is processed in memory (no streaming)
  - No SQL support; use DB::Handy for SQL-style queries
  - LTSV parsing is line-oriented; multi-line values are not supported
  - Lazy evaluation defers errors until a terminal method is called

AUTHOR

  INABA Hitoshi <ina@cpan.org>

COPYRIGHT AND LICENSE

lib/LTSV/LINQ.pm  view on Meta::CPAN

  time:2026-02-13T10:00:00	host:192.0.2.1	status:200	url:/index.html	bytes:1024

=head3 LTSV Characteristics

=over 4

=item * B<One record per line>

A complete record is always a single newline-terminated line. This makes
streaming processing trivial: read a line, parse it, process it, discard it.
There is no multi-line quoting problem, no block parser required.

=item * B<Tab as field delimiter>

Fields are separated by a single horizontal tab character (C<0x09>).
The tab is a C0 control character in the ASCII range (C<0x00>-C<0x7F>),
which has an important consequence for multibyte character encodings.

=item * B<Colon as label-value separator>

lib/LTSV/LINQ.pm  view on Meta::CPAN

=item * B<Self-describing fields>

Every field carries its own label. A record is human-readable without a
separate schema or header line. Fields can appear in any order, and
optional fields can simply be omitted. Adding a new field to some records
does not break parsers that do not know about it.

=item * B<Streaming-friendly>

Because each record is one line, LTSV files can be processed with line-by-line
streaming. Memory usage is proportional to the longest single record, not
the total file size. This is why C<FromLTSV> in this module uses a lazy
iterator rather than loading the whole file.

=item * B<Grep- and awk-friendly>

Standard Unix text tools (C<grep>, C<awk>, C<sed>, C<sort>, C<cut>) work
naturally on LTSV files. A field can be located with a pattern like
C<status:5[0-9][0-9]> without any special parser. This makes ad-hoc
analysis and shell scripting straightforward.

lib/LTSV/LINQ.pm  view on Meta::CPAN

rules differ between implementations (RFC 4180 vs. Microsoft Excel vs. others).
LTSV has no quoting: the tab delimiter and the colon separator do not appear
inside values in any of the supported encodings (by the multibyte-safety
argument above), so no escaping mechanism is needed.

=item * B<Wide adoption in server logging>

LTSV originated in the Japanese web industry as a structured log format for
HTTP access logs. Many web servers (Apache, Nginx) and log aggregation tools
support LTSV output or parsing. The format is particularly popular for
application and infrastructure logging where grep-ability and streaming
analysis matter.

=back

For the formal LTSV specification, see L<http://ltsv.org/>.

=head2 What is LINQ?

LINQ (Language Integrated Query) is a set of query capabilities introduced
in the .NET Framework 3.5 (C# 3.0, 2007) by Microsoft. It defines a

lib/LTSV/LINQ.pm  view on Meta::CPAN

A: OrderBy must load all elements into memory to sort them.

  # Slow on 1GB file - loads everything
  ->OrderBy(sub { $_[0]{timestamp} })->Take(10)

  # Faster - limit before sorting (if possible)
  ->Where(status => '500')->OrderBy(...)->Take(10)

=item B<Q: How do I process files larger than memory?>

A: Use ForEach or streaming terminal operations:

  # Process 100GB file with 1KB memory
  my $error_count = 0;
  LTSV::LINQ->FromLTSV("100gb.log")
      ->Where(sub { $_[0]{level} eq 'ERROR' })
      ->ForEach(sub { $error_count++ });

  print "Errors: $error_count\n";

=back