LTSV-LINQ
view release on metacpan or search on metacpan
lib/LTSV/LINQ.pm view on Meta::CPAN
=item * L</DIAGNOSTICS> -- Error messages
=item * L</FAQ> -- Common questions and answers
=item * L</COOKBOOK> -- Common patterns
=item * L</DESIGN PHILOSOPHY>
=item * L</LIMITATIONS AND KNOWN ISSUES>
=item * L</BUGS>
=item * L</SUPPORT>
=item * L</SEE ALSO>
=back
=head1 DESCRIPTION
LTSV::LINQ provides a LINQ-style query interface for LTSV (Labeled
Tab-Separated Values) files. It offers a fluent, chainable API for
filtering, transforming, and aggregating LTSV data.
Key features:
=over 4
=item * B<Lazy evaluation> - O(1) memory usage for most operations
=item * B<Method chaining> - Fluent, readable query composition
=item * B<DSL syntax> - Simple key-value filtering
=item * B<60 LINQ methods> - Comprehensive query capabilities
=item * B<Pure Perl> - No XS dependencies
=item * B<Perl 5.005_03+> - Works on ancient and modern Perl
=back
=head2 What is LTSV?
LTSV (Labeled Tab-Separated Values) is a text format for structured logs and
data records. Each line consists of tab-separated fields, where each field is
a C<label:value> pair. A single LTSV record occupies exactly one line.
B<Format example:>
time:2026-02-13T10:00:00 host:192.0.2.1 status:200 url:/index.html bytes:1024
=head3 LTSV Characteristics
=over 4
=item * B<One record per line>
A complete record is always a single newline-terminated line. This makes
streaming processing trivial: read a line, parse it, process it, discard it.
There is no multi-line quoting problem, no block parser required.
=item * B<Tab as field delimiter>
Fields are separated by a single horizontal tab character (C<0x09>).
The tab is a C0 control character in the ASCII range (C<0x00>-C<0x7F>),
which has an important consequence for multibyte character encodings.
=item * B<Colon as label-value separator>
Within each field, the label and value are separated by a single colon
(C<0x3A>, US-ASCII C<:>). This is also a plain ASCII character with the same
multibyte-safety guarantees as the tab.
=back
=head3 LTSV Advantages
=over 4
=item * B<Multibyte-safe delimiters (Tab and Colon)>
This is perhaps the most important technical advantage of LTSV over formats
such as CSV (comma-delimited) or TSV without labels.
In many multibyte character encodings used across Asia and beyond, a
single logical character is represented by a sequence of two or more bytes.
The danger in older encodings is that a byte within a multibyte sequence can
coincidentally equal the byte value of an ASCII delimiter, causing a naive
byte-level parser to split the field in the wrong place.
The following table shows well-known encodings and their byte ranges:
Encoding First byte range Following byte range
---------- -------------------- -------------------------------
Big5 0x81-0xFE 0x40-0x7E, 0xA1-0xFE
Big5-HKSCS 0x81-0xFE 0x40-0x7E, 0xA1-0xFE
CP932X 0x81-0x9F, 0xE0-0xFC 0x40-0x7E, 0x80-0xFC
EUC-JP 0x8E-0x8F, 0xA1-0xFE 0xA1-0xFE
GB 18030 0x81-0xFE 0x30-0x39, 0x40-0x7E, 0x80-0xFE
GBK 0x81-0xFE 0x40-0x7E, 0x80-0xFE
Shift_JIS 0x81-0x9F, 0xE0-0xFC 0x40-0x7E, 0x80-0xFC
RFC 2279 0xC2-0xF4 0x80-0xBF
UHC 0x81-0xFE 0x41-0x5A, 0x61-0x7A, 0x81-0xFE
UTF-8 0xC2-0xF4 0x80-0xBF
WTF-8 0xC2-0xF4 0x80-0xBF
The tab character is C<0x09>. The colon is C<0x3A>. Both values are
strictly below C<0x40>, the lower bound of any following byte in the encodings
listed above. Neither C<0x09> nor C<0x3A> appears anywhere as a first byte
either. Therefore:
TAB (0x09) never appears as a byte within any multibyte character
in Big5, Big5-HKSCS, CP932X, EUC-JP, GB 18030, GBK, Shift_JIS,
RFC 2279, UHC, UTF-8, or WTF-8.
':' (0x3A) never appears as a byte within any multibyte character
in the same set of encodings.
This means that LTSV files containing values in B<any> of those encodings
can be parsed correctly by a B<simple byte-level split> on tab and colon,
with no knowledge of the encoding whatsoever. There is no need to decode
the text before parsing, and no risk of a misidentified delimiter.
By contrast, CSV has encoding problems of a different kind.
The comma (C<0x2C>) and the double-quote (C<0x22>) do B<not> appear as
following bytes in Shift_JIS or Big5, so they are not directly confused with
multibyte character content. However, the backslash (C<0x5C>) B<does>
appear as a valid following byte in both Shift_JIS (following byte range
C<0x40>-C<0x7E> includes C<0x5C>) and Big5 (same range). Many CSV
parsers and the C runtime on Windows use backslash or backslash-like
sequences for escaping, so a naive byte-level search for the escape
character can be misled by a multibyte character whose second byte is
C<0x5C>. Beyond this, CSV's quoting rules are underspecified (RFC 4180
vs. Excel vs. custom dialects differ), which makes writing a correct,
encoding-aware CSV parser considerably harder than parsing LTSV.
LTSV sidesteps all of these issues by choosing delimiters (tab and colon)
that fall below C<0x40>, outside every following-byte range of every traditional
multibyte encoding.
UTF-8 is safe for all ASCII delimiters because continuation bytes are
always in the range C<0x80>-C<0xBF>, never overlapping ASCII. But LTSV's
choice of tab and colon also makes it safe for the traditional multibyte
encodings that predate Unicode, which is critical for systems that still
operate on traditional-encoded data.
=item * B<Self-describing fields>
Every field carries its own label. A record is human-readable without a
separate schema or header line. Fields can appear in any order, and
optional fields can simply be omitted. Adding a new field to some records
does not break parsers that do not know about it.
=item * B<Streaming-friendly>
Because each record is one line, LTSV files can be processed with line-by-line
streaming. Memory usage is proportional to the longest single record, not
the total file size. This is why C<FromLTSV> in this module uses a lazy
iterator rather than loading the whole file.
=item * B<Grep- and awk-friendly>
Standard Unix text tools (C<grep>, C<awk>, C<sed>, C<sort>, C<cut>) work
naturally on LTSV files. A field can be located with a pattern like
C<status:5[0-9][0-9]> without any special parser. This makes ad-hoc
analysis and shell scripting straightforward.
=item * B<No quoting rules>
CSV requires quoting fields that contain commas or newlines, and the quoting
rules differ between implementations (RFC 4180 vs. Microsoft Excel vs. others).
LTSV has no quoting: the tab delimiter and the colon separator do not appear
inside values in any of the supported encodings (by the multibyte-safety
argument above), so no escaping mechanism is needed.
=item * B<Wide adoption in server logging>
LTSV originated in the Japanese web industry as a structured log format for
HTTP access logs. Many web servers (Apache, Nginx) and log aggregation tools
support LTSV output or parsing. The format is particularly popular for
application and infrastructure logging where grep-ability and streaming
analysis matter.
=back
For the formal LTSV specification, see L<http://ltsv.org/>.
=head2 What is LINQ?
LINQ (Language Integrated Query) is a set of query capabilities introduced
in the .NET Framework 3.5 (C# 3.0, 2007) by Microsoft. It defines a
unified model for querying and transforming data from diverse sources --
in-memory collections, relational databases (LINQ to SQL), XML documents
(LINQ to XML), and more -- using a single, consistent API.
This module brings LINQ-style querying to Perl, applied specifically to
LTSV data sources.
=head3 LINQ Characteristics
=over 4
=item * B<Unified query model>
LINQ provides a single set of operators that works uniformly across
data sources. Whether the source is an array, a file, or a database,
the same C<Where>, C<Select>, C<OrderBy>, C<GroupBy> methods apply.
LTSV::LINQ follows this principle: the same methods work on in-memory
arrays (C<From>) and LTSV files (C<FromLTSV>) alike.
=item * B<Declarative style>
LINQ queries express I<what> to retrieve, not I<how> to retrieve it.
A query like C<-E<gt>Where(sub { $_[0]{status} >= 400 })-E<gt>Select(...)>
describes the intent clearly, without explicit loop management.
This reduces cognitive overhead and makes queries easier to read and verify.
=item * B<Composability>
Each LINQ operator takes a sequence and returns a new sequence (or a
scalar result for terminal operators). Because operators are ordinary
method calls that return objects, they compose naturally:
$query->Where(...)->Select(...)->OrderBy(...)->GroupBy(...)->ToArray()
Any intermediate result is itself a valid query object, ready for
further transformation or immediate consumption.
=item * B<Lazy evaluation (deferred execution)>
Intermediate operators (C<Where>, C<Select>, C<Take>, etc.) do not
execute immediately. They construct a chain of iterator closures.
Evaluation is deferred until a terminal operator (C<ToArray>, C<Count>,
C<First>, C<Sum>, C<ForEach>, etc.) pulls items through the chain.
This means:
=over 4
=item - Memory usage is bounded by the window of data in flight, not by the
total data size. A C<Where-E<gt>Select-E<gt>Take(10)> over a million-line
file reads at most 10 records past the first matching one.
lib/LTSV/LINQ.pm view on Meta::CPAN
my $query1 = LTSV::LINQ->FromLTSV("file.ltsv");
my @first = $query1->ToArray();
my $query2 = LTSV::LINQ->FromLTSV("file.ltsv");
my @second = $query2->ToArray();
=item B<Q: How do I do OR conditions in Where?>
A: Use code reference form with C<||>:
# OR condition requires code reference
->Where(sub {
$_[0]{status} == 200 || $_[0]{status} == 304
})
# DSL only supports AND
->Where(status => '200') # Single condition only
=item B<Q: Why does my query seem to run multiple times?>
A: Some operations require multiple passes:
# This reads the file TWICE
my $avg = $query->Average(...); # Pass 1: Calculate
my @all = $query->ToArray(); # Pass 2: Collect (iterator reset!)
# Save result instead
my @all = $query->ToArray();
my $avg = LTSV::LINQ->From(\@all)->Average(...);
=back
=head2 Performance Questions
=over 4
=item B<Q: How can I process a huge file efficiently?>
A: Use lazy operations and avoid materializing:
# Good - constant memory
LTSV::LINQ->FromLTSV("huge.log")
->Where(status => '500')
->ForEach(sub { print $_[0]{message}, "\n" });
# Bad - loads everything into memory
my @all = LTSV::LINQ->FromLTSV("huge.log")->ToArray();
=item B<Q: Why is OrderBy slow on large files?>
A: OrderBy must load all elements into memory to sort them.
# Slow on 1GB file - loads everything
->OrderBy(sub { $_[0]{timestamp} })->Take(10)
# Faster - limit before sorting (if possible)
->Where(status => '500')->OrderBy(...)->Take(10)
=item B<Q: How do I process files larger than memory?>
A: Use ForEach or streaming terminal operations:
# Process 100GB file with 1KB memory
my $error_count = 0;
LTSV::LINQ->FromLTSV("100gb.log")
->Where(sub { $_[0]{level} eq 'ERROR' })
->ForEach(sub { $error_count++ });
print "Errors: $error_count\n";
=back
=head2 DSL Questions
=over 4
=item B<Q: Can DSL do numeric comparisons?>
A: No. DSL uses string equality (C<eq>). Use code reference for numeric:
# DSL - string comparison
->Where(status => '200') # $_[0]{status} eq '200'
# Code ref - numeric comparison
->Where(sub { $_[0]{status} == 200 })
->Where(sub { $_[0]{bytes} > 1000 })
=item B<Q: How do I do case-insensitive matching in DSL?>
A: DSL doesn't support it. Use code reference:
# Case-insensitive requires code reference
->Where(sub { lc($_[0]{method}) eq 'get' })
=item B<Q: Can I use regular expressions in DSL?>
A: No. Use code reference:
# Regex requires code reference
->Where(sub { $_[0]{url} =~ m{^/api/} })
=back
=head2 Compatibility Questions
=over 4
=item B<Q: Does this work on Perl 5.6?>
A: Yes. Tested on Perl 5.005_03 through 5.40+.
=item B<Q: Do I need to install any CPAN modules?>
A: No. Pure Perl with no dependencies beyond core.
=item B<Q: Can I use this on Windows?>
A: Yes. Pure Perl works on all platforms.
=item B<Q: Why support such old Perl versions?>
( run in 0.890 second using v1.01-cache-2.11-cpan-600a1bdf6e4 )