App-Greple-xlate
view release on metacpan or search on metacpan
lib/App/Greple/xlate.pm view on Meta::CPAN
of multiple lines of non-empty text, they are converted together into
a single line. This operation is performed as follows:
=over 2
=item *
Remove white space at the beginning and end of each line.
=item *
If a line ends with a full-width punctuation character, concatenate
with next line.
=item *
If a line ends with a full-width character and the next line begins
with a full-width character, concatenate the lines.
=item *
If either the end or the beginning of a line is not a full-width
character, concatenate them by inserting a space character.
=back
Cache data is managed based on the normalized text, so even if
modifications are made that do not affect the normalization results,
the cached translation data will still be effective.
This normalization process is performed only for the first (0th) and
even-numbered pattern. Thus, if two patterns are specified as
follows, the text matching the first pattern will be processed after
normalization, and no normalization process will be performed on the
text matching the second pattern.
greple -Mxlate -E normalized -E not-normalized
Therefore, use the first pattern for text that is to be processed by
combining multiple lines into a single line, and use the second
pattern for pre-formatted text. If there is no text to match in the
first pattern, use a pattern that does not match anything, such as
C<(?!)>.
=head1 MASKING
Occasionally, there are parts of text that you do not want translated.
For example, tags in markdown files. DeepL suggests that in such
cases, the part of the text to be excluded be converted to XML tags,
translated, and then restored after the translation is complete. To
support this, it is possible to specify the parts to be masked from
translation.
--xlate-setopt maskfile=MASKPATTERN
This will interpret each line of the file C<MASKPATTERN> as a regular
expression, translate strings matching it, and revert after
processing. Lines beginning with C<#> are ignored.
Complex pattern can be written on multiple lines with backslash
escaped newline.
How the text is transformed by masking can be seen by B<--xlate-mask>
option.
Mask placeholders are well-formed self-closing XML tags such as
C<< <m id="1" /> >>. JSON-based LLM engines receive the tags in their
input arrays. For DeepL, a request containing marker tags is escaped
and enclosed in a temporary C<< <xlate> >> root, with XML tag handling
enabled and each marker category registered as a non-splitting tag.
The wrapper is removed before the placeholders are validated and
restored.
Masking protects markup from being translated. To conceal sensitive
strings from the translation service itself, see L</ANONYMIZATION AND
TEMPLATES>; both can be used together.
This interface is experimental and subject to change in the future.
=head1 ANONYMIZATION AND TEMPLATES
Sensitive strings can be concealed before they are sent to the
translation API and restored in the output. Three sources of
anonymization rules are available: a dictionary file
(B<--xlate-anonymize>), inline marks in the document itself
(B<--xlate-anonymize-mark>), and YAML front matter values
(B<--xlate-frontmatter>). Each string is replaced by a category tag
such as C<< <person id="1" /> >> during transmission. The concealment
target is API transmission only: local cache files store restored
plain text. Use B<--xlate-dryrun> to inspect exactly what would be
transmitted.
For form documents (quarterly reports and the like), define the
actors up front and reference them in the body:
---
å ±å‘Šè€…: 山田太郎
発注会社: ã‚¢ã‚¯ãƒ¡æ ªå¼ä¼šç¤¾
---
本件ã«ã¤ã„㦠{{ å ±å‘Šè€… }} ãŒèª¿æŸ»ã‚’è¡Œã£ãŸã€‚
Translate the template once per language with C<--xlate-template>
(and C<--xlate-frontmatter> when the values are kept in the file),
then render each case with B<pandoc-embedz> standalone mode --
values under C<global:> in an external config never reach the
translation API at all:
greple -Mxlate --xlate --xlate-engine=gpt5 --xlate-to=EN-US \
--xlate-template= --xlate-format=xtxt \
--match-paragraph --all --need=0 \
report-template.md > report-template.EN.md
pandoc-embedz --standalone report-template.EN.md \
-c case-123.yaml -o report-123.EN.md < /dev/null
For inline marks, providing a macro definition config makes the same
translated template render either the real names or a redacted
version:
# macros.yaml # macros-redacted.yaml
preamble: | preamble: |
{% macro person(name) %}{{ name }}{% endmacro %}
{% macro person(name) %}(関係者){% endmacro %}
Exclude embedz blocks from translation when a document contains them:
--exclude '^```embedz\n(?s:.*?)^```\n'
=head1 OPTIONS
lib/App/Greple/xlate.pm view on Meta::CPAN
next if ! $formatter{$_} or ref $formatter{$_};
$formatter{$_} = $formatter{$formatter{$_}} // die;
}
my %cache;
use App::Greple::xlate::Mask;
my $maskobj;
my $anonobj;
sub setup {
return if state $once_called++;
if (defined $cache_method) {
if ($cache_method eq '') {
$cache_method = 'auto';
}
if ($cache_method =~ /^(no|never)/i) {
$cache_method = '';
}
}
if ($xlate_engine) {
# Resolve the engine module. Backend-based engines live under a
# backend namespace (e.g. llm::gpt5, gpty::gpt5); others live
# directly under App::Greple::xlate (e.g. deepl, null). Try
# backend namespaces FIRST, in order of preference, so that
# --xlate-engine=gpt5 binds to llm::gpt5 even if a stale
# top-level App::Greple::xlate::gpt5 lingers in @INC from an
# older install. Use --xlate-setopt backend=NAME to force a
# specific backend (e.g. backend=gpty for comparison with the
# old gpty engine).
my @backend = length($engine_backend // '') ? $engine_backend : qw(llm gpty);
my $mod;
for my $cand ((map __PACKAGE__ . "::$_\::$xlate_engine", @backend),
__PACKAGE__ . "::$xlate_engine") {
if (eval "require $cand; 1") { $mod = $cand; last }
# Fall through only when the candidate itself is missing;
# a syntax error or a missing dependency inside an existing
# module must be reported, not silently skipped.
(my $path = $cand) =~ s{::}{/}g;
die $@ unless $@ =~ /^Can't locate \Q$path.pm\E /;
}
$mod or die "Engine $xlate_engine is not available.\n";
$mod->import;
no strict 'refs';
${"$mod\::lang_from"} = $lang_from;
${"$mod\::lang_to"} = $lang_to;
*XLATE = \&{"$mod\::xlate"};
$engine_supports_context = ${"$mod\::XLATE_CONTEXT"};
if (not defined &XLATE) {
die "No \"xlate\" function in $mod.\n";
}
}
if (my $pat = opt('mask')) {
$maskobj = App::Greple::xlate::Mask->new(pattern => $pat);
}
if (my $patfile = opt('maskfile')) {
$maskobj = App::Greple::xlate::Mask->new(file => $patfile);
}
if (defined $anonymize_file or defined $anonymize_mark) {
$anonobj = App::Greple::xlate::Mask->new(STABLE => 1);
$anonobj->add_escape_rule;
$anonobj->load_anonymize_file($anonymize_file)
if defined $anonymize_file;
}
}
use App::Greple::xlate::Text;
sub postgrep {
my $grep = shift;
my @blocks;
my %pending;
for my $r ($grep->result) {
my($b, @match) = @$r;
for my $m (@match) {
my($s, $e, $i) = @$m;
my $key = App::Greple::xlate::Text
->new($grep->cut(@$m), paragraph => ($i % 2 == 0))
->normalized;
my $hit = !$pending{$key} && exists $cache{$key};
if (not $hit and not $pending{$key}++) {
$cache{$key} = undef;
}
push @blocks, { key => $key, s => $s, e => $e, hit => $hit };
}
}
my @regions;
my $i = 0;
while ($i < @blocks) {
if ($blocks[$i]{hit}) { $i++; next }
my $j = $i;
$j++ while $j < @blocks and not $blocks[$j]{hit};
push @regions, [ $i, $j - 1 ];
$i = $j;
}
return if not @regions;
my $with_context = $engine_supports_context
&& $context_window > 0
&& grep { $_->{hit} } @blocks;
if ($with_context) {
my %queued;
for my $region (@regions) {
my @texts = grep { not $queued{$_}++ }
map $blocks[$_]{key}, $region->[0] .. $region->[1];
next unless @texts;
cache_update({
texts => \@texts,
context => region_context(\@blocks, @$region),
});
}
} else {
my %seen;
my @texts = grep { not $seen{$_}++ }
map $blocks[$_]{key},
map { $_->[0] .. $_->[1] } @regions;
cache_update({ texts => \@texts, context => undef });
}
}
our $CONTEXT_SOURCE_MAX = 2000; # per-side raw source slice limit
lib/App/Greple/xlate.pm view on Meta::CPAN
return $s;
}
}
sub callback { goto &xlate }
sub mask_string {
my($s) = +{ @_ }->{match};
if ($anonobj) {
$anonobj->mask($s);
}
if ($maskobj) {
$maskobj->mask($s);
}
$s;
}
sub cache_file {
my $file = sprintf("%s.xlate-%s-%s.json",
$current_file, $xlate_engine, $lang_to);
if ($cache_method eq 'auto') {
# Seeding targets a document whose cache does not exist yet,
# so a seed implies cache creation even in auto mode.
(-f $file or defined $cache_seed) ? $file : undef;
} else {
if ($cache_method and -f $current_file) {
$file;
} else {
undef;
}
}
}
sub begin {
setup if not (state $done++);
my %args = @_;
$current_file = delete $args{&::FILELABEL} or die;
s/\z/\n/ if /.\z/;
$current_text = $_;
$frontmatter_len = 0;
if ($use_frontmatter
and $current_text =~ /\A(---\n(?s:.*?)^---\n)/m) {
my $fm = $1;
$frontmatter_len = length $fm;
# A paragraph-style match pattern joins the front matter with
# the first body paragraph unless a blank line separates them,
# and a straddling match defeats the --exclude region.
if (substr($current_text, $frontmatter_len, 1) ne "\n") {
warn "$current_file: no blank line after front matter; " .
"it may be caught by paragraph matching.\n";
}
my @values;
for my $line (split /\n/, $fm) {
next if $line =~ /^---/;
my($k, $v) = $line =~ /^([^\s:#][^:]*):\s*(.+?)\s*$/ or next;
$v =~ s/\A(["'])(.*)\1\z/$2/s; # strip surrounding quotes
push @values, $v;
}
if (@values) {
if (not $anonobj) {
$anonobj = App::Greple::xlate::Mask->new(STABLE => 1);
$anonobj->add_escape_rule;
}
$anonobj->add_rule(var => quotemeta($_)) for @values;
}
}
if ($anonobj and defined $anonymize_mark) {
my $regex = length($anonymize_mark)
? $anonymize_mark : $App::Greple::xlate::Mask::DEFAULT_MARK;
$anonobj->file_rules(
App::Greple::xlate::Mask::extract_marks($current_text, $regex));
}
if (not defined $xlate_engine) {
die "Select translation engine.\n";
}
if ($output_format =~ /^(:+)$/) {
$colon_count = length($1);
$output_format = 'colon';
}
if (my $file = cache_file) {
my @opt;
if ($cache_method =~ /create|clear/i) {
push @opt, clear => 1;
}
if ($cache_method =~ /accumulate/i) {
push @opt, accumulate => 1;
}
if ($force_update) {
push @opt, force_update => 1;
}
if (defined $cache_seed) {
push @opt, seed => $cache_seed;
}
if ($dryrun) {
push @opt, readonly => 1;
}
require App::Greple::xlate::Cache;
tie %cache, 'App::Greple::xlate::Cache', $file, @opt;
die "skip $current_file" if $cache_method eq 'create';
}
}
sub end {
# if (my $obj = tied %cache) {
# $obj->update;
# }
}
sub set {
while (my($key, $val) = splice @_, 0, 2) {
next if $key eq &::FILELABEL;
die "$key: Invalid option.\n" if not exists $opt{$key};
opt($key) = $val;
}
}
1;
__DATA__
builtin xlate-debug! $debug
builtin xlate-progress! $show_progress
( run in 2.100 seconds using v1.01-cache-2.11-cpan-aadc1410aed )