view release on metacpan or  search on metacpan

README.md  view on Meta::CPAN

[![Actions Status](https://github.com/kaz-utashiro/App-Greple-xlate/actions/workflows/test.yml/badge.svg?branch=main)](https://github.com/kaz-utashiro/App-Greple-xlate/actions?workflow=test)
# NAME

App::Greple::xlate - translation support module for greple

# SYNOPSIS

    greple -Mxlate --xlate-engine gpt5 --xlate pattern target-file

    greple -Mxlate --xlate-engine deepl --xlate pattern target-file

# VERSION

Version 2.02

# DESCRIPTION

**Greple** **xlate** module find desired text blocks and replace them by
the translated text.  The primary engine is GPT-5.6 Terra (`llm/gpt5.pm`),
which calls the [llm](https://llm.datasette.io/) command; DeepL
(`deepl.pm`) and legacy **gpty**-based engines are also included.

Translations are cached per file, so re-running a command costs
nothing for unchanged text.  When a document is edited, only the
changed paragraphs are sent to the API again; a context-aware engine
also receives the surrounding translations, the raw source text
around the change, and the previous version of the edited paragraph,
so the new translation keeps the established wording (see
**--xlate-context-window**).  Sensitive strings can be concealed
before transmission (see ["ANONYMIZATION AND TEMPLATES"](#anonymization-and-templates)).

If you want to translate normal text blocks in a document written in
the Perl's pod style, use **greple** command with `--xlate-engine gpt5`
and `perl` module like this:

    greple -Mxlate --xlate-engine gpt5 -Mperl --pod --re '^([\w\pP].*\n)+' --all foo.pm

In this command, pattern string `^([\w\pP].*\n)+` means consecutive
lines starting with alpha-numeric and punctuation letter.  This
command show the area to be translated highlighted.  Option **--all**
is used to produce entire text.

<div>
    <p>
    <img width="750" src="https://raw.githubusercontent.com/kaz-utashiro/App-Greple-xlate/main/images/select-area.png">
    </p>
</div>

Then add `--xlate` option to translate the selected area.  Then, it
will find the desired sections and replace them by the translation
engine's output.

By default, original and translated text is printed in the "conflict
marker" format compatible with [git(1)](http://man.he.net/man1/git).  Using `ifdef` format, you
can get desired part by [unifdef(1)](http://man.he.net/man1/unifdef) command easily.  Output format
can be specified by **--xlate-format** option.

<div>
    <p>
    <img width="750" src="https://raw.githubusercontent.com/kaz-utashiro/App-Greple-xlate/main/images/format-conflict.png">
    </p>
</div>

If you want to translate entire text, use **--match-all** option.  This
is a short-cut to specify the pattern `(?s).+` which matches entire
text.

Conflict marker format data can be viewed in side-by-side style by
[sdif](https://metacpan.org/pod/App%3A%3Asdif) command with `-V` option.  Since it makes no sense
to compare on a per-string basis, the `--no-cdif` option is
recommended.  If you do not need to color the text, specify
`--no-textcolor` (or `--no-tc`).

    sdif -V --no-filename --no-tc --no-cdif data_shishin.deepl-EN-US.cm

<div>
    <p>
    <img width="750" src="https://raw.githubusercontent.com/kaz-utashiro/App-Greple-xlate/main/images/sdif-cm-view.png">
    </p>
</div>

# NORMALIZATION

Processing is done in specified units, but in the case of a sequence
of multiple lines of non-empty text, they are converted together into
a single line.  This operation is performed as follows:

- Remove white space at the beginning and end of each line.
- If a line ends with a full-width punctuation character, concatenate
with next line.
- If a line ends with a full-width character and the next line begins
with a full-width character, concatenate the lines.
- If either the end or the beginning of a line is not a full-width
character, concatenate them by inserting a space character.

Cache data is managed based on the normalized text, so even if
modifications are made that do not affect the normalization results,
the cached translation data will still be effective.

This normalization process is performed only for the first (0th) and
even-numbered pattern.  Thus, if two patterns are specified as
follows, the text matching the first pattern will be processed after
normalization, and no normalization process will be performed on the
text matching the second pattern.

    greple -Mxlate -E normalized -E not-normalized

Therefore, use the first pattern for text that is to be processed by
combining multiple lines into a single line, and use the second
pattern for pre-formatted text.  If there is no text to match in the
first pattern, use a pattern that does not match anything, such as
`(?!)`.

# MASKING

Occasionally, there are parts of text that you do not want translated.
For example, tags in markdown files. DeepL suggests that in such
cases, the part of the text to be excluded be converted to XML tags,
translated, and then restored after the translation is complete.  To
support this, it is possible to specify the parts to be masked from
translation.

    --xlate-setopt maskfile=MASKPATTERN

This will interpret each line of the file `MASKPATTERN` as a regular
expression, translate strings matching it, and revert after
processing.  Lines beginning with `#` are ignored.

Complex pattern can be written on multiple lines with backslash
escaped newline.

How the text is transformed by masking can be seen by **--xlate-mask**
option.

Mask placeholders are well-formed self-closing XML tags such as
`<m id="1" />`.  JSON-based LLM engines receive the tags in their
input arrays.  For DeepL, a request containing marker tags is escaped
and enclosed in a temporary `<xlate>` root, with XML tag handling

README.md  view on Meta::CPAN

    - `always`, `yes`, `1`

        Maintain cache anyway as far as the target is normal file.

    - `clear`

        Clear the cache data first.

    - `never`, `no`, `0`

        Never use cache file even if it exists.

    - `accumulate`

        By default behavior, unused data is removed from the cache file.  If
        you don't want to remove them and keep in the file, use `accumulate`.
- **--xlate-update**

    This option forces to update cache file even if it is not necessary.

# COMMAND LINE INTERFACE

You can easily use this module from the command line by using the
`xlate` command included in the distribution.  See the `xlate` man
page for usage.

The `xlate` command supports GNU-style long options such as
`--to-lang`, `--from-lang`, `--engine`, and `--file`.  Use
`xlate -h` to see all available options.

The `xlate` command works in concert with the Docker environment, so
even if you do not have anything installed on hand, you can use it as
long as Docker is available.  Use `-D` or `-C` option.

Docker operations are handled by [App::dozo](https://metacpan.org/pod/App%3A%3Adozo), which can also be
used as a standalone command.  The `dozo` command supports the
`.dozorc` configuration file for persistent container settings.

Also, since makefiles for various document styles are provided,
translation into other languages is possible without special
specification.  Use `-M` option.

You can also combine the Docker and `make` options so that you can
run `make` in a Docker environment.

Running like `xlate -C` will launch a shell with the current working
git repository mounted.

Read Japanese article in ["SEE ALSO"](#see-also) section for detail.

# EMACS

Load the `xlate.el` file included in the repository to use `xlate`
command from Emacs editor.  `xlate-region` function translate the
given region.  Default language is `EN-US` and you can specify
language invoking it with prefix argument.

<div>
    <p>
    <img width="750" src="https://raw.githubusercontent.com/kaz-utashiro/App-Greple-xlate/main/images/emacs.png">
    </p>
</div>

# ENVIRONMENT

- DEEPL\_AUTH\_KEY

    Set your authentication key for DeepL service.

- OPENAI\_API\_KEY

    OpenAI authentication key, used by the legacy **gpty** engines.  The
    `llm`-based **gpt5** engine reads this variable too, but keys stored
    with `llm keys set openai` also work.

- GREPLE\_XLATE\_CACHE

    Set the default cache strategy (see ["CACHE OPTIONS"](#cache-options)).

# INSTALL

## CPANMINUS

    $ cpanm App::Greple::xlate

## TOOLS

Install the command line tool for the engine you use: `llm` for the
**gpt5** engine, `deepl` for DeepL, `gpty` for the legacy GPT
engines.

[https://llm.datasette.io/](https://llm.datasette.io/)

[https://github.com/DeepLcom/deepl-python](https://github.com/DeepLcom/deepl-python)

[https://github.com/tecolicom/App-gpty](https://github.com/tecolicom/App-gpty)

# SEE ALSO

## MODULES

[App::Greple::xlate::llm](https://metacpan.org/pod/App%3A%3AGreple%3A%3Axlate%3A%3Allm),
[App::Greple::xlate::deepl](https://metacpan.org/pod/App%3A%3AGreple%3A%3Axlate%3A%3Adeepl)

[App::dozo](https://metacpan.org/pod/App%3A%3Adozo) - Generic Docker runner used by xlate for container operations

## RELATED MODULES

- [App::Greple](https://metacpan.org/pod/App%3A%3AGreple)

    See the **greple** manual for the detail about target text pattern.
    Use **--inside**, **--outside**, **--include**, **--exclude** options to
    limit the matching area.

- [App::Greple::update](https://metacpan.org/pod/App%3A%3AGreple%3A%3Aupdate)

    You can use `-Mupdate` module to modify files by the result of
    **greple** command.

- [App::sdif](https://metacpan.org/pod/App%3A%3Asdif)