view release on metacpan or  search on metacpan

agent/skills/chorus-pdf.md  view on Meta::CPAN

After the layout analysis (`analyse_pages`), the script knows the **exact number of
`LTFigure` elements** to send to Claude. Each figure = 1 API call ≈ 30 s average.
The IDE has a hard timeout of ~10 minutes.

**Decision rule — applied inside the generated script:**

| Figures detected (`n_figs`) | Estimated time | Action |
|----------------------------:|---------------:|--------|
| ≤ 15 | ≤ 7.5 min | proceed directly |
| 16–19 | 8–9.5 min | borderline — script still exits with a warning |
| ≥ 16 | > 8 min | **exit(2)** + print `nohup` command |

The script prints the exact `nohup` command to copy-paste, then exits with code **2**
(not an error — "nohup required"). Exit code 0 is reserved for successful completion.

```
[chorus-pdf]   → 31 pages, 12 with figures, 18 figure(s) total
[chorus-pdf] ⚠️  18 figures detected — estimated time ~9–13 min (threshold: 15).
[chorus-pdf]    Run with nohup to avoid a 10-min IDE timeout:
[chorus-pdf]    CHORUS_PDF_FORCE=1 nohup python3 /path/to/agent/extract-pdf-<slug>.py > corpus/<NNN>-<slug>-vision.md.log 2>&1 &
```

The threshold constant `NOHUP_THRESHOLD = 15` is defined at the top of the script
and can be adjusted if the API is consistently faster (lower latency environment).

**Bypass `CHORUS_PDF_FORCE=1`:** when the script is launched via `nohup` (detached from
the IDE), set `CHORUS_PDF_FORCE=1` to skip the gate and proceed unconditionally.
Without this flag, the gate fires even under `nohup` because the script cannot detect
whether it is running inside the IDE or not.

> ⚠️ **instruction:** when executing the script and receives exit code 2, it must **not** retry automatically. Instead, it must
> set `CHORUS_PDF_FORCE=1` and relaunch with `nohup` — **launch the nohup command**
> (pre-approved, no confirmation required).

---

## Phase 2 — Generate the extraction script

Create `$SANDBOX/agent/` if it does not exist.
writes `$SANDBOX/agent/extract-pdf-<slug>.py`, then executes it

### Vision extraction prompt (used verbatim in `--auto` and `--images` scripts)

```
You are a technical document extraction engine.
Your task is to produce a complete, faithful plain-text reconstruction of this PDF page.

Apply the following rules strictly:

TEXT
- Extract all text in reading order (top to bottom, left to right).
- For multi-column layouts: extract column 1 fully, then column 2. Insert a blank line between columns.
- Preserve section numbers, article numbers, and clause references exactly as printed.
- Preserve all footnote markers and footnote text (append footnotes at end of page output).
- Do not summarize, paraphrase, or omit any text.

TABLES
- Reconstruct every table in Markdown format (pipe syntax).
- Preserve all column headers, row labels, units, and footnote references inside the table.
- If a table spans multiple pages: output the fragment visible on this page; prefix it with
  [TABLE CONTINUED — <table title or number>] if this is a continuation.

FIGURES AND DIAGRAMS
- For every figure, diagram, or illustration: output a block of the form:
    [FIGURE <N> — <title or caption>]
    <Structured description of all visual content:>
    - Labeled dimensions, dimensions with units
    - Named components and their spatial relationships
    - Numerical values visible in or next to the figure
    - Arrows, load paths, connection points, hinge symbols, support symbols
    - Hatching patterns and what material or condition they represent
    - Scale bar if present
    [END FIGURE <N>]
- If there is no figure number or caption in the PDF, assign [FIGURE ?] and describe anyway.

EQUATIONS AND FORMULAS
- Render every equation in linearized form (e.g., σ = F / A).
- Preserve all variable names, subscripts, and units as printed.

HEADERS AND FOOTERS
- If a page has a running header or footer containing normative information (standard number,
  edition date, section title): include it once at the top of the page output as:
    [HEADER: <content>]
- Omit purely decorative headers/footers (page number alone, logo only).

OUTPUT FORMAT
- Begin each page with: === PAGE <N> ===
- End each page with: === END PAGE <N> ===
- Separate pages with a single blank line.
- Use UTF-8. Preserve all special characters (±, ≤, ≥, ×, °, ², ³, φ, σ, …).
- Do not add commentary outside the === PAGE === markers.
```

---

### Script template — Text mode (no flag + Claude unavailable, or forced text mode)

Uses `pdfminer.six` only. No API key, no network. Figures produce a placeholder.
Output: `<NNN>-<slug>-text.txt`

```python
#!/usr/bin/env python3
"""
chorus-pdf extraction script — text mode (default)
Generated by chorus-pdf skill
Sandbox : <sandbox-name>
Source  : <input-pdf-path>
Output  : <output-txt-path>   (e.g. corpus/003-uk-approved-doc-a-2013-text.txt)
"""

import sys
import os

PDF_PATH    = "<input-pdf-path>"
OUTPUT_PATH = "<output-txt-path>"

FIGURE_PLACEHOLDER = (
    "[FIGURE — not extracted]\n"
    "[Run chorus-pdf with --hybrid, --auto or --images to extract figures via LLM vision]"
)

agent/skills/chorus-pdf.md  view on Meta::CPAN

```
# No flag — hybrid activated automatically if API key present (DEFAULT)
chorus-pdf  <sandbox> corpus/002-uk-approved-doc-a-2013.pdf
→ corpus/003-uk-approved-doc-a-2013-vision.md   (hybrid mode)

# No API key — text mode fallback
chorus-pdf  <sandbox> corpus/002-uk-approved-doc-a-2013.pdf
→ corpus/003-uk-approved-doc-a-2013-text.txt

# API key available — text-dominant document, faster
chorus-pdf  <sandbox> corpus/002-uk-approved-doc-a-2013.pdf --auto
→ corpus/003-uk-approved-doc-a-2013-vision.md

# Mostly diagrams or scanned PDF
chorus-pdf  <sandbox> corpus/002-uk-approved-doc-a-2013.pdf --images
→ corpus/003-uk-approved-doc-a-2013-vision.md

# Then in all cases:
chorus-feed <sandbox> corpus/003-uk-approved-doc-a-2013-text.txt
            (or: corpus/003-uk-approved-doc-a-2013-vision.md)
```

If a `.txt` from `pdftotext` already exists alongside the PDF, prefer the `-text.txt`
(text mode) or `-vision.md` (hybrid/auto/images) for `chorus-feed`.
The `pdftotext` output can be kept for diff/audit purposes.

---

## Quick Reference — Naming Conventions

| Artifact | Convention | Example |
|----------|-----------|---------|
| Extraction script | `agent/extract-pdf-<slug>.py` | `agent/extract-pdf-uk-approved-doc-a.py` |
| Text mode output | `corpus/<NNN>-<slug>-text.txt` | `corpus/003-uk-approved-doc-a-2013-text.txt` |
| Auto/Hybrid/Images output | `corpus/<NNN>-<slug>-vision.md` | `corpus/003-uk-approved-doc-a-2013-vision.md` |
| Original PDF | kept as-is in `corpus/` | `corpus/002-uk-approved-doc-a-2013.pdf` |

---

## Troubleshooting

**"The output in text mode is identical to what pdftotext produced"**
→ Both tools read the same embedded text layer. The gain from pdfminer is the layout
  ordering (multi-column), not the character content. For richer extraction, use `--auto`.

**"Text mode output has garbled column order"**
→ `pdfminer` uses `boxes_flow=0.5` which handles most two-column layouts. For unusual
  layouts (three columns, overlapping regions), `--auto` or `--images` will be more
  reliable since Claude reconstructs column order visually.

**"Figures are described but values seem invented"**
→ LLMs can hallucinate values in dense technical diagrams. Always cross-check critical
  normative values against the original PDF. Mark uncertain values with
  `# TODO: verify against PDF §<N>` in `Helpers.pm`.

**"Auto mode: a text page was sent to vision unnecessarily"**
→ The threshold `len(text.strip()) > 50` in `classify_pages` may be too low for sparse
  pages (cover pages, blank pages, page numbers only). Increase to `> 200` if needed.

**"Chunk boundaries cut through a table"**
→ The fragment is prefixed `[TABLE CONTINUED — ...]`. `chorus-feed` treats both
  fragments as separate text blocks — rarely an issue for KB extraction.

**"Too slow — 54-page PDF with --auto takes a long time"**
→ Only the vision pages hit the API. If 16/54 pages have figures: 4 chunks × ~30s = ~2 min.
  The text pages (pdfminer) complete in seconds. Total: ~2–3 minutes for a 54-page standard.

**"Script exited with code 2 — nohup required"**
→ The layout analysis detected ≥ 16 figures. The script aborted before any API call.
  Copy the `nohup` command printed to stderr and run it in a terminal:
  ```bash
  nohup python3 $SANDBOX/agent/extract-pdf-<slug>.py > $SANDBOX/corpus/<NNN>-<slug>-vision.md.log 2>&1 &
  tail -f $SANDBOX/corpus/<NNN>-<slug>-vision.md.log
  ```
  The threshold `NOHUP_THRESHOLD = 15` can be raised in the script header if your
  API responses are consistently faster (< 20s/call on average).

**"I want higher quality on specific diagram pages"**
→ Increase `DPI = 200` in the script header. Keep `CHUNK_SIZE = 3` at 200 DPI
  (PNG ≈ 900 KB/page → 3 pages ≈ 2.7 MB payload).