view release on metacpan or  search on metacpan

agent/skills/chorus-import-project.md  view on Meta::CPAN

if word_mode == "hybrid" and all_figure_descs:
    annotated_descs, xref_index_block, xref_map = _xref_pass_word(
        all_figure_descs, block_texts, find_text_occurrences_word)
else:
    annotated_descs = all_figure_descs
    xref_index_block = ""
    xref_map = {}
# xref_map passed to Phase 3 as first-class matching candidates (same as PDF pipeline)
```

##### Step 3 — Assemble final output

XML order is the reading order — no Y-sort required. Append XREF INDEX at end.

```python
output_parts = [content for (_, content) in elements]
if xref_index_block:
    output_parts.append(xref_index_block)
extracted_text = "\n\n".join(output_parts)
```

##### nohup gate (hybrid mode — Word)

If the document contains **≥ 15 embedded images** → exit(2) + nohup command.
`CHORUS_WORD_FORCE=1` to bypass, identical to `chorus-pdf` nohup gate.

##### Figure-heavy domains warning (text mode)

When `word_mode == "text"` **and** ≥ 5 images detected, emit the same prominent
warning as the PDF pipeline — Phase 3 will be blind to image content.

##### Dependencies

| Package | Install | Notes |
|---------|---------|-------|
| `python-docx` | `pip install python-docx` | DOCX extraction |
| `Pillow` | `pip install Pillow` | Image conversion (hybrid mode) |

> ⛔ **If extraction tools are absent** → warn and offer Case A (inline paste).
> Never block the workflow over a missing optional tool.

---

#### Other formats — LibreOffice universal fallback

For formats not natively handled (`.doc`, `.odt`, `.pptx`, `.ods`, `.rtf`):

```bash
libreoffice --headless --convert-to pdf "<fichier>" --outdir /tmp/
# → produces /tmp/<basename>.pdf → feed to PDF hybrid pipeline above
```

If LibreOffice is absent → ask the engineer to provide copy-pasted content (Case A).
Never block the workflow over a missing tool — offer the inline alternative.

---

## Phase 1 — Read the KB (canonical terminology)

### 1.0 Sandbox inventory (first tool call — token keepalive)

**Before reading any file**, read the directory tree $SANDBOX/ immediately.

This serves two purposes:
1. Acquires the full sandbox structure early (agents list, rules dirs, existing JSON/report files)
2. Ensures at least one tool call happens before any long reading+thinking cycle,
   keeping the IDE token active from the very start.

Use this inventory to:
- Confirm the list of `<slug>.org` files to read in 1.2
- Know which `rules/<slug>/` directories exist (for the keepalive calls in 1.2)
- Detect `$SANDBOX/agent/thesaurus.org` if present (for 1.2b — highest priority source)
- Detect any existing `import-report-*.org` files (for 1.3 — secondary memory)

### 1.1 Pipeline Index

Read `$SANDBOX/agent/chorus/index.org`:
- Namespace + agent list (slug, pos)
- Global slot dictionary (if present in the index)

### 1.2 Per-agent terminology

For each agent, apply this two-step sequence:

1. **Read** `$SANDBOX/agent/chorus/<slug>.org` and extract:

| KB Section | What we extract |
|---|---|
| `Ontologie` | Domain concepts, synonyms, relationships (e.g. "entrait" = horizontal truss beam) |
| `Catalogue des Frames` | Exact types (`type_element`), mandatory slots per type |
| `Dictionnaire des slots` | Canonical names, value types, units, allowed domains |

2. **Immediately after** (no thinking between the two calls): read the directory tree $SANDBOX/rules/<slug>/
   to list the rule files for this agent.

> **Why the immediate tool call:** Opus extended thinking after reading a dense KB file
> can be long enough to expire the IDE token. Reading the directory tree right after
> each read resets the token TTL and produces a useful rules inventory at no extra cost.

Build an internal **terminology reference**:
```
concept_kb        → type_element / slot_kb       unit_kb     domain
────────────────────────────────────────────────────────────────────
montant porteur   → montant_porteur              —           —
lisse              → lisse_basse / lisse_haute   —           to clarify
classe résistance → classe_bois                 —           C14/C16/C18/C24/C30
épaisseur isolant → epaisseur_mm                mm          positive integer
conductivité λ    → classe_conductivite         —           "031"/"035"/"040"
hauteur libre     → hauteur_libre_m             m           decimal
section           → section_bois                —           "BxH" ex. "45x145"
```

### 1.2b Sandbox thesaurus (highest priority source)

If `$SANDBOX/agent/thesaurus.org` exists, **read it immediately after Phase 1.2**,
before any import-report.

The thesaurus is the **canonical project-terminology memory** for this sandbox. It is
separated from the normative KB (`<slug>.org`) and from import reports: it stores
project-specific synonyms validated by the engineer across all previous imports.

**Priority rule:**
```
thesaurus.org (1.2b)  >  import-report-*.org (1.3)  >  KB aliases (1.2)
```

A mapping present in the thesaurus is applied **at ✅ confidence without asking the
engineer again**, regardless of what the KB or previous import-reports say.

#### Thesaurus format

agent/skills/chorus-import-project.md  view on Meta::CPAN

* Pending — to confirm on next import
  Terms provisionally mapped (⚠️) in a previous import, not yet confirmed by the engineer.
  Re-raised on next import if the same term appears — engineer decision upgrades to ✅ or rejects.

  | Project term   | Proposed KB mapping            | Flag              | Source import     |
  |---|---|---|---|
  | isolant soufflé | type_element: isolant_vrac ?  | ⚠️ _a_confirmer  | import-report-003 |

* Out-of-scope terms (⬜)
  Terms explicitly identified as outside this sandbox's KB scope.
  Silently excluded on future imports — not re-raised to the engineer.

  | Project term  | Reason                      | Recommended sandbox  | Last seen         |
  |---|---|---|---|
  | bardage zinc  | hors périmètre sandbox-structurel | sandbox-bardage | import-report-001 |
```

#### How the thesaurus is used in Phase 3

When building the alignment table (Phase 3), **check the thesaurus first** for every
term in the raw inventory:

```
For each project term T:
  1. Look up T in thesaurus → Aliases type_element  : hit → apply ✅, skip KB lookup
  2. Look up T in thesaurus → Aliases slot values   : hit → apply ✅, skip KB lookup
  3. Look up T in thesaurus → Out-of-scope          : hit → mark ⬜, skip KB lookup
  4. Look up T in thesaurus → Pending               : hit → re-raise ⚠️ to engineer
  5. Not in thesaurus → proceed with KB lookup (Phase 3 standard flow)
```

> **Rule:** a thesaurus hit at step 1–3 is **final** — do not re-ask the engineer,
> do not consult the KB, do not propose alternatives. The engineer already decided.
>
> A thesaurus hit at step 4 (Pending) re-raises the question exactly once. If the
> engineer confirms → move the entry to Aliases and record ✅. If rejected → move to
> Out-of-scope and record ⬜.

#### Thesaurus initialisation

If `thesaurus.org` does not yet exist, it is created automatically at the end of the
first import that produces at least one ✅ or ⚠️ alignment (Phase 6 — see below).
No manual creation is required.

---

### 1.3 Previous alignment decisions

If `$SANDBOX/agent/import-report-*.org` exists, read the **latest report** as a
secondary memory source — complementary to the thesaurus, not a substitute.

- Retrieve mappings not yet promoted to the thesaurus → reapply without asking
- Retrieve pending questions not yet in thesaurus → re-raise if the same terms reappear
- **Skip any entry already covered by thesaurus.org** (1.2b takes priority)

---

## Phase 2 — Raw Inventory of Project Elements

### Keepalive checkpoint (token refresh before long thinking phases)

**Before starting the inventory**, if the source is a filesystem file, call:
```bash
wc -l "<fichier-source-extrait>"
```
or, if working from inline/already-extracted text, read the directoy tree $SANDBOX/agent/
to confirm the report directory.

> **Why:** Phases 2, 3 and 4 are pure thinking phases with no tool calls.
> On a complex project (many element types, many ambiguous terms), the combined
> thinking time across these three phases can expire the IDE token before Phase 5
> triggers the next tool call (JSON write). This checkpoint resets the TTL just
> before entering the silent zone.

Scan the source data and produce a **raw inventory**:
an uninterpreted list of what the engineer has provided.

```
Source line / cell              Term identified      Associated values
────────────────────────────────────────────────────────────────────────
"Poteau porteur 45×145 C24"    "poteau porteur"     dim=45×145, classe=C24
"h libre 2,5m, entraxe 40cm"  "h libre"            2.5m / "entraxe"=40cm
"Laine de verre λ035, e=20cm" "laine de verre"     λ=0.035, e=200mm
"panneau OSB 12mm, CE"         "panneau OSB"        ep=12mm, CE=oui
```

> **Rule:** do not map at this stage — inventory first, align later.
> Preserve the original source text in the inventory for traceability.

---

## Phase 3 — Terminology Alignment

### KB Coverage Gauge (pre-alignment check)

**Before starting the term-by-term alignment**, compute a coverage indicator from the
raw inventory (Phase 2) against the KB reference (Phase 1.2):

```
📊 KB Coverage Gauge
   Distinct types detected in inventory : N  (e.g. 8)
   Types recognised in KB               : n / N  (e.g. 5 / 8 = 62%)
   Critical slots covered               : n / total  (e.g. 11 / 17 = 65%)
   KB Aliases section present           : yes / no
```

| Coverage | Level | Action |
|---|---|---|
| ≥ 80% types + ≥ 80% critical slots | 🟢 Good | Proceed directly |
| 60–79% on either axis | 🟡 Moderate | Proceed with a warning — flag `_couverture_kb: moderate` in the JSON `_import` block |
| < 60% on either axis | 🔴 Low | Emit the warning below and ask whether to continue |

**🔴 Low-coverage warning (display before alignment):**

```
⚠️  KB Coverage Gauge — LOW COVERAGE DETECTED
    Types recognised   : n/N (XX%)
    Critical slots     : n/N (XX%)

    The KB may lack aliases for the project's terminology.