view release on metacpan or  search on metacpan

docs/EN.AmberDB_User-Guide.md  view on Meta::CPAN

```perl
# 1. Start Transaction
$adb->transact_start();

my $product_id = 42;
my $quantity   = 2;
my $user_id    = 1001;

# Read product and check inventory
my @product = $adb->read_id("catalog_product", $product_id);
my $current_stock = $product[8]; # Block 8 = Stock count

if ($current_stock < $quantity) {
    # Insufficient stock: report transaction error (transact_end will trigger automatic rollback)
    $adb->transact_error("catalog_product", "Insufficient stock ($current_stock < $quantity)");
} else {
    # Deduct stock and update product (@product[0] contains $product_id)
    $product[8] -= $quantity;
    $adb->modify_id("catalog_product", @product);

    # Create order record
    my @order = ( $user_id, $product_id, $quantity, time(), "confirmed" );
    my $order_id = $adb->insert_id("orders", undef, @order);
}

# Finalize transaction (commits if clean, automatically rolls back on error)
my $res = $adb->transact_end();

if ($res->{status} eq "commit") {
    print "Order placed successfully and stock deducted!\n";
} else {
    warn "Transaction aborted! All changes were automatically rolled back.\n";
}
```

### 7.5 Durability and Crash Recovery

- **IO::Handle Buffer Flushing & Sync:** Every journal entry is immediately flushed with `$fh->flush`. When configured with `cfg => { txn_sync => 1 }`, AmberDB enforces physical OS/disk-level synchronization (`$fh->sync` / `fsync`).
- **`flock`-Based Ownership:** Active transactions hold an exclusive non-blocking lock (`LOCK_EX | LOCK_NB`) on their active undo journal. If a process crashes unexpectedly, the lock is automatically released by the operating system.
- **Orphan Recovery (`transact_recover`):** If a worker process terminates abruptly, stale journal files in `dbstore/journal/` are scanned. By verifying that the file lock has dropped and the process is no longer active, the journal is safely rolled ...

### 7.6 Core Architectural Philosophy: Authoritative Data vs. Rebuildable Indexes

AmberDB's storage and transaction architecture is organized around a strict hierarchy of data authority:

1. **Authoritative Master Files (Non-Reconstructible Source of Truth):**
   - **`.db` (Master Document Data):** Primary storage for all active records and documents.
   - **`.del` (Soft-Deleted Archive):** Preserves deleted records under `keep_deleted`. Once moved here, deleted data cannot be reconstructed from `.db`.
   - **`.aut` (User Audit Trail):** Chronological, time-series history of who created, edited, or deleted records (`log_owner`). This historical data cannot be generated from any other source.

2. **Derived & Rebuildable Indexes (Disposable Secondary Projections):**
   - **`.inx` (Record Index + Sort), `.fld` (Match), `.src` (Full-Text), `.fac` (Facet), `.slg` (URL Slug):** All these index files are deterministic projections derived directly from `.db`.
   - If any secondary index is corrupted, deleted, or incomplete, running `AmberDB::Tools->set_index($table)` reconstructs all indexes from scratch within seconds with **zero data loss**.

> **Rationale Behind Transaction Design:** `AmberDB::Transact` was deliberately engineered around this principle. A failure writing to the authoritative `.db` file (`is_index == 0`) triggers an immediate automatic `rollback`. However, if the master d...

### 7.7 Exempting Auxiliary Tables from Failure Cascades (`no_transact`)

In multi-table business operations (e.g. creating an order, updating inventory, and charging accounts), some tables represent **core transactional entities** (orders, payments, inventory), while others serve as **auxiliary or secondary records** (cus...

AmberDB allows declaring tables with `no_transact => 1` (either in schema `.table` or dynamically at runtime) to **exempt them from transaction abort cascades**:

1. **Static Schema Definition (`.table` file):**
   ```perl
   # order_customer_summary.table
   {
       name        => "Customer Order Summary",
       no_transact => 1,   # Failures here do NOT abort the main transaction
       schema      => [qw(user_id order_id amount created_at)],
   }
   ```

2. **Dynamic Runtime Configuration (`table_attr`):**
   ```perl
   # Temporarily exempt an auxiliary table during a specific workflow:
   $adb->table_attr("order_customer_summary", no_transact => 1);
   ```

> **How It Works:**  
> - If an error occurs on a table marked `no_transact => 1`, the error is treated as non-critical (like index errors), and `transact_end` proceeds to `commit`.  
> - However, if a primary operation fails and triggers a `rollback`, all changes on `no_transact` tables are **still safely reverted in LIFO order via the undo journal (`dbstore/journal/txn_*`)** to ensure complete database consistency without ghost ...

### 7.8 Multi-Process Concurrency, Lock Isolation, and Stress Verification

AmberDB is engineered for high-concurrency production deployments (Apache, Plack/PSGI, Starman, FastCGI, Starlet) and background worker pools (cron jobs, async queues) where **dozens of independent processes simultaneously read and write to the same ...

#### Operating System-Level Lock & Platform Isolation:
1. **Linux / POSIX Environments:** Leveraging POSIX `fork()` and kernel-level `flock(LOCK_EX)` / `flock(LOCK_SH)` locks, every worker process operates within an isolated memory address space. Strict Two-Phase Locking (Strict 2PL) guarantees serializa...
2. **Windows / MSYS2 Environments:** On Windows NT architectures, full OS-level lock integrity and file descriptor isolation are maintained across independent worker processes (`perl.exe`).
3. **Record-Level Locking (`flock_open` / `flock_close`):** For critical concurrent updates on individual records (such as high-demand stock decrements or shared counters), `$adb->flock_open($table, "write", $id)` eliminates race conditions and lost ...

#### Concurrency & Stress Test Suite (`xt/amberdb_concurrency_stress.t`):
The database engine's resilience under extreme parallel load is verified by the author/release stress test suite:
```bash
# Run multi-process concurrency stress tests directly:
perl -Ilib xt/amberdb_concurrency_stress.t
```
This test suite validates 5 mission-critical concurrency scenarios:
- **1. Parallel Writers:** Multi-worker concurrent inserts verifying zero ID collisions, exact table counts, and synchronized secondary index compilation (`.inx`, `.fld`, `.src`, `.fac`, `.slg`).
- **2. Interleaved Reads & Writes:** Concurrent reader processes executing streaming scans and index queries while writers continuously insert new data without deadlocks or corruption.
- **3. Concurrent Transactions & Crash Recovery:** Simulated sudden process termination mid-transaction, verifying that orphaned undo journals (`journal/txn_*`) are safely rolled back by `transact_recover` without interfering with active concurrent t...
- **4. Concurrent URL Slug Collisions:** Dozens of processes simultaneously inserting identical product titles, confirming deterministic `-1`, `-2` suffix generation and 100% bidirectional bijection (`_0.slg` $\leftrightarrow$ `_1.slg`).
- **5. High-Concurrency Inventory Decrements:** Multiple workers decrementing stock on the same product record under `flock_open` write locks, verifying atomic final inventory consistency.

---

## 8. High-Throughput Batch Operations (Batch ETL & Ingestion)

AmberDB provides a dedicated **2-Phase Batch Pipeline** for ingesting and updating large volumes of records (ETL from CSV, JSON, XML, or REST APIs) at maximum throughput.

### 8.1 Why Use `insert_list` Instead of `insert_id` in a Loop?

Executing `insert_id` in a loop forces the operating system to perform $N$ independent file opens (`open/tie`), lock acquisitions (`flock`), auto-increment sequence mutations, and secondary index writes (`.inx`, `.src`, `.fld`, `.fac`). For $N$ recor...

`insert_list` splits the ingestion workflow into 2 unified phases, reducing I/O complexity to $O(K)$:
1. **Phase 1 (Single I/O Master Table Write):** The `.db` Berkeley DB file is opened exactly **once** (`table_write`). Auto-increment IDs are allocated contiguously (`table_autoid`), field formatters and schema rules are evaluated, and all records ar...
2. **Phase 2 (Batched Secondary Index Merge):** Each secondary index file (`.inx`, `.src`, `.fld`, `.fac`, and junk tier) is opened exactly **once** and the entire batch is compiled into binary bitsets and B-tree branches via unified merges (`records...

> [!TIP]
> On a batch of 10,000 records, `insert_list` finishes **50x to 100x faster** than a standard `insert_id` loop.