AmberDB
view release on metacpan or search on metacpan
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
4, # Block 4: Title (String sorting)
{ blk => 10, type => 'num' }, # Block 10: Price (Numeric sorting)
{ blk => 12, type => 'date' }, # Block 12: Timestamp sorting (YYYYMMDDHHMMSS)
],
}
```
#### 6.4.2 Using Sort in Query Methods
Pass the `sort` option to `read_all`, `field_fetch`, or `search_table` to retrieve sorted datasets immediately:
```perl
# 1. Default Direction: Descending / Highest First (DESC: 99->0, Z->A)
my @products = $adb->read_all("catalog_product", sort => 10);
my @products = $adb->read_all("catalog_product", sort => { blk => 10 });
# 2. Reverse Direction: Ascending / Lowest First (ASC: 0->99, A->Z)
my @products = $adb->read_all("catalog_product", sort => -10);
my @products = $adb->read_all("catalog_product", sort => { blk => 10, reverse => 1 });
# 3. Primary Key (ID) Ascending Order:
my @products = $adb->read_all("catalog_product", sort => { reverse => 1 }); # 1..N oldest first
# 4. Sorting with field_fetch and search_table:
my @cat_items = $adb->field_fetch("catalog_product", 1, "electronics", sort => { blk => 10, reverse => 1 });
my ($count, @search) = $adb->search_table("catalog_product", "headphone", 0, 20, sort => -10);
```
---
## 7. Transaction Safety, ACID Guarantees, and Crash Recovery (Transactions)
`AmberDB::Transact` provides full **ACID-compliant transactions** and **Strict Two-Phase Locking (Strict 2PL)** concurrency control for multi-table updates (e.g., creating an order, updating inventory, and charging accounts).
### 7.1 Transactional Integrity & The Single Transaction Spine
In modern e-commerce and enterprise workflows, a single high-level user action (such as "Complete Checkout") triggers an interdependent semantic operation chain spanning multiple tables and sub-systems:
```text
Checkout Operation Chain:
├─ Order Confirmation (creating entry in orders table)
├─ Cart Cleared (purging items from cart table)
├─ Customer Account (balance deduction or card charge record)
├─ Company Account (revenue entry in general ledger)
├─ Stock Inventory (deducting counts in catalog_product table)
└─ Supplier Dispatch (writing work item to supplier_queue table)
```
These operations are **semantically coupled and mutually dependent**. If one operation fails while the others persist, the database falls into an inconsistent state:
- If the customer's payment is processed and the order record is created, but inventory deduction fails or crashes;
- Or if stock is deducted and the cart is emptied, but the revenue entry fails to record;
the system state becomes corrupted. To eliminate these anomalies, the entire sequence must be unified within a **single transaction spine (`transact_start` $\rightarrow$ `transact_end`)**. If any step encounters an error or if the process crashes, Am...
### 7.2 ACID Guarantees in AmberDB
AmberDB guarantees the four classical ACID properties through embedded flat-file database mechanics:
| ACID Property | Implementation Mechanism & Guarantees |
| :--- | :--- |
| **Atomicity** | **Disk-Backed Undo-Journaling:** When `transact_start()` is called, a microsecond-stamped `.txn` journal is created. Every `insert_id`, `modify_id`, and `delete_id` call appends reverse undo instructions. If a critical base error oc...
| **Consistency** | **Schema, Index, and State Integrity:** Inbound records are validated against schema field rules, data types, and byte limits. Primary keys (`autoid`), inverted word indexes, columnar facets, and URL slugs are synchronized in real...
| **Isolation** | **Strict Two-Phase Locking (Strict 2PL):** Every record modified within an active transaction acquires an exclusive OS-level lock (`flock LOCK_EX`). Locks are held throughout the entire transaction duration, preventing concurrent wo...
| **Durability** | **Synchronous Journaling & Crash Recovery (`transact_recover`):** All journal writes invoke `$fh->flush`. When configured with `cfg => { txn_sync => 1 }`, AmberDB triggers OS/kernel `fsync` (`$fh->sync`) and Berkeley DB cache flush...
> **Architectural Note: Batch ETL Imports vs. Business Transactions**
> Methods such as `insert_list`, `modify_list`, and `delete_list` are specialized for high-throughput batch imports (e.g., ingesting large XML/JSON product catalogs). Since list records are typically independent entities without cross-dependencies, d...
### 7.3 Transaction Workflow
1. **`transact_start()`**: Opens a microsecond-stamped undo journal (`.txn`) in `$dbase_dir/txn/` and recovers any orphaned transactions left by dead processes (`transact_recover`).
2. **CRUD Operations**: `insert_id`, `modify_id`, `delete_id` write updates to the base `.db` file, acquire record write locks (`flock`), and record reverse undo entries in the `.txn` journal.
3. **`transact_end()`**: Finalizes the transaction.
- If clean: Deletes journal, releases held locks, and commits changes (`status => "commit"`).
- If base errors occurred: Evaluates journal in reverse (LIFO) order, restoring base records and index states to their pre-transaction snapshot, releases locks (`status => "rollback"`).
4. **`transact_rollback()`**: Manually triggers immediate rollback based on business logic.
### 7.4 Practical Example: Checkout & Inventory Transaction
```perl
# 1. Start Transaction
$adb->transact_start();
my $product_id = 42;
my $quantity = 2;
my $user_id = 1001;
# Read product and check inventory
my @product = $adb->read_id("catalog_product", $product_id);
my $current_stock = $product[8]; # Block 8 = Stock count
if ($current_stock < $quantity) {
# Insufficient stock: report transaction error (transact_end will trigger automatic rollback)
$adb->transact_error("catalog_product", "Insufficient stock ($current_stock < $quantity)");
} else {
# Deduct stock and update product (@product[0] contains $product_id)
$product[8] -= $quantity;
$adb->modify_id("catalog_product", @product);
# Create order record
my @order = ( $user_id, $product_id, $quantity, time(), "confirmed" );
my $order_id = $adb->insert_id("orders", undef, @order);
}
# Finalize transaction (commits if clean, automatically rolls back on error)
my $res = $adb->transact_end();
if ($res->{status} eq "commit") {
print "Order placed successfully and stock deducted!\n";
} else {
warn "Transaction aborted! All changes were automatically rolled back.\n";
}
```
### 7.5 Durability and Crash Recovery
- **IO::Handle Buffer Flushing & Sync:** Every journal entry is immediately flushed with `$fh->flush`. When configured with `cfg => { txn_sync => 1 }`, AmberDB enforces physical OS/disk-level synchronization (`$fh->sync` / `fsync`).
- **`flock`-Based Ownership:** Active transactions hold an exclusive non-blocking lock (`LOCK_EX | LOCK_NB`) on their `.txn` file. If a process crashes unexpectedly, the lock is automatically released by the operating system.
- **Orphan Recovery (`transact_recover`):** If a worker process terminates abruptly, stale `.txn` files in `txn/` are scanned. By verifying that the file lock has dropped and the process is no longer active, the journal is safely rolled back to resto...
### 7.6 Core Architectural Philosophy: Authoritative Data vs. Rebuildable Indexes
AmberDB's storage and transaction architecture is organized around a strict hierarchy of data authority:
1. **Authoritative Master Files (Non-Reconstructible Source of Truth):**
- **`.db` (Master Document Data):** Primary storage for all active records and documents.
- **`.del` (Soft-Deleted Archive):** Preserves deleted records under `keep_deleted`. Once moved here, deleted data cannot be reconstructed from `.db`.
- **`.aut` (User Audit Trail):** Chronological, time-series history of who created, edited, or deleted records (`log_owner`). This historical data cannot be generated from any other source.
2. **Derived & Rebuildable Indexes (Disposable Secondary Projections):**
- **`.inx` (Record Index), `.fld` (Match), `.src` (Full-Text), `.srt` (Sort), `.fac` (Facet), `.slg` (URL Slug):** All these index files are deterministic projections derived directly from `.db`.
- If any secondary index is corrupted, deleted, or incomplete, running `AmberDB::Tools->set_index($table)` reconstructs all indexes from scratch within seconds with **zero data loss**.
> **Rationale Behind Transaction Design:** `AmberDB::Transact` was deliberately engineered around this principle. A failure writing to the authoritative `.db` file (`is_index == 0`) triggers an immediate automatic `rollback`. However, if the master d...
### 7.7 Exempting Auxiliary Tables from Failure Cascades (`no_transact`)
In multi-table business operations (e.g. creating an order, updating inventory, and charging accounts), some tables represent **core transactional entities** (orders, payments, inventory), while others serve as **auxiliary or secondary records** (cus...
AmberDB allows declaring tables with `no_transact => 1` (either in schema `.table` or dynamically at runtime) to **exempt them from transaction abort cascades**:
1. **Static Schema Definition (`.table` file):**
```perl
# order_customer_summary.table
{
name => "Customer Order Summary",
no_transact => 1, # Failures here do NOT abort the main transaction
schema => [qw(user_id order_id amount created_at)],
}
```
2. **Dynamic Runtime Configuration (`table_attr`):**
```perl
# Temporarily exempt an auxiliary table during a specific workflow:
$adb->table_attr("order_customer_summary", no_transact => 1);
```
> **How It Works:**
> - If an error occurs on a table marked `no_transact => 1`, the error is treated as non-critical (like index errors), and `transact_end` proceeds to `commit`.
> - However, if a primary operation fails and triggers a `rollback`, all changes on `no_transact` tables are **still safely reverted in LIFO order via the `.txn` journal** to ensure complete database consistency without ghost records.
### 7.8 Multi-Process Concurrency, Lock Isolation, and Stress Verification
AmberDB is engineered for high-concurrency production deployments (Apache, Plack/PSGI, Starman, FastCGI, Starlet) and background worker pools (cron jobs, async queues) where **dozens of independent processes simultaneously read and write to the same ...
#### Operating System-Level Lock & Platform Isolation:
1. **Linux / POSIX Environments:** Leveraging POSIX `fork()` and kernel-level `flock(LOCK_EX)` / `flock(LOCK_SH)` locks, every worker process operates within an isolated memory address space. Strict Two-Phase Locking (Strict 2PL) guarantees serializa...
2. **Windows / MSYS2 Environments:** On Windows NT architectures, full OS-level lock integrity and file descriptor isolation are maintained across independent worker processes (`perl.exe`).
3. **Record-Level Locking (`flock_open` / `flock_close`):** For critical concurrent updates on individual records (such as high-demand stock decrements or shared counters), `$adb->flock_open($table, "write", $id)` eliminates race conditions and lost ...
#### Concurrency & Stress Test Suite (`xt/amberdb_concurrency_stress.t`):
The database engine's resilience under extreme parallel load is verified by the author/release stress test suite:
```bash
# Run multi-process concurrency stress tests directly:
perl -Ilib xt/amberdb_concurrency_stress.t
```
This test suite validates 5 mission-critical concurrency scenarios:
- **1. Parallel Writers:** Multi-worker concurrent inserts verifying zero ID collisions, exact table counts, and synchronized secondary index compilation (`.inx`, `.fld`, `.src`, `.fac`, `.srt`, `.slg`).
- **2. Interleaved Reads & Writes:** Concurrent reader processes executing streaming scans and index queries while writers continuously insert new data without deadlocks or corruption.
- **3. Concurrent Transactions & Crash Recovery:** Simulated sudden process termination mid-transaction, verifying that orphaned `.txn` journals are safely rolled back by `transact_recover` without interfering with active concurrent transactions.
- **4. Concurrent URL Slug Collisions:** Dozens of processes simultaneously inserting identical product titles, confirming deterministic `-1`, `-2` suffix generation and 100% bidirectional bijection (`_0.slg` $\leftrightarrow$ `_1.slg`).
- **5. High-Concurrency Inventory Decrements:** Multiple workers decrementing stock on the same product record under `flock_open` write locks, verifying atomic final inventory consistency.
---
## 8. High-Throughput Batch Operations (Batch ETL & Ingestion)
AmberDB provides a dedicated **2-Phase Batch Pipeline** for ingesting and updating large volumes of records (ETL from CSV, JSON, XML, or REST APIs) at maximum throughput.
### 8.1 Why Use `insert_list` Instead of `insert_id` in a Loop?
Executing `insert_id` in a loop forces the operating system to perform $N$ independent file opens (`open/tie`), lock acquisitions (`flock`), auto-increment sequence mutations, and secondary index writes (`.inx`, `.src`, `.fld`, `.fac`, `.srt`). For $...
`insert_list` splits the ingestion workflow into 2 unified phases, reducing I/O complexity to $O(K)$:
1. **Phase 1 (Single I/O Master Table Write):** The `.db` Berkeley DB file is opened exactly **once** (`table_write`). Auto-increment IDs are allocated contiguously (`table_autoid`), field formatters and schema rules are evaluated, and all records ar...
2. **Phase 2 (Batched Secondary Index Merge):** Each secondary index file (`.inx`, `.src`, `.fld`, `.fac`, `.srt`, and junk tier) is opened exactly **once** and the entire batch is compiled into binary bitsets and B-tree branches via unified merges (...
> [!TIP]
> On a batch of 10,000 records, `insert_list` finishes **50x to 100x faster** than a standard `insert_id` loop.
### 8.2 Batch Insert (`insert_list`)
```perl
# Array of record column tuples.
# Pass 0 or undef for ID to automatically allocate 64-bit auto-increment IDs.
my @new_products = (
[ 0, "5", "3", "Wireless Headphones", "149.90", "2026-08-28", "1" ],
[ 0, "5,12", "8", "Mechanical Keyboard", "299.00", "2026-08-28", "1" ],
[ 0, "12", "3", "Gaming Mouse", "89.50", "2026-08-28", "1" ],
# ... hundreds or thousands of records ...
);
my $status = $adb->insert_list("catalog_product", @new_products);
# Returns hashref of created IDs: { 101 => 1, 102 => 1, 103 => 1, ... }
```
### 8.3 Bulk Modify (`modify_list`)
```perl
my @updates = (
[ 101, "5", "3", "Wireless Headphones Pro", "179.90", "2026-08-28", "1" ],
[ 102, "5,12", "8", "Mechanical Keyboard RGB", "329.00", "2026-08-28", "1" ],
);
my $status = $adb->modify_list("catalog_product", @updates);
```
### 8.4 Bulk Delete (`delete_list`)
```perl
# Target IDs can be passed as a flat list or array reference
my $status = $adb->delete_list("catalog_product", 101, 102, 103);
# or:
# $adb->delete_list("catalog_product", [101, 102, 103]);
```
### 8.5 Chunking Strategy for Large Datasets (ETL Ingestion)
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
---
### 9.6 Schema Configuration Parameters Reference (Table Level)
The following reference table details all top-level parameters supported in `.table` schema definitions, along with default values and legacy alias equivalents:
| Parameter | Type | Default | Legacy / Alias | Description |
| :--- | :--- | :--- | :--- | :--- |
| `name` | `string` | `"Table"` | — | Human-readable table title. |
| `id_type` | `string` | `"num"` | — | Primary key format: `"num"` (64-bit unsigned int) or `"ascii"` (max 8-byte alphanumeric string). |
| `record_index` | `0 / 1` | `0` | `readall` | When `1`, enables the `.inx` primary binary index, `table_count`, `table_lastid`, and auto-increment. |
| `search_block` | `ARRAY` | `[]` | — | Block numbers indexed in `.src` for full-text inverted search. |
| `match_block` | `ARRAY` | `[]` | `fields` | Block numbers indexed in `.fld` for exact field-to-ID matching and relational lookup. |
| `sort_block` | `ARRAY` | `[]` | — | Pre-computed `.srt` binary sort indexes (`[ 4, { blk => 10, type => 'num' } ]`). |
| `facet_block` | `ARRAY` | `[]` | `filter_block` | Block numbers indexed in `.fac` for columnar faceted category navigation. |
| `slug_block` | `ARRAY` | `[]` | `rwlink` | Block numbers combined for automated bidirectional `.slg` URL slug generation (e.g. `[2, 4]`). |
| `use_facet` | `0 / 1` | `0` | — | Enables the facet counting engine and `field_fltkeys` / `facet_menu` on the table. |
| `facet_rules` | `ARRAY` | `[]` | — | Scoping rules for facet counting (e.g., displaying only in-stock items in filter menus). |
| `use_junk` | `0 / 1` | `0` | — | Enables dual-tier indexing by segregating inactive/out-of-stock records to Cold Tier B. |
| `junk_rules` | `ARRAY` | `[]` | — | Business rules determining automatic routing of records between active and junk tiers. |
| `use_cache` | `0 / 1 / 2` | `0` | `usecache` | `0`: Disabled, `1`: Soft (.inx metadata), `2`: Hard (Full shared RAM-Disk mirror). |
| `cache_ttl` | `integer` | `3600` | — | Table-specific RAM cache time-to-live in seconds. |
| `keep_deleted` | `0 / 1` | `0` | `nodelete` | Preserves deleted records in `.del` soft-delete archive instead of permanent deletion. |
| `log_owner` | `0 / 1` | `0` | `authority` | Records user modification audit trails in `.aut` files. |
| `use_alias` | `0 / 1` | `0` | `uselnk` | Enables `.lnk` alias routing table for merged records or legacy URL redirections. |
| `use_counter` | `0 / 1` | `0` | `usecnt` | Enables automated hit/view read counters in `.cnt` files. |
| `parent_table` | `string` | `""` | — | Parent table name for vertical partitioning (child table shares the same primary ID). |
| `force` | `0 / 1` | `0` | — | When `1`, `insert_id` overwrites existing records rather than failing (Replace mode). |
| `min_char` | `integer` | `2` | `minchar` | Minimum word length for full-text search indexing (1, 2, or 3). |
| `stop_word` | `string` | `""` | `nextkey` | Stop-words excluded from full-text search indexing (e.g., `"the and for with"`). |
| `repeat_ids` | `integer` | `undef` | — | Target block number where extracted child item IDs are consolidated. |
| `repeat_start` | `integer` | `undef` | — | Starting block index for dynamic repeating child rows (order items, cart lines). |
| `view_block` | `ARRAY` | `[]` | — | Priority block numbers displayed in UI / CMS listing views. |
| `use_menu` | `0 / 1` | `1` | — | Controls display of the table in admin panel navigation menus. |
| `no_transact` | `0 / 1` | `0` | — | Exempts table from transactional rollback error propagation. |
| `no_backup` | `0 / 1` | `0` | — | Disables daily CSV user audit logging for this table. |
---
### 9.7 Block (Field) Definitions, 8 Core Field Types, UI Inputs, and Validation Reference
Each block definition inside the `blocks` array supports the following attributes:
#### 9.7.1 Core Block Attributes
| Attribute | Type | Description | Example |
| :--- | :--- | :--- | :--- |
| `id` | `string` | Programmatic field identifier | `id => "email"` |
| `name` | `string` | Display label for UI forms and table headers | `name => "Email Address"` |
| `type` | `string` | Data storage, type validation, and indexing type | `type => "text"` |
| `input` | `string` | HTML/UI Form input component type | `input => "select"` |
| `valid` | `string` | Automated data validation rule | `valid => "not_null;email"` |
| `option` | `string` | Enumerated choice options (`value:label` pairs) | `option => "1:Active,0:Inactive"` |
| `rdbm` | `string / HASH`| Foreign table lookup mapping (`foreign_table;display_block`) | `rdbm => "catalog_category;2"` |
| `extend` | `HASH` | 1:1 vertical table extension | `extend => { table => "catalog_price", join => "id" }` |
#### 9.7.2 Supported 8 Core Field Types (`type`)
AmberDB uses **8 unified core storage types** across serialization (`db_encode`/`db_decode`), indexing, and sorting layers:
| Field Type (`type`) | Description | `enc_validate` (Write Phase) | `dec_validate` (Read Phase) | Indexing & Sorting Behavior |
| :--- | :--- | :--- | :--- | :--- |
| **`auto_id`** | Auto-increment ID (Block 0) | Primary key format validation | ID scalar return | Primary key index (`.inx`) |
| **`text`** | Standard UTF-8 Text | UTF-8 string validation | String scalar (`$val // ''`) | Inverted index (`.src`), dictionary (`.str`) |
| **`num`** / **`number`** | Numeric (Integer / Float / Boolean) | Numeric validation (`^[+-]?[0-9]+(?:\.[0-9]+)?$`), defaults empty to `0` | Numeric scalar cast (`0 + $val`) | Numerical sorting (`<=>`) in `.srt`, `.fld` filters |
| **`ascii`** | ASCII-Only Text | ASCII normalization via `to_ascii` | Clean ASCII text | URL slug map (`.slg`), ASCII `.srt` sorting |
| **`date`** | Date and Time | Assigns system date if `auto_date` is active | Date string | Chronological sort in `.srt` via `str2dateid` |
| **`array`** / **`repeat`** | List / Repeating Rows | ARRAY ref or `[split /,/]` | Perl `ARRAY` ref (`[]`) | Multi-value matching (`field_fetch`) |
| **`hash`** | Dictionary / Object (HASH ref) | HASH ref validation | Perl `HASH` ref (`{}`) | Schemaless nested key-value store |
| **`binary`** | Binary Payload / Base64 | Raw bytes or Base64 string | Raw binary scalar | Direct flat file storage |
> [!NOTE]
> **Numeric and Boolean Management:** The `num` (or `number`) type handles positive (`150`, `+25`), negative (`-50`, `-12.75`), floating-point values, and `0 / 1` boolean flags. Unchecked HTML checkboxes or empty numerical inputs are automatically no...
#### 9.7.3 UI Form Input Components (`input`)
Determines how the field is rendered in UI forms and administration panels:
| Component (`input`) | UI Element | Description |
| :--- | :--- | :--- |
| `text` | Text Input | Standard single-line text field `<input type="text">`. |
| `textarea` | Textarea | Multi-line plain text box `<textarea>`. |
| `summernote` | Summernote | Rich WYSIWYG HTML visual editor for articles/descriptions. |
| `select` | Dropdown Select | Single-selection dropdown list `<select>`. |
| `checkbox` | Checkbox | Multi-selection checkboxes `<input type="checkbox">` (Use `type => "num"` for Boolean). |
| `radio` | Radio Buttons | Single-selection radio options `<input type="radio">`. |
| `file` | File Upload | Attachment or image file uploader `<input type="file">`. |
| `hidden` | Hidden Field | Hidden form element `<input type="hidden">` (for primary IDs). |
| `email` | Email Input | HTML5 email input field `<input type="email">`. |
| `ascii` | ASCII Field | User/code input box constrained to ASCII charset. |
| `number` | Number Input | Numeric stepper `<input type="number">`. |
| `date` | Date Picker | Interactive date calendar selector `<input type="date">`. |
| `password` | Password Field | Obscured security input `<input type="password">`. |
| `repeat` / `repeats` | Repeater Table | Dynamic sub-row table input with add/remove row buttons (Order items, invoice lines). |
| `search_block` | Search Box | Search-assisted dynamic filter input. |
| `selectbyfind` | SelectByFind | Foreign relation selector populated via dynamic search. |
| `selectbylist` | SelectByList | Multi-item picker component from list. |
#### 9.7.4 Repeating Child Row Blocks (`repeat_start` and `repeat_ids`)
AmberDB natively supports dynamic repeating child rows (e.g. order line items, invoice product rows) horizontally across the flat parent record without relational child tables or `JOIN` operations:
- **Horizontal Row Slicing (`@record[15..$#record]`):** Repeating items, each field index beyond fixed blocks (`$record[15]`, `$record[16]`, `$record[17]`, ...) holds an individual repeating record item (e.g. `[ 101, 'Book', 2, '150.00' ]`).
- **`repeat_start`**: Specifies the starting block index where dynamic repeating rows begin (e.g. `repeat_start => 15`). In the schema, block 15 acts as the prototype template for all succeeding indices.
- **`repeat_ids`**: The engine (`repeat_fields`) scans `@record[15..$#record]`, extracts the first element (numeric item ID) of each repeating row, joins them with commas (`"101,102,103"`), and stores the string in `repeat_ids` (e.g. block 12). Inclu...
```perl
# Example Schema Definition (Order Table):
repeat_ids => 12, # Aggregated item IDs block (e.g. "101,102,103")
repeat_start => 15, # Repeating child rows start at block 15
blocks => [
{ id => "id", name => "Order ID", type => "auto_id", input => "hidden" }, # 0
# ... fixed header fields (date, customer, address) ...
{ id => "prod_ids", name => "Product IDs", type => "text", input => "hidden" }, # 12 (repeat_ids target)
# ...
{ id => "products", name => "Order Items", type => "repeat", input => "repeats" }, # 15 (repeat_start template)
];
# In-Memory Record Layout:
# $record[0] = 1001; # Order Primary ID (Numeric primary key)
# $record[12] = "101,102,103"; # Auto-populated by engine via repeat_fields
# $record[15] = [ 101, 'Book', 2, '150.00' ]; # 1st Product Row
# $record[16] = [ 102, 'Pad', 1, '85.00' ]; # 2nd Product Row
# $record[17] = [ 103, 'Pen', 5, '20.00' ]; # 3rd Product Row
```
#### 9.7.5 Automated Validation Rules (`valid`)
Multiple validation rules can be chained using semicolon (`;`) (e.g. `valid => "not_null;email"`):
| Rule (`valid`) | Description | Validation Check |
| :--- | :--- | :--- |
| `none` | No Validation | Field accepts any input without validation (default). |
| `not_null` | Required | Field cannot be null, undefined, or empty string. |
| `unique` | Unique Value | Asserts that no other record in the table contains this value. |
| `email` | Email Format | Validates RFC-compliant email pattern. |
| `telefon` | Phone Number | Validates national/international phone format. |
| `ascii` | ASCII Only | Restricts character set strictly to ASCII [0-127]. |
| `numeric` | Numeric Only | Enforces that value is a valid numeric scalar. |
| `regex` | Regular Expression | Tests against custom regex pattern rule. |
| `auto_num` | Auto Number | Automatically assigns an incrementing numerical sequence. |
| `auto_pass` | Auto Password | Generates random secure password and stores salted hash. |
| `auto_date` | Auto Date | Automatically populates with current system timestamp. |
| `auto_str` | Template String | Pre-populates predefined template text. |
#### 9.7.6 Unique Constraints & Bidirectional String/ID Dictionary (`.unq`)
AmberDB uses `.unq` (Unique & Dictionary) index files (`${table}_${block}.unq`) to manage both **uniqueness validation** and **relational string $\leftrightarrow$ numeric ID translation** with $O(1)$ disk lookup speed:
1. **Extension Clarity:** Renamed from legacy `.str` to `.unq` to eliminate any visual ambiguity with `.srt` (Sort indexes).
2. **$O(1)$ Duplicate Enforcement (`valid => "unique"`):**
- When `valid => "unique"` is specified (e.g. `username`, `email`, `barcode`), `insert_id` and `modify_id` perform an instantaneous $O(1)$ check on `s:$value` in `${table}_${blk}.unq`.
- If another record holds this value, the transaction is rejected with a unique constraint error.
- Successfully written records store bidirectional mappings (`s:$value => $rid` and `n:$rid => $value`), which are automatically cleaned up when records are deleted.
3. **RDBM & `match_block` String-to-ID Auto-Resolution:**
- When a string name is passed to a relational field (e.g. `"Can Publishing"` for `rdbm => "catalog_brand;1"`), AmberDB queries `s:Can Publishing` in `catalog_brand_1.unq`.
- If present, it resolves to the existing numeric ID; if absent in `write` mode, it auto-registers the entry in `.unq` and the foreign table with an incremented ID.
- The inverted filter index (`.fld`) always stores **pure numeric IDs**, ensuring lightweight index storage and fast integer comparisons.
---
### 9.8 Schema-Driven Type Validation & Casting (`enc_validate` & `dec_validate`)
AmberDB enforces two-way data integrity between Perl runtime types and database storage:
1. **Write Phase Validation (`enc_validate`):**
- Invoked in `insert_id`, `modify_id`, `insert_list`, and `modify_list` right before records are written to disk and secondary indexes.
- Cleans numeric fields, trims whitespace, and converts empty/invalid inputs to `0`.
- Normalizes non-ASCII characters for `ascii` fields using `to_ascii`.
- Fills empty `valid => "auto_date"` fields with the current ISO date.
- Converts comma-separated strings into Perl `ARRAY` refs for `array` fields and enforces `HASH` refs for `hash` fields.
2. **Read Phase Casting (`dec_validate`):**
- Invoked in `read_id`, `read_list`, and `read_all` immediately after `db_decode`.
- Casts numeric fields to numeric scalars (`0 + $val`), eliminating uninitialized value warnings in mathematical expressions.
- Guarantees `array` fields return `[]` and `hash` fields return `{}` even when empty.
3. **Simple Mode Performance:**
- In Schemaless (`simple => 1`) mode or for tables without block definitions, `enc_validate` and `dec_validate` return input data immediately with zero CPU overhead.
### 9.8.1 How Schemas Coordinate with CRUD Operations
When a record is added or modified via `insert_id` or `modify_id`, the passed array elements map directly to block indices:
```perl
# Block Mapping:
# Block 0 : ID (PrimaryKey - auto-generated by the engine or passed as 0)
# Block 1 : @record[0] -> Category ID ("5")
# Block 2 : @record[1] -> Brand ID ("12")
# Block 3 : @record[2] -> Author ID ("")
# Block 4 : @record[3] -> Title ("Wireless Headphones")
# Block 5 : @record[4] -> Subtitle ("Active Noise Cancelling")
# Block 6 : @record[5] -> Supplier ("Sony")
# Block 7 : @record[6] -> Description ("<p>Detailed product description...</p>")
# Block 8 : @record[7] -> Stock ("150")
# Block 9 : @record[8] -> Barcode ("8690123456789")
# Block 10: @record[9] -> Price ("2499.90")
# Block 11: @record[10]-> Status ("1")
my @product = (
"5", "12", "", "Wireless Headphones", "Active Noise Cancelling",
"Sony", "<p>Detailed product description...</p>", 150, "8690123456789", 2499.90, "1"
);
my $new_id = $adb->insert_id("catalog_product", 0, @product);
```
In a single atomic pass, the engine consults the schema and:
1. Validates and normalizes field types via `enc_validate`.
2. Writes the raw record to `catalog_product.db`.
3. Updates `catalog_product.inx` primary index (since `record_index => 1`).
4. Indexes Category (5), Brand (12), and Status (1) in `catalog_product_*.fld` match indexes (since `match_block => [1, 2, 3, 11]`).
5. Extracts, tokenizes, normalizes, and indexes Title, Subtitle, Description, and Barcode in `catalog_product_*.src` inverted search indexes.
6. Generates the URL slug `sony-wireless-headphones` into `catalog_product.slg` (since `slug_block => [2, 4]`).
---
### 9.9 Dynamic Runtime Schema Manipulation (`table_attr`)
AmberDB schemas are mutable at runtime without database recreation or migrations:
```perl
# Scenario 1: Narrow full-text search scope dynamically for barcode POS scanners
$adb->table_attr("catalog_product", { search_block => [ 4, 9 ] });
# Scenario 2: Include soft-deleted records or enable audit logging dynamically
$adb->table_attr("catalog_product", { keep_deleted => 1 });
# Scenario 3: Temporarily disable cache during heavy batch ETL or reporting
$adb->table_attr("catalog_product", { use_cache => 0 });
```
---
### 9.10 Dynamic Expanding Tables and Repeating Blocks (`repeat_ids` & `repeat_start`)
AmberDB breaks free from fixed column width constraints by allowing a variable number of child items (e.g. order line items, cart items, invoice rows) to be appended dynamically at the end of a single parent document record. This feature eliminates c...
#### 9.10.1 Schema Configuration (`order_active.table` Example)
```perl
# dbstore/schema/order_active.table
{
name => "Active Orders",
record_index => 1,
match_block => [ 1, 2, 12, 14 ], # 12: Product Loop (repeat_ids) is automatically indexed
keep_deleted => 1,
log_owner => 1,
repeat_ids => 12, # Block where extracted child IDs are consolidated
repeat_start => 15, # Starting index where variable child blocks begin
blocks => [
{ id => "id", name => "ID", type => "auto_id" }, # 0
{ id => "member_id", name => "Member ID", type => "text" }, # 1
{ id => "invoice_no", name => "Invoice No", type => "text" }, # 2
{ id => "amounts", name => "Amounts", type => "array" }, # 3
{ id => "timestamps", name => "Timestamps", type => "array" }, # 4
{ id => "status", name => "Status", type => "option" }, # 5
{ id => "session_id", name => "Session ID", type => "text" }, # 6
{ id => "delivery_address", name => "Delivery Address", type => "array" }, # 7
{ id => "invoice_address", name => "Invoice Address", type => "array" }, # 8
{ id => "cargo", name => "Shipping Info", type => "array" }, # 9
{ id => "payment_info", name => "Payment Method", type => "array" }, # 10
{ id => "credit_card_info", name => "Card Info", type => "array" }, # 11
{ id => "product_ids", name => "Product Loop", type => "text" }, # 12 (repeat_ids)
{ id => "member_notes", name => "Customer Notes", type => "array" }, # 13
{ id => "gift_products", name => "Gift Products", type => "text" }, # 14
{ id => "products", name => "Order Items", type => "repeat" }, # 15 (repeat_start)
]
}
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
### 11.4 Automatic State Migration
When updating a product, AmberDB evaluates the schema rules in real time:
* Setting `sales_status` to `0` or disabling a vendor automatically **demotes the product from storefront to archive**.
* Restocking the item and setting status back to `1` automatically **restores the product to the active storefront**.
* Zero manual data maintenance or migration scripts required.
---
## 12. Automated URL Slug Management
When `slug_block => [2, 4]` is configured (Brand + Title), AmberDB generates and manages clean URL slugs automatically:
```perl
# Retrieve URL Slug by Record ID
my $slug_map = $adb->get_slug("catalog_product", 0, 5001);
my $slug = $slug_map->{5001};
print "URL: /product/$slug\n"; # Output: /product/acme-wireless-headphones
# Resolve Record ID from URL Slug (Router lookup)
my $id_map = $adb->get_slug("catalog_product", 1, "acme-wireless-headphones");
my $id = $id_map->{"acme-wireless-headphones"};
print "Resolved Product ID: $id\n";
```
### 12.1 Automatic Slug Collision Resolution (Numeric Suffixes)
When multiple records generate identical base slugs (e.g. two distinct products named "Wireless Headphones"), AmberDB automatically appends deterministic incrementing numeric suffixes (`_2`, `_3`) to ensure strict uniqueness:
* 1st Record: `wireless-headphones`
* 2nd Record: `wireless-headphones_2`
* 3rd Record: `wireless-headphones_3`
---
## 13. Unified Shared RAM Cache (.db / .inx) & Persistent Buffer
`AmberDB::Cache` provides a unified shared RAM cache mirroring AmberDB's native `.db` and `.inx` formats:
```text
┌────────────────────────────────────────────────â”
│ dbstore/cache/ (tmpfs RAM-Disk) │
├──────────────────────┬─────────────────────────┤
│ cache/${table}.db │ cache/${table}.inx │
│ (Records) │ (lastid, keys, meta...) │
└──────────────────────┴─────────────────────────┘
```
### Cache Levels (`use_cache`)
* **`0` (Disabled):** No caching.
* **`1` (Soft Cache):** Caches `lastid`, `keys`, and `count` metadata in `cache/${table}.inx`, and supports manual `$adb->cache_write` / `$adb->cache_read`.
* **`2` (Hard Cache - Full Table RAM Mirror):** Table records are cached in `cache/${table}.db` and `cache/${table}.inx` in RAM. Reads (`read_id`, `read_list`) are served directly from RAM.
```perl
# 1. Manual Cache Write (stores in cache/${table}.inx)
$adb->cache_write("catalog_product", "featured_items", @featured_list);
# 2. Cache Read
my @featured = $adb->cache_read("catalog_product", "featured_items");
# 3. Hard Cache Table Preload
$adb->cache_preload("catalog_category");
# 4. Invalidate Cache (Automatically purged on modify / delete_id)
$adb->cache_delete("catalog_product", "featured_items"); # Single key
$adb->cache_delete("catalog_product"); # Entire table cache (.db and .inx)
# 5. Inspect RAM-Disk Diagnostics & Mount Status
my $cache_diag = $adb->cache_setup();
# Returns hashref: { is_mounted => 1, mount_desc => "...", cache_dir => "...", cache_size => "512M" }
```
### Cache TTL (`cache_ttl`) & Runtime Overrides
The `cache_ttl` expiration time is defined per-table directly inside its schema (e.g. `cache_ttl => 1800`). Ephemeral data structures like session tokens or process locks can have their expiration configured in the schema or dynamically tuned at runt...
```perl
# Dynamically configure session table cache TTL to 30 minutes (1800 seconds)
$adb->table_attr("session", { use_cache => 1, cache_ttl => 1800 });
```
### Temporary Disk Buffer
For large reporting queries or intermediate batch jobs:
```perl
$adb->buffer_write("temp_report", @large_data);
my @data = $adb->buffer_read("temp_report");
$adb->buffer_delete("temp_report");
```
---
## 14. Configuration and Deterministic Flag Management (`config`)
Runtime behavior can be tuned and safely configured via the `$adb->config()` method:
```perl
# Bulk or single configuration assignment (Recommended)
$adb->config(
no_write => 1, # Read-only maintenance mode: block all writes
no_backup => 1, # Disable daily CSV audit logging for all tables
simple => 1, # Direct unindexed mode: bypasses secondary index generation
keys_only => 1, # read_all returns IDs only
cache_size => '1024M', # RAM-Disk / tmpfs cache size (Default: 512M)
);
# Single scalar getter:
my $no_write = $adb->config('no_write');
# Bulk getter (returns a safe shallow copy):
my $cfg = $adb->config();
```
---
## 15. Data Structures, Low-Level Table and Stream Operations
Beneath the standard CRUD layer, AmberDB provides direct access to optimized `DB_File` C-level primitives and raw streaming methods:
### 15.1 Data Structures and Serialization (`db_encode`, `db_decode`)
AmberDB encodes and decodes complex nested Perl structures:
```perl
# Encode: Native Perl Data → String
my $encoded = $adb->db_encode("Text", [ 1, 2, 3 ], { key => "val" });
( run in 1.723 second using v1.01-cache-2.11-cpan-d01c6094234 )