AmberDB
view release on metacpan or search on metacpan
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
[ðŸ Home](index.html) • [📖 About](EN.About_AmberDB.html) • [🚀 Quick Start](index.html#-quick-start) • [📘 Tutorial](EN.AmberDB_User-Guide.html) • [🌠Locale](EN.AmberDB-Locale_User-G...
---
# Developer Guide and Comprehensive Documentation
> **Version:** 5.23.1 · **Initial Design:** 2005 · **Last Updated:** 2026
> **Namespace:** `AmberDB`
> **Built-in Modules:** `Base`, `Index`, `Transact`, `Cache`, `Array`, `String`, `Date`, `Locale`, `Tools`
---
## Table of Contents
1. [What is AmberDB?](#1-what-is-amberdb)
2. [Quick Start](#2-quick-start)
3. [CRUD Operations (Core Data Management)](#3-crud-operations-core-data-management)
4. [Reading, Filtering, and Sorting](#4-reading-filtering-and-sorting)
5. [Simple Mode and Direct Schemaless Access (Simple Mode)](#5-simple-mode-and-direct-schemaless-access-simple-mode)
6. [Indexing and Search Engine](#6-indexing-and-search-engine)
7. [Transaction Safety, ACID Guarantees, and Crash Recovery (Transactions)](#7-transaction-safety-acid-guarantees-and-crash-recovery-transactions)
8. [High-Throughput Batch Operations (Batch ETL & Ingestion)](#8-high-throughput-batch-operations-batch-etl--ingestion)
9. [Schema Configuration (.table & In-Memory)](#9-schema-configuration-table--in-memory)
10. [Database Group Structure (.dbase)](#10-database-group-structure-dbase)
11. [Smart Tiered (Hot / Cold Junk) Indexing](#11-smart-tiered-hot--cold-junk-indexing)
12. [Automated URL Slug Management](#12-automated-url-slug-management)
13. [Unified Shared RAM Cache (.db / .inx) & Persistent Buffer](#13-unified-shared-ram-cache-db--inx--persistent-buffer)
14. [Configuration and Deterministic Flag Management (`config`)](#14-configuration-and-deterministic-flag-management-config)
15. [Data Structures, Low-Level Table and Stream Operations](#15-data-structures-low-level-table-and-stream-operations)
16. [Faceted Search & Category Filters (Facet Engine)](#16-faceted-search--category-filters-facet-engine)
17. [User Audit Trail and Backup](#17-user-audit-trail-and-backup)
18. [Maintenance and Repair Tools (AmberDB::Tools)](#18-maintenance-and-repair-tools-amberdbtools)
19. [File Extensions Map](#19-file-extensions-map)
20. [Directory Structure](#20-directory-structure)
21. [Developer Best Practices and Recommendations](#21-developer-best-practices-and-recommendations)
22. [Full Working Example (Checkout & Stock Transaction Scenario)](#22-full-working-example-checkout--stock-transaction-scenario)
23. [Method Quick Reference Table](#23-method-quick-reference-table)
24. [Why Use AmberDB? (Comparison with SQL and SQLite)](#24-why-use-amberdb-comparison-with-sql-and-sqlite)
25. [Boundaries and Debated Topics (Physical Constraints vs. Conscious Architectural Choices)](#25-boundaries-and-debated-topics-physical-constraints-vs-conscious-architectural-choices)
---
## 1. What is AmberDB?
`AmberDB` is a **high-performance, schema-driven NoSQL database engine for Perl**, featuring **precomputed inverted indexing, ACID-compliant transactions with Strict Two-Phase Locking (Strict 2PL), and automatic crash recovery on top of Berkeley DB (...
From a developer's perspective, AmberDB eliminates the overhead of provisioning and maintaining external database servers. A single CRUD call automatically updates and synchronizes all associated full-text search, field-match, facet filter, binary so...
### Built-in Modular Architecture
AmberDB is self-contained and does not rely on heavy external dependencies:
```text
┌─────────────────────────────────────────────────────────────────────────â”
│ AmberDB │
├─────────────────────────────────────────────────────────────────────────┤
│ AmberDB::Base → Schema parsing, paths, data serialization │
│ AmberDB::Index → Binary indexes (.inx, .fld, .src, .fac, .srt) │
│ AmberDB::Transact → Undo-log transactions, rollback & recovery │
│ AmberDB::Cache → Native RAM-Disk (tmpfs) Shared Cache & TTL │
│ AmberDB::Array → High-speed array utilities (nodup, crop) │
│ AmberDB::String → String utilities, HTML formatting & cleaning │
│ AmberDB::Date → Date calculations, timestamps, formatting │
│ AmberDB::Locale → Built-in multilingual collation & word search │
├─────────────────────────────────────────────────────────────────────────┤
│ AmberDB::Tools → Standalone reindexing, vacuum & repair tools │
└─────────────────────────────────────────────────────────────────────────┘
```
> **Note:** Collation-aware multilingual sorting and searching are powered by the integrated `AmberDB::Locale` module and require no external services or third-party packages.
---
## 2. Quick Start
### 2.1 Instantiating the Database Object
```perl
use AmberDB;
my $adb = AmberDB->new(
cfg => {
language => "en", # Built-in Locale language ("en", "tr", "de" etc.)
},
path => {
dbase_dir => "./dbstore", # Database root directory
},
);
```
> [!TIP]
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
# 4. Read View / Hit Counter from .cnt File
my $views = $adb->read_count("catalog_product", 5001);
print "Product 5001 viewed $views times.\n";
```
---
## 5. Simple Mode and Direct Schemaless Access (Simple Mode)
In AmberDB, **Simple Mode (`simple => 1`)** represents the entirely schemaless, lightweight, direct flat-file NoSQL operational mode where no `.table` or `.dbase` schema files and no secondary binary indexes (`.inx`, `.src`, `.fld`, `.fac`, `.srt`, `...
In Simple Mode, records can store rich, nested data structures directly, including array and hash references (`ARRAY`/`HASH`). The index generation and maintenance overhead is completely eliminated; single-key read and write operations (`read_id`, `i...
---
### 5.1 Initializing and Activating Simple Mode
Simple Mode can be activated in four distinct ways:
1. **Constructor Initialization via `cfg`:**
```perl
my $adb = AmberDB->new(
path => { dbase_dir => "/var/data/sessions" },
cfg => { simple => 1 },
);
```
2. **Quick Shortcut Helper via `AmberDB::Tools` (`db_simple`):**
```perl
use AmberDB::Tools;
my $tools = AmberDB::Tools->new();
my $adb = $tools->db_simple("/var/data/sessions");
```
3. **Dynamic Runtime Switch via `config`:**
```perl
$adb->config( simple => 1 );
```
4. **Automatic Simple Mode Trigger via Custom Extensions (`db_ext`):**
AmberDB defaults to `.db`. If `db_ext` is configured with any extension other than `"db"` (e.g. `"dat"`, `"cache"`, `"session"`), the engine **automatically switches into Simple Mode**:
```perl
my $adb = AmberDB->new(
path => { dbase_dir => "/var/data/cache" },
cfg => { db_ext => "dat" }, # Automatically activates simple => 1
);
```
> **Directory Layout Note:** In standard mode, tables reside under `$dbase_dir/tables/`. In Simple Mode, the engine creates and reads database files directly inside the root of `dbase_dir` (`$dbase_dir/<table_name>.<ext>`). To open existing standard-...
---
### 5.2 Flexible & Arbitrary Record IDs (No 8-Byte Limit)
The standard mode **8-byte limit** and **strict ASCII/numeric format constraints** are relaxed in Simple Mode (`id_check` accepts arbitrary scalar keys and applies safe key sanitization):
- **Emails and Special Characters:** `user@example.com`, `api:v1:user:1005`
- **Long Tokens and UUIDs:** `sess_99999_abcdef_1234567890_extra_long_token` (up to 255 bytes)
- **Hyphenated Codes and Prefixes:** `TR-2026-08-31-INVOICE-001`
- **Unicode / Multilingual Keys:** `prod_özellik_kırmızı_xl`
- **Safe Key Sanitization:** Automatically trims leading/trailing whitespace (`trim_space`); strictly rejects NUL bytes (`\0`), control characters (`\r`, `\n`, `\t`), and references (ARRAY/HASH refs) to protect Berkeley DB C layers and CSV backup int...
- **Auto-ID Flexibility:** Custom IDs are not constrained to be strictly greater than `lastid`.
```perl
$adb->insert_id( 'sessions', 'user@example.com', 'Active', 'Chrome', time() );
my @sess = $adb->read_id( 'sessions', 'user@example.com' );
```
---
### 5.3 Data Operations (CRUD & Bulk)
All standard CRUD and bulk methods operate seamlessly in Simple Mode:
```perl
# Single Insert, Read, Modify, Delete
$adb->insert_id( 'orders', 'order_101', 'Pending', '150.00' );
my @order = $adb->read_id( 'orders', 'order_101' );
$adb->modify_id( 'orders', 'order_101', 'Completed', '175.50' );
$adb->delete_id( 'orders', 'order_101' );
my $exists = $adb->exist_id( 'orders', 'order_101' );
# Bulk Operations (Bulk CRUD)
my $ins_status = $adb->insert_list( 'orders', [ 'o_1', 'A', 50 ], [ 'o_2', 'B', 75 ] );
my $mod_status = $adb->modify_list( 'orders', [ 'o_1', 'A+', 55 ] );
my $del_status = $adb->delete_list( 'orders', 'o_1', 'o_2' );
```
---
### 5.4 Unindexed Direct Queries & Filtering
Since secondary index files are omitted, queries stream sequentially across the raw database file (`recs_scan`):
1. **Table Scan and Pagination (`read_all`):**
```perl
# Paged scan (start => 0, limit => 10)
my ( $total_count, @records ) = $adb->read_all( 'items', 0, 10 );
# Retrieve keys only
my @keys = $adb->read_all( 'items', keys_only => 1 );
# In-memory sorting (Block 3 ASC: -3, DESC: 3)
my @sorted = $adb->read_all( 'items', sort => -3, keys_only => 1 );
```
2. **Field Value Fetching (`field_fetch`):**
```perl
# Block 2: Category = 'Apparel'
my @apparel = $adb->field_fetch( 'catalog', 2, 'Apparel' );
# Multi-value matching (Block 3: Color in ['Blue', 'Black'])
my ( $cnt, @results ) = $adb->field_fetch( 'catalog', 3, [ 'Blue', 'Black' ], 0, 20, sort => -4 );
```
3. **Full-Text Word Search (`search_table`):**
```perl
# Collation-aware word search (AND logic)
my @articles = $adb->search_table( 'articles', 'market economy' );
# Combined search with field filter (Block 2: Category = 'Finance')
my ( $cnt, @filtered ) = $adb->search_table( 'articles', 'rates', 0, 10, filter => [ 2, 'Finance' ] );
```
---
### 5.5 ACID Transactions
In Simple Mode, `transact_start`, `transact_commit`, and `transact_rollback` provide full ACID transaction safety. When a `rollback` is triggered, raw modifications in the `.db` file are restored:
```perl
$adb->transact_start();
eval {
$adb->insert_id( 'sessions', 'token_123', 'TempData', time() );
die "Critical error" if $failed;
$adb->transact_end();
};
if ($@) {
$adb->transact_rollback(); # token_123 is cleanly reverted from the .db file
}
```
---
### 5.6 Continuous Daily Backup Logs (`recs_back`)
Because text backup is schema-independent, **daily audit and continuous recovery streaming (`recs_back`)** is fully active in Simple Mode.
In accordance with Simple Mode's flat directory structure, no separate `backup/` or `YYYY/` subfolder is created. Every `insert_id` (`add`), `modify_id` (`edit`), and `delete_id` (`del`) operation is logged directly to **`$dbase_dir/YYYY-MM-DD.csv`**...
```text
2026-08-31 14:30:00 admin add sessions sess_token_99999 Active\x1f192.168.1.50
2026-08-31 14:31:15 admin edit sessions sess_token_99999 Closed\x1f192.168.1.50
2026-08-31 14:32:00 admin del sessions sess_token_99999
```
- To disable backup logging for volatile caches, configure `cfg => { no_backup => 1 }` or `$adb->config(no_backup => 1)`.
- Custom backup targets can be set via `path => { backup_dir => "/custom/backup/path" }`.
---
### 5.7 RAM-Disk Architecture & Caching in Simple Mode
In standard mode, AmberDB manages RAM-disk staging via schema `use_cache => 2` rules.
**In Simple Mode, RAM-disk utilization is direct and flexible:**
Since Simple Mode requires no schema files, creating a high-performance in-memory cache or session store simply involves binding a second AmberDB instance directly to the RAM-disk / tmpfs mount:
```perl
# 1. Persistent disk instance (For durable storage)
my $db_disk = AmberDB->new(
path => { dbase_dir => "/var/data/app/dbstore/tables" },
cfg => { simple => 1 },
);
# 2. RAM-Disk instance (Zero-latency in-memory cache/session store)
# (Linux: /dev/shm or tmpfs, Windows: ImDisk / RamDisk volume)
my $db_ramdisk = AmberDB->new(
path => { dbase_dir => "/dev/shm/amber_cache" },
cfg => { simple => 1, no_backup => 1 }, # Disable backup for pure transient cache
);
# In-memory reads and writes at nanosecond speed:
$db_ramdisk->insert_id( "sessions", $session_token, $user_id, time() );
my @sess = $db_ramdisk->read_id( "sessions", $session_token );
```
Benefits of this dual-instance design:
- In-memory tables run without disk I/O bottlenecks.
- Persistent tables remain safely on durable physical storage.
- Dynamic temporary tables can be spun up in seconds without schema files.
---
### 5.8 Feature Comparison: Standard vs. Simple Mode
| Feature / Subsystem | Standard Mode (`simple => 0`) | Simple Mode (`simple => 1`) |
| :--- | :---: | :---: |
| **Schema Files (`.table`, `.dbase`)** | Required & Enforced | None / Schemaless |
| **Arbitrary & Long Record IDs** | 8-Byte / Strict ASCII Limits | **Completely Unrestricted** |
| **Direct CRUD (`insert_id`, `read_id`)** | $O(1)$ | **$O(1)$ (Max Throughput)** |
| **Bulk Operations (`insert_list`, etc.)** | Supported | Supported |
| **Table Scan (`read_all`)** | Binary `.inx` or Direct | Direct Streaming Scan |
| **Pagination (`limit`) & `keys_only`** | Supported | Supported |
| **In-Memory Sorting (`sort => 2`)** | Supported | Supported |
| **Field Matching (`field_fetch`)** | Indexed `.fld` $O(1)$ | Sequential Streaming Scan |
| **Word Search (`search_table`)** | Inverted Index `.src` | Collation Streaming Scan |
| **ACID Transactions (`transact_*`)** | Supported (Index Undo) | **Supported (Raw Undo)** |
| **Continuous Daily Backup (`recs_back`)** | Supported (`backup/YYYY/`) | **Supported (Same Directory `YYYY-MM-DD.csv`)** |
| **Secondary Indexes (`.inx, .fld, .src, .srt, .fac`)** | Generated & Maintained | **Disabled (Zero Index Cost)** |
| **URL Slug Mapping (`.slg`)** | Auto Generated | Disabled |
| **Audit Logs (`.aut`) & Archive (`.del`)** | Schema-Driven | Disabled |
| **Directory Hierarchy** | `tables/`, `schema/`, `backup/`, etc. | **Flat Single Directory (`$dbase_dir/<table_name>.db`)** |
| **Secondary Indexes (`.inx, .fld, .src, .srt, .fac`)** | Generated & Maintained | **Disabled (Zero Index Cost)** |
| **URL Slug Mapping (`.slg`)** | Auto Generated | Disabled |
| **Audit Logs (`.aut`) & Archive (`.del`)** | Schema-Driven | Disabled |
| **Directory Hierarchy** | `tables/`, `schema/`, `backup/`, etc. | **Flat Single Directory (`$dbase_dir/<table_name>.db`)** |
---
## 6. Indexing and Search Engine
AmberDB maintains structured binary index files based on the schema configuration.
### 6.1 Index Types
| Extension | Index Type | Description |
|---|---|---|
| `.inx` | Record Index | Packed binary array of all active IDs, total count, and highest ID. |
| `.fld` | Match Index | Block-level key-to-IDs inverted index (`field_fetch`). |
| `.str` | Field Dictionary | Bidirectional string-to-numeric ID dictionary companion for `.fld` (`_${blk}.str`). |
| `.src` | Full-Text Index | Word-level token inverted index (`search_table`). |
| `.srt` | Sort Index | Pre-sorted binary array of record IDs for `sort_block` definitions. |
| `.fac` | Facet Index | Fast forward index for faceted filter navigation. |
| `.slg` | URL Slug Index | Bidirectional map: `_0.slg` (ID → Slug) and `_1.slg` (Slug → ID). |
### 6.2 Unified 8-Byte Binary Packing Standard
AmberDB achieves high throughput and compact disk storage through uniform **8-byte binary packing**:
- **Numeric IDs (`id_type => "num"`):** Packed as `Q*` (64-bit unsigned integers, native endian).
- **ASCII IDs (`id_type => "ascii"`):** Packed as `a8*` (fixed 8-byte null-padded ASCII).
This binary layout enables zero-copy slicing for pagination (`LIMIT/OFFSET`) directly through raw byte offsets ($O(1)$ `substr` slicing) without decoding full record buffers into memory.
### 6.3 Inverted Match Index (`.fld`) and Bidirectional Dictionary (`.str`)
For fields declared under `match_block`, AmberDB indexes data across two complementary tiers:
1. **Packed Binary Inverted Match Index (`.fld`):**
Maintains a dedicated `<table_name>_<blk>.fld` file per block. Keys map directly to 8-byte packed binary arrays (`Q*` / `a8*`) containing matching record IDs. Queries via `field_fetch` perform direct $O(1)$ key lookups into this file.
2. **Bidirectional String-to-ID Dictionary (`.str`):**
For non-relational free-text attributes (Category Name, Brand Name, Author, Status Tags), the engine automatically manages a companion `<table_name>_<blk>.str` dictionary:
* **Forward Lookup (`s:<term>` $\rightarrow$ `$nid`):** Assigns an incremental numeric token ID to each unique textual string.
* **Reverse Lookup (`n:$nid` $\rightarrow$ `<term>`):** Enables $O(1)$ reverse label translation from numeric IDs back to human-readable text.
* **Transparent Resolution:** When calling `field_fetch` or `field_filter`, developers can pass either the canonical numeric ID (`12`) or the textual label (`"Sony"`). The engine automatically resolves text terms via `.str` and retrieves the match...
### 6.4 Sorting Mechanism & Developer Guide
AmberDB provides high-performance, pre-indexed sorting across specific table blocks.
#### 6.4.1 Schema Configuration (`sort_block`)
Define sortable blocks in your `.table` schema file. Specify a simple block index (`4`), or declare explicit types (`type`) for numeric and date fields:
```perl
# dbstore/schema/catalog_product.table
{
id_type => 'num',
sort_block => [
4, # Block 4: Title (String sorting)
{ blk => 10, type => 'num' }, # Block 10: Price (Numeric sorting)
{ blk => 12, type => 'date' }, # Block 12: Timestamp sorting (YYYYMMDDHHMMSS)
],
}
```
#### 6.4.2 Using Sort in Query Methods
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
my @new_products = (
[ 0, "5", "3", "Wireless Headphones", "149.90", "2026-08-28", "1" ],
[ 0, "5,12", "8", "Mechanical Keyboard", "299.00", "2026-08-28", "1" ],
[ 0, "12", "3", "Gaming Mouse", "89.50", "2026-08-28", "1" ],
# ... hundreds or thousands of records ...
);
my $status = $adb->insert_list("catalog_product", @new_products);
# Returns hashref of created IDs: { 101 => 1, 102 => 1, 103 => 1, ... }
```
### 8.3 Bulk Modify (`modify_list`)
```perl
my @updates = (
[ 101, "5", "3", "Wireless Headphones Pro", "179.90", "2026-08-28", "1" ],
[ 102, "5,12", "8", "Mechanical Keyboard RGB", "329.00", "2026-08-28", "1" ],
);
my $status = $adb->modify_list("catalog_product", @updates);
```
### 8.4 Bulk Delete (`delete_list`)
```perl
# Target IDs can be passed as a flat list or array reference
my $status = $adb->delete_list("catalog_product", 101, 102, 103);
# or:
# $adb->delete_list("catalog_product", [101, 102, 103]);
```
### 8.5 Chunking Strategy for Large Datasets (ETL Ingestion)
For massive imports (e.g. 50,000+ records), chunking records into batches of 500 to 1,000 optimizes memory allocation and balances disk cache flushing:
```perl
my $chunk_size = 1000;
for (my $i = 0; $i < @huge_dataset; $i += $chunk_size) {
my $end = $i + $chunk_size - 1;
$end = $#huge_dataset if $end > $#huge_dataset;
my @chunk = @huge_dataset[$i .. $end];
$adb->insert_list("catalog_product", @chunk);
}
```
---
## 9. Schema Configuration (.table & In-Memory)
AmberDB is a schema-driven database engine. Table schemas define primary key constraints, field data types, multi-dimensional indexes, automatic URL slug generation, facet filters, lifecycle junk rules, data validation constraints, and variable repea...
### 9.1 Database and Table Directory Layout
AmberDB stores tables, indexes, and schema definitions in dedicated physical directories under the configured `dbstore` root:
| Directory | Purpose |
|---|---|
| `dbstore/tables/` | Base data (`.db`) and binary indexes (`.inx`, `.fld`, `.src`, `.fac`, `.srt`, `.slg`) |
| `dbstore/schema/` | Schema files (`.table`) and group configs (`.dbase`) |
| `dbstore/conf/` | Plain-text `.conf` configuration and property files |
| `dbstore/backup/` | Daily CSV audit backups (`dbgun/YYYYMMDD/`) |
| `dbstore/cache/` | **Unified Shared RAM-Disk (ImDisk/tmpfs) Root:** |
| `dbstore/cache/tables/` | Mirrored hot `.db` and `.inx` tables in RAM for `use_cache => 1 & 2` |
| `dbstore/cache/conf/` | Compiled high-speed config cache (`*.pl` hash references) |
| `dbstore/cache/schema/` | Cached / pre-compiled table schemas in RAM (`*.table`, `*.dbase`) |
| `dbstore/cache/lock/` | Process and table-level `flock` lock files in RAM (`*.lock`) |
| `dbstore/cache/pids/` | Process lock files and login error state logs (`*.pid`, `*.error`) |
> [!IMPORTANT]
> **Version 5.21.0 Migration Notice:** The only manual action required when upgrading existing projects is to rename your database directory's `dbstore/scheme/` folder to **`dbstore/schema/`**. All programmatic path resolutions and API calls are auto...
### 9.2 Schema Role & Flexibility: Optional vs. Full Definition
Schema design in AmberDB is **modular, tiered, and highly flexible**:
* **Minimalist / Lightweight Usage:** Defining the `blocks` array in the schema file is **not mandatory**. You can define an ultra-fast, lightweight schema specifying only the indexing directives: `record_index`, `match_block`, `search_block`, and `s...
```perl
# dbstore/schema/catalog_product.table
{
name => "Product Catalog",
id_type => "num", # "num" (64-bit uint) or "ascii" (max 8 bytes)
record_index => 1, # Enable .inx primary record index & auto-increment counter
match_block => [ 1, 2, 3, 11 ], # .fld Exact field match indexes (Category, Brand, Author, Status)
search_block => [ 4, 5, 7, 9 ], # .src Full-text search fields (Title, Subtitle, Description, Barcode)
sort_block => [ 4, { blk => 10, type => 'num' } ], # .srt Pre-sorted binary ID buffers
keep_deleted => 1, # Preserve soft-deleted record timestamps in .del
log_owner => 1, # Write operator audit trails to .aut log
}
```
* **Advanced / Form-Driven & Validated Usage:** When the `blocks` array is specified, field data types (`type`), HTML form widgets (`input`), mandatory/custom validation rules (`valid`), and relational lookups (`rdbm`) are automatically enforced by t...
---
### 9.3 Schema Definition & Retrieval Methods (`table_info` & `table_attr`)
1. **Disk-Based Schemas (Recommended):**
Placed in `dbstore/schema/<table_name>.table`. AmberDB automatically parses and caches them on first access.
2. **In-Memory Dynamic Schemas:**
Programmatically assigned at runtime via `$adb->table_attr("table_name", { ... })`.
3. **Retrieving Active Schema (`table_info`):**
To inspect the parsed configuration hash reference for any table, call `$adb->table_info($table_name)`:
```perl
my $schema = $adb->table_info("catalog_product");
print "Table Name: $schema->{name}\n";
print "Search Blocks: " . join(", ", @{ $schema->{search_block} || [] }) . "\n";
```
> [!IMPORTANT]
> **Schema Files (`.table` and `.dbase`) Are Native Perl Code (Hash References)**
> In AmberDB, `.table` and `.dbase` files are not static JSON or YAML documents; they are native Perl hash references (`{ ... }`) dynamically evaluated at runtime via Perl's built-in `do` statement.
>
> * **Syntax Error Safety:** If a schema file contains any Perl syntax error (such as a missing comma `,`, unclosed bracket `}` or `]`, bad quote, or illegal character), `do` fails and returns `undef`. Consequently, the engine **will not be able to l...
> * **Validation Tip:** Validate schema files before deployment using the Perl compilation check: `perl -c dbstore/schema/table_name.table`.
### 9.4 Table Naming Conventions
* **Format:** Tables must follow lowercase alphanumeric `snake_case`: `<database>_<table_name>` (e.g. `catalog_product`, `member_user`).
* **Database Prefix:** The segment before the first underscore defines the database group (`<database>.dbase`).
* **Schema File Resolution:** For example, `catalog_product` maps to schema file `dbstore/schema/catalog_product.table` and its database configuration `dbstore/schema/catalog.dbase`.
### 9.5 Example Schema (`catalog_product.table`)
```perl
# dbstore/schema/catalog_product.table
{
name => "Product Catalog",
id_type => "num",
record_index => 1,
match_block => [ 1, 2, 3 ],
search_block => [ 4, 5 ],
}
```
---
### 9.6 Schema Configuration Parameters Reference (Table Level)
The following reference table details all top-level parameters supported in `.table` schema definitions, along with default values and legacy alias equivalents:
| Parameter | Type | Default | Legacy / Alias | Description |
| :--- | :--- | :--- | :--- | :--- |
| `name` | `string` | `"Table"` | — | Human-readable table title. |
| `id_type` | `string` | `"num"` | — | Primary key format: `"num"` (64-bit unsigned int) or `"ascii"` (max 8-byte alphanumeric string). |
| `record_index` | `0 / 1` | `0` | `readall` | When `1`, enables the `.inx` primary binary index, `table_count`, `table_lastid`, and auto-increment. |
| `search_block` | `ARRAY` | `[]` | — | Block numbers indexed in `.src` for full-text inverted search. |
| `match_block` | `ARRAY` | `[]` | `fields` | Block numbers indexed in `.fld` for exact field-to-ID matching and relational lookup. |
| `sort_block` | `ARRAY` | `[]` | — | Pre-computed `.srt` binary sort indexes (`[ 4, { blk => 10, type => 'num' } ]`). |
| `facet_block` | `ARRAY` | `[]` | `filter_block` | Block numbers indexed in `.fac` for columnar faceted category navigation. |
| `slug_block` | `ARRAY` | `[]` | `rwlink` | Block numbers combined for automated bidirectional `.slg` URL slug generation (e.g. `[2, 4]`). |
| `use_facet` | `0 / 1` | `0` | — | Enables the facet counting engine and `field_fltkeys` / `facet_menu` on the table. |
| `facet_rules` | `ARRAY` | `[]` | — | Scoping rules for facet counting (e.g., displaying only in-stock items in filter menus). |
| `use_junk` | `0 / 1` | `0` | — | Enables dual-tier indexing by segregating inactive/out-of-stock records to Cold Tier B. |
| `junk_rules` | `ARRAY` | `[]` | — | Business rules determining automatic routing of records between active and junk tiers. |
| `use_cache` | `0 / 1 / 2` | `0` | `usecache` | `0`: Disabled, `1`: Soft (.inx metadata), `2`: Hard (Full shared RAM-Disk mirror). |
| `cache_ttl` | `integer` | `3600` | — | Table-specific RAM cache time-to-live in seconds. |
| `keep_deleted` | `0 / 1` | `0` | `nodelete` | Preserves deleted records in `.del` soft-delete archive instead of permanent deletion. |
| `log_owner` | `0 / 1` | `0` | `authority` | Records user modification audit trails in `.aut` files. |
| `use_alias` | `0 / 1` | `0` | `uselnk` | Enables `.lnk` alias routing table for merged records or legacy URL redirections. |
| `use_counter` | `0 / 1` | `0` | `usecnt` | Enables automated hit/view read counters in `.cnt` files. |
| `parent_table` | `string` | `""` | — | Parent table name for vertical partitioning (child table shares the same primary ID). |
| `force` | `0 / 1` | `0` | — | When `1`, `insert_id` overwrites existing records rather than failing (Replace mode). |
| `min_char` | `integer` | `2` | `minchar` | Minimum word length for full-text search indexing (1, 2, or 3). |
| `stop_word` | `string` | `""` | `nextkey` | Stop-words excluded from full-text search indexing (e.g., `"the and for with"`). |
| `repeat_ids` | `integer` | `undef` | — | Target block number where extracted child item IDs are consolidated. |
| `repeat_start` | `integer` | `undef` | — | Starting block index for dynamic repeating child rows (order items, cart lines). |
| `view_block` | `ARRAY` | `[]` | — | Priority block numbers displayed in UI / CMS listing views. |
| `use_menu` | `0 / 1` | `1` | — | Controls display of the table in admin panel navigation menus. |
| `no_transact` | `0 / 1` | `0` | — | Exempts table from transactional rollback error propagation. |
| `no_backup` | `0 / 1` | `0` | — | Disables daily CSV user audit logging for this table. |
---
### 9.7 Block (Field) Definitions, 8 Core Field Types, UI Inputs, and Validation Reference
Each block definition inside the `blocks` array supports the following attributes:
#### 9.7.1 Core Block Attributes
| Attribute | Type | Description | Example |
| :--- | :--- | :--- | :--- |
| `id` | `string` | Programmatic field identifier | `id => "email"` |
| `name` | `string` | Display label for UI forms and table headers | `name => "Email Address"` |
| `type` | `string` | Data storage, type validation, and indexing type | `type => "text"` |
| `input` | `string` | HTML/UI Form input component type | `input => "select"` |
| `valid` | `string` | Automated data validation rule | `valid => "not_null;email"` |
| `option` | `string` | Enumerated choice options (`value:label` pairs) | `option => "1:Active,0:Inactive"` |
| `rdbm` | `string / HASH`| Foreign table lookup mapping (`foreign_table;display_block`) | `rdbm => "catalog_category;2"` |
| `extend` | `HASH` | 1:1 vertical table extension | `extend => { table => "catalog_price", join => "id" }` |
#### 9.7.2 Supported 8 Core Field Types (`type`)
AmberDB uses **8 unified core storage types** across serialization (`db_encode`/`db_decode`), indexing, and sorting layers:
| Field Type (`type`) | Description | `enc_validate` (Write Phase) | `dec_validate` (Read Phase) | Indexing & Sorting Behavior |
| :--- | :--- | :--- | :--- | :--- |
| **`auto_id`** | Auto-increment ID (Block 0) | Primary key format validation | ID scalar return | Primary key index (`.inx`) |
| **`text`** | Standard UTF-8 Text | UTF-8 string validation | String scalar (`$val // ''`) | Inverted index (`.src`), dictionary (`.str`) |
| **`num`** / **`number`** | Numeric (Integer / Float / Boolean) | Numeric validation (`^[+-]?[0-9]+(?:\.[0-9]+)?$`), defaults empty to `0` | Numeric scalar cast (`0 + $val`) | Numerical sorting (`<=>`) in `.srt`, `.fld` filters |
| **`ascii`** | ASCII-Only Text | ASCII normalization via `to_ascii` | Clean ASCII text | URL slug map (`.slg`), ASCII `.srt` sorting |
| **`date`** | Date and Time | Assigns system date if `auto_date` is active | Date string | Chronological sort in `.srt` via `str2dateid` |
| **`array`** / **`repeat`** | List / Repeating Rows | ARRAY ref or `[split /,/]` | Perl `ARRAY` ref (`[]`) | Multi-value matching (`field_fetch`) |
| **`hash`** | Dictionary / Object (HASH ref) | HASH ref validation | Perl `HASH` ref (`{}`) | Schemaless nested key-value store |
| **`binary`** | Binary Payload / Base64 | Raw bytes or Base64 string | Raw binary scalar | Direct flat file storage |
> [!NOTE]
> **Numeric and Boolean Management:** The `num` (or `number`) type handles positive (`150`, `+25`), negative (`-50`, `-12.75`), floating-point values, and `0 / 1` boolean flags. Unchecked HTML checkboxes or empty numerical inputs are automatically no...
#### 9.7.3 UI Form Input Components (`input`)
Determines how the field is rendered in UI forms and administration panels:
| Component (`input`) | UI Element | Description |
| :--- | :--- | :--- |
| `text` | Text Input | Standard single-line text field `<input type="text">`. |
| `textarea` | Textarea | Multi-line plain text box `<textarea>`. |
| `summernote` | Summernote | Rich WYSIWYG HTML visual editor for articles/descriptions. |
| `select` | Dropdown Select | Single-selection dropdown list `<select>`. |
| `checkbox` | Checkbox | Multi-selection checkboxes `<input type="checkbox">` (Use `type => "num"` for Boolean). |
| `radio` | Radio Buttons | Single-selection radio options `<input type="radio">`. |
| `file` | File Upload | Attachment or image file uploader `<input type="file">`. |
| `hidden` | Hidden Field | Hidden form element `<input type="hidden">` (for primary IDs). |
| `email` | Email Input | HTML5 email input field `<input type="email">`. |
| `ascii` | ASCII Field | User/code input box constrained to ASCII charset. |
| `number` | Number Input | Numeric stepper `<input type="number">`. |
| `date` | Date Picker | Interactive date calendar selector `<input type="date">`. |
| `password` | Password Field | Obscured security input `<input type="password">`. |
| `repeat` / `repeats` | Repeater Table | Dynamic sub-row table input with add/remove row buttons (Order items, invoice lines). |
| `search_block` | Search Box | Search-assisted dynamic filter input. |
| `selectbyfind` | SelectByFind | Foreign relation selector populated via dynamic search. |
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
```text
┌────────────────────────────────────────────────â”
│ dbstore/cache/ (tmpfs RAM-Disk) │
├──────────────────────┬─────────────────────────┤
│ cache/${table}.db │ cache/${table}.inx │
│ (Records) │ (lastid, keys, meta...) │
└──────────────────────┴─────────────────────────┘
```
### Cache Levels (`use_cache`)
* **`0` (Disabled):** No caching.
* **`1` (Soft Cache):** Caches `lastid`, `keys`, and `count` metadata in `cache/${table}.inx`, and supports manual `$adb->cache_write` / `$adb->cache_read`.
* **`2` (Hard Cache - Full Table RAM Mirror):** Table records are cached in `cache/${table}.db` and `cache/${table}.inx` in RAM. Reads (`read_id`, `read_list`) are served directly from RAM.
```perl
# 1. Manual Cache Write (stores in cache/${table}.inx)
$adb->cache_write("catalog_product", "featured_items", @featured_list);
# 2. Cache Read
my @featured = $adb->cache_read("catalog_product", "featured_items");
# 3. Hard Cache Table Preload
$adb->cache_preload("catalog_category");
# 4. Invalidate Cache (Automatically purged on modify / delete_id)
$adb->cache_delete("catalog_product", "featured_items"); # Single key
$adb->cache_delete("catalog_product"); # Entire table cache (.db and .inx)
# 5. Inspect RAM-Disk Diagnostics & Mount Status
my $cache_diag = $adb->cache_setup();
# Returns hashref: { is_mounted => 1, mount_desc => "...", cache_dir => "...", cache_size => "512M" }
```
### Cache TTL (`cache_ttl`) & Runtime Overrides
The `cache_ttl` expiration time is defined per-table directly inside its schema (e.g. `cache_ttl => 1800`). Ephemeral data structures like session tokens or process locks can have their expiration configured in the schema or dynamically tuned at runt...
```perl
# Dynamically configure session table cache TTL to 30 minutes (1800 seconds)
$adb->table_attr("session", { use_cache => 1, cache_ttl => 1800 });
```
### Temporary Disk Buffer
For large reporting queries or intermediate batch jobs:
```perl
$adb->buffer_write("temp_report", @large_data);
my @data = $adb->buffer_read("temp_report");
$adb->buffer_delete("temp_report");
```
---
## 14. Configuration and Deterministic Flag Management (`config`)
Runtime behavior can be tuned and safely configured via the `$adb->config()` method:
```perl
# Bulk or single configuration assignment (Recommended)
$adb->config(
no_write => 1, # Read-only maintenance mode: block all writes
no_backup => 1, # Disable daily CSV audit logging for all tables
simple => 1, # Direct unindexed mode: bypasses secondary index generation
keys_only => 1, # read_all returns IDs only
cache_size => '1024M', # RAM-Disk / tmpfs cache size (Default: 512M)
);
# Single scalar getter:
my $no_write = $adb->config('no_write');
# Bulk getter (returns a safe shallow copy):
my $cfg = $adb->config();
```
---
## 15. Data Structures, Low-Level Table and Stream Operations
Beneath the standard CRUD layer, AmberDB provides direct access to optimized `DB_File` C-level primitives and raw streaming methods:
### 15.1 Data Structures and Serialization (`db_encode`, `db_decode`)
AmberDB encodes and decodes complex nested Perl structures:
```perl
# Encode: Native Perl Data → String
my $encoded = $adb->db_encode("Text", [ 1, 2, 3 ], { key => "val" });
# Decode: String → Native Perl Data
my ($text, $arr_ref, $hash_ref) = $adb->db_decode($encoded);
```
### 15.2 Low-Level Table and Stream Management (`table_read`, `table_write`, `table_close`)
Used for direct batch processing sessions or custom streaming tasks:
```perl
my $table_path = $adb->table_path("catalog_product") . ".db";
# 1. Open Table in Read/Write Mode with Exclusive Lock (flock LOCK_EX)
my $db_obj = $adb->table_write($table_path);
# 2. Open Table in Read-Only Mode (O_RDONLY)
my $db_ro = $adb->table_read($table_path);
# 3. Synchronize (sync), Unlock, and Close Table Session
$adb->table_close($table_path);
```
### 15.3 Raw Record Manipulation (`recs_get`, `recs_put`, `recs_del`, `recs_exist`, `recs_keys`, `recs_scan`, `table_readid`)
Executes direct `$db->get()`, `$db->put()`, and `$db->del()` calls on open or dynamically resolved table handles:
```perl
# 1. Bulk Read Raw Values (recs_get)
my $raw_data = $adb->recs_get($table_path, 5001, 5002);
# Returns: { 5001 => "raw_encoded_string", 5002 => "..." }
# 2. Single Record Direct Read with Auto-Session (table_readid)
my ($rid, @record) = $adb->table_readid($table_path, 5001);
# 3. Bulk Put Raw Records (recs_put)
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
\%selected_filters,
$table_info->{facet_block},
{ limit => 10, sort => "count" } # Display top 10 options per group sorted by product count
);
# $menu structure is ready to pass directly to your template:
# {
# count => 42, # Total matching products
# ids => [ 101, 105, 120, ... ], # IDs of matching products for product grid
# active_counts => { 1 => 1, 2 => 2 }, # Active filters count per block
# groups => [ # Ready-to-render sidebar groups:
# {
# blk => 2,
# name => "Brand",
# active => "1",
# active_count => 2,
# records => [
# { uid => "fc_2_12", param => "f2", val => 12, label => "Apple", count => 28, checked => "1" },
# { uid => "fc_2_14", param => "f2", val => 14, label => "Samsung", count => 14, checked => "1" },
# { uid => "fc_2_19", param => "f2", val => 19, label => "Sony", count => 6, checked => "" },
# ]
# },
# ...
# ]
# }
```
#### B. Dynamic Filters on Search Result Pages
Pass the list of search result IDs as `base_ids` so sidebar filters apply strictly to search results:
```perl
# 1. Search catalog for user query (keys_only returns unpaginated ID list)
my @found_ids = $adb->search_table("catalog_product", "sci-fi", keys_only => 1);
# 2. Generate facet menu scoped exclusively to the search results
my $search_facets = $adb->facet_menu(
"catalog_attributes",
\%selected_filters,
$table_info->{facet_block},
{ base_ids => \@found_ids }
);
```
---
## 17. User Audit Trail and Backup
### 17.1 User Action History (`log_owner`)
When `log_owner => 1` is enabled in the schema, record modification history is stored in `.aut`:
```perl
# Retrieve user audit history as formatted HTML
my $history_html = $adb->auth_view("catalog_product", 5001);
print $history_html;
# Output:
# add 2026-08-14 10:15 admin_user
# edit 2026-08-14 11:30 editor_user
```
### 17.2 Continuous Recovery Stream (`YYYY-MM-DD.csv`)
AmberDB automatically appends every `insert`, `modify`, and `delete` operation into a clean, chronological time-series stream in `backup/YYYY/YYYY-MM-DD.csv`.
Each entry is tab-separated (`\t`) using the standard format:
`[Timestamp] \t [User] \t [Action] \t [Table] \t [Record ID] \t [Packed Values]`
To disable this backup stream:
* **In Table Schema (Per-Table):** Add `no_backup => 1` in the table schema to disable logging for that specific table only.
* **Globally via Config (All Tables):** Set `$adb->config(no_backup => 1);` to disable logging across all tables.
### 17.3 Native Database Archive (`.amberdb` Dump & Restore)
AmberDB packages all schemas (`schema/*.table`, `schema/*.dbase`) and authoritative data files (`tables/*.db`, `tables/*.del`, `tables/*.aut`, `tables/*.cnt`) alongside cryptographically verified SHA-256 checksums in a single compressed, portable **`...
Derived index files (`.inx`, `.src`, `.fld`, `.fac`, `.srt`) are intentionally excluded to keep archives compact and ensure future-proof portability; `restore` deterministically rebuilds all indexes via `set_index`.
```perl
use AmberDB;
use AmberDB::Tools;
my $adb = AmberDB->new(path => { dbase_dir => "./dbstore" });
my $tools = AmberDB::Tools->new($adb);
# 1. Create full database backup archive (.amberdb)
my $archive = $tools->dump();
# Output: dbstore/backup/2026/amberdb_2026-08-28_180000.amberdb
# 2. Export specific tables as a focused snapshot archive
$tools->dump(
file => "backup/2026/catalog_backup.amberdb",
tables => ["catalog_product", "catalog_category"]
);
# 3. Restore database archive and automatically rebuild all indexes
$tools->restore(
file => "backup/2026/catalog_backup.amberdb",
force => 1, # Overwrite confirmation for non-empty target directories
reindex => 1 # Automatically reconstruct binary indexes from source data
);
```
#### CLI Command-Line Utility (`bin/amberdb_backup.pl`)
```bash
# Dump entire database to default archive
perl bin/amberdb_backup.pl --dump --file backup/2026/full_backup.amberdb
# Dump specific tables only
perl bin/amberdb_backup.pl --dump --tables products,orders
# Restore database archive with integrity checks and automatic reindexing
perl bin/amberdb_backup.pl --restore --file backup/2026/full_backup.amberdb --force
```
---
## 18. Maintenance and Repair Tools (AmberDB::Tools)
`AmberDB::Tools` provides utilities for reindexing, table vacuuming, and data migration:
```perl
use AmberDB;
use AmberDB::Tools;
my $adb = AmberDB->new(path => { dbase_dir => "./dbstore" });
my $tools = AmberDB::Tools->new($adb);
# 1. Rebuild all indexes for a table
$tools->set_index("catalog_product");
# 2. Rebuild indexes across all tables in database
$tools->index_alltables();
# 3. Verify index consistency
my @records = $adb->read_all("catalog_product", 0, 0, no_index => 1);
my $diff = $tools->check_readall("catalog_product", @records);
# 4. Vacuum Table (Removes fragmentation and shrinks .db file)
$tools->vacuum("catalog_product", 1); # 1 = automatically reindex after vacuum
# 5. Export / Import CSV
$tools->tie2csv("catalog_product");
$tools->csv2tie("catalog_product");
# 6. Batch Reindex / Convert All Database Tables
my $converted_report = $tools->convert_tables();
# 7. Delete Table and All Secondary Index Files from Disk
$tools->del_table("obsolete_table");
# 8. Lightweight Ad-Hoc AmberDB Instance for Temporary/Standalone Dirs
my $simple_adb = $tools->db_simple("/path/to/data/dir");
```
---
## 19. File Extensions Map
AmberDB file extensions are classified into 3 operational tiers based on their authority and reconstructibility:
| Extension | Role / Classification | Reconstructible? | Description |
|---|---|---|---|
| **Authoritative Master Data** | | | |
| `.db` | Primary Data (Source of Truth) | ⌠**No** (Authoritative) | Berkeley DB master document table (`DB_File` Hash). |
| `.del` | Soft-Deleted Archive | ⌠**No** (Authoritative) | Archive of soft-deleted records (`keep_deleted`). |
| `.aut` | User Audit Trail | ⌠**No** (Authoritative) | Chronological user action log (`log_owner`). |
| `.str` | String Dictionary Mapping | ⌠**No** (Authoritative) | Bidirectional string-to-foreign-key dictionary file (`_${blk}.str`). |
| **Derived Secondary Indexes** | | | |
| `.inx` | Record Index | **Yes** (`set_index`) | Binary array of all active IDs, total count, highest ID. |
| `.fld` | Inverted Match Index | **Yes** (`set_index`) | Block-level key-to-IDs inverted index (`match_block`). |
| `.src` | Full-Text Search Index | **Yes** (`set_index`) | Word-level token inverted index (`search_block`). |
| `.srt` | Sort Index | **Yes** (`set_index`) | Pre-sorted binary array of record IDs (`sort_block`). |
| `.fac` | Facet Navigation Index | **Yes** (`set_index`) | Forward index for faceted filter navigation (`facet_block`). |
| `.slg` | URL Slug Map | **Yes** (`set_index`) | Bidirectional map: `_0.slg` (ID→Slug) and `_1.slg` (Slug→ID). |
| `.jinx`| Junk Record Index | **Yes** (`set_index`) | Binary primary index for cold/archived records (`use_junk`). |
| `.jfld`| Junk Match Index | **Yes** (`set_index`) | Field match index for cold records (`jnktype => 'B'/'AB'`). |
| `.jsrc`| Junk Full-Text Search | **Yes** (`set_index`) | Word-level inverted index for cold records (`jnktype => 'B'/'AB'`). |
| **Runtime & Transient Files** | | | |
| `.cnt` | View / Hit Counter | âš ï¸ Counter state | Hit/read counter file (`use_counter`). |
| `.txn` | Transaction Undo Journal | âš ï¸ Transient (Runtime) | Active transaction rollback journal file (`txn/`). |
| `.cache`| Shared RAM-Disk Cache | Yes (RAM-Disk) | RAM-Disk shared cache file (`cache/`). |
| `.tmp` | Disk Buffer File | âš ï¸ Transient (Staging) | Disk staging buffer file under `dbstore/buffer/` (`buffer_write`). |
| `.lock` | Process Mutex Lock | âš ï¸ Transient (Mutex) | OS `flock` process synchronization lock file. |
---
## 20. Directory Structure
```text
dbstore/
├── schema/ ↠Schema and Group Configurations
│ ├── catalog.dbase ↠Group definition
│ ├── catalog_product.table ↠Product table schema
│ └── catalog_category.table ↠Category table schema
├── tables/ ↠Main Data and Index Files
│ ├── catalog_product.db ↠Main data file
│ ├── catalog_product.inx ↠Binary record index
│ ├── catalog_product_1.fld ↠Category match index
│ ├── catalog_product_4.src ↠Title search index
│ ├── catalog_product_10.srt ↠Price sort index
│ ├── catalog_product.fac ↠Facet index
│ ├── catalog_product_0.slg ↠ID → Slug Map
│ ├── catalog_product_1.slg ↠Slug → ID Map
│ ├── catalog_product.aut ↠Audit trail
│ └── catalog_product.del ↠Soft-deleted records
├── cache/ ↠Shared RAM-Disk Cache Files
├── buffer/ ↠Transient Disk Buffer / Staging Files
├── txn/ ↠Active Transaction Journals
├── pids/ ↠Lock Files
└── backup/ ↠Daily CSV Backups
```
---
## 21. Developer Best Practices and Recommendations
1. **Use `insert_list` for Bulk Ingestion:** When adding hundreds of records, use `insert_list` instead of looping over `insert_id`. Batch mode writes all records in a single file session and rebuilds indexes in one pass.
2. **Wrap Multi-Step Writes in `transact_start`:** Always wrap inventory deductions, checkout sequences, or multi-table balance updates inside transactions.
3. **Index Only Required Fields:** Only assign fields to `match_block` or `search_block` if they are actively queried to minimize disk write overhead.
4. **Always Handle Pagination Return Signatures Correctly:** When passing `$limit > 0` to `read_all`, `field_fetch`, or `search_table`, remember that the first returned value is `$total_count` integer. Never unpack into a single array (`my @records =...
5. **Choose Numeric IDs Where Possible:** Standardize on `id_type => "num"` for optimal 64-bit binary packing performance.
6. **Standardize on Record Array ID at Index 0:** Always maintain the Primary Key ID at Index 0 (`$record[0]`) within record arrays (`@record`). For new records, initialize with `0` and assign the returned ID via `my $id = $record[0] = $adb->insert_i...
---
## 22. Full Working Example (Checkout & Stock Transaction Scenario)
The following example demonstrates creating master entity tables, inserting a product with referenced foreign IDs and multi-category indexing, querying with sorting, and executing an atomic checkout transaction:
```perl
use strict;
use warnings;
use AmberDB;
# 1. Initialize Engine
my $adb = AmberDB->new(
cfg => { language => "en", user => "cashier_1" },
path => { dbase_dir => "./dbstore" }
);
# 2. Populate Master Entity Tables
my $cat_computers = $adb->insert_id("catalog_category", undef, "Computers & IT", 1); # ID: 5
my $cat_portable = $adb->insert_id("catalog_category", undef, "Portable Devices", 1);# ID: 12
my $brand_apple = $adb->insert_id("catalog_brand", undef, "Apple", "USA"); # ID: 8
my $author_team = $adb->insert_id("catalog_author", undef, "Hardware R&D", "Core"); # ID: 7
# 3. Add New Product (Relational fields receive IDs; multi-category stored as "5,12")
my @product = (
"5,12", # [1] Category IDs (5: Computers, 12: Portable)
"8", # [2] Brand ID: Apple (8)
"7", # [3] Author / Contributor ID: 7
"MacBook Pro M3", # [4] Product Title
"16GB RAM 512GB SSD Space Gray",# [5] Subtitle
"", "", "",
10, # [8] Stock Count: 10 units
"195949123456", # [9] Barcode
"1999.00", # [10] Price
"1" # [11] Status: Active
);
my $product_id = $adb->insert_id("catalog_product", undef, @product);
print "1. Product created -> ID: $product_id\n";
# 4. Read Auto-Generated URL Slug
my $slug_map = $adb->get_slug("catalog_product", 0, $product_id);
print "2. Product URL -> /product/$slug_map->{$product_id}\n";
# 5. Query Multi-Category (e.g. Category 12) Sorted by Price
my ($total, @items) = $adb->field_fetch(
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
[ # [3] Nested ARRAY: Order Items (Product IDs: 101, 102)
[ 101, "Laptop", 1, 35000 ],
[ 102, "Wireless Mouse", 2, 750 ]
],
{ status => "confirmed", tracking_code => "TR12345" } # [4] Nested HASH: Metadata
);
$adb->insert_id("orders", 1001, @order);
```
This entire document is written to the `.db` file as a **single key-value pair**. When read via `$adb->read_id("orders", 1001)`, it is instantly returned as native Perl array and hash references ready for immediate use, completely avoiding JSON deser...
### 24.2 Resolving Relationships with Low I/O via `match_block`
In SQL, answering *"Which orders contain Product 101?"* requires scanning the `order_items` index/table, joining with `orders`, and executing multiple disk/cache seeks across separate tables.
**In AmberDB:**
The order record contains the array of product items in Block 3. When `match_block => [3]` is defined in the schema, the engine automatically extracts each product ID using `field_to_list` and indexes it into `orders_3.fld`.
```perl
# Fetch all order records containing Product 101:
my @orders = $adb->field_fetch("orders", 3, 101);
```
This operation executes a **single direct key lookup** from `orders_3.fld`, retrieving all Order IDs matching the key `101` directly (with O(1) average-time lookup per indexed key):
```text
# Inside orders_3.fld:
# 101 => [ 1001, 1005, 1023 ] (Packed binary RID array)
```
After retrieving the keys, the engine reads their record values in a single pass and returns all detailed information belonging to the matching orders.
While SQL engines traverse multiple tables, B-Trees, and relational joins; AmberDB resolves the query directly via precomputed inverted indexes, **eliminating redundant disk I/O and query-planning overhead**.
### 24.3 Schema-Driven Automated Multi-Indexing on CRUD
In SQL, you must manually manage `CREATE INDEX` statements, full-text indexes, and trigger logic or application glue code to keep search indexes synchronized.
In AmberDB, you declare indexes once in the table's `.table` schema file:
```perl
{
match_block => [1, 3], # Customer ID & Product ID match index (.fld)
search_block => [4], # Full-text search index (.src)
facet_block => [1, 2], # Faceted navigation index (.fac)
sort_block => [10], # Binary sorted price index (.srt)
slug_block => [1, 4], # Bidirectional URL slug index (.slg)
log_owner => 1, # User audit trail (.aut)
keep_deleted => 1, # Soft-delete archive (.del)
}
```
Whenever you execute `$adb->insert_id(...)`, `$adb->modify_id(...)`, or `$adb->delete_id(...)`, the engine automatically synchronizes the base table and all corresponding index files in one atomic step.
### 24.4 Direct Inverted Key Lookups (Zero Query Planner Overhead)
In SQL, running `SELECT id FROM orders WHERE customer_id = 'A'` requires parsing, query plan evaluation, cost optimization, and virtual machine execution.
In AmberDB, `field_fetch` is a direct hash key lookup on Berkeley DB returning packed binary buffers. Query planning overhead is zero.
### 24.5 Built-in Lifecycle and Domain Features
- **Automatic URL Slug Management:** When titles or categories change, clean slugs like `/products/laptop-pro-m3` and conflict resolution suffixes are generated automatically.
- **Audit Trails (.aut):** User identity, action type (`add`, `edit`, `del`), and timestamps are recorded without extra tables.
- **Safe Soft Deletion (.del):** Deleted records are archived safely and can be inspected or restored.
- **Zero Configuration & Portability:** Copying the database directory creates a complete, standalone backup that can run on any Perl-enabled system.
---
## 25. Boundaries and Debated Topics (Physical Constraints vs. Conscious Architectural Choices)
In database design, every architectural decision serves a specific optimization goal. Certain characteristics that developers coming from traditional SQL environments might initially perceive as "constraints" or "omissions" are, in fact, **deliberate...
### 25.1 Physical and Environmental Boundaries (Out-of-Scope Scenarios)
The following scenarios lie outside the intended operational scope of an embedded, file-based database engine like AmberDB:
#### 25.1.1 High-Concurrency Parallel Write-Heavy Workloads
AmberDB relies on `DB_File` (Berkeley DB). Write operations enforce a file-level exclusive lock (`flock`).
- **Out of Scope:** Workloads where hundreds or thousands of concurrent clients continuously write or update the same table file in parallel (e.g., high-frequency financial exchange order books, distributed real-time telemetry counters).
- **Ideal Scenarios:** Read-heavy architectures, e-commerce product catalogs, content management systems (CMS), order processing, customer directories, and mid-scale enterprise data management.
#### 25.1.2 Distributed Multi-Node Concurrent Network Writes (Multi-Master Clustering)
AmberDB is optimized for high-speed local filesystem storage. Multiple physical servers writing concurrently to the same database files over shared network storage (e.g., NFS, SMB shares) can encounter lock latency and filesystem cache invalidation d...
---
### 25.2 Debated Topics: Omission or Conscious Performance Advantage?
The following architectural choices might appear restrictive from an ad-hoc SQL mindset, but they are the exact reasons why AmberDB delivers superior throughput and latency:
#### 25.2.1 Full-Table Ad-Hoc Queries on Unindexed Fields: Omission or Performance Guarantee?
- **Common Perception:** *"In SQL, I can execute ad-hoc filters on any arbitrary column without declaring an index first."*
- **Reality & Advantage:** Unindexed column queries in SQL trigger unconstrained **full table scans**, spiking server CPU and saturating disk I/O in production. AmberDB encourages developers to declare queryable fields upfront in the schema (`match_b...
#### 25.2.2 Bulk Methods Bypass Undo Journals: Limitation or Maximum I/O Throughput?
- **Common Perception:** *"Why don't `insert_list` and `modify_list` record an automatic undo transaction log?"*
- **Reality & Advantage:** Appending individual undo-journal entries during ingestion of hundreds of thousands of records introduces severe disk I/O bottlenecks. AmberDB opens a single file session and streams data directly to memory and disk buffers...
> **Developer Flexibility:** When a batch of operations strictly requires transactional atomicity and rollback capability, simply execute a standard loop of single-record CRUD calls (`insert_id`, `modify_id`, `delete_id`) inside a `transact_start()` ...
#### 25.2.3 Fixed Binary Key Lengths: Limitation or Zero-Copy Slicing Speed?
- **Common Perception:** *"Why are ASCII primary keys limited to a maximum of 8 bytes?"*
- **Reality & Advantage:** AmberDB defaults to 64-bit unsigned integers (`id_type => "num"`, `Q*`). When ASCII is explicitly configured, the 8-byte fixed-width standard (`a8*`) eliminates the need for dynamic variable-length string parsing in index m...
---
*This documentation is maintained for `AmberDB` v5.23.1 and aligns with active codebase architecture and developer practices.*
( run in 0.557 second using v1.01-cache-2.11-cpan-4ef0a570458 )