AmberDB
view release on metacpan or search on metacpan
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
# dbstore/schema/catalog_product.table
{
name => "Product Catalog",
record_index => 1,
match_block => [ 1, 2, 3 ],
search_block => [ 4, 5 ],
}
```
---
### 9.6 Schema Configuration Parameters Reference (Table Level)
The following reference table details all top-level parameters supported in `.table` schema definitions, along with default values and legacy alias equivalents:
| Parameter | Type | Default | Legacy / Alias | Description |
| :--- | :--- | :--- | :--- | :--- |
| `name` | `string` | `"Table"` | - | Human-readable table title. |
| `use_simple` | `0 / 1` | `0` | `simple` | When `1`, enables key-value mode allowing arbitrary string keys up to 255 bytes (UUIDs, slugs, tokens) with zero `.inx` index overhead. |
| `record_index` | `0 / 1` | `0` | `readall` | When `1`, enables the `.inx` primary binary index, `table_count`, `table_lastid`, and auto-increment. |
| `search_block` | `ARRAY` | `[]` | - | Block numbers indexed in `.src` for full-text inverted search. |
| `match_block` | `ARRAY` | `[]` | `fields` | Block numbers indexed in `.fld` for exact field-to-ID matching and relational lookup. |
| `sort_block` | `ARRAY` | `[]` | - | Pre-computed `.srt` binary sort indexes (`[ 4, { blk => 10, type => 'num' } ]`). |
| `facet_block` | `ARRAY` | `[]` | `filter_block` | Block numbers indexed in `.fac` for columnar faceted category navigation. |
| `slug_block` | `ARRAY` | `[]` | `rwlink` | Block numbers combined for automated bidirectional `.slg` URL slug generation (e.g. `[2, 4]`). |
| `use_facet` | `0 / 1` | `0` | - | Enables the facet counting engine and `field_fltkeys` / `facet_menu` on the table. |
| `facet_rules` | `ARRAY` | `[]` | - | Scoping rules for facet counting (e.g., displaying only in-stock items in filter menus). |
| `use_junk` | `0 / 1` | `0` | - | Enables dual-tier indexing by segregating inactive/out-of-stock records to Cold Tier B. |
| `junk_rules` | `ARRAY` | `[]` | - | Business rules determining automatic routing of records between active and junk tiers. |
| `use_ramdisk` | `0 / 1 / 2 / 3` | `0` | - | `0`: Disabled, `1`: RAM index mirror, `2`: Full RAM-Disk mirror (dual-write), `3`: Volatile pure RAM-disk (.db only, unindexed simple mode). |
| `ramdisk_ttl` | `integer` | `300` | - | Time-to-live in seconds, strictly applicable to `use_ramdisk => 3`. |
| `table_dir` | `string` | `""` | - | Custom storage subfolder (e.g., `table_dir => 'orders'`, `table_dir => ''` for root directory). |
| `keep_deleted` | `0 / 1` | `0` | `nodelete` | Preserves deleted records in `.del` soft-delete archive instead of permanent deletion. |
| `log_owner` | `0 / 1` | `0` | `authority` | Records user modification audit trails in `.aut` files. |
| `use_alias` | `0 / 1` | `0` | `uselnk` | Enables `.lnk` alias routing table for tables where duplicate records are deleted and merged. |
| `use_counter` | `0 / 1` | `0` | `usecnt` | Enables automated hit/view read counters in `.cnt` files. |
| `parent_table` | `string` | `""` | - | Parent table name for vertical partitioning (child table shares the same primary ID). |
| `force` | `0 / 1` | `0` | - | When `1`, `insert_id` overwrites existing records rather than failing (Replace mode). |
| `min_char` | `integer` | `2` | `minchar` | Minimum word length for full-text search indexing (1, 2, or 3). |
| `stop_word` | `string` | `""` | `nextkey` | Stop-words excluded from full-text search indexing (e.g., `"the and for with"`). |
| `repeat_ids` | `integer` | `undef` | - | Target block number where extracted child item IDs are consolidated. |
| `repeat_start` | `integer` | `undef` | - | Starting block index for dynamic repeating child rows (order items, cart lines). |
| `view_block` | `ARRAY` | `[]` | - | Priority block numbers displayed in UI / CMS listing views. |
| `use_menu` | `0 / 1` | `1` | - | Controls display of the table in admin panel navigation menus. |
| `no_transact` | `0 / 1` | `0` | - | Exempts table from transactional rollback error propagation. |
| `no_backup` | `0 / 1` | `0` | - | Disables daily CSV user audit logging for this table. |
---
### 9.7 Block (Field) Definitions, 8 Core Field Types, UI Inputs, and Validation Reference
Each block definition inside the `blocks` array supports the following attributes:
#### 9.7.1 Core Block Attributes
| Attribute | Type | Description | Example |
| :--- | :--- | :--- | :--- |
| `id` | `string` | Programmatic field identifier | `id => "email"` |
| `name` | `string` | Display label for UI forms and table headers | `name => "Email Address"` |
| `type` | `string` | Data storage, type validation, and indexing type | `type => "text"` |
| `input` | `string` | HTML/UI Form input component type | `input => "select"` |
| `valid` | `string` | Automated data validation rule | `valid => "not_null;email"` |
| `option` | `string` | Enumerated choice options (`value:label` pairs) | `option => "1:Active,0:Inactive"` |
| `rdbm` | `string / HASH`| Foreign table lookup mapping (`foreign_table;display_block`) | `rdbm => "catalog_category;2"` |
| `extend` | `HASH` | 1:1 vertical table extension | `extend => { table => "catalog_price", join => "id" }` |
#### 9.7.2 Supported 8 Core Field Types (`type`)
AmberDB uses **8 unified core storage types** across serialization (`db_encode`/`db_decode`), indexing, and sorting layers:
| Field Type (`type`) | Description | `enc_validate` (Write Phase) | `dec_validate` (Read Phase) | Indexing & Sorting Behavior |
| :--- | :--- | :--- | :--- | :--- |
| **`auto_id`** | Auto-increment ID (Block 0) | Primary key format validation | ID scalar return | Primary key index (`.inx`) |
| **`text`** | Standard UTF-8 Text | UTF-8 string validation | String scalar (`$val // ''`) | Inverted index (`.src`), dictionary (`.str`) |
| **`num`** / **`number`** | Numeric (Integer / Float / Boolean) | Numeric validation (`^[+-]?[0-9]+(?:\.[0-9]+)?$`), defaults empty to `0` | Numeric scalar cast (`0 + $val`) | Numerical sorting (`<=>`) in `.srt`, `.fld` filters |
| **`ascii`** | ASCII-Only Text | ASCII normalization via `to_ascii` | Clean ASCII text | URL slug map (`.slg`), ASCII `.srt` sorting |
| **`date`** | Date and Time | Assigns system date if `auto_date` is active | Date string | Chronological sort in `.srt` via `str2dateid` |
| **`array`** / **`repeat`** | List / Repeating Rows | ARRAY ref or `[split /,/]` | Perl `ARRAY` ref (`[]`) | Multi-value matching (`field_fetch`) |
| **`hash`** | Dictionary / Object (HASH ref) | HASH ref validation | Perl `HASH` ref (`{}`) | Schemaless nested key-value store |
| **`binary`** | Binary Payload / Base64 | Raw bytes or Base64 string | Raw binary scalar | Direct flat file storage |
> [!NOTE]
> **Numeric and Boolean Management:** The `num` (or `number`) type handles positive (`150`, `+25`), negative (`-50`, `-12.75`), floating-point values, and `0 / 1` boolean flags. Unchecked HTML checkboxes or empty numerical inputs are automatically no...
#### 9.7.3 UI Form Input Components (`input`)
Determines how the field is rendered in UI forms and administration panels:
| Component (`input`) | UI Element | Description |
| :--- | :--- | :--- |
| `text` | Text Input | Standard single-line text field `<input type="text">`. |
| `textarea` | Textarea | Multi-line plain text box `<textarea>`. |
| `summernote` | Summernote | Rich WYSIWYG HTML visual editor for articles/descriptions. |
| `select` | Dropdown Select | Single-selection dropdown list `<select>`. |
| `checkbox` | Checkbox | Multi-selection checkboxes `<input type="checkbox">` (Use `type => "num"` for Boolean). |
| `radio` | Radio Buttons | Single-selection radio options `<input type="radio">`. |
| `file` | File Upload | Attachment or image file uploader `<input type="file">`. |
| `hidden` | Hidden Field | Hidden form element `<input type="hidden">` (for primary IDs). |
| `email` | Email Input | HTML5 email input field `<input type="email">`. |
| `ascii` | ASCII Field | User/code input box constrained to ASCII charset. |
| `number` | Number Input | Numeric stepper `<input type="number">`. |
| `date` | Date Picker | Interactive date calendar selector `<input type="date">`. |
| `password` | Password Field | Obscured security input `<input type="password">`. |
| `repeat` / `repeats` | Repeater Table | Dynamic sub-row table input with add/remove row buttons (Order items, invoice lines). |
| `search_block` | Search Box | Search-assisted dynamic filter input. |
| `selectbyfind` | SelectByFind | Foreign relation selector populated via dynamic search. |
| `selectbylist` | SelectByList | Multi-item picker component from list. |
#### 9.7.4 Repeating Child Row Blocks (`repeat_start` and `repeat_ids`)
AmberDB natively supports dynamic repeating child rows (e.g. order line items, invoice product rows) horizontally across the flat parent record without relational child tables or `JOIN` operations:
- **Horizontal Row Slicing (`@record[15..$#record]`):** Repeating items, each field index beyond fixed blocks (`$record[15]`, `$record[16]`, `$record[17]`, ...) holds an individual repeating record item (e.g. `[ 101, 'Book', 2, '150.00' ]`).
- **`repeat_start`**: Specifies the starting block index where dynamic repeating rows begin (e.g. `repeat_start => 15`). In the schema, block 15 acts as the prototype template for all succeeding indices.
- **`repeat_ids`**: The engine (`repeat_fields`) scans `@record[15..$#record]`, extracts the first element (numeric item ID) of each repeating row, joins them with commas (`"101,102,103"`), and stores the string in `repeat_ids` (e.g. block 12). Inclu...
```perl
# Example Schema Definition (Order Table):
repeat_ids => 12, # Aggregated item IDs block (e.g. "101,102,103")
repeat_start => 15, # Repeating child rows start at block 15
blocks => [
{ id => "id", name => "Order ID", type => "auto_id", input => "hidden" }, # 0
# ... fixed header fields (date, customer, address) ...
{ id => "prod_ids", name => "Product IDs", type => "text", input => "hidden" }, # 12 (repeat_ids target)
# ...
{ id => "products", name => "Order Items", type => "repeat", input => "repeats" }, # 15 (repeat_start template)
];
# In-Memory Record Layout:
# $record[0] = 1001; # Order Primary ID (Numeric primary key)
# $record[12] = "101,102,103"; # Auto-populated by engine via repeat_fields
# $record[15] = [ 101, 'Book', 2, '150.00' ]; # 1st Product Row
# $record[16] = [ 102, 'Pad', 1, '85.00' ]; # 2nd Product Row
# $record[17] = [ 103, 'Pen', 5, '20.00' ]; # 3rd Product Row
```
#### 9.7.5 Automated Validation Rules (`valid`)
Multiple validation rules can be chained using semicolon (`;`) (e.g. `valid => "not_null;email"`):
| Rule (`valid`) | Description | Validation Check |
| :--- | :--- | :--- |
| `none` | No Validation | Field accepts any input without validation (default). |
| `not_null` | Required | Field cannot be null, undefined, or empty string. |
| `unique` | Unique Value | Asserts that no other record in the table contains this value. |
| `email` | Email Format | Validates RFC-compliant email pattern. |
| `telefon` | Phone Number | Validates national/international phone format. |
| `ascii` | ASCII Only | Restricts character set strictly to ASCII [0-127]. |
| `numeric` | Numeric Only | Enforces that value is a valid numeric scalar. |
| `regex` | Regular Expression | Tests against custom regex pattern rule. |
| `auto_num` | Auto Number | Automatically assigns an incrementing numerical sequence. |
| `auto_pass` | Auto Password | Generates random secure password and stores salted hash. |
| `auto_date` | Auto Date | Automatically populates with current system timestamp. |
| `auto_str` | Template String | Pre-populates predefined template text. |
#### 9.7.6 Unique Constraints & Bidirectional String/ID Dictionary (`.unq`)
AmberDB uses `.unq` (Unique & Dictionary) index files (`${table}_${block}.unq`) to manage both **uniqueness validation** and **relational string $\leftrightarrow$ numeric ID translation** with $O(1)$ disk lookup speed:
1. **Extension Clarity:** Renamed from legacy `.str` to `.unq` to eliminate any visual ambiguity with `.srt` (Sort indexes).
2. **$O(1)$ Duplicate Enforcement (`valid => "unique"`):**
- When `valid => "unique"` is specified (e.g. `username`, `email`, `barcode`), `insert_id` and `modify_id` perform an instantaneous $O(1)$ check on `s:$value` in `${table}_${blk}.unq`.
- If another record holds this value, the transaction is rejected with a unique constraint error.
- Successfully written records store bidirectional mappings (`s:$value => $rid` and `n:$rid => $value`), which are automatically cleaned up when records are deleted.
3. **RDBM & `match_block` String-to-ID Auto-Resolution:**
- When a string name is passed to a relational field (e.g. `"Can Publishing"` for `rdbm => "catalog_brand;1"`), AmberDB queries `s:Can Publishing` in `catalog_brand_1.unq`.
- If present, it resolves to the existing numeric ID; if absent in `write` mode, it auto-registers the entry in `.unq` and the foreign table with an incremented ID.
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
# 6. Stream/Iterate Over All Records without High Memory Overhead (recs_scan)
$adb->recs_scan($table_path, sub {
my ($key, $raw_val) = @_;
# Process record stream lazily
});
# 7. Bulk Delete Raw Records (recs_del)
$adb->recs_del($table_path, 5001, 5002);
```
### 15.4 Table Metadata and ID Helpers (`table_keys`, `table_count`, `table_lastid`, `table_autoid`, `table_create`)
```perl
# Retrieve array of all active primary keys
my @all_ids = $adb->table_keys("catalog_product");
# Total active record count
my $total = $adb->table_count("catalog_product");
# Highest (latest) primary key
my $last_id = $adb->table_lastid("catalog_product");
# Generate or format next auto-increment ID
my $new_id = $adb->table_autoid("catalog_product");
# Initialize empty .db file for table
$adb->table_create("catalog_product");
```
### 15.5 String & Text Processing Utilities (`Amber::Util::String`)
Since `AmberDB` inherits from `Amber::Util::String`, a suite of fast string sanitization, formatting, and classification helpers are directly accessible on `$adb`:
```perl
# 1. Whitespace Normalization & Flattener (trim_space)
my $clean = $adb->trim_space(" hello \n\t world "); # Preserves line breaks
my $flat = $adb->trim_space(" hello \n\t world ", 1); # Flattens all whitespace to single space
```
```perl
# 2. HTML Tag Stripping (remove_tags)
my $text = $adb->remove_tags("<p>Description with <br/>line break</p>");
# 3. Text Truncation with Ellipsis Preservation (truncate_text / sub_str / short_title)
my $summary = $adb->truncate_text($long_body, 120); # Word-boundary safe truncation
my $short = $adb->short_title($product_title, 32); # ASCII-normalized short slug/title
# 4. Data Pattern Classifier (what_isthis)
my $type = $adb->what_isthis("user@example.com"); # Returns: 'email'
# Recognizes: email, barcode, gsm, phone, tcno, number, ascii, letter, domain, other
# 5. HTML Entity Conversion (html_ascode / code_ashtml / text2html / html2text)
my $encoded_html = $adb->html_ascode('<a href="test">'); # Encodes special characters to HTML entities
my $plain_text = $adb->html2text($html_document);
```
---
## 16. Faceted Search & Category Filters (Facet Engine)
The Facet Engine powers e-commerce sidebar filter menus (Brand, Category, Author, Price Range, Color, etc.), designed for high-performance, low-latency multi-select faceted filtering across large product catalogs.
### 16.1 Key Benefits & Features
* **Low-Latency Columnar Aggregation:** Instead of scanning full records across the entire database on every page view, the engine reads only the targeted columnar forward index files (`.fac`), aggregating filter menus with minimal I/O overhead.
* **Counts In-Stock & Active Items Only:** Discontinued, out-of-stock, or disabled products never inflate filter counts; shoppers see only genuine, purchasable options and accurate item counts.
* **Smart Multi-Select (Disjunctive Counting):** When a shopper selects multiple brands (e.g., both *Apple* and *Samsung*), remaining brand counts stay visible and accurate (OR logic within the group, AND logic across groups).
* **Search-Scoped Filters (`base_ids`):** When a visitor searches for a keyword (e.g., "wireless headphones"), the sidebar filter displays attributes only for the matching search results, rather than the entire store.
* **Automatic Label Resolution:** Numeric IDs and free-text attributes (e.g., Color names) are automatically resolved into human-readable UI labels without requiring manual join queries.
### 16.2 Schema Configuration (`.table`)
Enable the facet engine by adding `use_facet => 1` and your `facet_block` specifications to your table schema:
```perl
# dbstore/schema/catalog_attributes.table
{
name => "Product Attributes",
use_facet => 1, # Enables the facet filtering engine on this table
# Define which blocks to expose as sidebar filters:
facet_block => [
# Relational Filters (Category, Brand, Author from foreign tables):
{ blk => 1, id => "category", label => "Category", table => "catalog_category", name_idx => 2 },
{ blk => 2, id => "brand", label => "Brand", table => "catalog_producer", name_idx => 2 },
{ blk => 3, id => "author", label => "Author", table => "catalog_contributor", name_idx => 2 },
# Numeric / Range Filters:
{ blk => 4, id => "price", label => "Price Range" },
# Free-Text Attributes (Color, Size, etc.):
{ blk => 6, id => "color", label => "Color" },
],
}
```
### 16.3 Usage & Practical Examples
#### A. Building Category Sidebar Menus
Generate complete filter groups and matching product counts in a single method call:
```perl
# User selections from URL query string: Category 5, Brand 12 or 14 selected
my %selected_filters = ( 1 => "5", 2 => ["12", "14"] );
my $menu = $adb->facet_menu(
"catalog_attributes",
\%selected_filters,
$table_info->{facet_block},
{ limit => 10, sort => "count" } # Display top 10 options per group sorted by product count
);
# $menu structure is ready to pass directly to your template:
# {
# count => 42, # Total matching products
# ids => [ 101, 105, 120, ... ], # IDs of matching products for product grid
# active_counts => { 1 => 1, 2 => 2 }, # Active filters count per block
# groups => [ # Ready-to-render sidebar groups:
# {
# blk => 2,
# name => "Brand",
# active => "1",
# active_count => 2,
# records => [
# { uid => "fc_2_12", param => "f2", val => 12, label => "Apple", count => 28, checked => "1" },
# { uid => "fc_2_14", param => "f2", val => 14, label => "Samsung", count => 14, checked => "1" },
# { uid => "fc_2_19", param => "f2", val => 19, label => "Sony", count => 6, checked => "" },
# ]
# },
# ...
# ]
# }
```
#### B. Dynamic Filters on Search Result Pages
Pass the list of search result IDs as `base_ids` so sidebar filters apply strictly to search results:
```perl
# 1. Search catalog for user query (keys_only returns unpaginated ID list)
my @found_ids = $adb->search_table("catalog_product", "sci-fi", keys_only => 1);
# 2. Generate facet menu scoped exclusively to the search results
my $search_facets = $adb->facet_menu(
"catalog_attributes",
\%selected_filters,
$table_info->{facet_block},
{ base_ids => \@found_ids }
);
```
---
## 17. User Audit Trail and Backup
### 17.1 User Action History (`log_owner`)
When `log_owner => 1` is enabled in the schema, record modification history is stored in `.aut`:
```perl
# Retrieve user audit history as formatted HTML
my $history_html = $adb->auth_view("catalog_product", 5001);
print $history_html;
# Output:
# add 2026-08-14 10:15 admin_user
# edit 2026-08-14 11:30 editor_user
```
### 17.2 Continuous Recovery Stream (`YYYY-MM-DD.csv`)
AmberDB automatically appends every `insert`, `modify`, and `delete` operation into a clean, chronological time-series stream in `backup/YYYY/YYYY-MM-DD.csv`.
Each entry is tab-separated (`\t`) using the standard format:
`[Timestamp] \t [User] \t [Action] \t [Table] \t [Record ID] \t [Packed Values]`
To disable this backup stream:
* **In Table Schema (Per-Table):** Add `no_backup => 1` in the table schema to disable logging for that specific table only.
* **Globally via Config (All Tables):** Set `$adb->config(no_backup => 1);` to disable logging across all tables.
### 17.3 Native Database Archive (`.amberdb` Dump & Restore)
AmberDB packages all schemas (`schema/*.table`, `schema/*.dbase`) and authoritative data files (`tables/*.db`, `tables/*.del`, `tables/*.aut`, `tables/*.cnt`) alongside cryptographically verified SHA-256 checksums in a single compressed, portable **`...
Derived index files (`.inx`, `.src`, `.fld`, `.fac`, `.srt`) are intentionally excluded to keep archives compact and ensure future-proof portability; `restore` deterministically rebuilds all indexes via `set_index`.
```perl
use AmberDB;
use AmberDB::Tools;
my $adb = AmberDB->new(path => { dbase_dir => "./dbstore" });
my $tools = AmberDB::Tools->new($adb);
# 1. Create full database backup archive (.amberdb)
my $archive = $tools->dump();
# Output: dbstore/backup/2026/amberdb_2026-08-28_180000.amberdb
# 2. Export specific tables as a focused snapshot archive
$tools->dump(
file => "backup/2026/catalog_backup.amberdb",
tables => ["catalog_product", "catalog_category"]
);
# 3. Restore database archive and automatically rebuild all indexes
$tools->restore(
file => "backup/2026/catalog_backup.amberdb",
force => 1, # Overwrite confirmation for non-empty target directories
reindex => 1 # Automatically reconstruct binary indexes from source data
);
```
( run in 1.519 second using v1.01-cache-2.11-cpan-e623d60df62 )