AmberDB
view release on metacpan or search on metacpan
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
AmberDB provides flexible methods for listing, filtering, and sorting records.
> [!CRITICAL]
> **PAGINATION RETURN SIGNATURE & ARCHITECTURAL RATIONALE:**
> For `read_all`, `field_fetch`, and `search_table`, the presence or absence of `$limit` governs the structure of the returned list:
>
> * **1. Unpaginated Calls (`$limit == 0` or omitted):**
> The method reads **all matching records**. Since the total count is intrinsically available via `scalar @records`, no separate count variable is prepended. The list consists solely of record array references:
> `my @records = $adb->read_all("catalog_product");`
> *(Every item in `@records` is a record arrayref: `$records[0]->[1]`)*
>
> * **2. Paginated Calls (`$limit > 0` e.g. `0, 20` or `start => 0, limit => 20`):**
> Instead of reading thousands of records into RAM, the engine only deserializes the requested page slice (e.g. 20 records). However, web UIs require the total matched count to render pagination bars (e.g. *"Showing 1-20 of 1,250 products"*). Amber...
> `my ($total_count, @page_records) = $adb->read_all("catalog_product", 0, 20);`
>
> âš ï¸ **FATAL ERROR WARNING:**
> If you assign paginated results to a single array (`my @records = $adb->read_all("catalog_product", 0, 20);`), the first element `$records[0]` will be the **integer total** (e.g. `1250`), not a record reference. Attempting `$records[0]->[1]` or `$r...
> **Rule:** Whenever `$limit > 0`, always unpack results as `my ($total, @records)`.
### 4.1 `read_all` — Reading All Records with Pagination
```perl
# 1. Read all records in default order (newest first - descending ID)
my @all_records = $adb->read_all("catalog_product");
# 2. Unpaginated with Extra Options (start: 0, limit: 0 returns @records / @ids directly)
# 2.1 Retrieve record IDs only (Zero deserialization, ultra memory-efficient — keys_only)
my @all_ids = $adb->read_all("catalog_product", 0, 0, keys_only => 1);
# 2.2 Tiered Query Mode (jnktype => 'A' [Active only] | 'B' [Junk only] | 'AB' [Active + Junk])
my @active_only = $adb->read_all("catalog_product", 0, 0, jnktype => 'A');
my @active_and_jnk= $adb->read_all("catalog_product", 0, 0, jnktype => 'AB');
# 2.3 Bypass index for direct table scan (no_index)
my @raw_records = $adb->read_all("catalog_product", 0, 0, no_index => 1);
# 2.4 Unpaginated sorting (sort => 10 [descending] or sort => -10 [ascending])
my @all_price_asc = $adb->read_all("catalog_product", 0, 0, sort => -10); # Cheapest first
my @all_price_desc= $adb->read_all("catalog_product", 0, 0, sort => 10); # Highest first
my @all_alpha = $adb->read_all("catalog_product", 0, 0, sort => { blk => 4, reverse => 1 });
# 3. Paginated Queries (limit > 0 always returns ($total_count, @page))
# 3.1 First 20 records
my ($total, @page1) = $adb->read_all("catalog_product", 0, 20);
print "Total records: $total, Retrieved on this page: " . scalar(@page1) . "\n";
# 3.2 Paginated ID list (keys_only)
my ($total, @page_ids) = $adb->read_all("catalog_product", 0, 50, keys_only => 1);
# 3.3 Paginated and sorted
my ($total, @sorted_alpha) = $adb->read_all("catalog_product", 0, 20, sort => { blk => 4, reverse => 1 });
my ($total, @highest_price)= $adb->read_all("catalog_product", 0, 10, sort => 10);
my ($total, @lowest_price) = $adb->read_all("catalog_product", 0, 10, sort => -10);
# 3.4 Paginated and tiered (Active + Junk)
my ($total, @tiered_page) = $adb->read_all("catalog_product", 0, 20, jnktype => 'AB');
```
### 4.2 `field_fetch` — Inverted Match Index (.fld) and Multi-Value Querying
Fields defined in `match_block` are retrieved via inverted match indexes (`.fld`) with O(1) average lookup time per indexed key (when querying multiple values, cost scales with the number of keys). Even if a record stores multiple comma-separated IDs...
```perl
# 1. Fetch all products where Category ID (Block 1) matches "5"
my @products = $adb->field_fetch("catalog_product", 1, "5");
# 2. Fetch all products by Author ID (Block 3) "9" (Matches even if record has "7,9")
my @author_prods = $adb->field_fetch("catalog_product", 3, "9");
# 3. Paginated & sorted: Category 5 products sorted by Price (Block 10) ascending
my ($count, @sorted_prods) = $adb->field_fetch(
"catalog_product",
1, "5", # Block 1 == "5"
0, 12, # Start: 0, Limit: 12
sort => { blk => 10, reverse => 1 } # Price ascending
);
# 4. Multi-value matching (ARRAY ref, comma-separated string, or semicolon-separated)
my @multi = $adb->field_fetch("catalog_product", 1, ["5", "8"]);
my @multi = $adb->field_fetch("catalog_product", 1, "5, 8");
# 5. Fetch scalar record IDs only (Memory-efficient pipeline)
my ($total, @id_list) = $adb->field_fetch("catalog_product", 1, "5", 0, 50, keys_only => 1);
my @all_ids = $adb->field_fetch("catalog_product", 1, "5", keys_only => 1);
```
> **Deduplication Guarantee:** Even if a record matches multiple query values simultaneously, `array_nodup` guarantees that each record ID appears exactly once in the result set.
# 2. Multi-value Match: Fetch products where Block 1 matches 5 OR 12
my @products = $adb->field_fetch("catalog_product", 1, "5,12");
# 3. Paginated and Sorted Match: First 10 items in Category 5 sorted by price ascending
my ($total, @paged) = $adb->field_fetch(
"catalog_product", 1, "5",
0, 10,
sort => { blk => 10, reverse => 1 }
);
```
### 4.3 `field_filter` — Multi-Criteria Faceted Filtering
Executes compound boolean queries (AND / OR) across multiple block conditions with automated bitmask intersection:
```perl
my $filter_query = {
1 => "5", # Category ID == 5
2 => [ "8", "14" ], # Brand ID IN (8, 14)
10 => "100..500", # Price between $100 and $500
11 => "1", # In Stock == 1
};
my $result = $adb->field_filter("catalog_product", $filter_query, {
start => 0,
limit => 20,
sort => { blk => 10, reverse => 1 }
});
print "Filtered Count: $result->{count}\n";
my @record_ids = @{ $result->{ids} };
```
### 4.4 `search_table` — Full-Text & Phonetic Keyword Search
Performs intelligent locale-aware token search across fields defined in `search_block`. Runs against `.src` inverted index files for indexed tables via direct token lookups, or performs a full table scan with identical normalization parity for uninde...
```perl
# 1. Search for products matching "headphones bluetooth" (Default: AND logic)
my @results = $adb->search_table("catalog_product", "headphones bluetooth");
# 2. Paginated search with OR logic, sorted by price
my ($count, @results) = $adb->search_table(
"catalog_product",
"wireless headphones",
0, 20, # First 20 results
"or", # Match any keyword
sort => { blk => 10, reverse => 1 } # Sort by price ascending
);
# 3. Retrieve only matching record IDs (keys_only)
my ($count, @id_list) = $adb->search_table("catalog_product", "sony", 0, 50, keys_only => 1);
my @all_ids = $adb->search_table("catalog_product", "sony", keys_only => 1);
```
#### Key Highlights of AmberDB Search Normalization:
- **Apostrophe / Suffix Handling:** In records containing `"Türkiye'nin"`, queries for `"Türkiye"`, `"Türkiye'nin"`, and `"Türkiyenin"` all match. Suffixes following apostrophes (`"nin"`, `"da"`, `"in"`) are stripped as stop-words.
- **Final Consonant Devoicing (Phonetic Assimilation):** Automatic phonetic mapping for word-final consonants (`b$ => p`, `d$ => t`, `g$ => k`), seamlessly matching queries like `"tevhid"` $\leftrightarrow$ `"tevhit"`, `"gazab"` $\leftrightarrow$ `"g...
- **Circumflex Vowels:** Accented vowels (`â, î, û`) match standard vowels: `"kârın"` $\leftrightarrow$ `"karın"`, `"ÂLÎM"` $\leftrightarrow$ `"alim"`.
- **Character & ASCII Equivalence:** Full case-insensitive and Turkish/ASCII folding (`"ığdır"` $\leftrightarrow$ `"IĞDIR"` $\leftrightarrow$ `"igdir"`, `"ÇARŞI"` $\leftrightarrow$ `"çarşı"` $\leftrightarrow$ `"carsi"`, `"ÇÖPÇÜ"` $\leftr...
### 4.5 `read_list` — Reading Specific IDs in Specified Sequence
`read_list` is AmberDB's high-throughput batch record resolution engine. It plays an essential role both in the engine's internal query pipeline and in developer application code:
#### 1. Internal Engine Pipeline:
All high-level listing and querying methods in AmberDB (`read_all`, `field_fetch`, `search_table`, `field_filter`, etc.) operate in two decoupled stages:
1. **Index Filtering Stage:** The query method first reads lightweight record keys (`@ids`) from inverted index files (`.inx`, `.fld`, `.src`, `.srt`), evaluating Boolean logic (AND/OR), sorting, and pagination slicing (`recs_cutting`).
2. **Batch Document Resolution Stage:** Once the final matched ID list is finalized, it is forwarded in a single call to **`read_list`**. `read_list` opens the data table in a single batch session (or leverages the RAM-Disk cache) to deserialize all ...
#### 2. Developer API Usage & Relational Traversal (SQL JOIN Alternative):
Developers can use `read_list` directly to retrieve full document records for arbitrary collections of IDs efficiently in a single operation.
**Example Scenario: Fetching Full Profiles of Customers with Active Orders**
```perl
# 1. Retrieve all active order records
my @orders = $adb->read_all("order_active");
# 2. Assume Block 2 of each order record ($order[N]->[2]) holds the Customer ID.
# Extract unique Customer IDs using map:
my %customer_ids = map { $_->[2] => 1 } @orders;
# 3. Fetch full profile records for all matching customers in a single batch call:
my @customer_records = $adb->read_list("customers", [ keys %customer_ids ]);
foreach my $customer (@customer_records) {
my $c_id = $customer->[0]; # Customer ID
my $c_name = $customer->[1]; # Full Name
my $c_email = $customer->[2]; # Email
my $c_address = $customer->[3]; # Delivery Address (Shipping label / dispatch list)
print "Shipping Label -> ID: $c_id | Name: $c_name | Email: $c_email | Address: $c_address\n";
}
```
> [!TIP]
> `read_list` accepts an array reference (`\@ids`) or a flat array. It guarantees that the returned records preserve the **exact sequential order** of the input ID list.
### 4.6 Existence Check Functions
Quickly check whether a record or table exists without pulling full data into memory:
```perl
# 1. Single Record Existence (O(1) direct key check)
if ($adb->exist_id("catalog_product", 5001)) {
print "Product 5001 exists in database.\n";
}
# 2. Bulk Existence Check
my $presence_map = $adb->exist_list("catalog_product", 5001, 5002, 9999);
# Returns: { 5001 => 1, 5002 => 1, 9999 => 0 }
# 3. Physical Table / File Existence
if ($adb->exist_table("catalog_product")) {
print "catalog_product.db exists on disk.\n";
}
# Check specific file extension (e.g. .slg slug map)
if ($adb->exist_table("catalog_product", "slg")) {
print "Slug index file exists.\n";
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
---
### 5.1 Initializing and Activating Simple Mode
Simple Mode can be activated in four distinct ways:
1. **Constructor Initialization via `cfg`:**
```perl
my $adb = AmberDB->new(
path => { dbase_dir => "/var/data/sessions" },
cfg => { simple => 1 },
);
```
2. **Quick Shortcut Helper via `AmberDB::Tools` (`db_simple`):**
```perl
use AmberDB::Tools;
my $tools = AmberDB::Tools->new();
my $adb = $tools->db_simple("/var/data/sessions");
```
3. **Dynamic Runtime Switch via `config`:**
```perl
$adb->config( simple => 1 );
```
4. **Automatic Simple Mode Trigger via Custom Extensions (`db_ext`):**
AmberDB defaults to `.db`. If `db_ext` is configured with any extension other than `"db"` (e.g. `"dat"`, `"cache"`, `"session"`), the engine **automatically switches into Simple Mode**:
```perl
my $adb = AmberDB->new(
path => { dbase_dir => "/var/data/cache" },
cfg => { db_ext => "dat" }, # Automatically activates simple => 1
);
```
> **Directory Layout Note:** In standard mode, tables reside under `$dbase_dir/tables/`. In Simple Mode, the engine creates and reads database files directly inside the root of `dbase_dir` (`$dbase_dir/<table_name>.<ext>`). To open existing standard-...
---
### 5.2 Flexible & Arbitrary Record IDs (No 8-Byte Limit)
The standard mode **8-byte limit** and **strict ASCII/numeric format constraints** are relaxed in Simple Mode (`id_check` accepts arbitrary scalar keys and applies safe key sanitization):
- **Emails and Special Characters:** `user@example.com`, `api:v1:user:1005`
- **Long Tokens and UUIDs:** `sess_99999_abcdef_1234567890_extra_long_token` (up to 255 bytes)
- **Hyphenated Codes and Prefixes:** `TR-2026-08-31-INVOICE-001`
- **Unicode / Multilingual Keys:** `prod_özellik_kırmızı_xl`
- **Safe Key Sanitization:** Automatically trims leading/trailing whitespace (`trim_space`); strictly rejects NUL bytes (`\0`), control characters (`\r`, `\n`, `\t`), and references (ARRAY/HASH refs) to protect Berkeley DB C layers and CSV backup int...
- **Auto-ID Flexibility:** Custom IDs are not constrained to be strictly greater than `lastid`.
```perl
$adb->insert_id( 'sessions', 'user@example.com', 'Active', 'Chrome', time() );
my @sess = $adb->read_id( 'sessions', 'user@example.com' );
```
---
### 5.3 Data Operations (CRUD & Bulk)
All standard CRUD and bulk methods operate seamlessly in Simple Mode:
```perl
# Single Insert, Read, Modify, Delete
$adb->insert_id( 'orders', 'order_101', 'Pending', '150.00' );
my @order = $adb->read_id( 'orders', 'order_101' );
$adb->modify_id( 'orders', 'order_101', 'Completed', '175.50' );
$adb->delete_id( 'orders', 'order_101' );
my $exists = $adb->exist_id( 'orders', 'order_101' );
# Bulk Operations (Bulk CRUD)
my $ins_status = $adb->insert_list( 'orders', [ 'o_1', 'A', 50 ], [ 'o_2', 'B', 75 ] );
my $mod_status = $adb->modify_list( 'orders', [ 'o_1', 'A+', 55 ] );
my $del_status = $adb->delete_list( 'orders', 'o_1', 'o_2' );
```
---
### 5.4 Unindexed Direct Queries & Filtering
Since secondary index files are omitted, queries stream sequentially across the raw database file (`recs_scan`):
1. **Table Scan and Pagination (`read_all`):**
```perl
# Paged scan (start => 0, limit => 10)
my ( $total_count, @records ) = $adb->read_all( 'items', 0, 10 );
# Retrieve keys only
my @keys = $adb->read_all( 'items', keys_only => 1 );
# In-memory sorting (Block 3 ASC: -3, DESC: 3)
my @sorted = $adb->read_all( 'items', sort => -3, keys_only => 1 );
```
2. **Field Value Fetching (`field_fetch`):**
```perl
# Block 2: Category = 'Apparel'
my @apparel = $adb->field_fetch( 'catalog', 2, 'Apparel' );
# Multi-value matching (Block 3: Color in ['Blue', 'Black'])
my ( $cnt, @results ) = $adb->field_fetch( 'catalog', 3, [ 'Blue', 'Black' ], 0, 20, sort => -4 );
```
3. **Full-Text Word Search (`search_table`):**
```perl
# Collation-aware word search (AND logic)
my @articles = $adb->search_table( 'articles', 'market economy' );
# Combined search with field filter (Block 2: Category = 'Finance')
my ( $cnt, @filtered ) = $adb->search_table( 'articles', 'rates', 0, 10, filter => [ 2, 'Finance' ] );
```
---
### 5.5 ACID Transactions
In Simple Mode, `transact_start`, `transact_error`, and `transact_end` provide full ACID transaction safety. When an error is logged (`transact_error`) or an operation fails, `transact_end` automatically triggers rollback, restoring raw modifications...
```perl
$adb->transact_start();
eval {
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
* **Full Back-Office & Invoice Access:** Back-office admins and invoice systems can query archived or historical records at any time using a single parameter (`jnktype => "AB"` or `"B"`).
### 11.2 Schema Configuration (`.table`)
Enable dual-tier indexing and declare your business rules in `junk_rules`:
```perl
# dbstore/schema/catalog_product.table
{
name => "Products",
record_index => 1,
use_junk => 1, # Enables smart hot/cold indexing
# Define conditions that qualify a record as "Junk / Archive":
junk_rules => [
# 1. Product's own sales status (Block 20) is not 1 (Active) -> ARCHIVE
[ 20, "ne", 1 ],
# 2. Relational Vendor Rule: Publisher (Block 2) status is disabled in catalog_producer -> ARCHIVE
[ "2->14", "ne", 1 ],
],
jnktype => "AB", # Default query mode (Active first, then archive)
search_block => [ 4, 5 ],
match_block => [ 1, 2, 3 ],
}
```
### 11.3 Usage Scenarios & Code Examples
Select the optimal query tier using the `jnktype` parameter:
#### A. Storefront & Category Pages (Active Items Only - Mode `A`)
Keep category listings and customer browsing clean of obsolete items:
```perl
# Read active products for category listing:
my @storefront_items = $adb->read_all("catalog_product", jnktype => "A");
# Customer search:
my @results = $adb->search_table("catalog_product", "headphones", jnktype => "A");
```
#### B. Storewide Search (Active First, Archived Items Appended - Mode `AB`)
Ensure rare or older items remain discoverable without burying in-stock products:
```perl
# Active products rank first, discontinued items appear at the end:
my ($total, @results) = $adb->search_table("catalog_product", "clean code", 0, 20, jnktype => "AB");
```
#### C. Back-Office Admin & Reports (Archived Items Only - Mode `B`)
Inspect discontinued, out-of-stock, or passive catalog items:
```perl
# List all archived/junk product IDs:
my @archived_ids = $adb->read_all("catalog_product", jnktype => "B", keys_only => 1);
```
#### D. Order & Invoice Processing (Direct ID Access)
Past orders access product details seamlessly regardless of whether the item is active or archived:
```perl
# Fetch product details directly by ID (Works instantly for both active and archived products):
my @product = $adb->read_id("catalog_product", $old_product_id);
```
### 11.4 Automatic State Migration
When updating a product, AmberDB evaluates the schema rules in real time:
* Setting `sales_status` to `0` or disabling a vendor automatically **demotes the product from storefront to archive**.
* Restocking the item and setting status back to `1` automatically **restores the product to the active storefront**.
* Zero manual data maintenance or migration scripts required.
---
## 12. Automated URL Slug Management
When `slug_block => [2, 4]` is configured (Brand + Title), AmberDB generates and manages clean URL slugs automatically:
```perl
# Retrieve URL Slug by Record ID
my $slug_map = $adb->get_slug("catalog_product", 0, 5001);
my $slug = $slug_map->{5001};
print "URL: /product/$slug\n"; # Output: /product/acme-wireless-headphones
# Resolve Record ID from URL Slug (Router lookup)
my $id_map = $adb->get_slug("catalog_product", 1, "acme-wireless-headphones");
my $id = $id_map->{"acme-wireless-headphones"};
print "Resolved Product ID: $id\n";
```
### 12.1 Automatic Slug Collision Resolution (Numeric Suffixes)
When multiple records generate identical base slugs (e.g. two distinct products named "Wireless Headphones"), AmberDB automatically appends deterministic incrementing numeric suffixes (`_2`, `_3`) to ensure strict uniqueness:
* 1st Record: `wireless-headphones`
* 2nd Record: `wireless-headphones_2`
* 3rd Record: `wireless-headphones_3`
---
## 13. Unified Shared RAM Cache (.db / .inx) & Persistent Buffer
`AmberDB::Cache` provides a unified shared RAM cache mirroring AmberDB's native `.db` and `.inx` formats:
```text
┌────────────────────────────────────────────────â”
│ dbstore/cache/ (tmpfs RAM-Disk) │
├──────────────────────┬─────────────────────────┤
│ cache/${table}.db │ cache/${table}.inx │
│ (Records) │ (lastid, keys, meta...) │
└──────────────────────┴─────────────────────────┘
```
### Cache Levels (`use_cache`)
* **`0` (Disabled):** No caching.
* **`1` (Soft Cache):** Caches `lastid`, `keys`, and `count` metadata in `cache/${table}.inx`, and supports manual `$adb->cache_write` / `$adb->cache_read`.
* **`2` (Hard Cache - Full Table RAM Mirror):** Table records are cached in `cache/${table}.db` and `cache/${table}.inx` in RAM. Reads (`read_id`, `read_list`) are served directly from RAM.
```perl
# 1. Manual Cache Write (stores in cache/${table}.inx)
$adb->cache_write("catalog_product", "featured_items", @featured_list);
( run in 0.347 second using v1.01-cache-2.11-cpan-aadc1410aed )