AmberDB
view release on metacpan or search on metacpan
# AmberDB
[](https://metacpan.org/pod/AmberDB)
[](https://www.perl.org)
[](LICENSE)
[](https://github.com/marufcetin/amberdb/actions)
[](https://marufcetin.github.io/amberdb/)
[](https://github.com/marufcetin/amberdb)
**AmberDB** is a high-performance, schema-driven NoSQL database engine for Perl, featuring ACID transactions and precomputed inverted indexing on top of Berkeley DB (`DB_File`). It delivers zero-overhead schema management, extensible JSON-like block ...
---
## Key Features
- **Ultra High-Performance**: Leverages Berkeley DB (`DB_File`) hash storage with $O(1)$ binary slicing and configurable in-memory buffers.
- **JOIN-Free JSON-like Extensible Block Records**: Eliminates complex relational SQL `JOIN` overhead by storing hierarchical, extensible block records. Newly added blocks and attributes are automatically indexed on the fly for low-latency multi-dime...
- **Schema-Driven Dynamic Runtime Manipulation**: Table-specific schemas govern field validations, encodings, and index mappings. Schemas and values are fully mutable and can be modified dynamically at runtime without requiring table recreation or mi...
- **8-Byte Packed Binary Indexing**: Primary and secondary indexes use unified 8-byte packed binary buffers (`Q>*`), enabling $O(1)$ substring slicing, sub-millisecond pagination, and memory-efficient `keys_only` scalar pipelines (with `use_simple =>...
- **Intelligent & Locale-Aware Accent Search**: Advanced full-text search engine (`.src`) equipped with regional language and accent intelligence, phonetic devoicing (`b/d/g -> p/t/k`), circumflex/accent unfolding (`â/î/û -> a/i/u`), apostrophe su...
- **Columnar Facet Indexing (`.fac`)**: High-performance multi-dimensional facet filtering with index-level bitwise intersections and bidirectional string dictionaries (`.unq`) for e-commerce, catalogs, and large categorical datasets.
- **Multi-Tier Junk & Lifecycle Management**: Segregates active records from historical/archived data (`.db` master vs `.jnk` tier) with seamless single-pass hybrid queries (`jnktype => 'A' | 'B' | 'AB' | 'BA'`).
- **ACID-Compliant Undo-Journal Transactions**: Full ACID multi-table transactions with disk-backed undo journaling (`dbstore/journal/txn_*`), Strict Two-Phase Locking (Strict 2PL), automatic LIFO rollback upon failure or abnormal process exit, and o...
- **2-Pillar Disaster Recovery & Native `.amberdb` Archiving**:
- **Pillar 1 (Continuous Recovery Stream):** Automatic append-only audit stream in `backup/YYYY/YYYY-MM-DD.csv` capturing every `insert`, `modify`, and `delete`.
- **Pillar 2 (Native Portable Archive):** Compressed, portable `.amberdb` archives containing schemas (`schema/*.table`, `schema/*.dbase`) and authoritative data files (`table/*.db`, `table/*.del`, `table/*.aut`, `table/*.cnt`, `table/*.unq`) with ...
- **Multi-Granularity Concurrency Control**: Non-blocking shared reads and exclusive writes at both table-level and individual record-level using OS-native `flock`.
- **ORM & Data Hydration (`inflate` / `deflate`)**: Native transformation between flat storage arrays and schema-mapped hash structures (`$adb->inflate` and `$adb->deflate`), including automatic RDBM foreign relationship resolution and repeating chil...
- **Granular Field Operations**: Direct field mutation without full record rewriting via `update_field`, positional child block insertion via `insert_field`, and safe targeted child deletion via `delete_field`.
- **Multilingual Locale Engine**: Out-of-the-box support for 10 languages (`gb` [default Global Base], `en`, `tr`, `de`, `fr`, `es`, `ja`, `ru`, `ar`, `az`) with language-specific case folding (e.g. Turkish `ı/I` and `i/İ`), cross-lingual accent fo...
- **High-Throughput 2-Phase Batch Operations**: High-performance batch ingestion pipeline (`insert_list`, `modify_list`, `delete_list`) opens master `.db` once for batch writing and executes single-pass index merging (`.inx`, `.src`, `.fld`, `.fac`, ...
- **Transparent Physical RAM-Disk Acceleration**: Integrated cross-platform orchestration (`tmpfs` Linux, `APFS` macOS, `ImDisk` Windows) across 4 operational tiers (0: Disk, 1: Hybrid Index-only, 2: Full RAM Mirror with dual-write, 3: Volatile pure ...
---
## File System & Storage Architecture
AmberDB organizes database files into a clean, deterministic physical directory structure:
```text
dbstore/
âââ schema/ â Database Group & Table Schemas
â âââ catalog.dbase â Database group configuration
â âââ catalog_product.table â Product table schema
âââ table/ â Master Data & Derived Index Files
â âââ catalog_product.db â Primary key-value data table (DB_File Hash)
â âââ catalog_product.del â Soft-deleted records archive (keep_deleted)
â âââ catalog_product.aut â User audit trail log (log_owner)
â âââ catalog_product.cnt â View/hit counters (use_counter)
â âââ catalog_product.unq â Bidirectional string-to-ID dictionary
â âââ catalog_product.inx â Primary 8-byte packed ID and sort index
â âââ catalog_product.fld â Inverted exact-match field index ("$blk:$val")
â âââ catalog_product.src â Full-text keyword search index
â âââ catalog_product.fac â Columnar facet filter index
âââ backup/ â Disaster Recovery & Archives
âââ 2026/
âââ 2026-08-28.csv â Continuous time-series audit stream (Pillar 1)
âââ full_backup.amberdb â Compressed native database archive (Pillar 2)
```
### File Extension Reference
AmberDB classifies physical file extensions into three operational tiers:
#### Authoritative Master Data (Source of Truth â Non-Reconstructible)
- **`.db`** â **Primary Document Table**: Berkeley DB master document storage (`DB_File` Hash).
- **`.del`** â **Soft-Deleted Archive**: Historical archive of soft-deleted records (`keep_deleted`).
- **`.aut`** â **User Audit Trail**: Chronological user action and modification audit log (`log_owner`).
- **`.unq`** â **Unique & Dictionary Index**: Bidirectional string-to-foreign-key dictionary and unique constraint enforcement (`.unq`).
#### Derived Secondary Indexes (Reconstructible via `set_index`)
- **`.inx`** â **Primary & Sorted Record Index**: Binary array of active IDs, record count, highest ID, and pre-sorted order arrays (`sort_block`).
- **`.fld`** â **Inverted Match Index**: Block-level key-to-IDs inverted index (`match_block`).
- **`.src`** â **Full-Text Search Index**: Word-level token inverted index (`search_block`).
- **`.fac`** â **Facet Navigation Index**: Forward bitset index for faceted filter navigation (`facet_block`).
- **`.slg`** â **URL Slug Map**: Flat unified bidirectional slug map: `0:$rid` (IDâSlug) and `1:$slug` (SlugâID).
#### Runtime & Backup Files
- **`.amberdb`** â **Native Database Archive**: Compressed tar archive with schemas, data files, and SHA-256 manifest.
- **`.csv`** â **Continuous WAL Stream**: Daily chronological append-only audit stream (`backup/YYYY/YYYY-MM-DD.csv`).
- **`.cnt`** â **View / Hit Counter**: High-throughput concurrent atomic counter store (`use_counter`).
- **`.tmp`** â **Disk Buffer File**: Disk staging buffer file under `dbstore/buffer/` (`buffer_write`).
- **`.lock`** â **Process Mutex Lock**: OS `flock` process synchronization lock file.
---
## Installation
### Standalone Windows Executable (`amberdb.exe`)
For Windows users who do not have Perl installed, a self-contained portable executable is available:
1. Download the latest **`amberdb-win64.zip`** from [GitHub Releases](https://github.com/marufcetin/amberdb/releases).
2. Extract to a directory of your choice (e.g. `C:\amberdb`).
3. Add `C:\amberdb` to your `PATH` via PowerShell:
```powershell
[Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:\amberdb", "User")
```
*(Or via Windows GUI: Environment Variables â Edit User `Path` â New â `C:\amberdb`)*
4. Run `amberdb` from any terminal prompt!
### Via CPAN (Recommended for Perl Environments)
AmberDB can be installed directly from CPAN across Linux, macOS, and Windows (Strawberry Perl / MSYS2 / MSYS64):
```bash
cpanm AmberDB
# or
cpan AmberDB
```
### Manual Build from Source
```bash
git clone https://github.com/marufcetin/amberdb.git
cd amberdb
perl Makefile.PL
make
make test
make install
```
*(On Windows, you can also install locally via `cpanm .` or `cpan .`)*
---
## Quick Start
### 1. Initialization
```perl
use strict;
use warnings;
use AmberDB;
# Initialize AmberDB instance handle ($adb)
my $adb = AmberDB->new(
cfg => { user => 'admin_user', language => 'gb' },
path => { dbase_dir => './dbstore' }
);
{
offset => 0,
limit => 20,
}
);
```
### 5. Multi-Block Field & Facet Filtering
```perl
my $res = $adb->field_filter("catalog_product", {
type => "and",
filter => {
2 => "Electronics",
5 => 1 # Active status
},
sort => { blk => 3, reverse => 0 }, # Ascending price
offset => 0,
limit => 10,
});
print "Found $res->{count} matching products.\n";
```
### 6. ACID Transactions & Strict 2PL (Undo-Journal)
AmberDB provides full **ACID-compliant transactions** via disk-backed undo-journaling and Strict Two-Phase Locking (Strict 2PL):
```perl
# Start atomic multi-table transaction
$adb->transact_start();
# 1. Check & deduct balance (acquires record lock, writes undo log)
my @account = $adb->read_id("user_account", $user_id);
if ($account[2] < 100.00) {
$adb->transact_error("user_account", "Insufficient balance");
} else {
$account[2] -= 100.00;
$adb->modify_id("user_account", @account);
# 2. Create order
my $order_id = $adb->insert_id("order_master", 0, $user_id, 100.00, "COMPLETED");
}
# 3. Finalize transaction (commits if clean, automatically rolls back on error)
my $status = $adb->transact_end();
if ($status->{status} eq 'commit') {
print "Order created and balance deducted successfully.\n";
} else {
warn "Transaction aborted and changes rolled back automatically.\n";
}
```
### 7. Native Database Backup & Restore (`.amberdb`)
```perl
use AmberDB::Tools;
my $tools = AmberDB::Tools->new($adb);
# Create a compressed .amberdb backup archive
my $archive = $tools->dump();
# Archive created at: dbstore/backup/2026/amberdb_2026-08-28_180000.amberdb
# Restore archive with SHA-256 verification and automatic index rebuilding
my $result = $tools->restore(
file => "backup/2026/full_backup.amberdb",
force => 1, # Overwrite authorization for non-empty directories
reindex => 1 # Automatically reconstruct .inx, .src, .fld, .fac, .slg
);
```
### 8. High-Throughput Batch Operations (Batch ETL & Ingestion)
When importing or updating hundreds or thousands of records from CSV, JSON, or external APIs, use the 2-phase batch methods (`insert_list`, `modify_list`, `delete_list`). These methods open the master `.db` file once and update all secondary indexes ...
```perl
# --- BATCH INSERT ---
# Array of record tuples: [ [ID (0 for auto-assign), Title, Category, Price, Date, Status], ... ]
my @batch_products = (
[ 0, "Mechanical Keyboard RGB", "Accessories", 129.99, "2026-08-28", 1 ],
[ 0, "Ergonomic Office Chair", "Furniture", 349.50, "2026-08-28", 1 ],
[ 0, "4K Ultra-Wide Monitor", "Electronics", 799.00, "2026-08-28", 1 ],
);
# Ingest batch in a single I/O pass with automatic index compilation
my $status = $adb->insert_list("catalog_product", @batch_products);
# Returns hashref of created IDs: { 101 => 1, 102 => 1, 103 => 1 }
# --- BATCH UPDATE ---
my @updates = (
[ 101, "Mechanical Keyboard RGB v2", "Accessories", 139.99, "2026-08-28", 1 ],
[ 102, "Ergonomic Office Chair XL", "Furniture", 369.50, "2026-08-28", 1 ],
);
$adb->modify_list("catalog_product", @updates);
# --- BATCH DELETE ---
$adb->delete_list("catalog_product", 101, 102, 103);
```
---
## CLI Utilities & Standalone Tools
### Standalone Windows CLI (`amberdb.exe`)
For Windows environments without Perl, download the self-contained `amberdb-win64.zip` from [GitHub Releases](https://github.com/marufcetin/amberdb/releases). It runs natively with **zero dependencies** (no Perl installation required):
```powershell
# Extract to C:\amberdb and add to User PATH (one-line PowerShell command):
[Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:\amberdb", "User")
```
After restarting your terminal, use `amberdb` from any folder:
```cmd
amberdb setup
amberdb tables
amberdb read catalog_product 10
amberdb search catalog_product "laptop"
amberdb dump backup/catalog.amberdb
```
---
### 1. Unified Management CLI (`amberdb` / `amberdb.exe`)
The consolidated administrative CLI for infrastructure provisioning, database setup, RAM-disk orchestration, backups, and maintenance:
```bash
# Infrastructure setup (creates physical directories and baseline config)
amberdb setup
amberdb setup /path/to/dbstore
# RAM-disk management (Linux tmpfs, macOS APFS, Windows ImDisk)
amberdb setup ramdisk start 512M
amberdb setup ramdisk status
amberdb setup ramdisk stop
# Native backup (.amberdb dump and restore)
amberdb dump catalog file=backup/catalog.amberdb
amberdb restore file=backup/catalog.amberdb
# Storage layout & engine migrations
amberdb update storage --all
amberdb update version --check
# Re-index secondary binary indexes (.inx, .fld, .src, .slg)
amberdb reindex catalog
```
### 2. `bin/amberdb_daemon.pl` (Service Supervisor & Sync Daemon)
Unified process controller, self-healing cron watchdog, and Tier 4 background write-behind sync engine:
```bash
# Start background write-behind sync daemon
perl bin/amberdb_daemon.pl start
# Inspect running daemon, RAM-disk status, and journal queue
perl bin/amberdb_daemon.pl status
# Synchronous flush of all pending journal events
perl bin/amberdb_daemon.pl flush
# Gracefully stop daemon process
perl bin/amberdb_daemon.pl stop
# Cron watchdog (exits in <1ms if healthy, auto-restarts if dead)
perl bin/amberdb_daemon.pl watchdog
```
---
## Multilingual Locale Engine
AmberDB includes a built-in localization and text processing engine (`AmberDB::Locale`):
```perl
my $locale = AmberDB::Locale->new('tr');
# Correct Turkish case folding
print $locale->uc('ıÅık'); # "IÅIK"
print $locale->uc('istanbul'); # "İSTANBUL"
print $locale->lc('İZMİR'); # "izmir"
# Word normalization and phonetic devoicing
print $locale->normalize("Ahmet'in kitabı"); # "ahmet kitabi"
# Currency and number formatting
print $locale->format_currency(1250.50, 'TRY'); # "âº1.250,50"
```
Supported Languages: **Global Base (`gb`)**, **English (`en`)**, **Turkish (`tr`)**, **German (`de`)**, **French (`fr`)**, **Spanish (`es`)**, **Japanese (`ja`)**, **Russian (`ru`)**, **Arabic (`ar`)**, **Azerbaijani (`az`)**.
---
## Documentation
Full comprehensive guides are available in the [`docs/`](docs/) and [`wiki/`](wiki/) directories:
- **English Documentation**:
- [Command-Line Interface (CLI) User Guide](wiki/Guide-CLI.md)
- [AmberDB Database System & Architecture Guide](docs/EN.AmberDB_User-Guide.md)
- [AmberDB::Locale User Guide](docs/EN.AmberDB-Locale_User-Guide.md)
- [AmberDB vs SQL Comparison Guide](docs/EN.AmberDB-vs-SQL_User-Guide.md)
- [AmberDB vs SQLite Benchmark Report (600K Movies)](docs/EN.AmberDB-vs-SQLite_Benchmark.md)
- **Türkçe Dokümantasyon**:
- [Komut Satırı Arayüzü (CLI) Kullanım Rehberi](wiki/TR-Guide-CLI.md)
- [AmberDB Veritabanı Sistemi & Mimari Rehberi](docs/TR.AmberDB_Veritabani_Sistemi.md)
- [AmberDB::Locale Kullanım Rehberi](docs/TR.AmberDB-Locale_Kullanim_Rehberi.md)
- [AmberDB vs SQL KarÅılaÅtırmalı Kullanım Rehberi](docs/TR.AmberDB-vs-SQL_Kullanim_Rehberi.md)
- [AmberDB vs SQLite Kıyaslama Raporu (600K Film)](docs/TR.AmberDB-vs-SQLite_Benchmark.md)
---
## Running Tests
AmberDB includes an exhaustive test suite covering core operations, indexing, transactions, search, facets, locales, and backups, along with multi-process concurrency stress tests:
```bash
# Run standard unit & integration test suite (66 test files, 650+ assertions)
prove -l t/
# Run author & extended integration test suite (RAM-disk & multi-process stress)
prove -l xt/
# or directly:
perl -Ilib xt/amberdb_ramdisk.t
perl -Ilib xt/amberdb_concurrency_stress.t
```
---
## Contributing
Contributions, bug reports, and pull requests are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
---
## Author
**Maruf Cetin**
Email: [marufcetin@gmail.com](mailto:marufcetin@gmail.com)
GitHub: [@marufcetin](https://github.com/marufcetin)
---
## License and Copyright
Copyright (C) 2005-2026 Maruf Cetin.
This library is free software; you can redistribute it and/or modify it under the terms of the Artistic License 2.0. See [LICENSE](LICENSE) for details.
( run in 1.393 second using v1.01-cache-2.11-cpan-99fa8243a42 )