AmberDB
view release on metacpan or search on metacpan
* Updated main AmberDB abstract to "High-performance embedded NoSQL database engine for Perl".
5.23.0 2026-09-02
- [ARCHITECTURE & ID SIMPLIFICATION] Pure 64-bit Binary Engine & Deprecation of 8-Byte ASCII IDs:
* Deprecated 8-byte ASCII IDs (a8) across binary index structures in favor of pure 64-bit Big-Endian unsigned integer packing (Q>*), guaranteeing O(1) binary slicing with fixed 8-byte record strides.
* Simplified bin_encode() and bin_decode() by removing fragile \0 byte auto-detection and string unpacking heuristics.
* Removed legacy id_type schema attribute across all modules and test suites; standard relational tables strictly enforce positive integer numeric IDs.
- [STORAGE & PER-TABLE SIMPLE MODE] Hybrid Multi-Model Architecture with use_simple => 1:
* Introduced per-table use_simple => 1 attribute allowing key-value tables with arbitrary string keys up to 255 bytes (UUIDs, session tokens, emails, slugs, etc.) alongside standard relational tables.
* Preserved canonical physical table path ($dbase_dir/tables/$table.db) for use_simple tables in standard database mode.
* Selective schema sanitization: tables with use_simple => 1 strip indexing, columnar, and caching definitions (blocks, match_block, search_block, facet_block, sort_block, record_index, use_cache, cache_ttl) to achieve zero index/cache I/O ...
* Enabled keep_deleted => 1 archiving (.del) on use_simple tables.
- [RDBM & INTEGRITY] Foreign Key and Cross-Table Isolation:
* Prohibited standard relational tables from binding foreign keys (RDBM) to use_simple tables in rdbm_target() and _resolve_field_value().
- [TESTS & VERIFICATION] Dedicated Test Coverage:
* Added comprehensive test suite t/amberdb_table_use_simple.t verifying arbitrary string keys, schema sanitization, zero index file creation, keep_deleted archiving, and RDBM isolation.
* Cleaned up legacy id_type occurrences across all 41 test files and concurrency stress tests (xt/amberdb_concurrency_stress.t).
- [DISTRIBUTION & METADATA] Decoupled Internal Module Versions and CPAN no_index:
* Stripped redundant $VERSION definitions from 15 internal/engine-only modules (Base, Cache, Transact, Index, Index::Facet, Index::Junk, Locale::Currency, Locale::Lang::*) to streamline release maintenance.
* Retained explicit $VERSION in public, standalone modules (AmberDB, AmberDB::Array, AmberDB::Date, AmberDB::Locale, AmberDB::String, AmberDB::Tools).
* Added comprehensive META_MERGE no_index configuration in Makefile.PL covering internal directories, namespaces, and packages.
5.22.0 2026-08-31
- [REFACTOR] Codebase Cleanliness:
* Removed redundant single-line helper is_rdbm_block() in favor of direct rdbm_target() calls across AmberDB::Index, AmberDB::Index::Junk, and AmberDB::Tools.
- [RDBM & INTEGRITY] Hardened Foreign Table Auto-Registration in field_to_list():
* Replaced decoupled table_autoid() and recs_put() sequence with atomic insert_id($target_table, 0, @record) to prevent ID race conditions.
* Corrected foreign record column alignment so auto-registered values are accurately placed at block index $target_blk rather than hardcoded column 1.
* Guaranteed that foreign table primary indexes (.inx) and secondary indexes (.unq, .src, .fac, .slg) are fully and consistently constructed upon auto-registration.
- [SECURITY & VALIDATION] Universal ID Verification and Simple Mode Key Sanitization:
* Eliminated opt-in config('id_check') requirement: every schema table now strictly enforces its schema id_type (positive integers for num, safe 8-byte chars for ascii).
* Enforced scalar ID requirement: any reference (ARRAY ref, HASH ref, etc.) passed as a record ID is strictly rejected across all modes.
* Introduced Safe Key Sanitization in Simple Mode: automatically trims leading/trailing whitespace, strictly rejects NUL bytes (\0) and control characters (\r, \n, \t, \x00-\x1F, \x7F) that corrupt Berkeley DB or CSV backups, and enforces a...
* Integrated unconditional id_check across table_autoid(), read_id(), modify_id(), delete_id(), and exist_id().
- [REFACTOR] Terminology and Codebase Refactoring: SEO -> Slug:
* Completely replaced SEO terminology with Slug across the entire codebase, test suite, and documentation without legacy aliases:
- Renamed methods: set_seourl() -> set_slug(), get_seourl() -> get_slug().
- Renamed schema attributes: seo_block -> slug_block, seo_max_len -> slug_max_len.
* Renamed test suite t/amberdb_seo_facet_bulk.t to t/amberdb_slug_facet_bulk.t.
- [STORAGE] Standardized URL Slug Map File Extension: .rwt -> .slg:
* Renamed binary slug map files from _0.rwt / _1.rwt to _0.slg (ID -> Slug) and _1.slg (Slug -> ID) across AmberDB, Index, Transact, and Tools.
* Updated Tools->set_index() to reconstruct .slg files with zero data loss.
- [DOCS] Caching Terminology Unification:
- Upgraded transaction engine specification to full ACID-Compliance with Strict Two-Phase Locking (Strict 2PL):
* Enforced Lock-Before-Write and Lock-Before-Read ordering across insert_id, modify_id, and delete_id for true serializable isolation.
* Introduced 'no_transact => 1' schema attribute and table_attr() support to exempt auxiliary tables from abort cascades while preserving LIFO rollback consistency.
- Added comprehensive ACID architectural guarantees section to documentation (README.md, Turkish and English User Guides).
- Clarified architectural distinction between high-throughput batch ETL imports and atomic business transactions.
- Standardized file open error diagnostics and OS-level reporting ($!) across all core modules:
* Added explicit OS error reporting ($!) to all open and tie failures in AmberDB, Base, Cache, Transact, and Tools.
* Replaced silent schema open failure in AmberDB::Base::table_write with diagnostic cluck and graceful return.
* Improved audit log error handling in AmberDB::auth_insert with cluck and record skipping.
- Redesigned 2-Pillar Disaster Recovery and Native Backup Architecture:
* Upgraded recs_back to continuous chronological time-series stream in 'backup/YYYY/YYYY-MM-DD.csv' eliminating folder clutter and ensuring zero-data-loss logging.
* Added Tools->dump() for creating portable, compressed '.amberdb' archives packaging schemas (schema/*.table, schema/*.dbase), authoritative data files (tables/*.db, tables/*.del, tables/*.aut, tables/*.cnt, tables/*_*.str), and cryptograp...
* Preserved native physical directory layout (schema/ and tables/) in .amberdb archives for 1-to-1 extraction and portability.
* Added Tools->restore() with SHA-256 checksum verification, non-empty database safety checks, and automated deterministic binary index reconstruction via set_index.
* Enhanced Tools->all_tables() with dual scalar/list context (grouped hashref vs. flat list) and automated 4-digit year directory discovery (e.g. 2024/, 2025/, 2026/).
* Upgraded 'bin/convert_dbstore.pl' table discovery and added side-file reporting for .str string dictionaries alongside .del, .aut, and .cnt.
* Introduced CLI utility 'bin/amberdb_backup.pl' for command-line database dump and disaster recovery operations.
* Added comprehensive unit test suite t/amberdb_backup.t covering WAL streaming, .amberdb archiving, .dbase/.str preservation, and full database restore.
- Updated POD documentation across AmberDB, AmberDB::Transact, and AmberDB::Tools modules.
5.02 2026-08-25
- Initial public release prepared for CPAN and GitHub.
- High-performance Berkeley DB (DB_File) flat-file database engine.
- Packed 8-byte binary indexing pipeline for O(1) substr slicing.
- High-speed index-assisted full-text search with phonetic and language normalization.
- Tiered indexing (Active, Junk/Archived, and Hybrid AB/BA query modes).
- Multi-dimensional Columnar Facet indexing (.fac) with high-efficiency bitsets.
- Undo-journal transaction engine (transact_start, transact_end, transact_rollback) with automatic LIFO rollback.
AGENTS.md
bin/amberdb_backup.pl
bin/convert_dbstore.pl
bin/push_wiki.pl
bin/setup_ramdisk.bat
bin/setup_ramdisk.pl
bin/setup_ramdisk.ps1
bin/setup_ramdisk.sh
Changes
CONTRIBUTING.md
cpanfile
docs/_config.yml
t/amberdb-locale_02_ascii.t
t/amberdb-locale_03_num2text.t
t/amberdb-locale_04_sort.t
t/amberdb-locale_05_formatting.t
t/amberdb-locale_06_currency.t
t/amberdb-locale_07_search_regex.t
t/amberdb-locale_08_word_normalization.t
t/amberdb.t
t/amberdb_array.t
t/amberdb_array_sort.t
t/amberdb_backup.t
t/amberdb_cache.t
t/amberdb_date.t
t/amberdb_ecommerce_facet.t
t/amberdb_encapsulation.t
t/amberdb_escape_encode.t
t/amberdb_facet_columnar.t
t/amberdb_index.t
t/amberdb_junk_facet_integration.t
t/amberdb_junk_tiered.t
t/amberdb_lock.t
MANIFEST.SKIP view on Meta::CPAN
# Version control
\B\.git\b
\B\.github\b
\B\.gitignore\b
\B\.gitattributes\b
# Temp, backup and editor files
~$
\.bak$
\.old$
\.tmp$
\.swp$
\#.*#$
\.\#.*
\b\.vscode\b
\b\.idea\b
\.code-workspace$
Makefile.PL view on Meta::CPAN
WriteMakefile(
NAME => 'AmberDB',
AUTHOR => 'Maruf Cetin <marufcetin@gmail.com>',
VERSION_FROM => 'lib/AmberDB.pm',
ABSTRACT_FROM => 'lib/AmberDB.pm',
LICENSE => 'artistic_2',
MIN_PERL_VERSION => '5.016',
PL_FILES => {},
EXE_FILES => [
'bin/amberdb_backup.pl',
'bin/convert_dbstore.pl',
'bin/setup_ramdisk.pl',
],
CONFIGURE_REQUIRES => {
'ExtUtils::MakeMaker' => '0',
},
BUILD_REQUIRES => {
'Test::More' => '0.98',
'File::Temp' => '0',
'FindBin' => '0',
- **Ultra High-Performance**: Leverages Berkeley DB (`DB_File`) hash storage with $O(1)$ binary slicing and configurable in-memory buffers.
- **JOIN-Free JSON-like Extensible Block Records**: Eliminates complex relational SQL `JOIN` overhead by storing hierarchical, extensible block records. Newly added blocks and attributes are automatically indexed on the fly for low-latency multi-dime...
- **Schema-Driven Dynamic Runtime Manipulation**: Table-specific schemas govern field validations, encodings, and index mappings. Schemas and values are fully mutable and can be modified dynamically at runtime without requiring table recreation or mi...
- **8-Byte Packed Binary Indexing**: Primary and secondary indexes use unified 8-byte packed binary buffers (`Q*` / `a8*`), enabling $O(1)$ substring slicing, sub-millisecond pagination, and memory-efficient `keys_only` scalar pipelines.
- **Intelligent & Locale-Aware Accent Search**: Advanced full-text search engine (`.src`) equipped with regional language and accent intelligence, phonetic devoicing (`b/d/g -> p/t/k`), circumflex/accent unfolding (`â/î/û -> a/i/u`), apostrophe su...
- **Columnar Facet Indexing (`.fac`)**: High-performance multi-dimensional facet filtering with index-level bitwise intersections and bidirectional string dictionaries (`.str`) for e-commerce, catalogs, and large categorical datasets.
- **Multi-Tier Junk & Lifecycle Management**: Segregates active records from historical/archived data (`.db` master vs `.jnk` tier) with seamless single-pass hybrid queries (`jnktype => 'A' | 'B' | 'AB' | 'BA'`).
- **ACID-Compliant Undo-Journal Transactions**: Full ACID multi-table transactions with disk-backed journaling (`.txn`), Strict Two-Phase Locking (Strict 2PL), automatic LIFO rollback upon failure or abnormal process exit, and orphaned journal recove...
- **2-Pillar Disaster Recovery & Native `.amberdb` Archiving**:
- **Pillar 1 (Continuous Recovery Stream):** Automatic append-only audit stream in `backup/YYYY/YYYY-MM-DD.csv` capturing every `insert`, `modify`, and `delete`.
- **Pillar 2 (Native Portable Archive):** Compressed, portable `.amberdb` archives containing schemas (`schema/*.table`, `schema/*.dbase`) and authoritative data files (`tables/*.db`, `tables/*.del`, `tables/*.aut`, `tables/*.cnt`, `tables/*_*.str`...
- **Multi-Granularity Concurrency Control**: Non-blocking shared reads and exclusive writes at both table-level and individual record-level using OS-native `flock`.
- **Multilingual Locale Engine**: Out-of-the-box support for 9 languages (`en`, `tr`, `de`, `fr`, `es`, `ja`, `ru`, `ar`, `az`) with language-specific case folding (e.g. Turkish `ı/I` and `i/İ`), collation, currency, and date formatting.
- **High-Throughput 2-Phase Batch Operations**: High-performance batch ingestion pipeline (`insert_list`, `modify_list`, `delete_list`) opens master `.db` once for batch writing and executes single-pass index merging (`.inx`, `.src`, `.fld`, `.fac`, ...
- **RAM-Disk Acceleration**: Integrated CLI tools and automation for mounting `tmpfs` (Linux) or `ImDisk` (Windows) for sub-microsecond in-memory table access.
---
## File System & Storage Architecture
│ ├── catalog_products.db ↠Primary key-value data table (DB_File Hash)
│ ├── catalog_products.del ↠Soft-deleted records archive (keep_deleted)
│ ├── catalog_products.aut ↠User audit trail log (log_owner)
│ ├── catalog_products.cnt ↠View/hit counters (use_counter)
│ ├── catalog_products_1.str ↠Bidirectional string-to-ID dictionary
│ ├── catalog_products.inx ↠Primary 8-byte packed ID index
│ ├── catalog_products_1.fld ↠Inverted exact-match field index
│ ├── catalog_products_2.src ↠Full-text keyword search index
│ ├── catalog_products_3.fac ↠Columnar facet filter bitset index
│ └── catalog_products_4.srt ↠Monotonic binary pre-sorted index
└── backup/ ↠Disaster Recovery & Archives
└── 2026/
├── 2026-08-28.csv ↠Continuous time-series audit stream (Pillar 1)
└── full_backup.amberdb ↠Compressed native database archive (Pillar 2)
```
### File Extension Reference
| File Extension | Classification | Reconstructible? | Description |
| :--- | :--- | :--- | :--- |
| **Authoritative Master Data** | | | |
| `.db` | **Primary Data (Source of Truth)** | ⌠**No** (Authoritative) | Berkeley DB master document table (`DB_File` Hash) |
| `.del` | **Soft-Deleted Archive** | ⌠**No** (Authoritative) | Archive of soft-deleted records (`keep_deleted`) |
| `.aut` | **User Audit Trail** | ⌠**No** (Authoritative) | Chronological user action log (`log_owner`) |
| `.fld` | **Inverted Match Index** | **Yes** (`set_index`) | Block-level key-to-IDs inverted index (`match_block`) |
| `.src` | **Full-Text Search Index** | **Yes** (`set_index`) | Word-level token inverted index (`search_block`) |
| `.srt` | **Sorted Index** | **Yes** (`set_index`) | Pre-sorted binary array of record IDs (`sort_block`) |
| `.fac` | **Facet Navigation Index** | **Yes** (`set_index`) | Forward bitset index for faceted filter navigation (`facet_block`) |
| `.slg` | **URL Slug Map** | **Yes** (`set_index`) | Bidirectional map: `_0.slg` (ID→Slug) and `_1.slg` (Slug→ID) |
| `.jinx`| **Junk Record Index** | **Yes** (`set_index`) | Binary primary index for cold/archived records (`use_junk`) |
| `.jfld`| **Junk Match Index** | **Yes** (`set_index`) | Field match index for cold records (`jnktype => 'B'/'AB'`) |
| `.jsrc`| **Junk Full-Text Search** | **Yes** (`set_index`) | Word-level inverted index for cold records (`jnktype => 'B'/'AB'`) |
| **Runtime & Backup Files** | | | |
| `.amberdb` | **Native Database Archive** | Portable Archive | Compressed tar archive with schemas, data files, and SHA-256 manifest |
| `.csv` | **Continuous WAL Stream** | Append-Only Log | Daily chronological audit stream (`backup/YYYY/YYYY-MM-DD.csv`) |
| `.cnt` | **View / Hit Counter** | Counter State | High-throughput concurrent counter store (`use_counter`) |
| `.txn` | **Transaction Undo Journal** | Transient (Runtime) | Active transaction rollback journal file (`txn/`) |
| `.cache` | **Shared RAM-Disk Cache** | Yes (RAM-Disk) | RAM-Disk shared cache file (`cache/`) |
| `.tmp` | **Disk Buffer File** | Transient (Staging) | Disk staging buffer file under `dbstore/buffer/` (`buffer_write`) |
| `.lock` | **Process Mutex Lock** | Transient (Mutex) | OS `flock` process synchronization lock file |
---
## Installation
}
```
### 7. Native Database Backup & Restore (`.amberdb`)
```perl
use AmberDB::Tools;
my $tools = AmberDB::Tools->new($adb);
# Create a compressed .amberdb backup archive
my $archive = $tools->dump();
# Archive created at: dbstore/backup/2026/amberdb_2026-08-28_180000.amberdb
# Restore archive with SHA-256 verification and automatic index rebuilding
my $result = $tools->restore(
file => "backup/2026/full_backup.amberdb",
force => 1, # Overwrite authorization for non-empty directories
reindex => 1 # Automatically reconstruct .inx, .src, .fld, .fac, .srt
);
```
### 8. High-Throughput Batch Operations (Batch ETL & Ingestion)
When importing or updating hundreds or thousands of records from CSV, JSON, or external APIs, use the 2-phase batch methods (`insert_list`, `modify_list`, `delete_list`). These methods open the master `.db` file once and update all secondary indexes ...
```perl
# --- BATCH DELETE ---
$adb->delete_list("catalog_products", 101, 102, 103);
```
---
## CLI Utilities
AmberDB ships with standalone command-line tools in `bin/`:
### 1. `bin/amberdb_backup.pl` (Native Backup & Restore)
Create and restore compressed, portable `.amberdb` database archives:
```bash
# Dump entire database to default archive
perl bin/amberdb_backup.pl --dump
# Dump specific tables to custom archive
perl bin/amberdb_backup.pl --dump --file backup/catalog.amberdb --tables products,orders
# Restore archive into database with automatic index rebuilding
perl bin/amberdb_backup.pl --restore --file backup/catalog.amberdb --force
```
### 2. `bin/convert_dbstore.pl` (Binary Re-Indexer)
Scans and rebuilds all table indexes (`.inx`, `.fld`, `.src`, `.srt`) into packed 8-byte binary format:
```bash
perl bin/convert_dbstore.pl --dbstore ./dbstore
```
### 3. `bin/setup_ramdisk.pl` (RAM-Disk Accelerator)
Mounts/unmounts ultra-fast RAM-disk caches for Linux (`tmpfs`) and Windows (`ImDisk`):
- [AmberDB Database System & Architecture Guide](docs/EN.AmberDB_User-Guide.md)
- [AmberDB::Locale User Guide](docs/EN.AmberDB-Locale_User-Guide.md)
- 📖 **Türkçe Dokümantasyon**:
- [AmberDB Veritabanı Sistemi & Mimari Rehberi](docs/TR.AmberDB_Veritabani_Sistemi.md)
- [AmberDB::Locale Kullanım Rehberi](docs/TR.AmberDB-Locale_Kullanim_Rehberi.md)
---
## Running Tests
AmberDB includes an exhaustive test suite covering core operations, indexing, transactions, search, facets, locales, and backups, along with multi-process concurrency stress tests:
```bash
# Run standard unit & integration test suite (39 test files, 390+ assertions)
prove -l t/
# Run multi-process concurrency & stress test suite (cross-platform Linux & Windows)
prove -l xt/
# or directly:
perl -Ilib xt/amberdb_concurrency_stress.t
```
bin/amberdb_backup.pl view on Meta::CPAN
#!/usr/bin/perl
# bin/amberdb_backup.pl - Native Archive & Disaster Recovery Utility for AmberDB
# Creates (.dump) and restores (.restore) portable, compressed .amberdb archives.
use 5.016;
use strict;
use warnings;
use Getopt::Long qw(GetOptions);
use FindBin;
use lib "$FindBin::Bin/../lib";
use AmberDB;
use AmberDB::Tools;
bin/amberdb_backup.pl view on Meta::CPAN
die "\n[ERROR] Database restore failed. Target directory may not be empty (use --force to overwrite).\n";
}
}
print "=================================================================\n";
sub usage {
my ($exit_code) = @_;
print <<"USAGE";
Usage:
perl bin/amberdb_backup.pl --dump [--file <outfile.amberdb>] [--tables <t1,t2>] [--dbase <dir>]
perl bin/amberdb_backup.pl --restore --file <archive.amberdb> [--force] [--no-reindex] [--tables <t1,t2>] [--dbase <dir>]
Actions:
-d, --dump Create a portable, compressed .amberdb archive of the database
-r, --restore Restore a .amberdb archive into the target database directory
Options:
-f, --file=FILE Path to .amberdb archive (required for restore; optional for dump)
-t, --tables=LIST Comma-separated list of tables to dump/restore (default: all)
--dbase=DIR Database base directory (default: 'dbstore')
--force Allow restore to overwrite existing data in a non-empty target directory
--no-reindex Skip automated index reconstruction during restore
-h, --help Display this help message
Examples:
# Dump all tables to default location (backup/YYYY/amberdb_YYYY-MM-DD_time.amberdb):
perl bin/amberdb_backup.pl --dump
# Dump specific tables to custom file:
perl bin/amberdb_backup.pl --dump --file backup/catalog.amberdb --tables products,categories
# Restore backup archive into fresh or staging directory:
perl bin/amberdb_backup.pl --restore --file backup/catalog.amberdb --dbase dbstore_staging
# Force restore over existing database:
perl bin/amberdb_backup.pl --restore --file backup/catalog.amberdb --force
USAGE
exit($exit_code);
}
docs/EN.About_AmberDB.md view on Meta::CPAN
[ðŸ Home](index.html) • [📖 About](EN.About_AmberDB.html) • [🚀 Quick Start](index.html#-quick-start) • [📘 Tutorial](EN.AmberDB_User-Guide.html) • [🌠Locale](EN.AmberDB-Locale_User-G...
---
# About
**AmberDB** is an embedded NoSQL database engine for Perl that processes Array-based records supporting JSON-like nested and complex structures. Thanks to custom indexes generated through schema definitions, it performs block matching, querying, full...
It is built on the C code of Berkeley DB (`DB_File`) provided by the standard Perl package and does not use any other database engine. `DB_File` is Berkeley DB's low-level interface that provides key-value access via Perl. AmberDB constructs a data m...
---
## Why AmberDB?
AmberDB brings together diverse capabilities within a single Perl database engine:
* **No External Database Server:** Embedded directly into the application as an in-process object.
* **Array-Based Records:** Database records are Arrays (lists) supporting JSON-like nested structures.
* **Full CRUD Operations:** Fast, direct insert, read, update, and delete methods.
docs/EN.About_AmberDB.md view on Meta::CPAN
$adb->modify_list("products", @records); # batch modify
$adb->delete_list("products", @ids); # batch delete
```
For instance, when inserting a list of one thousand records using `insert_list`, AmberDB creates the records and their indexes in a single batch process. Similarly, a massive price list can be modified within a single batch operation. This makes batc...
---
## Portable Backups
Because AmberDB is file-based, it is fully portable. It also includes safer native database tools for backup and restoration.
A database can be exported into a portable `.amberdb` archive containing the authoritative database state and schema. Integrity is verified using SHA-256, and derived indexes are rebuilt during restoration. This makes it possible to migrate or archiv...
---
## Epilogue
AmberDB provides an experience close to **PostgreSQL + Elasticsearch + Redis**, achieving this at an extremely low footprint and cost. It requires no standalone server installation and embeds directly into your application. It consumes minimal system...
AmberDB is released under the **Artistic License 2.0**. Source code, documentation, and examples are available and actively maintained on CPAN and GitHub.
* **GitHub Issues:** [https://github.com/marufcetin/amberdb/issues](https://github.com/marufcetin/amberdb/issues)
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
7. [Transaction Safety, ACID Guarantees, and Crash Recovery (Transactions)](#7-transaction-safety-acid-guarantees-and-crash-recovery-transactions)
8. [High-Throughput Batch Operations (Batch ETL & Ingestion)](#8-high-throughput-batch-operations-batch-etl--ingestion)
9. [Schema Configuration (.table & In-Memory)](#9-schema-configuration-table--in-memory)
10. [Database Group Structure (.dbase)](#10-database-group-structure-dbase)
11. [Smart Tiered (Hot / Cold Junk) Indexing](#11-smart-tiered-hot--cold-junk-indexing)
12. [Automated URL Slug Management](#12-automated-url-slug-management)
13. [Unified Shared RAM Cache (.db / .inx) & Persistent Buffer](#13-unified-shared-ram-cache-db--inx--persistent-buffer)
14. [Configuration and Deterministic Flag Management (`config`)](#14-configuration-and-deterministic-flag-management-config)
15. [Data Structures, Low-Level Table and Stream Operations](#15-data-structures-low-level-table-and-stream-operations)
16. [Faceted Search & Category Filters (Facet Engine)](#16-faceted-search--category-filters-facet-engine)
17. [User Audit Trail and Backup](#17-user-audit-trail-and-backup)
18. [Maintenance and Repair Tools (AmberDB::Tools)](#18-maintenance-and-repair-tools-amberdbtools)
19. [File Extensions Map](#19-file-extensions-map)
20. [Directory Structure](#20-directory-structure)
21. [Developer Best Practices and Recommendations](#21-developer-best-practices-and-recommendations)
22. [Full Working Example (Checkout & Stock Transaction Scenario)](#22-full-working-example-checkout--stock-transaction-scenario)
23. [Method Quick Reference Table](#23-method-quick-reference-table)
24. [Why Use AmberDB? (Comparison with SQL and SQLite)](#24-why-use-amberdb-comparison-with-sql-and-sqlite)
25. [Boundaries and Debated Topics (Physical Constraints vs. Conscious Architectural Choices)](#25-boundaries-and-debated-topics-physical-constraints-vs-conscious-architectural-choices)
---
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
---
### 5.2 Flexible & Arbitrary Record IDs (No 8-Byte Limit)
The standard mode **8-byte limit** and **strict ASCII/numeric format constraints** are relaxed in Simple Mode (`id_check` accepts arbitrary scalar keys and applies safe key sanitization):
- **Emails and Special Characters:** `user@example.com`, `api:v1:user:1005`
- **Long Tokens and UUIDs:** `sess_99999_abcdef_1234567890_extra_long_token` (up to 255 bytes)
- **Hyphenated Codes and Prefixes:** `TR-2026-08-31-INVOICE-001`
- **Unicode / Multilingual Keys:** `prod_özellik_kırmızı_xl`
- **Safe Key Sanitization:** Automatically trims leading/trailing whitespace (`trim_space`); strictly rejects NUL bytes (`\0`), control characters (`\r`, `\n`, `\t`), and references (ARRAY/HASH refs) to protect Berkeley DB C layers and CSV backup int...
- **Auto-ID Flexibility:** Custom IDs are not constrained to be strictly greater than `lastid`.
```perl
$adb->insert_id( 'sessions', 'user@example.com', 'Active', 'Chrome', time() );
my @sess = $adb->read_id( 'sessions', 'user@example.com' );
```
---
### 5.3 Data Operations (CRUD & Bulk)
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
};
if ($@) {
$adb->transact_rollback(); # token_123 is cleanly reverted from the .db file
}
```
---
### 5.6 Continuous Daily Backup Logs (`recs_back`)
Because text backup is schema-independent, **daily audit and continuous recovery streaming (`recs_back`)** is fully active in Simple Mode.
In accordance with Simple Mode's flat directory structure, no separate `backup/` or `YYYY/` subfolder is created. Every `insert_id` (`add`), `modify_id` (`edit`), and `delete_id` (`del`) operation is logged directly to **`$dbase_dir/YYYY-MM-DD.csv`**...
```text
2026-08-31 14:30:00 admin add sessions sess_token_99999 Active\x1f192.168.1.50
2026-08-31 14:31:15 admin edit sessions sess_token_99999 Closed\x1f192.168.1.50
2026-08-31 14:32:00 admin del sessions sess_token_99999
```
- To disable backup logging for volatile caches, configure `cfg => { no_backup => 1 }` or `$adb->config(no_backup => 1)`.
- Custom backup targets can be set via `path => { backup_dir => "/custom/backup/path" }`.
---
### 5.7 RAM-Disk Architecture & Caching in Simple Mode
In standard mode, AmberDB manages RAM-disk staging via schema `use_cache => 2` rules.
**In Simple Mode, RAM-disk utilization is direct and flexible:**
Since Simple Mode requires no schema files, creating a high-performance in-memory cache or session store simply involves binding a second AmberDB instance directly to the RAM-disk / tmpfs mount:
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
# 1. Persistent disk instance (For durable storage)
my $db_disk = AmberDB->new(
path => { dbase_dir => "/var/data/app/dbstore/tables" },
cfg => { simple => 1 },
);
# 2. RAM-Disk instance (Zero-latency in-memory cache/session store)
# (Linux: /dev/shm or tmpfs, Windows: ImDisk / RamDisk volume)
my $db_ramdisk = AmberDB->new(
path => { dbase_dir => "/dev/shm/amber_cache" },
cfg => { simple => 1, no_backup => 1 }, # Disable backup for pure transient cache
);
# In-memory reads and writes at nanosecond speed:
$db_ramdisk->insert_id( "sessions", $session_token, $user_id, time() );
my @sess = $db_ramdisk->read_id( "sessions", $session_token );
```
Benefits of this dual-instance design:
- In-memory tables run without disk I/O bottlenecks.
- Persistent tables remain safely on durable physical storage.
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
| **Schema Files (`.table`, `.dbase`)** | Required & Enforced | None / Schemaless |
| **Arbitrary & Long Record IDs** | 8-Byte / Strict ASCII Limits | **Completely Unrestricted** |
| **Direct CRUD (`insert_id`, `read_id`)** | $O(1)$ | **$O(1)$ (Max Throughput)** |
| **Bulk Operations (`insert_list`, etc.)** | Supported | Supported |
| **Table Scan (`read_all`)** | Binary `.inx` or Direct | Direct Streaming Scan |
| **Pagination (`limit`) & `keys_only`** | Supported | Supported |
| **In-Memory Sorting (`sort => 2`)** | Supported | Supported |
| **Field Matching (`field_fetch`)** | Indexed `.fld` $O(1)$ | Sequential Streaming Scan |
| **Word Search (`search_table`)** | Inverted Index `.src` | Collation Streaming Scan |
| **ACID Transactions (`transact_*`)** | Supported (Index Undo) | **Supported (Raw Undo)** |
| **Continuous Daily Backup (`recs_back`)** | Supported (`backup/YYYY/`) | **Supported (Same Directory `YYYY-MM-DD.csv`)** |
| **Secondary Indexes (`.inx, .fld, .src, .srt, .fac`)** | Generated & Maintained | **Disabled (Zero Index Cost)** |
| **URL Slug Mapping (`.slg`)** | Auto Generated | Disabled |
| **Audit Logs (`.aut`) & Archive (`.del`)** | Schema-Driven | Disabled |
| **Directory Hierarchy** | `tables/`, `schema/`, `backup/`, etc. | **Flat Single Directory (`$dbase_dir/<table_name>.db`)** |
| **Secondary Indexes (`.inx, .fld, .src, .srt, .fac`)** | Generated & Maintained | **Disabled (Zero Index Cost)** |
| **URL Slug Mapping (`.slg`)** | Auto Generated | Disabled |
| **Audit Logs (`.aut`) & Archive (`.del`)** | Schema-Driven | Disabled |
| **Directory Hierarchy** | `tables/`, `schema/`, `backup/`, etc. | **Flat Single Directory (`$dbase_dir/<table_name>.db`)** |
---
## 6. Indexing and Search Engine
AmberDB maintains structured binary index files based on the schema configuration.
### 6.1 Index Types
| Extension | Index Type | Description |
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
### 9.1 Database and Table Directory Layout
AmberDB stores tables, indexes, and schema definitions in dedicated physical directories under the configured `dbstore` root:
| Directory | Purpose |
|---|---|
| `dbstore/tables/` | Base data (`.db`) and binary indexes (`.inx`, `.fld`, `.src`, `.fac`, `.srt`, `.slg`) |
| `dbstore/schema/` | Schema files (`.table`) and group configs (`.dbase`) |
| `dbstore/conf/` | Plain-text `.conf` configuration and property files |
| `dbstore/backup/` | Daily CSV audit backups (`dbgun/YYYYMMDD/`) |
| `dbstore/cache/` | **Unified Shared RAM-Disk (ImDisk/tmpfs) Root:** |
| `dbstore/cache/tables/` | Mirrored hot `.db` and `.inx` tables in RAM for `use_cache => 1 & 2` |
| `dbstore/cache/conf/` | Compiled high-speed config cache (`*.pl` hash references) |
| `dbstore/cache/schema/` | Cached / pre-compiled table schemas in RAM (`*.table`, `*.dbase`) |
| `dbstore/cache/lock/` | Process and table-level `flock` lock files in RAM (`*.lock`) |
| `dbstore/cache/pids/` | Process lock files and login error state logs (`*.pid`, `*.error`) |
> [!IMPORTANT]
> **Version 5.21.0 Migration Notice:** The only manual action required when upgrading existing projects is to rename your database directory's `dbstore/scheme/` folder to **`dbstore/schema/`**. All programmatic path resolutions and API calls are auto...
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
| `use_counter` | `0 / 1` | `0` | `usecnt` | Enables automated hit/view read counters in `.cnt` files. |
| `parent_table` | `string` | `""` | — | Parent table name for vertical partitioning (child table shares the same primary ID). |
| `force` | `0 / 1` | `0` | — | When `1`, `insert_id` overwrites existing records rather than failing (Replace mode). |
| `min_char` | `integer` | `2` | `minchar` | Minimum word length for full-text search indexing (1, 2, or 3). |
| `stop_word` | `string` | `""` | `nextkey` | Stop-words excluded from full-text search indexing (e.g., `"the and for with"`). |
| `repeat_ids` | `integer` | `undef` | — | Target block number where extracted child item IDs are consolidated. |
| `repeat_start` | `integer` | `undef` | — | Starting block index for dynamic repeating child rows (order items, cart lines). |
| `view_block` | `ARRAY` | `[]` | — | Priority block numbers displayed in UI / CMS listing views. |
| `use_menu` | `0 / 1` | `1` | — | Controls display of the table in admin panel navigation menus. |
| `no_transact` | `0 / 1` | `0` | — | Exempts table from transactional rollback error propagation. |
| `no_backup` | `0 / 1` | `0` | — | Disables daily CSV user audit logging for this table. |
---
### 9.7 Block (Field) Definitions, 8 Core Field Types, UI Inputs, and Validation Reference
Each block definition inside the `blocks` array supports the following attributes:
#### 9.7.1 Core Block Attributes
| Attribute | Type | Description | Example |
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
---
## 14. Configuration and Deterministic Flag Management (`config`)
Runtime behavior can be tuned and safely configured via the `$adb->config()` method:
```perl
# Bulk or single configuration assignment (Recommended)
$adb->config(
no_write => 1, # Read-only maintenance mode: block all writes
no_backup => 1, # Disable daily CSV audit logging for all tables
simple => 1, # Direct unindexed mode: bypasses secondary index generation
keys_only => 1, # read_all returns IDs only
cache_size => '1024M', # RAM-Disk / tmpfs cache size (Default: 512M)
);
# Single scalar getter:
my $no_write = $adb->config('no_write');
# Bulk getter (returns a safe shallow copy):
my $cfg = $adb->config();
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
```perl
# Retrieve user audit history as formatted HTML
my $history_html = $adb->auth_view("catalog_product", 5001);
print $history_html;
# Output:
# add 2026-08-14 10:15 admin_user
# edit 2026-08-14 11:30 editor_user
```
### 17.2 Continuous Recovery Stream (`YYYY-MM-DD.csv`)
AmberDB automatically appends every `insert`, `modify`, and `delete` operation into a clean, chronological time-series stream in `backup/YYYY/YYYY-MM-DD.csv`.
Each entry is tab-separated (`\t`) using the standard format:
`[Timestamp] \t [User] \t [Action] \t [Table] \t [Record ID] \t [Packed Values]`
To disable this backup stream:
* **In Table Schema (Per-Table):** Add `no_backup => 1` in the table schema to disable logging for that specific table only.
* **Globally via Config (All Tables):** Set `$adb->config(no_backup => 1);` to disable logging across all tables.
### 17.3 Native Database Archive (`.amberdb` Dump & Restore)
AmberDB packages all schemas (`schema/*.table`, `schema/*.dbase`) and authoritative data files (`tables/*.db`, `tables/*.del`, `tables/*.aut`, `tables/*.cnt`) alongside cryptographically verified SHA-256 checksums in a single compressed, portable **`...
Derived index files (`.inx`, `.src`, `.fld`, `.fac`, `.srt`) are intentionally excluded to keep archives compact and ensure future-proof portability; `restore` deterministically rebuilds all indexes via `set_index`.
```perl
use AmberDB;
use AmberDB::Tools;
my $adb = AmberDB->new(path => { dbase_dir => "./dbstore" });
my $tools = AmberDB::Tools->new($adb);
# 1. Create full database backup archive (.amberdb)
my $archive = $tools->dump();
# Output: dbstore/backup/2026/amberdb_2026-08-28_180000.amberdb
# 2. Export specific tables as a focused snapshot archive
$tools->dump(
file => "backup/2026/catalog_backup.amberdb",
tables => ["catalog_product", "catalog_category"]
);
# 3. Restore database archive and automatically rebuild all indexes
$tools->restore(
file => "backup/2026/catalog_backup.amberdb",
force => 1, # Overwrite confirmation for non-empty target directories
reindex => 1 # Automatically reconstruct binary indexes from source data
);
```
#### CLI Command-Line Utility (`bin/amberdb_backup.pl`)
```bash
# Dump entire database to default archive
perl bin/amberdb_backup.pl --dump --file backup/2026/full_backup.amberdb
# Dump specific tables only
perl bin/amberdb_backup.pl --dump --tables products,orders
# Restore database archive with integrity checks and automatic reindexing
perl bin/amberdb_backup.pl --restore --file backup/2026/full_backup.amberdb --force
```
---
## 18. Maintenance and Repair Tools (AmberDB::Tools)
`AmberDB::Tools` provides utilities for reindexing, table vacuuming, and data migration:
```perl
use AmberDB;
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
│ ├── catalog_product_10.srt ↠Price sort index
│ ├── catalog_product.fac ↠Facet index
│ ├── catalog_product_0.slg ↠ID → Slug Map
│ ├── catalog_product_1.slg ↠Slug → ID Map
│ ├── catalog_product.aut ↠Audit trail
│ └── catalog_product.del ↠Soft-deleted records
├── cache/ ↠Shared RAM-Disk Cache Files
├── buffer/ ↠Transient Disk Buffer / Staging Files
├── txn/ ↠Active Transaction Journals
├── pids/ ↠Lock Files
└── backup/ ↠Daily CSV Backups
```
---
## 21. Developer Best Practices and Recommendations
1. **Use `insert_list` for Bulk Ingestion:** When adding hundreds of records, use `insert_list` instead of looping over `insert_id`. Batch mode writes all records in a single file session and rebuilds indexes in one pass.
2. **Wrap Multi-Step Writes in `transact_start`:** Always wrap inventory deductions, checkout sequences, or multi-table balance updates inside transactions.
3. **Index Only Required Fields:** Only assign fields to `match_block` or `search_block` if they are actively queried to minimize disk write overhead.
4. **Always Handle Pagination Return Signatures Correctly:** When passing `$limit > 0` to `read_all`, `field_fetch`, or `search_table`, remember that the first returned value is `$total_count` integer. Never unpack into a single array (`my @records =...
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
### 24.4 Direct Inverted Key Lookups (Zero Query Planner Overhead)
In SQL, running `SELECT id FROM orders WHERE customer_id = 'A'` requires parsing, query plan evaluation, cost optimization, and virtual machine execution.
In AmberDB, `field_fetch` is a direct hash key lookup on Berkeley DB returning packed binary buffers. Query planning overhead is zero.
### 24.5 Built-in Lifecycle and Domain Features
- **Automatic URL Slug Management:** When titles or categories change, clean slugs like `/products/laptop-pro-m3` and conflict resolution suffixes are generated automatically.
- **Audit Trails (.aut):** User identity, action type (`add`, `edit`, `del`), and timestamps are recorded without extra tables.
- **Safe Soft Deletion (.del):** Deleted records are archived safely and can be inspected or restored.
- **Zero Configuration & Portability:** Copying the database directory creates a complete, standalone backup that can run on any Perl-enabled system.
---
## 25. Boundaries and Debated Topics (Physical Constraints vs. Conscious Architectural Choices)
In database design, every architectural decision serves a specific optimization goal. Certain characteristics that developers coming from traditional SQL environments might initially perceive as "constraints" or "omissions" are, in fact, **deliberate...
### 25.1 Physical and Environmental Boundaries (Out-of-Scope Scenarios)
The following scenarios lie outside the intended operational scope of an embedded, file-based database engine like AmberDB:
docs/TR.AmberDB_Veritabani_Sistemi.md view on Meta::CPAN
$adb->transact_rollback(); # token_123 kaydı .db dosyasından tamamen silinir
}
```
---
### 5.6 Günlük Sürekli Yedekleme Günlükleri (Daily Backup Logs - `recs_back`)
Basit mod şemadan bağımsız olduğundan, **metin tabanlı sürekli denetim ve kurtarma akışı (`recs_back`)** basit modda da varsayılan olarak devrededir.
Basit modun düz dizin yapısı gereği ayrı bir `backup/` veya `YYYY/` alt klasörü oluşturulmaz; yapılan her `insert_id` (`add`), `modify_id` (`edit`) ve `delete_id` (`del`) işlemi doğrudan tablolarla aynı dizinde bulunan **`$dbase_dir/YYYY-...
```text
2026-08-31 14:30:00 admin add sessions sess_token_99999 Aktif\x1f192.168.1.50
2026-08-31 14:31:15 admin edit sessions sess_token_99999 Kapali\x1f192.168.1.50
2026-08-31 14:32:00 admin del sessions sess_token_99999
```
- İstenirse `cfg => { no_backup => 1 }` veya `$adb->config(no_backup => 1)` ile yedekleme günlükleri devre dışı bırakılabilir.
- Özel bir yedek dizini tanımlanmak istendiğinde `path => { backup_dir => "/harici/yedek/yolu" }` verilebilir.
---
### 5.7 Basit Modda RAM-Disk Mimarisi ve Önbellekleme
AmberDB standart modda RAM-disk önbelleğini şemadaki `use_cache => 2` kuralı ile `dbstore/cache` alt dizinine kopyalayarak yönetir.
**Basit modda ise RAM-disk kullanımı çok daha doğrudan ve esnektir:**
Basit mod şemaya ihtiyaç duymadığından, yüksek performanslı bir bellek önbelleği / geçici oturum deposu oluşturmak için ikinci bir AmberDB basit nesnesi doğrudan RAM-disk yoluna bağlanır:
docs/TR.AmberDB_Veritabani_Sistemi.md view on Meta::CPAN
# 1. Kalıcı disk nesnesi (Kalıcı veriler için)
my $db_kalici = AmberDB->new(
path => { dbase_dir => "/var/data/eticaret/dbstore/tables" },
cfg => { simple => 1 },
);
# 2. RAM-Disk nesnesi (Sıfır gecikmeli hızlı oturum/önbellek tabloları için)
# (Linux: /dev/shm veya tmpfs, Windows: ImDisk / RamDisk sürücüsü)
my $db_ramdisk = AmberDB->new(
path => { dbase_dir => "/dev/shm/amber_cache" },
cfg => { simple => 1, no_backup => 1 }, # Önbellek için yedekleme kapatılabilir
);
# RAM üzerinde nanosaniye hızında oturum okuma/yazma:
$db_ramdisk->insert_id( "oturumlar", $session_token, $user_id, time() );
my @oturum = $db_ramdisk->read_id( "oturumlar", $session_token );
```
Bu çift nesneli mimari sayesinde:
- RAM-disk üzerindeki tablolar disk I/O darboğazına takılmadan bellek hızında çalışır.
- Kalıcı tablolar ana depolama alanında güvenle tutulmaya devam eder.
docs/TR.AmberDB_Veritabani_Sistemi.md view on Meta::CPAN
| **Şema Dosyaları (`.table`, `.dbase`)** | Zorunlu / Kullanılır | Yok / Şemasız |
| **Özel ve Uzun ID'ler (UUID, E-posta, vb.)** | 8 Bayt / ASCII Kısıtlı | **Tamamen Serbest** |
| **Tekil CRUD (`insert_id`, `read_id`, vb.)** | $O(1)$ | **$O(1)$ (Maksimum Hız)** |
| **Toplu Ä°ÅŸlemler (`insert_list`, vb.)** | Desteklenir | Desteklenir |
| **Tüm Tablo Okuma (`read_all`)** | `.inx` veya doğrudan | Doğrudan Dosya Taraması |
| **Sayfalama (`start`/`limit`) ve `keys_only`** | Desteklenir | Desteklenir |
| **Bellek İçi Sıralama (`sort => 2`)** | Desteklenir | Desteklenir |
| **Alana Göre Filtre (`field_fetch`)** | `.fld` İndeksli $O(1)$ | Sıralı Dosya Taraması |
| **Tam Metin Arama (`search_table`)** | `.src` Ters İndeksli | Sıralı Dosya Taraması (Türkçe Normalizasyonlu) |
| **ACID Ä°ÅŸlemler (`transact_*`)** | Desteklenir (Ä°ndeks Geri Alma Dahil) | **Desteklenir (Ham Veri Geri Alma)** |
| **Sürekli Yedekleme Akışı (`recs_back`)** | Desteklenir (`backup/YYYY/`) | **Desteklenir (Aynı Dizinde `YYYY-MM-DD.csv`)** |
| **İkincil İndeksler (`.inx, .fld, .src, .srt, .fac`)** | Oluşturulur ve Güncellenir | **Oluşturulmaz (Sıfır İndeks Maliyeti)** |
| **URL Slug Rewrite (`.slg`)** | Otomatik Üretilir | Devre Dışı |
| **Denetim İzi (`.aut`) ve Arşiv (`.del`)** | Şema Kuralına Göre Tutulur | Devre Dışı |
| **Dizin Yapısı** | `tables/`, `schema/`, `backup/` vb. | **Düz Kök Dizin (`$dbase_dir/<tablo>.db`)** |
| **İkincil İndeksler (`.inx, .fld, .src, .srt, .fac`)** | Oluşturulur ve Güncellenir | **Oluşturulmaz (Sıfır İndeks Maliyeti)** |
| **URL Slug Rewrite (`.slg`)** | Otomatik Üretilir | Devre Dışı |
| **Denetim İzi (`.aut`) ve Arşiv (`.del`)** | Şema Kuralına Göre Tutulur | Devre Dışı |
| **Dizin Yapısı** | `tables/`, `schema/`, `backup/` vb. | **Düz Kök Dizin (`$dbase_dir/<tablo>.db`)** |
---
## 6. İndeksleme ve Arama Mekanizması
AmberDB, tablolara hızlı erişim sağlamak için veriyi şemada tanımlanan kurallara göre ikili (binary) indeks dosyalarına yazar.
### 6.1 İndeks Türleri
| Dosya Uzantısı | İndeks Türü | Açıklama |
docs/TR.AmberDB_Veritabani_Sistemi.md view on Meta::CPAN
### 9.1 Veritabanı ve Tablo Dizin Yapısı
AmberDB tabloları, şemaları ve geçici/kalıcı dosyaları, belirlenen `dbstore` ana veri dizini altında fiziksel klasörlere ayrılarak saklanır:
| Dizin | Görevi |
|---|---|
| `dbstore/tables/` | Kalıcı `.db` ana veri, `.inx` kayıt indeksi, `.fld` eşleştirme, `.src` arama, `.fac` facet, `.srt` sıralama ve `.slg` slug dosyaları |
| `dbstore/schema/` | Kalıcı `.table` tablo şemaları ve `.dbase` grup yapılandırma dosyaları |
| `dbstore/conf/` | Kalıcı `.conf` düz metin ayar ve konfigürasyon dosyaları |
| `dbstore/backup/` | Günlük CSV denetim yedekleri (`dbgun/YYYYMMDD/`) |
| `dbstore/cache/` | **Birleşik RAM-Disk (ImDisk/tmpfs) Kök Dizini:** |
| `dbstore/cache/tables/` | `use_cache => 1 & 2` için RAM'e aynalanmış sıcak `.db` ve `.inx` tabloları |
| `dbstore/cache/conf/` | Derlenmiş hızlı yapılandırma önbelleği (`*.pl` hash referansları) |
| `dbstore/cache/schema/` | RAM'de önbelleğe alınmış / derlenmiş tablo şemaları (`*.table`, `*.dbase`) |
| `dbstore/cache/lock/` | Yalnızca RAM'de yaşayan kayıt ve tablo seviyesi `flock` kilitleri (`*.lock`) |
| `dbstore/cache/pids/` | Yalnızca RAM'de yaşayan süreç kilitleri, login attempt hataları (`*.pid`, `*.error`) |
> [!IMPORTANT]
> **Sürüm 5.21.0 Geçiş Uyarısı:** Eski projelerden yükseltme yaparken yapmanız gereken tek fiziksel işlem; veritabanı dizininizdeki `dbstore/scheme/` klasörünün adını **`dbstore/schema/`** olarak yeniden adlandırmaktır. Kod ve API ta...
docs/TR.AmberDB_Veritabani_Sistemi.md view on Meta::CPAN
---
## 14. Yapılandırma ve Deterministik Bayrak Yönetimi (`config`)
AmberDB'nin çalışma modunu değiştirmek ve yapılandırma bayraklarını güvenli bir şekilde yönetmek için `$adb->config()` metodu kullanılır:
```perl
# Toplu veya tekli yapılandırma ataması (Önerilen)
$adb->config(
no_write => 1, # Bakım modu: Tüm yazma işlemlerini engelle
no_backup => 1, # Tüm tablolar için günlük CSV denetim yedeklerini kapat
simple => 1, # İndekssiz doğrudan yazım modu (İkincil indeksler devre dışı bırakılır)
keys_only => 1, # read_all çağrılarında sadece ID'leri döndür
cache_size => '1024M', # RAM-Disk / tmpfs önbellek boyutu (Varsayılan: 512M)
);
# Tekil okuma:
my $no_write = $adb->config('no_write');
# Toplu okuma (Güvenli kopya döner):
my $cfg = $adb->config();
docs/TR.AmberDB_Veritabani_Sistemi.md view on Meta::CPAN
```perl
# Bir kaydın kimler tarafından ne zaman değiştirildiğini HTML olarak alma
my $gecmis_html = $adb->auth_view("catalog_product", 5001);
print $gecmis_html;
# Çıktı:
# add 2026-08-14 10:15 admin_maruf
# edit 2026-08-14 11:30 editor_ali
```
### 17.2 Sürekli Değişiklik Akışı (Continuous Recovery Stream — `YYYY-MM-DD.csv`)
AmberDB yapılan her `insert`, `modify` ve `delete` işlemini kronolojik zaman-serisi olarak `backup/YYYY/YYYY-MM-DD.csv` dosyasına otomatik olarak ekler (append-only).
Her satır tab ayrılmış (`\t`) olarak şu sütun yapısında yazılır:
`[Zaman Damgası] \t [Kullanıcı] \t [İşlem] \t [Tablo] \t [Kayıt ID] \t [Paketlenmiş Değerler]`
Bu akışı devre dışı bırakmak için:
* **Tablo Şemasında (Tablo Bazlı):** Şema dosyasına `no_backup => 1` eklenirse sadece o tablo için yedekleme kapatılır.
* **Genel Düzeyde (Tüm Tablolar):** `$adb->config(no_backup => 1);` tanımlanırsa tüm tablolar için yedekleme kapatılır.
### 17.3 Native Veritabanı Arşivi (`.amberdb` Dump & Restore)
AmberDB, tüm şemaları (`schema/*.table`, `schema/*.dbase`) ve otoriter veri dosyalarını (`tables/*.db`, `tables/*.del`, `tables/*.aut`, `tables/*.cnt`) SHA-256 doğrulama özetleriyle birlikte fiziksel dizin yapısıyla birebir örtüşen sıkı...
Türetilmiş indeks dosyaları (`.inx`, `.src`, `.fld`, `.fac`, `.srt`) boyuttan tasarruf etmek için arşiv içine konmaz; `restore` esnasında şema kurallarına göre `set_index` ile deterministik olarak sıfırdan üretilir.
```perl
use AmberDB;
use AmberDB::Tools;
my $adb = AmberDB->new(path => { dbase_dir => "./dbstore" });
my $tools = AmberDB::Tools->new($adb);
# 1. Tüm veritabanının tam yedeğini alma (.amberdb)
my $arsiv = $tools->dump();
# Çıktı: dbstore/backup/2026/amberdb_2026-08-28_180000.amberdb
# 2. Belirli tabloların snapshot yedeğini alma
$tools->dump(
file => "backup/2026/katalog_yedek.amberdb",
tables => ["catalog_product", "catalog_category"]
);
# 3. Yedeği geri yükleme ve tüm indeksleri otomatik inşa etme
$tools->restore(
file => "backup/2026/katalog_yedek.amberdb",
force => 1, # Var olan tabloların üzerine yazma izni
reindex => 1 # İndeksleri sıfırdan üret
);
```
#### CLI Komut Satırı Aracı (`bin/amberdb_backup.pl`)
```bash
# Veritabanını yedekleme
perl bin/amberdb_backup.pl --dump --file backup/2026/tam_yedek.amberdb
# Belirli tabloları yedekleme
perl bin/amberdb_backup.pl --dump --tables products,orders
# Yedeği güvenli şekilde geri yükleme
perl bin/amberdb_backup.pl --restore --file backup/2026/tam_yedek.amberdb --force
```
---
## 18. Bakım ve Onarım Araçları (AmberDB::Tools)
Veritabanı indekslerini sıfırdan yeniden oluşturmak, veri doğrulaması yapmak veya tabloları optimize etmek için `AmberDB::Tools` kullanılır:
```perl
use AmberDB;
docs/TR.AmberDB_Veritabani_Sistemi.md view on Meta::CPAN
│ ├── catalog_product_10.srt ↠Fiyat sıralama indeksi
│ ├── catalog_product.fac ↠Facet indeksi
│ ├── catalog_product_0.slg ↠ID → Slug Haritası
│ ├── catalog_product_1.slg ↠Slug → ID Haritası
│ ├── catalog_product.aut ↠Denetim logu
│ └── catalog_product.del ↠Silinen kayıtlar
├── cache/ ↠RAM-Disk Paylaşımlı Önbellek Dosyaları
├── buffer/ ↠Geçici Disk Buffer / Staging Dosyaları
├── txn/ ↠Aktif Transaction Günlükleri
├── pids/ ↠Dosya ve Kayıt Kilitleri
└── backup/ ↠Günlük CSV Yedekleri
```
---
## 21. GeliÅŸtirici Tavsiyeleri ve En Ä°yi Pratikler
1. **Toplu Veri Girişinde `insert_list` Kullanın:** Yüzlerce kaydı tek tek döngüde `insert_id` ile eklemek yerine tek seferde `insert_list` ile ekleyin; disk I/O ve indeksleme süresi 10 kat hızlanacaktır.
2. **Kritik İş Mantıklarında `transact_start` Kullanın:** Stok düşme, bakiye güncelleme ve sipariş onaylama gibi adımları mutlaka transaction bloğu içine alın.
3. **Şemalarda Gereksiz Blokları İndekslemeyin:** Yalnızca filtrelenecek alanları `match_block`, aranacak alanları `search_block` olarak tanımlayın.
4. **Sayfalama Dönen Değer İmzasını Doğru Karşılayın:** `read_all`, `field_fetch` ve `search_table` metotlarında `$limit > 0` verildiğinde dönen listenin ilk elemanının `$toplam` tamsayısı olduğunu unutmayın. Asla `my @kayitlar = $a...
lib/AmberDB.pm view on Meta::CPAN
if ($is_txn) {
my $new_raw;
$self->{_db}->{$file_path}->get( $rid, $new_raw );
$self->_txn_log( $tableid, "add", $rid, $new_raw, "" );
}
$self->table_close($file_path);
unless ($is_txn) { $self->flock_close( $tableid, $rid ); }
# for index actions and backup
@record = ( $rid, @record );
# text backup record.
$self->recs_back( "add", $tableid, \@record );
( $self->config('simple') || ( $table_info && $table_info->{use_simple} ) ) and return $rid;
# update .inx / .jinx and secondary indexes
my @batch = ( \@record );
if ( $table_info->{use_junk} && $self->junk_rules( $table_info, @record ) ) {
$self->junk_records_add( $table_path, $table_info, $tableid, [$rid] );
$self->junk_search_add( $table_path, $table_info, $tableid, \@batch );
$self->junk_match_add( $table_path, $table_info, \@batch );
lib/AmberDB.pm view on Meta::CPAN
# Phase 2: bulk index updates (each file is opened once)
$self->search_add( $table_path, $table_info, $tableid, \@batch );
$self->match_add( $table_path, $table_info, \@batch );
$self->facet_add( $table_path, $table_info, \@batch );
$self->sort_add( $table_path, $table_info, \@batch );
}
$self->unique_add( $table_path, $table_info, \@batch );
# Per-record operations: slug, auth, backup
foreach my $rec (@batch) {
$self->set_slug( $tableid, $rec, 1 );
$self->auth_write( $tableid, $table_path, "add", $rec->[0] );
$self->recs_back( "add", $tableid, $rec );
}
return \%statu;
}
# Replace the DB record with new data.
lib/AmberDB.pm view on Meta::CPAN
}
$self->table_close($file_path);
unless ($is_txn) { $self->flock_close( $tableid, $rid ); }
# Cache invalidate
$self->cache_delete($tableid, $rid);
my @new_rec = ( $rid, @record );
# text backup record.
$self->recs_back( "edit", $tableid, \@new_rec )
or cluck "[DB_TIE] Backup error (edit). $tableid\n";
( $self->config('simple') || ( $table_info && $table_info->{use_simple} ) ) and return $rid;
my @old_rec = ( $rid, $self->db_decode($old_record) );
# Index update (search, match, facet, sort)
my @pairs = ( [ $rid, \@old_rec, \@new_rec ] );
if ( $table_info->{use_junk} ) {
lib/AmberDB.pm view on Meta::CPAN
$self->junk_transition( $table_path, $table_info, $tableid, \@pairs );
}
else {
$self->search_modify( $table_path, $table_info, $tableid, \@pairs );
$self->match_modify( $table_path, $table_info, \@pairs );
$self->facet_modify( $table_path, $table_info, \@pairs );
}
$self->sort_modify( $table_path, $table_info, \@pairs );
$self->unique_modify( $table_path, $table_info, \@pairs );
# Per-record operations: slug, auth, backup
foreach my $pair (@pairs) {
my ( $rid, $old_rec, $new_rec ) = @$pair;
if ( $table_info->{slug_block} ) {
my $slug_map = $self->get_slug( $tableid, 0, $rid );
my $old_slug = $slug_map->{$rid};
my $new_slug = $self->set_slug( $tableid, $new_rec, 1 );
if ( $old_slug && $new_slug && $old_slug ne $new_slug ) {
if ( $self->table_write("${table_path}_1.slg") ) {
$self->recs_del( "${table_path}_1.slg", $old_slug );
lib/AmberDB.pm view on Meta::CPAN
if ($is_txn) {
$self->_txn_log( $tableid, "del", $rid, "", $record );
}
$self->table_close($file_path);
unless ($is_txn) { $self->flock_close( $tableid, $rid ); }
# Cache invalidate
$self->cache_delete($tableid, $rid);
# Text backup record
$self->recs_back( "del", $tableid, [ $rid, "" ] )
or cluck "[DB_TIE] Backup error (del). $tableid\n";
# Move to archive if keep_deleted enabled
if ( $table_info->{keep_deleted} ) {
( $self->table_write($del_path)
and $self->recs_put( $del_path, [ $rid, $record ] )
and $self->table_close($del_path) )
or cluck "[DB_TIE] $del_path can't open.\n";
}
lib/AmberDB.pm view on Meta::CPAN
else {
$self->records_del( $table_path, $table_info, \@del_rids, $tableid );
$self->search_del( $table_path, $table_info, $tableid, \@batch );
$self->match_del( $table_path, $table_info, \@batch );
$self->facet_del( $table_path, $table_info, \@batch );
$self->sort_del( $table_path, $table_info, \@batch );
}
$self->unique_del( $table_path, $table_info, \@batch );
# Per-record operations: slug, auth, backup
foreach my $rec (@batch) {
my $rid = $rec->[0];
if ( $table_info->{slug_block} ) {
my $slug_map = $self->get_slug( $tableid, 0, $rid );
my $slug = $slug_map->{$rid};
if ($slug) {
if ( $self->table_write("${table_path}_0.slg") ) {
$self->recs_del( "${table_path}_0.slg", $rid );
$self->table_close("${table_path}_0.slg");
}
lib/AmberDB.pm view on Meta::CPAN
return unless $db;
my $k = $self->utf_encode("$key");
my $ret = $db->del($k);
warn "[DB_TIE] $table_path can't del key $k.\n" if $ret > 0;
return $ret == 0 ? 1 : 0;
}
# Writes add|edit|del operation to daily CSV backup audit stream (backup/YYYY/YYYY-MM-DD.csv).
# Exits silently if no_backup is set (globally or in table schema).
# my $ok = $adb->recs_back("add|edit|del", $tableid, @records);
# ------------------------------------------------
sub recs_back {
my ( $self, $action, $tableid, @records ) = @_;
( $action and $tableid and scalar @records ) or return;
# Global config check: disables backup for all tables
return if $self->config('no_backup');
# Table schema check: no_backup => 1 in table schema
my $table_info = $self->table_info($tableid);
return if $table_info->{no_backup};
my $user = $self->config('user') || 'system';
$tableid =~ s/[:\/\\]/--/g;
my $backup_base = $self->path('backup_dir')
|| ( $self->path('dbase_dir') ? $self->path('dbase_dir') . "/backup" : "backup" );
my $year = ( $self->{date} && $self->{date}->{year} ) ? $self->{date}->{year} : (localtime)[5] + 1900;
my $month = ( $self->{date} && $self->{date}->{month} ) ? $self->{date}->{month} : sprintf( "%02d", (localtime)[4] + 1 );
my $day = ( $self->{date} && $self->{date}->{day} ) ? $self->{date}->{day} : sprintf( "%02d", (localtime)[3] );
my $date_iso = "$year-$month-$day";
my $time_str = ( $self->{date} && $self->{date}->{str} ) ? $self->{date}->{str} : "$date_iso " . sprintf( "%02d:%02d:%02d", (localtime)[2], (localtime)[1], (localtime)[0] );
my $backup_file;
if ( $self->config('simple') ) {
$backup_file = "$backup_base/$date_iso.csv";
}
else {
my $year_dir = "$backup_base/$year";
unless ( -d $year_dir ) {
require File::Path;
File::Path::make_path($year_dir);
}
$backup_file = "$year_dir/$date_iso.csv";
}
open my $YAZ, ">>:encoding(UTF-8)", $backup_file
or do {
cluck "[DB_BACKUP] Cannot open backup file $backup_file: $!\n";
return;
};
foreach my $record (@records) {
ref($record) eq "ARRAY" or $record = [$record];
my $rid = $record->[0];
my $bac_val = $self->db_encode( @{$record}[ 1 .. $#$record ] );
print $YAZ "$time_str\t$user\t$action\t$tableid\t$rid\t$bac_val\n";
}
close $YAZ;
lib/AmberDB.pm view on Meta::CPAN
=head2 Key Characteristics of Simple Mode
=over 4
=item * B<Arbitrary & Flexible Keys:> The 8-byte ASCII limit and numeric constraints are bypassed. Keys can be emails (C<user@example.com>), UUIDs, long tokens, or Unicode/multilingual strings.
=item * B<Flat Directory Structure:> All tables reside directly under C<dbase_dir> (e.g. C<$dbase_dir/table.db>). No C<tables/> or C<schema/> subfolders are required.
=item * B<Rich Nested Structures:> Records can store nested array and hash references (ARRAY/HASH) directly.
=item * B<Continuous Daily Backup Logs:> Text-based continuous daily WAL/CSV logs (C<recs_back>) automatically record all C<add>, C<edit>, and C<del> operations directly into C<$dbase_dir/YYYY-MM-DD.csv> alongside database tables (can be silenced wit...
=item * B<ACID Transactions:> Full multi-table transaction support with atomic rollback (restoring raw records in the C<.db> file).
=item * B<Streaming Queries & Sorting:> Methods like C<read_all>, C<field_fetch>, and C<search_table> operate via direct sequential streaming scans with full support for pagination (C<start>/C<limit>), C<keys_only>, and in-memory sorting.
=item * B<Zero-Latency RAM-Disk Caching:> Simple mode instances can be initialized directly on RAM-disk / tmpfs mount points (e.g. C</dev/shm/cache>) to provide nanosecond-speed transient session and cache stores.
=back
=head2 Simple Mode Example
lib/AmberDB.pm view on Meta::CPAN
# 3. Read record (O(1))
my @sess = $adb->read_id('sessions', 'user@example.com');
# 4. Search and filter without indexes
my ($count, @active) = $adb->field_fetch('sessions', 1, 'Active', 0, 10);
# 5. Dual-instance RAM-Disk architecture
my $ram_db = AmberDB->new(
path => { dbase_dir => "/dev/shm/amber_cache" },
cfg => { simple => 1, no_backup => 1 },
);
$ram_db->insert_id('tokens', $token_id, $user_id, time());
=head1 METHODS
=head2 new(%options)
Instantiates a new C<AmberDB> object.
=head2 config([$key], [%options])
lib/AmberDB.pm view on Meta::CPAN
my @all_ids = $adb->field_fetch("products", 1, "5", keys_only => 1);
my ($total_count, @ids) = $adb->field_fetch("products", 1, "5", 0, 20, keys_only => 1);
# Tiered Junk query mode
my @active = $adb->field_fetch("products", 1, "5", jnkmode => 'A'); # Only Active records
C<field_fetch> uses the C<match_block> definition in the schema and accesses inverted match index files (C<.fld>), providing $O(1)$ average-time lookup per indexed key (total retrieval cost scales with the number of requested values and matching reco...
=head2 search_table($table_id, $query, [$start], [$limit], [$mode], [%options])
It performs searches matching query terms using the full-text C<.src> index (or a sorted table scan backup method if unindexed). C<search_table> uses the C<AmberDB::Locale> module. It features advanced language normalization according to the selected...
B<IMPORTANT (Return Signature Convention):>
When C<$limit> is passed and C<E<gt> 0> (paginated), C<search_table> returns C<($total_count, @records)> where the first scalar is the total matching count integer. When C<$limit> is omitted or C<0> (unpaginated), it returns C<@records> directly. Unp...
# 1. Unpaginated (returns array of record arrayrefs directly)
my @records = $adb->search_table("catalog_product", "kablosuz kulaklık");
my @sorted_records = $adb->search_table("catalog_product", "kulaklık", 0, 0, sort => -5);
# 2. Paginated (first element is total matching count integer)
my ($total_count, @search) = $adb->search_table( "catalog_product", "kulaklık", 0, 20 );
lib/AmberDB/Base.pm view on Meta::CPAN
$table_path =~ s/\//-/g;
my $table_str = "";
# Scalar keys
my @scalar_keys = qw(
name record_index keep_deleted log_owner parent_table
use_menu use_simple force use_cache use_alias
use_counter use_facet stop_word min_char
use_junk cache_ttl repeat_ids repeat_start
no_transact no_backup
);
foreach my $key (@scalar_keys) {
next unless exists $tbl->{$key};
next unless defined $tbl->{$key} && $tbl->{$key} ne "";
( my $val = $tbl->{$key} ) =~ s/"/\\"/g;
$table_str .= "\t$key => \"$val\",\n";
}
# Array keys
my @array_keys = qw(
lib/AmberDB/Base.pm view on Meta::CPAN
# $self->set_datadir("/path/to/dbase")
# ------------------------------------------------
sub set_datadir {
my ( $self, $dbase_dir ) = @_;
$dbase_dir or return;
# declarations
my @dirs = qw(
dbase_dir table_dir schema_dir backup_dir
cache_dir table_cache schema_cache lock_cache buffer_dir txn_dir
);
foreach my $dir (@dirs) {
$self->{_path}->{$dir} //= "";
}
$self->{_path}->{dbase_dir} = $dbase_dir;
# db_ext tanımlı ve "db" değil ise simple moduna al
if ( defined $self->config('db_ext') && $self->config('db_ext') ne "db" ) {
$self->config( simple => 1 );
}
# do not proceed if simple mode (simple modunda dbstore alt dizinleri yoktur, tum yollar dbase_dir ile esitlenir)
if ( $self->config('simple') ) {
$self->{_path}->{table_dir} = $dbase_dir;
$self->{_path}->{schema_dir} = $dbase_dir;
$self->{_path}->{backup_dir} = $dbase_dir;
$self->{_path}->{buffer_dir} = $dbase_dir;
$self->{_path}->{txn_dir} = $dbase_dir;
$self->{_path}->{cache_dir} = $dbase_dir;
$self->{_path}->{table_cache} = $dbase_dir;
$self->{_path}->{schema_cache} = $dbase_dir;
$self->{_path}->{lock_cache} = $dbase_dir;
return 1;
}
$self->{_path}->{txn_dir} ||= "$dbase_dir/txn";
$self->{_path}->{backup_dir} ||= "$dbase_dir/backup";
$self->{_path}->{buffer_dir} ||= "$dbase_dir/buffer";
$self->{_path}->{schema_dir} ||= "$dbase_dir/schema";
$self->{_path}->{table_dir} ||= "$dbase_dir/tables";
$self->{_path}->{cache_dir} ||= "$dbase_dir/cache";
$self->{_path}->{table_cache} ||= "$self->{_path}->{cache_dir}/tables";
$self->{_path}->{schema_cache} ||= "$self->{_path}->{cache_dir}/schema";
$self->{_path}->{lock_cache} ||= "$self->{_path}->{cache_dir}/lock";
# Centralized directory creation: create required base directories at initialization
# ONLY when not in test mode (cfg->{test}) and dbase_dir is a dedicated directory (not '.')
unless ( $self->config('test') ) {
if ( defined $dbase_dir && $dbase_dir ne "." && $dbase_dir ne "" ) {
require File::Path;
for my $dir (
$self->{_path}->{dbase_dir},
$self->{_path}->{table_dir},
$self->{_path}->{schema_dir},
$self->{_path}->{backup_dir},
$self->{_path}->{buffer_dir},
$self->{_path}->{cache_dir},
$self->{_path}->{table_cache},
$self->{_path}->{schema_cache},
$self->{_path}->{lock_cache},
$self->{_path}->{txn_dir},
) {
if ( defined $dir && length($dir) && !$self->dir_exist($dir) ) {
eval { File::Path::make_path($dir) };
}
lib/AmberDB/Tools.pm view on Meta::CPAN
sub tie2csv {
my ( $self, $tableid ) = @_;
my $adb = $self->{_adb} or return;
$tableid or return;
my $table_path = $adb->table_path($tableid);
my $i = 1;
if ( -e "$table_path.csv" ) {
my $day_id = $adb->{date}->{day_id} || 'backup';
rename( "$table_path.csv", "$table_path-$day_id.csv" );
}
# cevirme islemlerini yap
my $tie_path = "${table_path}.$adb->{db_ext}";
return 1 unless -e $tie_path;
$adb->table_read($tie_path) or return 1;
my %data;
$adb->recs_scan(
lib/AmberDB/Tools.pm view on Meta::CPAN
my $adb = $self->{_adb} or return;
$tableid or return;
# file paths
my $table_path = $adb->table_path($tableid);
my $file_path = "${table_path}.$adb->{db_ext}";
my $csv_path = "$table_path.csv";
return unless -e $csv_path;
# backup with timestamp if exists
if ( -e "${table_path}.$adb->{db_ext}" ) {
my $sec_id = $adb->{date}->{second_id} || time();
rename( "${table_path}.$adb->{db_ext}",
"${table_path}-$sec_id.$adb->{db_ext}" );
unlink("${table_path}.$adb->{db_ext}");
}
# perform conversion operations
my $i = 1;
my @records;
lib/AmberDB/Tools.pm view on Meta::CPAN
if ( $adb->config('simple') ) {
my $ext = $adb->{db_ext} || "db";
my @files = $adb->dir_files( $dbase_dir, "*.$ext", full_path => 0 );
@all_tables = map { /^([a-z0-9_]+)\.\Q$ext\E$/i ? $1 : () } @files;
}
# 2. Standard Structured Mode: Multi-directory scan (tables/ and year directories) for .db files
else {
my $tbl_dir = File::Spec->catdir( $dbase_dir, 'tables' );
push @all_tables, $adb->dir_files( $tbl_dir, "*.db", full_path => 0 );
my %seen_dirs = ( "tables" => 1, "schema" => 1, "backup" => 1 );
if ($year_dir) {
my $yd_path = File::Spec->catdir( $dbase_dir, $year_dir );
push @all_tables, $adb->dir_files( $yd_path, "*.db", full_path => 0 );
$seen_dirs{$year_dir} = 1;
}
# Auto-discover any 4-digit year directories under $dbase_dir (e.g. 2024, 2025, 2026)
if ( -d $dbase_dir ) {
my @year_dirs = $adb->dir_files( $dbase_dir, qr/^\d{4}$/, full_path => 0, files_only => 0 );
foreach my $yd (@year_dirs) {
lib/AmberDB/Tools.pm view on Meta::CPAN
$converted{$tableid} = 1;
$self->{say} .= " -> Rebuilt packed binary indexes (.inx, .fld, .src, .srt) for $tableid\n";
}
}
return \%converted;
}
# Creates a compressed, portable .amberdb archive containing table schemas,
# native data files (.db, .del, .aut, .cnt), and integrity manifest.
# my $archive = $tools->dump( [file => 'backup.amberdb'], [tables => ['t1', 't2']] );
# ---------------------------------------------------------------------
sub dump {
my ( $self, %opts ) = @_;
my $adb = $self->{_adb} or return;
require Archive::Tar;
require Digest::SHA;
require JSON::PP;
require File::Spec;
require File::Path;
lib/AmberDB/Tools.pm view on Meta::CPAN
# 2. Flush and close open table handles for read consistency
$adb->close_all();
# 3. Determine output file path
my $year = ( $adb->{date} && $adb->{date}->{year} ) ? $adb->{date}->{year} : (localtime)[5] + 1900;
my $month = ( $adb->{date} && $adb->{date}->{month} ) ? $adb->{date}->{month} : sprintf( "%02d", (localtime)[4] + 1 );
my $day = ( $adb->{date} && $adb->{date}->{day} ) ? $adb->{date}->{day} : sprintf( "%02d", (localtime)[3] );
my $date_iso = "$year-$month-$day";
my $time_id = ( $adb->{date} && $adb->{date}->{second_id} ) ? $adb->{date}->{second_id} : time();
my $backup_base = $adb->path('backup_dir')
|| ( $adb->path('dbase_dir') ? $adb->path('dbase_dir') . "/backup" : "backup" );
my $year_dir = "$backup_base/$year";
unless ( -d $year_dir ) {
File::Path::make_path($year_dir);
}
my $outfile = $opts{file} || "$year_dir/amberdb_${date_iso}_${time_id}.amberdb";
# Ensure parent directory for $outfile exists
if ( my ($outdir) = $outfile =~ m{^(.*)[/\\]} ) {
unless ( -d $outdir ) {
File::Path::make_path($outdir);
lib/AmberDB/Tools.pm view on Meta::CPAN
cluck "[DB_BACKUP] Failed to write archive $outfile: " . $tar->error() . "\n";
return;
}
$self->{say} .= "Archive successfully written to $outfile (" . ( -s $outfile ) . " bytes)\n";
return wantarray ? ( $outfile, $manifest ) : $outfile;
}
# Restores a .amberdb archive into target database, validates checksums,
# and deterministically rebuilds all binary indexes via set_index.
# my $res = $tools->restore( file => 'backup.amberdb', [force => 1], [reindex => 1] );
# ---------------------------------------------------------------------
sub restore {
my ( $self, %opts ) = @_;
my $adb = $self->{_adb} or return;
my $file = $opts{file} or do {
cluck "[DB_RESTORE] Missing required parameter 'file'.\n";
return;
};
return unless -e $file;
lib/AmberDB/Tools.pm view on Meta::CPAN
AmberDB::Tools - Database maintenance, CLI reindexing, and bulk conversion toolset
=head1 SYNOPSIS
use AmberDB;
use AmberDB::Tools;
my $adb = AmberDB->new(path => { dbase_dir => "/path/to/dbstore" });
my $tools = AmberDB::Tools->new($adb);
# 1. Create portable .amberdb database backup archive
my $archive = $tools->dump();
# 2. Restore database archive with integrity validation and reindexing
my $result = $tools->restore(file => "backup.amberdb", force => 1);
# 3. Rebuild all indexes for a single table (.inx, .src, .fld, .fac, .srt)
$tools->set_index("catalog_product");
# 4. Rebuild only specific index components
$tools->set_search("catalog_product"); # Rebuild full-text search index
$tools->set_fields("catalog_product"); # Rebuild field exact match index
$tools->set_filters("catalog_product"); # Rebuild facet forward filter index
$tools->set_sort("catalog_product"); # Rebuild binary sort index
lib/AmberDB/Tools.pm view on Meta::CPAN
my $tools = AmberDB::Tools->new($adb);
=head1 METHODS
=head2 dump([%options])
Creates a compressed, portable C<.amberdb> archive file (gzipped tar archive) containing table and database schemas (C<schema/*.table>, C<schema/*.dbase>), native database data files (C<tables/*.db>, C<tables/*.del>, C<tables/*.aut>, C<tables/*.cnt>)...
Options:
=over 4
=item * C<file>: Custom output file path (defaults to C<backup/YYYY/amberdb_YYYY-MM-DD_time.amberdb>).
=item * C<tables>: Array reference of table IDs to include (defaults to all tables in database).
=item * C<table>: Single table ID to export as a focused snapshot.
=back
my $archive = $tools->dump();
my $archive = $tools->dump(tables => ["catalog_product", "orders_cart"]);
=head2 restore(%options)
Restores a C<.amberdb> archive into the target database. Validates archive integrity via SHA-256 checksums in C<manifest.json>, extracts schemas and data files, and deterministically reconstructs all binary indexes (C<.inx>, C<.src>, C<.fld>, C<.fac>...
Options:
=over 4
=item * C<file>: Path to C<.amberdb> archive file (required).
=item * C<force>: Boolean (default 0). Must be set to 1 to overwrite existing tables in a non-empty database directory.
=item * C<reindex>: Boolean (default 1). Automatically executes C<set_index> for all restored tables.
=item * C<tables>: Array reference of specific table IDs to extract from the archive.
=back
my $res = $tools->restore(file => "backup.amberdb", force => 1);
=head2 set_index($table_id, [@records])
Rebuilds all secondary and primary indexes for C<$table_id> based on its schema definition:
=over 4
=item * Primary key index (C<.inx>) via C<set_readall>
=item * Full-text search inverted indexes (C<_${blk}.src>) via C<set_search>
=item * Inverted field match indexes (C<_${blk}.fld>) via C<set_fields>
=item * Columnar facet filter forward indexes (C<_${blk}.fac>) via C<set_filters>
=item * Monotonic binary pre-sorted record indexes (C<_${blk}.srt>) via C<set_sort>
t/amberdb_backup.t view on Meta::CPAN
use AmberDB::Tools;
my $tmpdir = tempdir( CLEANUP => 1 );
$tmpdir =~ s{\\}{/}g;
my $adb = AmberDB->new(
path => { dbase_dir => $tmpdir },
cfg => { user => "admin", language => "tr" },
);
ok( defined $adb, "AmberDB instance created for backup tests" );
# =========================================================================
# SUBTEST 1: recs_back Daily WAL CSV Stream (backup/YYYY/YYYY-MM-DD.csv)
# =========================================================================
subtest "1. Continuous Recovery Stream (recs_back -> YYYY-MM-DD.csv)" => sub {
plan tests => 6;
# Create a test table
my $schema = {
name => 'audit_items',
record_index => 1,
search_block => [ 1 ],
log_owner => 1,
t/amberdb_backup.t view on Meta::CPAN
# Insert a record
my $rid = $adb->insert_id( 'audit_items', undef, 'Keyboard', 450 );
ok( $rid > 0, "Record inserted with ID $rid" );
# Edit record
$adb->modify_id( 'audit_items', $rid, 'Mechanical Keyboard', 550 );
# Delete record
$adb->delete_id( 'audit_items', $rid );
# Verify backup directory structure
my $year = $adb->{date}->{year};
my $date_iso = "$adb->{date}->{year}-$adb->{date}->{month}-$adb->{date}->{day}";
my $csv_file = "$tmpdir/backup/$year/$date_iso.csv";
ok( -d "$tmpdir/backup/$year", "Year directory backup/$year created" );
ok( -e $csv_file, "Daily CSV stream file $csv_file created" );
# Read CSV lines
open my $fh, "<:encoding(UTF-8)", $csv_file or die "Cannot open $csv_file: $!";
my @lines = <$fh>;
close $fh;
ok( scalar(@lines) >= 3, "CSV contains at least 3 log entries (add, edit, del)" );
# Check line format: timestamp \t user \t action \t table \t rid \t values
t/amberdb_backup.t view on Meta::CPAN
$adb->insert_id( 'catalog_products', undef, 'Desk Lamp', 'Furniture', 350 );
$adb->insert_id( 'catalog_products', undef, 'Office Chair', 'Furniture', 1200 );
# Create a unique/dictionary index (.unq) for block 2
$adb->insert_strs( 'catalog_products', 2, [ 10, 'Electronics' ], [ 20, 'Furniture' ] );
$adb->insert_id( 'orders', undef, 'Alice', 15350 );
$adb->insert_id( 'orders', undef, 'Bob', 1200 );
# Run full dump
my $dump_file = "$tmpdir/backup/test_dump.amberdb";
my ($outfile, $manifest) = $tools->dump( file => $dump_file );
ok( defined $outfile, "dump() returned output path" );
is( $outfile, $dump_file, "dump() created requested archive path" );
ok( -e $dump_file, "Archive file exists on disk" );
ok( -s $dump_file > 0, "Archive file is non-empty" );
# Validate returned in-memory manifest structure
ok( defined $manifest, "dump() returned manifest hashref" );
is( $manifest->{format}, 'AmberDB Archive', "Manifest format is 'AmberDB Archive'" );
t/amberdb_backup.t view on Meta::CPAN
ok( exists $manifest->{tables}->{catalog_products}, "Manifest contains catalog_products table metadata" );
is( $manifest->{tables}->{catalog_products}->{records}, 3, "Manifest records 3 records for catalog_products" );
};
# =========================================================================
# SUBTEST 3: Archive Package Inspection (.tar Extraction & Index Exclusion)
# =========================================================================
subtest "3. Archive Package Inspection & Integrity" => sub {
plan tests => 8;
my $dump_file = "$tmpdir/backup/test_dump.amberdb";
my $tar = Archive::Tar->new();
ok( $tar->read($dump_file), "Archive is a valid tar archive" );
my @files = $tar->list_files();
ok( grep { $_ eq 'manifest.json' } @files, "Archive contains manifest.json" );
ok( grep { $_ eq 'schema/catalog.dbase' } @files, "Archive contains schema/catalog.dbase" );
ok( grep { $_ eq 'schema/catalog_products.table' } @files, "Archive contains schema/catalog_products.table" );
ok( grep { $_ eq 'tables/catalog_products.db' } @files, "Archive contains tables/catalog_products.db" );
ok( grep { $_ eq 'tables/catalog_products_2.unq' } @files, "Archive contains tables/catalog_products_2.unq" );
t/amberdb_backup.t view on Meta::CPAN
my $sha = $manifest->{tables}->{catalog_products}->{sha256}->{'tables/catalog_products.db'};
ok( defined $sha && length($sha) == 64, "Manifest contains valid 64-character SHA-256 hash" );
};
# =========================================================================
# SUBTEST 4: Tools->restore() and Deterministic Index Rebuilding
# =========================================================================
subtest "4. Database Restore and Index Reconstruction" => sub {
plan tests => 17;
my $dump_file = "$tmpdir/backup/test_dump.amberdb";
# 4.1 Safety check: Restore into non-empty database without force should fail
my $tools = AmberDB::Tools->new($adb);
my $blocked_res = $tools->restore( file => $dump_file, force => 0 );
ok( !defined $blocked_res, "restore() without force on non-empty database returns undef" );
# 4.2 Create a completely clean staging database directory
my $stagedir = tempdir( CLEANUP => 1 );
$stagedir =~ s{\\}{/}g;
t/amberdb_encapsulation.t view on Meta::CPAN
plan tests => 6;
# Getter
is( $adb->path('dbase_dir'), $tmpdir, "path('dbase_dir') returns expected directory" );
# Setter via key-value
$adb->path( custom_path => "/opt/data" );
is( $adb->path('custom_path'), "/opt/data", "path(custom_path => ...) sets path" );
# Setter via hashref
$adb->path({ backup_dir => "$tmpdir/backup" });
is( $adb->path('backup_dir'), "$tmpdir/backup", "path({ backup_dir => ... }) sets path" );
# Bulk copy
my $paths_copy = $adb->path();
ok( ref($paths_copy) eq 'HASH', "path() returns HASH ref" );
is( $paths_copy->{dbase_dir}, $tmpdir, "path copy contains dbase_dir" );
# Mutation protection
$paths_copy->{dbase_dir} = "/compromised/path";
is( $adb->path('dbase_dir'), $tmpdir, "Internal path is NOT affected by mutating copy" );
};
t/amberdb_simple_mode.t view on Meta::CPAN
# 2. Modify record (edit action)
$adb->modify_id( 'sessions', $sid, 'Closed', '192.168.1.50' );
# 3. Delete record (del action)
$adb->delete_id( 'sessions', $sid );
my $date_iso = "$adb->{date}->{year}-$adb->{date}->{month}-$adb->{date}->{day}";
my $csv_file = "$db_dir/$date_iso.csv";
ok( -e $csv_file, "Daily CSV backup file $csv_file created directly in same dbase_dir in simple mode" );
ok( !-d "$db_dir/backup", "No separate backup directory created in simple mode" );
open my $fh, "<:encoding(UTF-8)", $csv_file or die "Cannot open $csv_file: $!";
my @lines = <$fh>;
close $fh;
is( scalar(@lines), 3, "CSV contains exactly 3 entries (add, edit, del) in simple mode" );
# Line 1: add
chomp $lines[0];
my @cols_add = split /\t/, $lines[0];
t/amberdb_simple_mode.t view on Meta::CPAN
my @cols_edit = split /\t/, $lines[1];
is( $cols_edit[2], 'edit', "Col 2 is action 'edit'" );
is( $cols_edit[4], $sid, "Col 4 is custom string ID" );
# Line 3: del
chomp $lines[2];
my @cols_del = split /\t/, $lines[2];
is( $cols_del[2], 'del', "Col 2 is action 'del'" );
is( $cols_del[4], $sid, "Col 4 is custom string ID" );
# 4. Verify no_backup flag disables logging
my $db_dir_noback = "$tmp_dir/simple_db8_noback";
mkdir($db_dir_noback);
my $adb_noback = AmberDB->new(
path => { dbase_dir => $db_dir_noback },
cfg => { simple => 1, no_backup => 1 },
);
$adb_noback->insert_id( 'tbl', 'id1', 'val1' );
my $noback_csv = "$db_dir_noback/$date_iso.csv";
ok( !-e $noback_csv, "No backup CSV created when no_backup => 1" );
};
# ============================================================
# 9. Simple Mode Key Sanitization and Reference Rejection
# ============================================================
subtest '9. Simple Mode Key Sanitization and Reference Rejection' => sub {
my $db_dir = "$tmp_dir/simple_db9";
mkdir($db_dir);
my $adb = $tools->db_simple($db_dir);
( run in 0.622 second using v1.01-cache-2.11-cpan-4ef0a570458 )