AmberDB
view release on metacpan or search on metacpan
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
the system state becomes corrupted. To eliminate these anomalies, the entire sequence must be unified within a **single transaction spine (`transact_start` $\rightarrow$ `transact_end`)**. If any step encounters an error or if the process crashes, Am...
### 7.2 ACID Guarantees in AmberDB
AmberDB guarantees the four classical ACID properties through embedded flat-file database mechanics:
| ACID Property | Implementation Mechanism & Guarantees |
| :--- | :--- |
| **Atomicity** | **Disk-Backed Undo-Journaling:** When `transact_start()` is called, a microsecond-stamped `.txn` journal is created. Every `insert_id`, `modify_id`, and `delete_id` call appends reverse undo instructions. If a critical base error oc...
| **Consistency** | **Schema, Index, and State Integrity:** Inbound records are validated against schema field rules, data types, and byte limits. Primary keys (`autoid`), inverted word indexes, columnar facets, and URL slugs are synchronized in real...
| **Isolation** | **Strict Two-Phase Locking (Strict 2PL):** Every record modified within an active transaction acquires an exclusive OS-level lock (`flock LOCK_EX`). Locks are held throughout the entire transaction duration, preventing concurrent wo...
| **Durability** | **Synchronous Journaling & Crash Recovery (`transact_recover`):** All journal writes invoke `$fh->flush`. When configured with `cfg => { txn_sync => 1 }`, AmberDB triggers OS/kernel `fsync` (`$fh->sync`) and Berkeley DB cache flush...
> **Architectural Note: Batch ETL Imports vs. Business Transactions**
> Methods such as `insert_list`, `modify_list`, and `delete_list` are specialized for high-throughput batch imports (e.g., ingesting large XML/JSON product catalogs). Since list records are typically independent entities without cross-dependencies, d...
### 7.3 Transaction Workflow
1. **`transact_start()`**: Opens a microsecond-stamped undo journal (`.txn`) in `$dbase_dir/txn/` and recovers any orphaned transactions left by dead processes (`transact_recover`).
2. **CRUD Operations**: `insert_id`, `modify_id`, `delete_id` write updates to the base `.db` file, acquire record write locks (`flock`), and record reverse undo entries in the `.txn` journal.
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
1. **Linux / POSIX Environments:** Leveraging POSIX `fork()` and kernel-level `flock(LOCK_EX)` / `flock(LOCK_SH)` locks, every worker process operates within an isolated memory address space. Strict Two-Phase Locking (Strict 2PL) guarantees serializa...
2. **Windows / MSYS2 Environments:** On Windows NT architectures, full OS-level lock integrity and file descriptor isolation are maintained across independent worker processes (`perl.exe`).
3. **Record-Level Locking (`flock_open` / `flock_close`):** For critical concurrent updates on individual records (such as high-demand stock decrements or shared counters), `$adb->flock_open($table, "write", $id)` eliminates race conditions and lost ...
#### Concurrency & Stress Test Suite (`xt/amberdb_concurrency_stress.t`):
The database engine's resilience under extreme parallel load is verified by the author/release stress test suite:
```bash
# Run multi-process concurrency stress tests directly:
perl -Ilib xt/amberdb_concurrency_stress.t
```
This test suite validates 5 mission-critical concurrency scenarios:
- **1. Parallel Writers:** Multi-worker concurrent inserts verifying zero ID collisions, exact table counts, and synchronized secondary index compilation (`.inx`, `.fld`, `.src`, `.fac`, `.srt`, `.slg`).
- **2. Interleaved Reads & Writes:** Concurrent reader processes executing streaming scans and index queries while writers continuously insert new data without deadlocks or corruption.
- **3. Concurrent Transactions & Crash Recovery:** Simulated sudden process termination mid-transaction, verifying that orphaned `.txn` journals are safely rolled back by `transact_recover` without interfering with active concurrent transactions.
- **4. Concurrent URL Slug Collisions:** Dozens of processes simultaneously inserting identical product titles, confirming deterministic `-1`, `-2` suffix generation and 100% bidirectional bijection (`_0.slg` $\leftrightarrow$ `_1.slg`).
- **5. High-Concurrency Inventory Decrements:** Multiple workers decrementing stock on the same product record under `flock_open` write locks, verifying atomic final inventory consistency.
---
## 8. High-Throughput Batch Operations (Batch ETL & Ingestion)
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
| `input` | `string` | HTML/UI Form input component type | `input => "select"` |
| `valid` | `string` | Automated data validation rule | `valid => "not_null;email"` |
| `option` | `string` | Enumerated choice options (`value:label` pairs) | `option => "1:Active,0:Inactive"` |
| `rdbm` | `string / HASH`| Foreign table lookup mapping (`foreign_table;display_block`) | `rdbm => "catalog_category;2"` |
| `extend` | `HASH` | 1:1 vertical table extension | `extend => { table => "catalog_price", join => "id" }` |
#### 9.7.2 Supported 8 Core Field Types (`type`)
AmberDB uses **8 unified core storage types** across serialization (`db_encode`/`db_decode`), indexing, and sorting layers:
| Field Type (`type`) | Description | `enc_validate` (Write Phase) | `dec_validate` (Read Phase) | Indexing & Sorting Behavior |
| :--- | :--- | :--- | :--- | :--- |
| **`auto_id`** | Auto-increment ID (Block 0) | Primary key format validation | ID scalar return | Primary key index (`.inx`) |
| **`text`** | Standard UTF-8 Text | UTF-8 string validation | String scalar (`$val // ''`) | Inverted index (`.src`), dictionary (`.str`) |
| **`num`** / **`number`** | Numeric (Integer / Float / Boolean) | Numeric validation (`^[+-]?[0-9]+(?:\.[0-9]+)?$`), defaults empty to `0` | Numeric scalar cast (`0 + $val`) | Numerical sorting (`<=>`) in `.srt`, `.fld` filters |
| **`ascii`** | ASCII-Only Text | ASCII normalization via `to_ascii` | Clean ASCII text | URL slug map (`.slg`), ASCII `.srt` sorting |
| **`date`** | Date and Time | Assigns system date if `auto_date` is active | Date string | Chronological sort in `.srt` via `str2dateid` |
| **`array`** / **`repeat`** | List / Repeating Rows | ARRAY ref or `[split /,/]` | Perl `ARRAY` ref (`[]`) | Multi-value matching (`field_fetch`) |
| **`hash`** | Dictionary / Object (HASH ref) | HASH ref validation | Perl `HASH` ref (`{}`) | Schemaless nested key-value store |
| **`binary`** | Binary Payload / Base64 | Raw bytes or Base64 string | Raw binary scalar | Direct flat file storage |
> [!NOTE]
> **Numeric and Boolean Management:** The `num` (or `number`) type handles positive (`150`, `+25`), negative (`-50`, `-12.75`), floating-point values, and `0 / 1` boolean flags. Unchecked HTML checkboxes or empty numerical inputs are automatically no...
#### 9.7.3 UI Form Input Components (`input`)
Determines how the field is rendered in UI forms and administration panels:
| Component (`input`) | UI Element | Description |
| :--- | :--- | :--- |
| `text` | Text Input | Standard single-line text field `<input type="text">`. |
| `textarea` | Textarea | Multi-line plain text box `<textarea>`. |
| `summernote` | Summernote | Rich WYSIWYG HTML visual editor for articles/descriptions. |
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
- When `valid => "unique"` is specified (e.g. `username`, `email`, `barcode`), `insert_id` and `modify_id` perform an instantaneous $O(1)$ check on `s:$value` in `${table}_${blk}.unq`.
- If another record holds this value, the transaction is rejected with a unique constraint error.
- Successfully written records store bidirectional mappings (`s:$value => $rid` and `n:$rid => $value`), which are automatically cleaned up when records are deleted.
3. **RDBM & `match_block` String-to-ID Auto-Resolution:**
- When a string name is passed to a relational field (e.g. `"Can Publishing"` for `rdbm => "catalog_brand;1"`), AmberDB queries `s:Can Publishing` in `catalog_brand_1.unq`.
- If present, it resolves to the existing numeric ID; if absent in `write` mode, it auto-registers the entry in `.unq` and the foreign table with an incremented ID.
- The inverted filter index (`.fld`) always stores **pure numeric IDs**, ensuring lightweight index storage and fast integer comparisons.
---
### 9.8 Schema-Driven Type Validation & Casting (`enc_validate` & `dec_validate`)
AmberDB enforces two-way data integrity between Perl runtime types and database storage:
1. **Write Phase Validation (`enc_validate`):**
- Invoked in `insert_id`, `modify_id`, `insert_list`, and `modify_list` right before records are written to disk and secondary indexes.
- Cleans numeric fields, trims whitespace, and converts empty/invalid inputs to `0`.
- Normalizes non-ASCII characters for `ascii` fields using `to_ascii`.
- Fills empty `valid => "auto_date"` fields with the current ISO date.
- Converts comma-separated strings into Perl `ARRAY` refs for `array` fields and enforces `HASH` refs for `hash` fields.
2. **Read Phase Casting (`dec_validate`):**
- Invoked in `read_id`, `read_list`, and `read_all` immediately after `db_decode`.
- Casts numeric fields to numeric scalars (`0 + $val`), eliminating uninitialized value warnings in mathematical expressions.
- Guarantees `array` fields return `[]` and `hash` fields return `{}` even when empty.
3. **Simple Mode Performance:**
- In Schemaless (`simple => 1`) mode or for tables without block definitions, `enc_validate` and `dec_validate` return input data immediately with zero CPU overhead.
### 9.8.1 How Schemas Coordinate with CRUD Operations
When a record is added or modified via `insert_id` or `modify_id`, the passed array elements map directly to block indices:
```perl
# Block Mapping:
# Block 0 : ID (PrimaryKey - auto-generated by the engine or passed as 0)
# Block 1 : @record[0] -> Category ID ("5")
# Block 2 : @record[1] -> Brand ID ("12")
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
my @product = (
"5", "12", "", "Wireless Headphones", "Active Noise Cancelling",
"Sony", "<p>Detailed product description...</p>", 150, "8690123456789", 2499.90, "1"
);
my $new_id = $adb->insert_id("catalog_product", 0, @product);
```
In a single atomic pass, the engine consults the schema and:
1. Validates and normalizes field types via `enc_validate`.
2. Writes the raw record to `catalog_product.db`.
3. Updates `catalog_product.inx` primary index (since `record_index => 1`).
4. Indexes Category (5), Brand (12), and Status (1) in `catalog_product_*.fld` match indexes (since `match_block => [1, 2, 3, 11]`).
5. Extracts, tokenizes, normalizes, and indexes Title, Subtitle, Description, and Barcode in `catalog_product_*.src` inverted search indexes.
6. Generates the URL slug `sony-wireless-headphones` into `catalog_product.slg` (since `slug_block => [2, 4]`).
---
### 9.9 Dynamic Runtime Schema Manipulation (`table_attr`)
docs/EN.AmberDB_User-Guide.md view on Meta::CPAN
```perl
# 1. Manual Cache Write (stores in cache/${table}.inx)
$adb->cache_write("catalog_product", "featured_items", @featured_list);
# 2. Cache Read
my @featured = $adb->cache_read("catalog_product", "featured_items");
# 3. Hard Cache Table Preload
$adb->cache_preload("catalog_category");
# 4. Invalidate Cache (Automatically purged on modify / delete_id)
$adb->cache_delete("catalog_product", "featured_items"); # Single key
$adb->cache_delete("catalog_product"); # Entire table cache (.db and .inx)
# 5. Inspect RAM-Disk Diagnostics & Mount Status
my $cache_diag = $adb->cache_setup();
# Returns hashref: { is_mounted => 1, mount_desc => "...", cache_dir => "...", cache_size => "512M" }
```
### Cache TTL (`cache_ttl`) & Runtime Overrides
The `cache_ttl` expiration time is defined per-table directly inside its schema (e.g. `cache_ttl => 1800`). Ephemeral data structures like session tokens or process locks can have their expiration configured in the schema or dynamically tuned at runt...
docs/TR.AmberDB_Veritabani_Sistemi.md view on Meta::CPAN
| `input` | `string` | Form giriÅŸ bileÅŸeni (UI) tipi | `input => "select"` |
| `valid` | `string` | Otomatik doğrulama kuralı | `valid => "not_null;email"` |
| `option` | `string` | Seçenek listesi (`değer:etiket` çiftleri) | `option => "1:Aktif,0:Pasif"` |
| `rdbm` | `string / HASH`| Başka tablodan veri çekme (`hedef_tablo;gösterilecek_blok`) | `rdbm => "catalog_category;2"` |
| `extend` | `HASH` | 1:1 dikey geniÅŸletme tablosu | `extend => { table => "catalog_price", join => "id" }` |
#### 9.7.2 Desteklenen 8 Çekirdek Veri Tipi (`type`)
AmberDB motoru, serileştirme (`db_encode`/`db_decode`), indeksleme ve sıralama katmanlarında **8 temel çekirdek veri tipi** kullanır:
| Veri Tipi (`type`) | Tanım | `enc_validate` (Yazma Anı) | `dec_validate` (Okuma Anı) | İndeks ve Sıralama Davranışı |
| :--- | :--- | :--- | :--- | :--- |
| **`auto_id`** | Otomatik artan ID (Blok 0) | ID format kontrolü ve sıralama | ID skaler dönüş | Birincil anahtar dizini (`.inx`) |
| **`text`** | Standart UTF-8 Metin | UTF-8 kaçış / metin doğrulaması | Dize (`$val // ''`) | `.src` ters indeksinde aranır, `.str` sözlüğü |
| **`num`** / **`number`** | Sayısal (Tamsayı / Ondalık / Boolean) | Sayısal doğrulama (`^[+-]?[0-9]+(?:\.[0-9]+)?$`), boşsa `0` | Sayı dönüşümü (`0 + $val`) | `.srt` sayısal (`<=>`) sıralama, `.fld` filtre |
| **`ascii`** | Salt ASCII karakterli metin | `to_ascii` ile ASCII normalizasyonu | ASCII metin | `.slg` slug haritası, `.srt` ASCII sıralama |
| **`date`** | Tarih ve Zaman | `auto_date` ise sistem tarihi atama | Tarih dizesi | `str2dateid` ile tarihsel kronolojik sıralama |
| **`array`** / **`repeat`** | Dizi / Tekrarlayan Satırlar | ARRAY ref veya `[split /,/]` | Perl `ARRAY` ref (`[]`) | Çoklu değer eşleşmesi (`field_fetch` multi-value) |
| **`hash`** | Sözlük / Nesne (HASH ref) | HASH ref kontrolü | Perl `HASH` ref (`{}`) | İç içe şemasız nesne saklama |
| **`binary`** | İkili Veri / Base64 | Ham binary bayt veya Base64 | Ham / Base64 skaler | Doğrudan dosya depolaması |
> [!NOTE]
> **Sayı ve Boolean Yönetimi:** `num` (veya `number`) tipi hem pozitif (`150`, `+25`), negatif (`-50`, `-12.75`), ondalıklı sayıları hem de `0 / 1` boolean bayraklarını yönetir. HTML formlarında iÅŸaretlenmeyen (`checkbox`) alanlar veya boÅ...
#### 9.7.3 Form GiriÅŸ BileÅŸenleri (`input`)
UI ve yönetim paneli katmanında form elemanının nasıl görüntüleneceğini belirler:
| Bileşen (`input`) | UI Elemanı | Açıklama |
| :--- | :--- | :--- |
| `text` | Metin Kutusu | Standart tek satırlık metin alanı `<input type="text">`. |
| `textarea` | Metin Alanı | Çok satırlı düz metin kutusu `<textarea>`. |
| `summernote` | Summernote | Zengin WYSIWYG görsel HTML editörü. |
docs/TR.AmberDB_Veritabani_Sistemi.md view on Meta::CPAN
- Bir alanda `valid => "unique"` tanımlandığında (örn. `username`, `email`, `barkod`), motor `insert_id` veya `modify_id` anında `.unq` dosyasından `s:$değer` anahtarını kontrol eder.
- Değer başka bir kayda aitse işlem anında durdurulur ve hata fırlatılır.
- Başarılı ekleme ve güncellemelerde çift yönlü anahtarlar (`s:$değer => $rid` ve `n:$rid => $değer`) kaydedilir. Kayıt silindiğinde bu anahtarlar `.unq` dosyasından temizlenir.
3. **RDBM ve `match_block` Metin $\leftrightarrow$ Sayısal ID Dönüşümü:**
- İlişkisel bir alana (`rdbm => "catalog_brand;1"`) veya metin filtre bloğuna string geldiğinde (örn. `"Can Yayınları"`), motor hedef tablonun `catalog_brand_1.unq` dosyasından `s:Can Yayınları` anahtarını sorgular.
- Kayıtlıysa mevcut sayısal ID'yi alır; kayıtlı değilse yeni otomatik ID üreterek `.unq` sözlüğüne ve hedef tabloya ekler.
- Ters indeks dosyası (`.fld`) içerisine **daima saf sayısal ID** yazılarak indekslerin hafif ve hızlı taranması sağlanır.
---
### 9.8 CRUD İşlemlerinde Tip Doğrulama ve Dönüşümü (`enc_validate` & `dec_validate`)
AmberDB, veri tutarlılığını sağlamak için iki yönlü şema doğrulama ve dönüşüm mekanizması uygular:
1. **Yazma Anında Doğrulama (`enc_validate`):**
- `insert_id`, `modify_id`, `insert_list` ve `modify_list` metotlarında veri diske ve ikincil indekslere yazılmadan **hemen önce** çalışır.
- `num` alanları için sayısal temizlik yapılır, boşluklar ayıklanır ve boş değerlere `0` atanır.
- `ascii` alanlarında Türkçe ve özel karakterler `to_ascii` ile normalize edilir.
- `valid => "auto_date"` kuralı olan boş tarih alanlarına otomatik güncel sistem tarihi atanır.
- `array` alanlarında virgüllü dizeler otomatik `ARRAY` referansına (`[ ... ]`), `hash` alanları `HASH` referansına (`{ ... }`) dönüştürülür.
2. **Okuma Anında Dönüşüm (`dec_validate`):**
- `read_id`, `read_list` ve `read_all` metotlarında diskten `db_decode` ile çözülen alanlar kullanıcıya dönmeden **hemen önce** çalışır.
- `num` alanları Perl'de `0 + $val` yapılarak sayısal skaler olarak döndürülür (`undef` uyarıları önlenir).
- `array` alanları boşsa `[]`, `hash` alanları `{}` olarak garanti edilir.
3. **Simple Mod Uyumu:**
- Şemasız (`simple => 1`) modda veya `blocks` tanımlanmamış tablolarda `enc_validate` ve `dec_validate` hiçbir ek döngü çalıştırmadan veriyi doğrudan döndürür (sıfır ek maliyet).
### 9.9 Çalışma Zamanında Dinamik Şema Manipülasyonu (`table_attr`)
AmberDB şemaları statik değildir. Şema dosyalarını diskte değiştirmeye veya migration çalıştırmaya gerek kalmadan, uygulama çalışma zamanında (runtime) tablo ayarlarını bellek üzerinde anlık olarak güncelleyebilir:
```perl
# Senaryo 1: Barkod POS cihazı veya hızlı kasa ekranı için arama kapsamını daraltma
# Tabloda normalde 2 (firma), 3 (yazar), 4 (başlık), 9 (barkod) aranırken,
# anlık olarak sadece Başlık (4) ve Barkod (9) bloklarında arama yaptırma:
$adb->table_attr("catalog_product", { search_block => [ 4, 9 ] });
lib/AmberDB.pm view on Meta::CPAN
}
# Transaction journal & record locking (Lock before write - Strict 2PL)
my $is_txn = ( $self->{_txn} && $self->{_txn}->{active} ) ? 1 : 0;
$self->flock_open( $tableid, "write", $rid );
if ($is_txn) {
$self->{_txn}->{locks}->{"${tableid}_${rid}"} = 1;
}
# Validate and normalize field values according to schema blocks
@record = $self->enc_validate( $tableid, \@record );
# Validate unique constraints across blocks
my ( $unq_ok, $unq_err ) = $self->unique_check( $table_path, $table_info, $rid, \@record );
if ( !$unq_ok ) {
$self->table_close($file_path);
unless ($is_txn) { $self->flock_close( $tableid, $rid ); }
$self->transact_error( $tableid, $unq_err // "Unique constraint violation" );
return;
}
lib/AmberDB.pm view on Meta::CPAN
my $file_path = "$table_path.$self->{db_ext}";
# Phase 1: raw writings (the file is opened once)
$self->table_write($file_path) or return {};
my ( %statu, @batch, @new_rids );
foreach my $record (@records) {
$record->[0] = $self->table_autoid( $tableid, $record->[0] );
next unless $record->[0];
my ( $rid, @fields ) = @$record;
@fields = $self->enc_validate( $tableid, \@fields );
my ( $unq_ok, $unq_err ) = $self->unique_check( $table_path, $table_info, $rid, \@fields );
if ( !$unq_ok ) {
cluck "[DB_UNIQUE] $unq_err\n";
next;
}
$record = [ $rid, @fields ];
$record = [ $self->repeat_fields( $table_info, @$record ) ];
lib/AmberDB.pm view on Meta::CPAN
if ( !$table_info->{force} ) {
if ( !$old_record ) {
$self->table_close($file_path);
unless ($is_txn) { $self->flock_close( $tableid, $rid ); }
$self->transact_error( $tableid, "Record not exist: $rid" );
return;
}
}
# Validate and normalize field values according to schema blocks
@record = $self->enc_validate( $tableid, \@record );
# Validate unique constraints across blocks
my ( $unq_ok, $unq_err ) = $self->unique_check( $table_path, $table_info, $rid, \@record );
if ( !$unq_ok ) {
$self->table_close($file_path);
unless ($is_txn) { $self->flock_close( $tableid, $rid ); }
$self->transact_error( $tableid, $unq_err // "Unique constraint violation" );
return;
}
lib/AmberDB.pm view on Meta::CPAN
if ($is_txn) {
my $new_raw;
$self->{_db}->{$file_path}->get( $rid, $new_raw );
$self->_txn_log( $tableid, "edit", $rid, $new_raw, $old_record // "" );
}
$self->table_close($file_path);
unless ($is_txn) { $self->flock_close( $tableid, $rid ); }
# Cache invalidate
$self->cache_delete($tableid, $rid);
my @new_rec = ( $rid, @record );
# text backup record.
$self->recs_back( "edit", $tableid, \@new_rec )
or cluck "[DB_TIE] Backup error (edit). $tableid\n";
$self->config('simple') and return $rid;
lib/AmberDB.pm view on Meta::CPAN
my $table_path = $self->table_path($tableid);
my $file_path = "$table_path.$self->{db_ext}";
# Phase 1: raw writings
$self->table_write($file_path) or return {};
my ( %statu, @pairs );
foreach my $record (@records) {
my ( $rid, @data ) = @$record;
$rid or next;
@data = $self->enc_validate( $tableid, \@data );
my ( $unq_ok, $unq_err ) = $self->unique_check( $table_path, $table_info, $rid, \@data );
if ( !$unq_ok ) {
cluck "[DB_UNIQUE] $unq_err\n";
next;
}
$record = [ $rid, @data ];
$record = [ $self->repeat_fields( $table_info, @$record ) ];
lib/AmberDB.pm view on Meta::CPAN
# Delete the record
$self->recs_del( $file_path, $rid );
if ($is_txn) {
$self->_txn_log( $tableid, "del", $rid, "", $record );
}
$self->table_close($file_path);
unless ($is_txn) { $self->flock_close( $tableid, $rid ); }
# Cache invalidate
$self->cache_delete($tableid, $rid);
# Text backup record
$self->recs_back( "del", $tableid, [ $rid, "" ] )
or cluck "[DB_TIE] Backup error (del). $tableid\n";
$self->config('simple') and return $rid;
# Move to archive if keep_deleted enabled
if ( $table_info->{keep_deleted} ) {
lib/AmberDB.pm view on Meta::CPAN
$rid = $self->id_check( $tableid, $rid );
return unless defined $rid && $rid ne '';
my $table_info = $self->table_info($tableid);
my $use_cache = $table_info->{use_cache} // 0;
# Read from cache (Hard Cache: use_cache == 2)
if ( $use_cache == 2 ) {
my @cached = $self->cache_read($tableid, $rid);
if (@cached) {
my @val_fields = $self->dec_validate( $tableid, \@cached );
return ( $rid, @val_fields );
}
}
# Resolve table path and read record
my $table_path = $self->table_path($tableid);
my $file_path = "$table_path.$self->{db_ext}";
-e $file_path
or do { cluck "[DB_TIE] $tableid id and $file_path file path not exist\n"; return; };
lib/AmberDB.pm view on Meta::CPAN
# Write to cache (Hard Cache: use_cache == 2)
if ( $use_cache == 2 && @fields > 1 ) {
$self->cache_write( $tableid, $rid, @fields[ 1 .. $#fields ] );
}
# Load access logs
$self->auth_read( $tableid, $table_path, $rid );
if ( @fields > 1 ) {
my $id_val = $fields[0];
my @val_fields = $self->dec_validate( $tableid, [ @fields[ 1 .. $#fields ] ] );
@fields = ( $id_val, @val_fields );
}
return @fields;
}
# Reads all records or range of records.
# my @records = $adb->read_all("tableID");
# my ($count, @records) = $adb->read_all("tableID", 0, 20);
# ------------------------------------------------
lib/AmberDB.pm view on Meta::CPAN
else {
$count = scalar @records;
}
# 4. Fetch full record data unless keys_only
unless ($keys_only) {
my $recs_data = $self->recs_get( $scan_path, @records );
foreach my $rec (@records) {
my $val = $recs_data ? $recs_data->{$rec} : undef;
my @decoded = defined $val ? $self->db_decode($val) : ();
my @clean = $self->dec_validate( $tableid, \@decoded );
$rec = [ $rec, @clean ];
}
}
$self->table_close($scan_path);
return $limit ? ( $count, @records ) : @records;
}
lib/AmberDB.pm view on Meta::CPAN
my %rec_by_id;
my @miss_ids;
if ( $use_cache == 2 ) {
foreach my $orig_id (@$ids) {
my $rid = $links->{$orig_id} || $orig_id;
next if $rec_by_id{$rid};
my @cached_rec = $self->cache_read( $tableid, $rid );
if (@cached_rec) {
my @val_fields = $self->dec_validate( $tableid, \@cached_rec );
$rec_by_id{$rid} = [ $rid, @val_fields ];
}
else {
push @miss_ids, $rid;
}
}
}
else {
foreach my $orig_id (@$ids) {
my $rid = $links->{$orig_id} || $orig_id;
lib/AmberDB.pm view on Meta::CPAN
$self->table_close($file_path);
if ($recs_data) {
foreach my $rid (@miss_ids) {
next if $rec_by_id{$rid};
my $key_esc = $esc_map{$rid};
my $value = $recs_data->{$rid} // ( defined $key_esc ? $recs_data->{$key_esc} : undef );
next unless defined $value && $value ne '';
my @decoded = $self->db_decode($value);
@decoded = $self->dec_validate( $tableid, \@decoded );
my $rec = [ $rid, @decoded ];
$rec_by_id{$rid} = $rec;
if ( $use_cache == 2 ) {
$self->cache_write( $tableid, $rid, @decoded );
}
}
}
}
$self->auth_read( $tableid, $table_path, @$ids );
lib/AmberDB.pm view on Meta::CPAN
if (@tmp) {
# If logic is OR
if ( lc($and_or) eq "or" ) {
$self->recs_scan(
$file_path,
sub {
my ( $key, $value ) = @_;
my %string = $self->get_words( $value, "write", $tableid );
foreach my $str (@tmp) {
if ( $string{$str} ) {
push( @records, [ $key, $self->dec_validate( $tableid, [ $self->db_decode($value) ] ) ] );
return;
}
}
}
);
}
# If logic is AND
else {
$self->recs_scan(
$file_path,
sub {
my ( $key, $value ) = @_;
my %string = $self->get_words( $value, "write", $tableid );
foreach my $str (@tmp) {
unless ( $string{$str} ) {
return;
}
}
push( @records, [ $key, $self->dec_validate( $tableid, [ $self->db_decode($value) ] ) ] );
}
);
}
}
$self->table_close($file_path);
}
# Apply field filter(s) if provided
if ( $filter && %filter_map && @records ) {
for my $fld ( keys %filter_map ) {
lib/AmberDB.pm view on Meta::CPAN
$self->table_read($file_path) or do { cluck "[DB_TIE] $file_path can't open.\n"; return; };
@keys = $self->recs_keys($file_path);
$self->table_close($file_path);
@keys = $self->db_sortid( $tableid, @keys );
$self->cache_write( $tableid, "keys", @keys );
return @keys;
}
# Sanitizes and validates a record ID according to table id_type (num or ascii).
# For id_type eq 'ascii': cleans non-ASCII chars and enforces deterministic 8-byte limit.
# For id_type eq 'num': enforces numeric digits.
# my $clean_id = $adb->id_check($tableid, $rid);
# ------------------------------------------------
sub id_check {
my ( $self, $tableid, $rid ) = @_;
return unless defined $rid && $rid ne '';
return if ref $rid;
lib/AmberDB/Base.pm view on Meta::CPAN
next if !$w;
next if ( $minchar && length($w) < $minchar );
next if $jump{$w};
$words{$w} = $w;
}
}
return %words;
}
# Normalizes and validates field values according to schema block definitions prior to encoding/writing.
# Usage:
# my @clean_fields = $adb->enc_validate($tableid, \@fields, $has_id);
# ------------------------------------------------
sub enc_validate {
my ( $self, $tableid, $fields_ref, $has_id ) = @_;
return wantarray ? () : [] unless defined $fields_ref;
my @fields = ( ref($fields_ref) eq 'ARRAY' ) ? @$fields_ref : ($fields_ref);
return wantarray ? @fields : \@fields unless @fields;
# Bypass in simple mode or when no tableid is given
return wantarray ? @fields : \@fields
if $self->config('simple') || !defined $tableid || $tableid eq '';
lib/AmberDB/Base.pm view on Meta::CPAN
}
push @cleaned, $val;
}
return wantarray ? @cleaned : \@cleaned;
}
# Normalizes and casts field values according to schema block definitions upon decoding/reading.
# Usage:
# my @decoded_fields = $adb->dec_validate($tableid, \@fields, $has_id);
# ------------------------------------------------
sub dec_validate {
my ( $self, $tableid, $fields_ref, $has_id ) = @_;
return wantarray ? () : [] unless defined $fields_ref;
my @fields = ( ref($fields_ref) eq 'ARRAY' ) ? @$fields_ref : ($fields_ref);
return wantarray ? @fields : \@fields unless @fields;
# Bypass in simple mode or when no tableid is given
return wantarray ? @fields : \@fields
if $self->config('simple') || !defined $tableid || $tableid eq '';
lib/AmberDB/Base.pm view on Meta::CPAN
$self->{_cfg}->{language} = $val;
$self->_load_locale($val) if $self->can('_load_locale');
},
db_ext => sub {
my $val = shift;
$self->{_cfg}->{db_ext} = $val;
$self->{db_ext} = $val;
if ( defined $val && $val ne "db" ) {
$self->{_cfg}->{simple} = 1;
}
$self->_invalidate_table_paths();
},
simple => sub {
my $val = shift;
$self->{_cfg}->{simple} = $val ? 1 : 0;
$self->_invalidate_table_paths();
},
};
while ( my ( $key, $val ) = each %pairs ) {
if ( exists $hooks->{$key} ) {
$hooks->{$key}->($val);
}
else {
$self->{_cfg}->{$key} = $val;
}
lib/AmberDB/Base.pm view on Meta::CPAN
if ( @args == 1 && !ref( $args[0] ) ) {
return $self->{_path}->{ $args[0] };
}
# 3. Setter: key-value list or hashref
my %pairs = ( @args == 1 && ref( $args[0] ) eq 'HASH' ) ? %{ $args[0] } : @args;
for my $key ( keys %pairs ) {
$self->{_path}->{$key} = $pairs{$key};
}
$self->_invalidate_table_paths();
return $self;
}
# Invalidate cached table paths if global path-affecting configurations change
# ------------------------------------------------
sub _invalidate_table_paths {
my ($self) = @_;
if ( $self->{_table} && ref( $self->{_table} ) eq 'HASH' ) {
for my $tbl ( keys %{ $self->{_table} } ) {
delete $self->{_table}->{$tbl}->{_path}
if ref( $self->{_table}->{$tbl} ) eq 'HASH';
}
}
}
lib/AmberDB/Base.pm view on Meta::CPAN
Tokenizes C<$string> into search index keywords, stripping punctuation, applying language-specific stopwords, and normalizing casing.
my %words = $adb->get_words("Kablosuz Kulaklık & Aksesuarlar");
=head2 bin_encode(\@record_ids, [$id_type])
Packs a list of record IDs into a compact, fixed 8-byte binary buffer:
=over 4
=item * Numeric IDs (C<id_type =E<gt> 'num'>): Packed as 64-bit unsigned Big-Endian integers (C<QE<gt>>).
=item * ASCII IDs (C<id_type =E<gt> 'ascii'>): Packed as fixed 8-byte ASCII strings (C<a8>), strictly validated against 8-byte limits.
=back
my $packed_buffer = $adb->bin_encode([ 1, 2, 3 ], 'num');
=head2 bin_decode($binary_buffer, [$start], [$limit], [$direction], [$id_type])
Decodes an 8-byte binary buffer using O(1) C<substr> byte-offset slicing without unpacking the entire buffer into memory. Returns C<($total_count, @slice_ids)>.
=over 4
=item * C<$start>: 0-based record offset.
lib/AmberDB/Cache.pm view on Meta::CPAN
return unless $table_info && $table_info->{use_cache};
my $cache_file = $self->cache_file_for( $tableid, $key );
my $encoded_val = $self->db_encode(@records);
$self->recs_put( $cache_file, [ $key, $encoded_val ] );
return 1;
}
# my $ok = $adb->cache_delete($tableid, [$key], [$type]);
# Invalidates entry from cache/$tableid.db / .inx or removes entire table cache files.
# ------------------------------------------------
sub cache_delete {
my ( $self, $tableid, $key, $type ) = @_;
$tableid or return;
my $table_info = $self->table_info($tableid);
return unless $table_info && $table_info->{use_cache};
if ( defined $key && $key ne '' ) {
lib/AmberDB/Cache.pm view on Meta::CPAN
my @cached_row = $adb->cache_read("catalog_product", "101");
=head2 cache_write($tableid, $key, @records)
Serializes and writes record data to the RAM-disk cache file.
$adb->cache_write("catalog_product", "top_sellers", [ 101, "Prod A" ], [ 102, "Prod B" ]);
=head2 cache_delete($tableid, [$key], [$type])
Invalidates cache entries. If C<$key> is provided, removes only that specific key. If C<$key> is omitted, removes and unlinks the entire table cache files (both C<.db> and C<.inx>).
$adb->cache_delete("catalog_product", "featured_items"); # Invalidate single entry
$adb->cache_delete("catalog_product"); # Clear entire table cache
=head2 cache_preload($tableid)
Preloads all records and metadata from the persistent storage tables directory into the RAM-disk cache directory. Uses atomic temporary files (C<.tmp.$$>) and file locking to prevent race conditions during live updates.
$adb->cache_preload("catalog_category");
=head2 cache_ensure($tableid)
lib/AmberDB/Index.pm view on Meta::CPAN
return $parts[$b2] // '';
}
}
# 2. Direct Field: 5
return $record->[$field_spec] // '';
}
# my @vals = $adb->field_to_list($value, $mode, $table_path, $table_info, $blk);
# Converts ARRAY ref, comma/semicolon delimited string or single value to a normalized list.
# In 'write' mode, registers text strings into .unq with auto-incrementing lastid, or validates rdbm.
# In 'read' mode, resolves existing string IDs from .unq without creating new entries.
# ------------------------------------------------
sub field_to_list {
my ( $self, $value, $mode, $table_path, $table_info, $blk ) = @_;
return () unless defined $value && $value ne '';
# 1. Normalize input using trim_space with flatten mode (1)
my @raw;
lib/AmberDB/Index.pm view on Meta::CPAN
Facet forward indexing (C<.fac>) is handled by C<AmberDB::Index::Facet>, and dual-tier cold record indexing (C<.jinx>, C<.jfld>, C<.jsrc>) is managed by C<AmberDB::Index::Junk>.
B<Inheritance Note:> C<AmberDB> inherits from C<AmberDB::Index> via C<use parent>. All indexing methods can be called directly on C<$adb>.
=head1 METHODS
=head2 field_to_list($value, [$mode], [$table_path], [$table_info], [$blk])
Converts ARRAY references, comma/semicolon-delimited strings, or single scalars into a normalized list of trimmed values.
=over 4
=item * In C<'write'> mode: Registers text values into the per-block unique/dictionary index (C<_${blk}.unq>) with auto-incrementing numeric IDs (or validates foreign key IDs for RDBM fields).
=item * In C<'read'> mode: Resolves existing string IDs from C<_${blk}.unq> without creating new dictionary entries.
=back
my @ids = $adb->field_to_list("Red, Blue, Green", 'write', $path, $info, 3);
=head2 normalize_sort_key($value, $type, [$length])
Normalizes an input value into a fixed-width byte key for fast monotonic sorting in binary C<.srt> files:
=over 4
=item * B<num / decimal>: Adds a C<1e12> (1,000,000,000,000) offset for signed float/integer monotonic sorting (C<%020.6f> format).
lib/AmberDB/Tools.pm view on Meta::CPAN
# Write tar.gz archive
unless ( $tar->write( $outfile, Archive::Tar::COMPRESS_GZIP() ) ) {
cluck "[DB_BACKUP] Failed to write archive $outfile: " . $tar->error() . "\n";
return;
}
$self->{say} .= "Archive successfully written to $outfile (" . ( -s $outfile ) . " bytes)\n";
return wantarray ? ( $outfile, $manifest ) : $outfile;
}
# Restores a .amberdb archive into target database, validates checksums,
# and deterministically rebuilds all binary indexes via set_index.
# my $res = $tools->restore( file => 'backup.amberdb', [force => 1], [reindex => 1] );
# ---------------------------------------------------------------------
sub restore {
my ( $self, %opts ) = @_;
my $adb = $self->{_adb} or return;
my $file = $opts{file} or do {
cluck "[DB_RESTORE] Missing required parameter 'file'.\n";
return;
t/amberdb_cache.t view on Meta::CPAN
ok( @cached_rec10, 'Record 10 found in hard cache' );
# read_id uses hard cache
my @rec20 = $adb->read_id( 'hard_table', 20 );
is( $rec20[1], 'Hard Item 20', 'read_id retrieved record 20' );
# read_all uses hard cache
my @all_recs = $adb->read_all('hard_table');
is( scalar(@all_recs), 2, 'read_all retrieved 2 records from hard cache' );
# Updating record invalidates/updates cache
$adb->modify_id( 'hard_table', 20, 'Hard Item 20 Updated' );
my @rec20_upd = $adb->read_id( 'hard_table', 20 );
is( $rec20_upd[1], 'Hard Item 20 Updated', 'read_id retrieved updated record after update_id' );
# Delete cache and verify auto-ensure on read_id
$adb->cache_delete('hard_table');
ok( !-e $cache_db, 'Cache deleted successfully' );
};
subtest 'Cache TTL Expiration' => sub {
t/amberdb_cache.t view on Meta::CPAN
# Populate keys cache
$adb->cache_write( 'test_table', 'keys', 100 );
is_deeply( [ $adb->cache_read( 'test_table', 'keys' ) ], [100], 'keys cached' );
# 2. Insert record with next ID (101)
$adb->insert_id( 'test_table', 101, 'Item 101' );
my ($lastid2) = $adb->cache_read( 'test_table', 'lastid' );
is( $lastid2, 101, 'LastID updated to 101 in cache' );
# 3. Verify keys cache was invalidated by insert_id -> records_add
my @keys_cached = $adb->cache_read( 'test_table', 'keys' );
is( scalar(@keys_cached), 0, 'keys cache automatically invalidated after record mutation' );
};
subtest 'Persistent Buffer Operations' => sub {
plan tests => 5;
# Write to persistent buffer
my @records = ( [ 1, 'Data 1' ], [ 2, 'Data 2' ] );
ok( $adb->buffer_write( 'test_table', @records ), 'Buffer write succeeded' );
# Verify buffer file created in $dbase_dir/buffer/ (not cache/)
t/amberdb_encapsulation.t view on Meta::CPAN
subtest "5. table_attr() path invalidation on path-affecting attributes" => sub {
plan tests => 3;
$adb->config( simple => 0, db_ext => 'db', use_section => 1 );
my $path1 = $adb->table_path("demo_table");
ok( length($path1) > 0, "Initial table path resolved: $path1" );
# Ensure target section directory exists before switching section
mkdir "$tmpdir/tables_north" unless -d "$tmpdir/tables_north";
# Change section -> should invalidate cached path and recalculate
$adb->table_attr("demo_table", section => "north");
my $path2 = $adb->table_path("demo_table");
ok( length($path2) > 0, "Updated table path resolved: $path2" );
isnt( $path1, $path2, "Path was refreshed after section attribute changed" );
};
subtest "6. table_info() shallow copy protection" => sub {
plan tests => 3;
my $info = $adb->table_info("demo_table");
t/amberdb_schema_types.t view on Meta::CPAN
};
my $ok = $adb->table_infset( "products", $schema );
ok( $ok, "table_infset created schema for products" );
my $info = $adb->table_info("products");
is( scalar( @{ $info->{blocks} } ), 10, "Schema has 10 block definitions" );
};
# ============================================================
# 2. enc_validate & dec_validate Unit Verification
# ============================================================
subtest '2. enc_validate & dec_validate Direct Unit Tests' => sub {
my $db_dir = "$tmp_dir/db_types";
my $adb = AmberDB->new( path => { dbase_dir => $db_dir } );
# Raw input fields (Blocks 1..9)
my @raw_input = (
"Türkçe Kitap & Başlık", # 1: text
" 150.75 ", # 2: num (float with whitespace)
"-25.50", # 3: number (negative float)
"invalid_num", # 4: num (invalid string -> 0)
"Ürün_Kodu_#105", # 5: ascii (Turkish chars -> ASCII)
"", # 6: date (empty + auto_date -> today)
"elektronik,telefon", # 7: array (comma string -> array ref)
{ color => "Mavi" }, # 8: hash (hash ref)
"BINARY_BLOB_XYZ", # 9: binary
);
my @enc = $adb->enc_validate( "products", \@raw_input );
is( $enc[0], "Türkçe Kitap & Başlık", "1. text preserved in enc_validate" );
is( $enc[1], 150.75, "2. num trimmed and cast to float" );
is( $enc[2], -25.50, "3. negative number cast correctly" );
is( $enc[3], 0, "4. invalid number defaulted to 0" );
like( $enc[4], qr/^Urun_Kodu_/, "5. ascii normalized to ASCII" );
like( $enc[5], qr/^\d{4}-\d{2}-\d{2}$/, "6. auto_date populated current ISO date" );
is_deeply( $enc[6], [ "elektronik", "telefon" ], "7. array string converted to ARRAY ref" );
is_deeply( $enc[7], { color => "Mavi" }, "8. hash ref preserved" );
is( $enc[8], "BINARY_BLOB_XYZ", "9. binary scalar preserved" );
# Test dec_validate
my @dec_input = (
"Türkçe Kitap", # 1: text
"200.5", # 2: num
"-50", # 3: number (negative)
"", # 4: num (empty string -> 0)
"CODE123", # 5: ascii
"2026-08-31", # 6: date
["a", "b"], # 7: array
{ a => 1 }, # 8: hash
"blob", # 9: binary
);
my @dec = $adb->dec_validate( "products", \@dec_input );
is( $dec[1], 200.5, "dec_validate casts num to number" );
is( $dec[2], -50, "dec_validate casts negative number to -50" );
is( $dec[3], 0, "dec_validate converts empty num to 0" );
is_deeply( $dec[6], ["a", "b"], "dec_validate ensures array ref" );
is_deeply( $dec[7], { a => 1 }, "dec_validate ensures hash ref" );
};
# ============================================================
# 3. CRUD Round-Trip with enc_validate & dec_validate
# ============================================================
subtest '3. Full CRUD Round-Trip via insert_id & read_id' => sub {
my $db_dir = "$tmp_dir/db_types";
my $adb = AmberDB->new( path => { dbase_dir => $db_dir } );
# Insert record with various types (including negative numbers and empty auto_date)
my $rid = $adb->insert_id(
"products", 101,
"Laptop Çantası", # 1: title (text)
"1250.50", # 2: price (num)
t/amberdb_schema_types.t view on Meta::CPAN
is( scalar(@all), 4, "read_all returned 4 records" );
my ($p201) = grep { $_->[0] == 201 } @all;
ok( $p201, "Record 201 present in read_all" );
is( $p201->[2], 50.25, "read_all record 201 has numeric type" );
is_deeply( $p201->[7], ["kat1"], "read_all record 201 has array ref" );
};
# ============================================================
# 5. Simple Mode Bypass Verification (Zero Overhead)
# ============================================================
subtest '5. Simple Mode Bypasses enc_validate & dec_validate' => sub {
my $db_dir = "$tmp_dir/db_types_simple";
mkdir($db_dir);
my $adb = AmberDB->new(
path => { dbase_dir => $db_dir },
cfg => { simple => 1 },
);
# In simple mode, arbitrary data passes untouched without schema block casting
my $raw_val = " custom string not forced to number ";
my @enc = $adb->enc_validate( "free_table", [$raw_val] );
is( $enc[0], $raw_val, "enc_validate in simple mode leaves data untouched" );
my @dec = $adb->dec_validate( "free_table", [$raw_val] );
is( $dec[0], $raw_val, "dec_validate in simple mode leaves data untouched" );
# Insert and read arbitrary key and data
$adb->insert_id( "free_table", "user\@test.com", "Active", "Data123" );
my @read = $adb->read_id( "free_table", "user\@test.com" );
is( $read[0], "user\@test.com", "Simple mode read_id returned exact key" );
is( $read[1], "Active", "Simple mode read_id returned field 1" );
};
done_testing();
( run in 2.161 seconds using v1.01-cache-2.11-cpan-d01c6094234 )