Storage Engine

Relata DB 3.0
Parts of this page still describe the 2.x line, which is kept in the 2.5.8 documentation snapshot. In 3.0 HTTP routes are under /v1/, the compatibility protocol doors (Postgres, Mongo, Redis, Neo4j, ClickHouse, S3, Flight, SPARQL) are retired, and clustering and multi-tenancy are Enterprise. See What is new in 3.0 and HTTP route restructure.
Storage tiering and cacheRows live in RAM (hot), spill to disk Parquet segments (warm) when the RAM cap is hit, and overflow to an S3-compatible object store (cold). The Foyer RAM cache (S3-FIFO admission) serves Parquet segment bytes back into the hot path.relata-cacheFoyer RAM · S3-FIFO admission · optional NVMe tierHOTRAM-resident rowsper-type interior locksμs reads / single-digit-ms writesbi-temporal current statespill at RAM capWARMDisk Parquet segmentsdisk-first pagingcolumnar Arrow IPCmutable tail + segmentsoverflowCOLDObject storeS3-compatiblecontent-addressed blobsmanifest-chainedserve segment bytes back to hot path

One S3-compatible bucket (or local directory on the free profile). Open formats throughout — Parquet, Arrow IPC, JSON manifests, content-addressed blobs. Hash-chained for tamper evidence. No proprietary on-disk format. A file written by RelataDB is readable by any Parquet reader, any S3 client, any Arrow consumer.

The bucket is the database

All durable data lives under one local directory (RELATA_DATA_DIR) or, when configured, in one S3-compatible bucket. The layout is identical; only the backend changes. Supported object stores: AWS S3, Cloudflare R2, Google Cloud Storage, Azure Blob (via S3 compatibility), MinIO, local disk.

# Local disk
RELATA_DATA_DIR=/var/lib/relata   cargo run -p relata-cli -- serve
# S3-compatible object store
AWS_ENDPOINT_URL=https://...  RELATA_S3_BUCKET=relata-prod   cargo run -p relata-cli -- serve

On-disk layout

relata-data/
├── ontology/          # schema-as-code, git-branched, one file per type
├── tables/            # one directory per ObjectType / EventType
│   ├── Profile/
│   │   ├── 2025-11-11.parquet
│   │   ├── 2025-11-12.parquet
│   │   └── _bloom/    # per-column bloom filters
│   └── CallEvent/
├── graph/             # CSR segments + incremental degree index, one dir per LinkType
├── vectors/           # HNSW graph + DiskANN warm tier, one dir per (type, column)
│   ├── Post.embedding/
│   └── MediaContent.clip/
├── fulltext/          # custom BM25 inverted index (integer posting lists)
├── identity_index/    # universal lookup MV: (CanonicalKind, bytes) → observations
├── provenance/        # PROV-O assertions, content-addressed by assertion hash
├── audit/             # append-only, hash-chained log
├── blobs/             # content-addressed by SHA-256 (media, attachments)
└── manifests/         # branch HEAD pointer + commit chain, one JSON per commit

In-memory store and per-type interior locking

The primary read path is an in-memory bi-temporal store built in relata-storage. Writes take &self, not &mut self. Each object type owns its own RwLock<TableState>, implemented with parking_lot::RwLock (PlRwLock type alias). A write to Profile does not block a read or write to Post.

parking_lot does not poison on panic, which eliminates the recover_poison() pattern.

Write-Ahead Log (WAL)

Every write goes through the WAL before it is applied in memory. The WAL is flushed before the write is acknowledged to the client. In-memory state is a cache of the WAL; on restart, the WAL is the source of truth.

Durability levelfsync timingProcess crashPower loss
async (default)Background flusher, ~10ms cadenceNo loss≤10ms loss
syncfsync before ackNo lossNo loss

The durability level is set per-write via the X-Relata-Durability request header, or forced cluster-wide with RELATA_WAL_SYNC=always (default is interval). A bulk loader uses async; a payment record uses sync. Same table, same writer, different durability per row.

The WAL is hash-chained: each WAL entry references the hash of the previous entry. A replay that produces a hash mismatch at any entry signals corruption or tampering.

Parquet segments

Each tables/<Type>/ directory holds Parquet data files. The default is one file per UTC day per type.

PropertyValueWhy
Row group size128 MBBalances scan throughput vs. zone-map granularity
CompressionZstd level 3Best speed/ratio for typed columns
Sort key(valid_from, cluster_key)Co-locates related rows in one row group
Bloom filtersIdentity-typed columnsSkip row groups that cannot contain the join key
Zone mapsvalid_from columnSkip row groups outside the temporal predicate
Page indexSorted columnsPage-level skipping within a row group

RELATA_FLUSH_SEGMENT_MAX_ROWS (default 250000) sets the max rows per flushed segment. A larger flush delta is split into ceil(delta/N) segments. 0 = unbounded, single segment per flush (legacy behavior).

The mutable tail is a small WAL-spilled segment kept in RAM and on local NVMe. On query, it is merged with the Parquet main store via copy-on-write. This is what lets writes land without waiting for a Parquet compaction cycle.

Three-tier cache

TierTechnologyLatencyScope
L1foyer (in-process, NVMe-backed, relata-cache)sub-msSingle reader node
L2Consistent-hash ring across reader nodes~1 msN=2 hot replicas
L3S3 / R2 / GCS / MinIO20–100 msDurable, all data
  • L1 admission uses S3-FIFO (scan-resistant — a full-table scan does not evict hot keys).
  • L2 fanout is bounded at N=2; reads beyond the hot working set go directly to L3.
  • Writes invalidate L1 and L2 on the owning shard synchronously, then propagate to replica shards asynchronously.

Vector storage — three tiers

The vector index has three tiers, each with different RAM/disk trade-offs:

TierStructureLocationRole
Hot (RAM)Custom HNSW (vector.rs)RAMFull graph + vectors resident; fastest recall
Warm (DiskANN)PagedAnnIndex — HNSW-backed with object-store segments (vector_diskann.rs)Disk / object storeRAM-resident graph, disk-resident vectors
Cold (IVF)IVF bucket staging areaObject storeOverflow; spills when RELATA_VECTOR_COLD_RESIDENT_MAX is exceeded

RELATA_VECTOR_COLD_RESIDENT_MAX (default 100000) is the soft cap on RAM-resident vectors in an IVF cold bucket's staging area before it spills to the PagedAnnIndex. Only active when an object store is configured.

RELATA_DISKANN_MAX_RESIDENT (soft cap on RAM-resident vectors before the index warns to shard or restart; 0/unset = unbounded).

The HNSW implementation is custom — it is not a wrapper around an external library. Supported distance metrics: cosine (primary), L2, dot product.

Per-column bloom filters

Each Identity-typed column has a bloom filter per Parquet row group. The query planner consults these before opening a row group. A point lookup on a phone number or IBAN typically skips 90–99% of row groups before reading a byte of actual data.

Summary index

A per-type summary index maintains COUNT and SUM aggregates incrementally as rows are written. COUNT(*) and SUM(col) on an unfiltered type resolve in O(1) without scanning any Parquet data.

Tamper-evident commit manifests

Every commit produces a manifest JSON. Manifests form an append-only hash chain — each one references the SHA-256 of the previous manifest.

{
  "commit_id": "01HQ8X3F9...",
  "branch": "main",
  "prev_commit_hash": "sha256:9f86d081...",
  "files_added": [
    "tables/Profile/2025-11-11.parquet",
    "audit/2025-11-11T00:00:01Z.ndjson"
  ],
  "manifest_root": "sha256:e3b0c442...",
  "writer_principal": "oidc:alice@example.com",
  "system_from": 1762828800000000000,
  "signatures": [
    { "kid": "writer-key-1", "alg": "Ed25519", "sig": "base64:..." }
  ]
}
  • Verified offline by relata doctor or online via GET /audit/count (returns chain_valid: true|false).
  • A missing manifest file is itself a tamper signal — the chain references a hash that no longer resolves.
  • Signatures are optional.

RAM budget and disk spill

RELATA_STORE_MAX_RAM_MB sets the row-store RAM budget before spill-to-disk:

Set it explicitly for any node whose data may outgrow RAM; the cap is byte-aware, so small datasets never spill. Check the default for your build in the environment variable reference. Every in-memory structure has a paged counterpart for when the budget is exceeded:

StructurePaged backend
Authoritative rowsSpill-to-disk mutable tail on local NVMe
Full-text postingsDiskIndexSource
Vector indexPagedAnnIndex + IVF cold tier
Graph adjacencyPagedCsrGraph
Identity indexLive-paged, incremental spill

Cold restart and lazy loading

PhaseDuration (10M rows, single node)
WAL + Parquet flush (graceful shutdown)~15 s
Cold-load from Parquet (eager)~55 s
Cold-load from manifest catalog only (lazy)under 1 s

RELATA_LAZY_RESTART=true loads only the manifest catalog at startup — O(manifest), not O(rows). Segments hydrate on demand when first queried. The eager default and the trade-off between start-up time and warm-cache latency are described in Capacity planning and WAL retention.

RELATA_HYDRATE_RECENT_SEGMENTS=N (default 0) hydrates the newest N segments into RAM at startup, trading startup time for warm-cache latency on the most recent data.

See also