Storage Engine#

One S3-compatible bucket (or local directory on the free profile). Open formats throughout — Parquet, Arrow IPC, JSON manifests, content-addressed blobs. Hash-chained for tamper evidence. No proprietary on-disk format. A file written by RelataDB is readable by any Parquet reader, any S3 client, any Arrow consumer.

The bucket is the database#

All durable data lives in one S3-compatible bucket or, on free, under one local directory (RELATA_DATA_DIR). The layout is identical across profiles; only the backend changes. Supported object stores: AWS S3, Cloudflare R2, Google Cloud Storage, Azure Blob (via S3 compatibility), MinIO, local disk.

RELATA_PROFILE=free    RELATA_DATA_DIR=/var/lib/relata   cargo run -p relata-cli -- serve
RELATA_PROFILE=server  AWS_ENDPOINT_URL=https://...      \
                        AWS_BUCKET=relata-prod            cargo run -p relata-cli -- serve

On-disk layout#

relata-data/
├── ontology/          # schema-as-code, git-branched, one file per type
├── tables/            # one directory per ObjectType / EventType
│   ├── Profile/
│   │   ├── 2025-11-11.parquet
│   │   ├── 2025-11-12.parquet
│   │   └── _bloom/    # per-column bloom filters
│   └── CallEvent/
├── graph/             # CSR segments + incremental degree index, one dir per LinkType
├── vectors/           # HNSW graph + DiskANN warm tier, one dir per (type, column)
│   ├── Post.embedding/
│   └── MediaContent.clip/
├── fulltext/          # custom BM25 inverted index (integer posting lists)
├── identity_index/    # universal lookup MV: (CanonicalKind, bytes) → observations
├── provenance/        # PROV-O assertions, content-addressed by assertion hash
├── audit/             # append-only, hash-chained log
├── blobs/             # content-addressed by SHA-256 (media, attachments)
└── manifests/         # branch HEAD pointer + commit chain, one JSON per commit

In-memory store and per-type interior locking#

The primary read path is an in-memory bi-temporal store built in relata-storage. Writes take &self, not &mut self. Each object type owns its own RwLock<TableState>, implemented with parking_lot::RwLock (PlRwLock type alias). A write to Profile does not block a read or write to Post.

parking_lot does not poison on panic, which eliminates the recover_poison() pattern.

Write-Ahead Log (WAL)#

Every write goes through the WAL before it is applied in memory. The WAL is flushed before the write is acknowledged to the client. In-memory state is a cache of the WAL; on restart, the WAL is the source of truth.

The durability level is set per-write via the X-Relata-Durability request header, or forced cluster-wide with RELATA_WAL_SYNC=always (default is interval). A bulk loader uses async; a payment record uses sync. Same table, same writer, different durability per row.

The WAL is hash-chained: each WAL entry references the hash of the previous entry. A replay that produces a hash mismatch at any entry signals corruption or tampering.

Parquet segments#

Each tables/<Type>/ directory holds Parquet data files. The default is one file per UTC day per type.

RELATA_FLUSH_SEGMENT_MAX_ROWS (default 250000) sets the max rows per flushed segment. A larger flush delta is split into ceil(delta/N) segments. 0 = unbounded, single segment per flush (legacy behavior).

The mutable tail is a small WAL-spilled segment kept in RAM and on local NVMe. On query, it is merged with the Parquet main store via copy-on-write. This is what lets writes land without waiting for a Parquet compaction cycle.

Three-tier cache#

• L1 admission uses S3-FIFO (scan-resistant — a full-table scan does not evict hot keys).
• L2 fanout is bounded at N=2; reads beyond the hot working set go directly to L3.
• Writes invalidate L1 and L2 on the owning shard synchronously, then propagate to replica shards asynchronously.

Vector storage — three tiers#

The vector index has three tiers, each with different RAM/disk trade-offs:

RELATA_VECTOR_COLD_RESIDENT_MAX (default 100000) is the soft cap on RAM-resident vectors in an IVF cold bucket's staging area before it spills to the PagedAnnIndex. Only active when an object store is configured.

RELATA_DISKANN_MAX_RESIDENT (soft cap on RAM-resident vectors before the index warns to shard or restart; 0/unset = unbounded).

The HNSW implementation is custom — it is not a wrapper around an external library. Supported distance metrics: cosine (primary), L2, dot product.

Per-column bloom filters#

Each Identity-typed column has a bloom filter per Parquet row group. The query planner consults these before opening a row group. A point lookup on a phone number or IBAN typically skips 90–99% of row groups before reading a byte of actual data.

Summary index#

A per-type summary index maintains COUNT and SUM aggregates incrementally as rows are written. COUNT(*) and SUM(col) on an unfiltered type resolve in O(1) without scanning any Parquet data.

Tamper-evident commit manifests#

Every commit produces a manifest JSON. Manifests form an append-only hash chain — each one references the SHA-256 of the previous manifest.

{
  "commit_id": "01HQ8X3F9...",
  "branch": "main",
  "prev_commit_hash": "sha256:9f86d081...",
  "files_added": [
    "tables/Profile/2025-11-11.parquet",
    "audit/2025-11-11T00:00:01Z.ndjson"
  ],
  "manifest_root": "sha256:e3b0c442...",
  "writer_principal": "oidc:alice@example.com",
  "system_from": 1762828800000000000,
  "signatures": [
    { "kid": "writer-key-1", "alg": "Ed25519", "sig": "base64:..." }
  ]
}

• Verified offline by relata doctor or online via GET /audit/count (returns chain_valid: true|false).
• A missing manifest file is itself a tamper signal — the chain references a hash that no longer resolves.
• Signatures are optional but recommended on cluster profile writers.

RAM budget and disk spill#

RELATA_STORE_MAX_RAM_MB sets the row-store RAM budget before spill-to-disk:

• server / cluster — defaults to 1024 MB. Small datasets never spill; the cap is byte-aware.
• free — unbounded, for local dev.

Explicit values override the profile default. Every in-memory structure has a paged counterpart for when the budget is exceeded:

Cold restart and lazy loading#

RELATA_LAZY_RESTART=true loads only the manifest catalog at startup — O(manifest), not O(rows). Segments hydrate on demand when first queried. This is the default on server and cluster profiles. free remains eager.

RELATA_HYDRATE_RECENT_SEGMENTS=N (default 0) hydrates the newest N segments into RAM at startup, trading startup time for warm-cache latency on the most recent data.

See also#

• Architecture Overview
• Data Model
• Backup & Restore
• Environment Variables