Scaling#

RelataDB scales from a laptop (free profile, ~1 B entities) to a multi-node cluster (cluster profile, 100 B–1 T+ entities). The key levers: RAM walls that engage automatically on server/cluster, paged backends that spill to object storage, streaming execution that never OOMs, and lazy cold-restart that brings large nodes back in seconds.

Choose the right profile#

# Local dev, CI, demos — unbounded RAM, eager restart
RELATA_PROFILE=free relata serve
 
# Single-node production — 1 GB RAM wall, disk-first walls, lazy restart on
RELATA_PROFILE=server RELATA_BEARER_TOKEN=<your-strong-token> relata serve
 
# Multi-node (alpha) — same as server plus coordination
RELATA_PROFILE=cluster RELATA_BEARER_TOKEN=<your-strong-token> relata serve

Single-node server is the production-recommended profile today. cluster is alpha — petabyte-scale sharding is still in progress.

All scaling knobs#

Set any of these to override the profile default. Explicit values always win.

Tune RAM walls for your node#

The 1 GB default is conservative. On a 32 GB node running only RelataDB, raise it:

RELATA_PROFILE=server \
RELATA_BEARER_TOKEN=<your-strong-token> \
RELATA_STORE_MAX_RAM_MB=16384 \
RELATA_GRAPH_RAM_BUDGET_MB=8192 \
RELATA_IDENTITY_RAM_BUDGET_MB=4096 \
relata serve

The caps are byte-aware — small datasets never spill. The wall only bites when you actually exceed the budget.

Disk-first walls and paged backends#

Every large structure has a paged backend. When the RAM budget is exceeded, data pages out to the object store and pages back in on demand. RAM becomes a cache; the object store is the truth.

Paging adds disk-read latency on cold paths but eliminates OOM. On hot paths, the cache hierarchy absorbs most reads.

Streaming execution#

The execution engine never materialises full intermediate results. Big joins, high-cardinality aggregates, and large result sets all stream:

# This query on a 50 M-row table streams results — does not OOM
relata query "SELECT dept, COUNT(*) FROM Person GROUP BY dept"

Hash-join and aggregate operators spill intermediate batches to the object store when they exceed the RAM budget, then merge-read in a streaming pass.

Columnar analytics#

Filter-free GROUP BY uses a vectorised columnar path that reads only the referenced columns:

-- Fast: only reads the "dept" column
SELECT dept, COUNT(*) FROM Person GROUP BY dept;
 
-- Slower: filter needs the full row to evaluate "active"
SELECT dept, COUNT(*) FROM Person WHERE active GROUP BY dept;

For analytical workloads on large types, design queries to avoid per-row filters where possible, or add a bloom-filtered column to narrow the scan before the filter.

Lazy restart#

RELATA_LAZY_RESTART=true (default on server/cluster) loads only the manifest catalog on startup. Rows hydrate on the first query that touches them. A 10 M-row node returns to ready in seconds instead of ~55 s.

# Pre-warm the 5 most recent segments at startup, stay lazy for the rest
RELATA_LAZY_RESTART=true \
RELATA_HYDRATE_RECENT_SEGMENTS=5 \
relata serve

This is the right setting for most production nodes: fast restart with warm cache for recent data.

Cache hierarchy#

S3-FIFO admission at L1 is scan-resistant — a one-off full-table scan does not evict hot working set. L2 consistent-hashing means a request routed to the right reader node hits warm cache without cross-node fetch.

Graph and vector scaling#

Graph — CSR adjacency stays RAM-resident up to RELATA_GRAPH_RAM_BUDGET_MB, then pages to PagedCsrGraph. The incremental degree index keeps DEGREE() queries O(1) regardless of graph size.

Vectors — HNSW is the in-memory graph for recall. DiskANN is the warm tier backed by object-store segments. The IVF cold bucket stages vectors in RAM up to RELATA_VECTOR_COLD_RESIDENT_MAX before spilling to PagedAnnIndex. Increase RELATA_DISKANN_MAX_RESIDENT on nodes with headroom to avoid premature spill warnings.

Vector tier routing (hot → warm → cold)#

The true on-disk beam search path (DiskAnnIndex::beam_search_true_disk, ADR-282) reads a node's graph adjacency + product-quantized (PQ) code as a single 4 KB sector fetch — never the whole segment — cached by RELATA_DISKANN_SECTOR_CACHE_MB (default 256 MB) under a per-query RELATA_DISKANN_IO_BUDGET (default 512 sector reads, fail-open), with only the top RELATA_DISKANN_RERANK_FACTOR · k candidates (default factor 6) paying a full-vector read for exact rerank. This is the mechanism that lets the warm tier bound RAM independently of corpus size instead of scaling with it.

Caveat (read before relying on this for capacity planning): this path is implemented and exercised at bench scale, but is not yet on relata-query's live query-dispatch path (vector search there is still synchronous; the true-disk beam search is async) and has not been benchmarked at the 100 M×768-d / recall@10 ≥ 0.95 target the design calls for. Until that follow-up lands, plan warm-tier capacity around the existing DiskANN object-store-segment behavior above, not the true-disk sector-paging numbers.

Cluster mode (alpha)#

Cluster adds coordinator/reader/writer/indexer roles, hash partitioning, and multi-region replication. Single-node server is production-recommended until the petabyte-scale sharding work lands; do not run cluster in production for data you cannot afford to lose until the alpha label is removed.

➡️ The canonical cluster recipe — every required env var (plain NODE_ID / CLUSTER_ROLE / CLUSTER_PEERS — not the RELATA_-prefixed names, which are different, unrelated vars), a tested local 3-node example, and the gotchas (gRPC port must match across nodes, CLUSTER_AUTH_TOKEN fails silently, three different bind-var conventions) — lives in Cluster Setup. Scaling-wise, cluster inherits the same RAM walls, paged backends, streaming execution, and lazy restart described above; the cluster-specific levers are the partition-count (RELATA_CLUSTER_SHARDS, default 8) and the cross-region merge cadence (RELATA_CROSS_REGION_MERGE_INTERVAL_SECS).

Performance characteristics#

Measured on representative hardware at 10 M rows. Use these as order-of-magnitude guides.

Cell masking is expensive because it must inspect every field of every returned row. Apply it to specific columns in specific policies, not globally.

Run the benchmark suite#

Before a capacity change, establish a baseline:

# Quick gate — under 1 min, run before every merge
cargo run -p relata-bench --release -- full --scale 100k --gate
 
# Full suite without HNSW builds (~8 min)
cargo run -p relata-bench --release -- full --scale 100k --no-ann
 
# Full suite including HNSW builds (~15 min+)
cargo run -p relata-bench --release -- full --scale 100k
 
# All scales
cargo run -p relata-bench --release -- full
 
# Memory recall at scale
cargo run -p relata-bench --release -- memory --scale 100k --all

Set RELATA_BENCH_NO_SAVE=1 to suppress writing relata-bench.json.

See also#

• Configuration — every scaling env var
• Backup & Restore — object-store setup, lazy restart
• Observability — RAM wall metrics, queue depth