⚙️ Understanding the Retrieval Pipeline#

Know what happens between POST /ingest and SELECT. This page explains the internal pipeline so you can reason about latency, eventual consistency windows, and why your search results appear 2–3 seconds after ingest.

The big picture#

WRITE PATH (milliseconds)                    READ PATH (milliseconds)
                                            
POST /ingest                                  
  │                                          
  ├─ validate purpose                        
  ├─ WAL-ahead (fsync) ← durability          
  ├─ store.insert (bi-temporal row)           
  ├─ index_secondary (sync: equality)         
  ├─ index_row_embeddings (sync: vector)      
  ├─ enqueue_pending_fts (deferred: BM25)     
  │                                          
  │  ~1–5ms ack returned to client            
  │                                          
  │   ┌── background (every 2s) ──┐           
  │   │ flush_pending_fts          │           
  │   │ → BM25 posting lists built │           
  │   │ → block-max WAND updated   │           
  │   │                            │           
  │   │ HNSW vector insert         │           POST /query
  │   │ → graph adjacency updated  │             │
  │   │ → identity index linked    │             ├─ Planner: parse + purpose + ACL
  │   └────────────────────────────┘             ├─ Executor: scan/filter/aggregate
  │                                               ├─ Index lookup: BM25 / HNSW / identity
  ▼                                               ├─ ACL: bitmap filter + cell masking
  Rows queryable via SQL (~instant)               ├─ Result cache (hit → skip execution)
  Rows searchable via HYBRID_SEARCH (~2–3s)       ▼
                                                  JSON / Arrow / SSE response

Write path — what happens when you ingest#

Step 1: Accept and acknowledge (~1ms)#

ingest("Person", [{"id": "alice", "name": "Alice Chen", "phone": "+14155550100"}])

The handler:

1. Validates PURPOSE — is "analytics" registered? Reject if not.
2. Validates schema — ontology check (is phone a valid field for Person?).
3. Pushes to IngestQueue — returns {rows_queued: 1, task_id: "itsk_..."} immediately.

The ack is a queue-ack, not a confirmed-write count. The background writer hasn't run yet. Poll GET /ingest/tasks/:task_id for rows_written vs rows_dropped.

Step 2: Background writer drain (every 10ms)#

A dedicated ingest-writer task (under supervise for crash recovery) drains the queue:

1. build_pre_batches — governance (ontology validation) + WAL line serialization. Runs across rayon (one thread per object type).
2. flush_confirmed_to_wal — WAL-ahead durability: every batch is written + fsynced to wal.jsonl BEFORE any store mutation. Crash here = clean replay, no data loss.
3. resolve_existing_for_drain — fresh identity lookup at drain time (#3824 fix). Determines insert-vs-update.
4. store.insert_with_tenant — writes the bi-temporal row under a per-type read guard (interior locking, &self).

Step 3: Synchronous indexing (during insert)#

These indexes are built synchronously — they're available for queries immediately after the writer drains:

Step 4: Deferred indexing (background, every 2s)#

A separate index-work-drain task runs every 2 seconds:

This is why HYBRID_SEARCH has a 2–3 second eventual-consistency window. SQL SELECT and WHERE work immediately (they use the synchronous secondary index). But BM25 full-text search and vector similarity need the background drain to build their indexes. This is by design — the ingest hot path stays fast (WAL + store write), and indexing happens off-thread.

Read path — what happens when you query#

Step 1: Planner (parse + validate + plan)#

query("SELECT name FROM Person WHERE company = 'Acme' AND risk = 'HIGH'")

1. Parse — SQL → AST (the restricted Relata grammar).
2. PURPOSE check — is the declared purpose registered? Strict mode rejects unknown purposes; open mode accepts any non-empty string.
3. ACL precompute — builds a bitmap: for this principal + this object type, which rows are visible? Compiled once, reused across the scan.
4. Plan cache — LRU keyed by (SQL template, principal, purpose, ontology version). A cache hit skips parsing + ACL precompute entirely.

Step 2: Executor (scan + filter + project)#

The executor dispatches on query type:

SELECT path#

scan_as_of_with_limit_and_bloom
  ├─ Bloom filter pushdown — skip segments that provably can't match WHERE
  ├─ Range index pruning — skip segments outside the WHERE range
  ├─ Stream live rows — bi-temporal filter: system_to == NOW (current version only)
  ├─ ACL bitmap filter — deny-wins: if any policy denies, row is excluded
  ├─ CellFilter — mask/redact individual columns per policy
  └─ Projection + LIMIT — only send requested columns, stop at N rows

Bloom + range pushdown (#3895) is why WHERE id = 'alice' on a million-row table returns in <1ms — the scan skips 99% of segments without reading them.

Aggregate path (GROUP BY, SUM, AVG)#

execute_grouped_columnar_pass
  ├─ Scan → typed ColumnBatch (columnar, not row-by-row)
  ├─ DataFusion bridge — vectorized hash aggregation
  └─ Merge partial results across shards (cluster mode)

The columnar aggregate path is ~1.3× faster than row-by-row for filter-free GROUP BY (#984). For filtered aggregates, the row path runs first, then the columnar pass aggregates the filtered subset.

HYBRID_SEARCH path#

execute_hybrid_search
  ├─ BM25 channel: FullTextIndex.search → WAND scoring → top-K candidates
  ├─ Vector channel: HNSW.graph_search → cosine similarity → top-K candidates
  ├─ Identity channel: IdentityIndex lookup → canonical matches
  └─ Reciprocal Rank Fusion (RRF): merge 3 ranked lists → unified top-K

RRF doesn't need score calibration between channels — it uses rank positions only. This is why BM25 (exact keyword) and vector (semantic) fuse well despite having completely different score scales.

Other operators#

Step 3: Result cache#

Before execution, the planner checks the result cache:

• Key: (SQL template, params, principal, purpose, tenant, ontology version)
• Hit: return cached rows instantly (sub-millisecond)
• Miss: execute → cache the result
• Invalidation: on any ingest to a type in the query's FROM clause

The cache is tenant-scoped — one tenant's cached results are never served to another. And it's ontology-versioned — a schema change (ALTER TYPE) invalidates all cached plans for that type.

Latency cheat sheet#

Why this design?#

The 2-second indexing window is deliberate. The alternative — indexing synchronously on the ingest hot path — would make every write pay the BM25 + HNSW + graph build cost. That's what caused the 107-second remember() stall (#3794) before the fix: the FTS block-max rebuild was O(n²) and ran inline.

By deferring indexing to a background drain:

• Ingest stays fast (WAL + store write only, ~1–5ms)
• Indexing is batched (one drain builds many rows' indexes at once)
• The hot path doesn't block on index build latency
• Crash recovery is clean (WAL replay + drain rebuilds any lost indexes)

The trade-off: eventual consistency for search (2–3s). SQL queries on indexed fields are immediate. This matches the operational reality — you ingest a batch, then search a few seconds later, not in the same millisecond.

Monitoring the pipeline#

# Queue depth — how many batches are waiting to drain
status = httpx.get(f"{BASE}/status", headers=H).json()
print(f"Queue depth: {status['queue_depth']}")
 
# Metrics — indexing lag
metrics = httpx.get(f"{BASE}/metrics", headers=H).text
# Look for:
#   relata_ingest_rows_total
#   relata_wal_fsync_total
#   relata_wal_fsync_latency_seconds_sum
#   relata_pending_fts_count  (should trend to 0)
#   relata_pending_index_work_count
 
# Health — is the drain task alive?
health = httpx.get(f"{BASE}/health/ready", headers=H).json()
# worker_health: {ingest-writer: "alive", index-work-drain: "alive"}

If queue_depth is climbing or pending_fts_count isn't trending to zero, the background tasks are falling behind — check CPU and disk I/O.

Key takeaways#

1. SQL queries (SELECT, WHERE, COUNT, GROUP BY) are immediately consistent — secondary indexes build synchronously.
2. Search (HYBRID_SEARCH, SIMILAR TO) has a 2–3 second eventual-consistency window — BM25 and HNSW build on the background drain.
3. Identity resolution (LOOKUP_IDENTITY) is immediate — the identity index is synchronous.
4. The result cache is tenant-scoped and ontology-versioned — safe, fast, self-invalidating.
5. The WAL-ahead design means zero data loss on crash — replay rebuilds the store + indexes.
6. All governance (PURPOSE, ACL, cell masking) is in the read path — not bolted on after.

Next: Hybrid Search — use the indexes this pipeline builds.

Every detail on this page is verified against the RelataDB source code. The pipeline is implemented, not designed.