Search and retrieval

Relata DB 3.0
In 3.0 the engine's LLM translator (RELATA_LLM_URL) is removed along with the nl_query tool; natural-language parsing belongs to your application, and embeddings for stored data arrive precomputed from OpenETL. Search routes are under /v1/search. The 2.5.8 line, kept in the 2.5.8 documentation snapshot, is described by the older material on this page.

RelataDB combines search with governance, time, identity, and provenance.

Search surfaces

SurfacePurpose
BM25/full-textkeyword search and snippets
Vector searchsimilarity over embeddings. Since v1.1 embeddings are caller-supplied — either pre-compute and send _emb_text in the row payload, or run the embedder sidecar so the media-worker drain cycle populates them post-ingest. SIMILAR ranks by multi-vector max-pool over all _emb_* slots and resolves the seed by id or _pk.
Image near-dupperceptual-hash (aHash+dHash) match. Images hash in-tree (pure-Rust decode, no sidecar), so a re-encoded/edited copy is found within a small Hamming distance out of the box; video/audio fingerprints still need the decoder sidecar.
Hybrid retrievalcombine lexical, vector, and graph signals
Identity lookupretrieve by canonical identifiers and identity variants
Graph traversalrelationship/path retrieval
Temporal filtersretrieve as of valid-time or system-time
Provenance filtersinspect where assertions came from

Ask in plain English (no LLM required)

The nl_query MCP tool / REST endpoint translates a plain-English question to governed SQL and executes it. With no RELATA_LLM_URL configured a deterministic in-tree translator handles a documented set of intents — so analysts get structured answers with zero external dependencies. Set RELATA_LLM_URL (Ollama / vLLM / LM Studio) for richer free-form parsing.

Supported deterministic phrasings:

IntentSayRuns
Graph path"paths between X and Y", "path from X to Y"PATHS_BETWEEN('X','Y', MAX_HOPS => 5)
Browseanything without an intent verbSELECT * FROM <Type> LIMIT 50

A recognised intent whose entities can't be extracted returns a clear "couldn't translate; try …" error — it never guesses and runs the wrong query. The active translator (local vs LLM) is reported in the startup posture block.

World-class search UX

RelataDB's search engine includes features inspired by Meilisearch and Typesense. The /search REST endpoint accepts JSON parameters for fine-grained control. Every parameter accepts either spelling — disable_on_attributes or disableOnAttributes, min_word_size or minWordSize — so a Meilisearch-compatible camelCase client body works unchanged alongside this repo's own snake_case convention.

Subword tokenization

camelCase, PascalCase, and letter↔digit transitions are split during tokenization so "iPhone14Max" produces tokens ["i", "phone", "14", "max"]. This means "phoneNumber" is findable by searching "phone". snake_case and kebab-case already split via the non-alphanumeric pass. Applied symmetrically at index and query time.

Matching strategy

Controls which query terms are required vs optional when scoring documents. All strategies still BM25-rank results; the difference is which documents are eligible.

StrategyBehaviourBest for
any (default)Every term is optional (OR) — pre-default behaviourHigh recall
allEvery term must be present (AND)Precision queries
lastOnly the last tokenised term is required; earlier terms optionalSearch-as-you-type
frequencyRarest terms required; common terms optionalMixed rare + common queries
booleanQuery text carries explicit AND/OR/NOT operatorsExact-combination queries
curl -s -X POST http://127.0.0.1:9090/search \
  -H 'Content-Type: application/json' \
  -d '{"query":"alice wonderland","type":"Person","limit":10,"matching_strategy":"all"}'

Boolean operators — with "matching_strategy": "boolean", uppercase AND / OR / NOT in the query text combine per-term BM25 matches as posting-list set operations (left-associative; a bare space means OR):

curl -s -X POST http://127.0.0.1:9090/search \
  -H 'Content-Type: application/json' \
  -d '{"query":"sanction OR ofac NOT expired","type":"Alert","limit":10,"matching_strategy":"boolean"}'

The same grammar is reachable from SQL as MATCH(col, 'q', BOOLEAN) and from the typed /search door as rank_by: ["boolean", "<column>", "<query>"].

Per-query typo tolerance

Override the global RELATA_SEARCH_PRESET per query. Gate fuzzy expansion by word length, disable specific words, or turn typos off entirely.

{
  "query": "john smith",
  "type": "Person",
  "limit": 10,
  "typo_tolerance": {
    "enabled": true,
    "min_word_size": 5,
    "disable_on_words": ["smith"],
    "disable_on_attributes": ["email"]
  }
}
FieldDefaultDescription
enabledtrueMaster toggle. false = exact-match only.
min_word_size0Words shorter than this stay exact (Meili default: 5 for 1-edit, 9 for 2-edit).
disable_on_words[]Specific query terms that should not be fuzzy-expanded.
disable_on_attributes[]Field names where typo tolerance is disabled.

Typo tolerance in SQL MATCH — POST /query

The same typo_tolerance object is accepted by POST /query alongside the SQL text. When set, SQL MATCH / PHRASE / SUFFIX / INFIX conditions apply per-token levenshtein fuzzy matching: 1 edit for words ≥ min_word_size (default: any length), 2 edits for words ≥ 9 chars.

POST /query
{
  "sql": "SELECT * FROM Person WHERE MATCH(name, 'jon')",
  "typo_tolerance": {
    "enabled": true,
    "min_word_size": 3
  }
}

This will also return rows where name contains "john" (1-edit match).

BM25F per-field weights

Boost results where a query term appears in high-priority fields (e.g. title outweighs body). Pass fieldWeights — a map of field names to f32 multipliers. The BM25 score is scaled by the highest weight of any named field that contains a query term. Fields absent from the map default to 1.0.

{
  "query": "database",
  "type": "Article",
  "limit": 10,
  "fieldWeights": {"title": 3.0, "body": 1.0, "tags": 5.0}
}

Rows whose query terms appear only in tags are scaled ×5; rows where the term appears in title are scaled ×3; rows with the term only in body are unaffected. Omitting fieldWeights (or passing {}) leaves all scores unchanged.

The FACETS SQL clause returns per-attribute value counts. Facets are computed over the full matching set (up to 100 000 rows), not just the top-k returned rows — so browse-UI facet counts are accurate even when LIMIT is small.

For numeric facet columns, the response also includes facetStats with min, max, sum, avg, and count:

{
  "facets": {"status": {"active": 42, "closed": 7}},
  "facetStats": {"amount": {"min": 100, "max": 50000, "sum": 125000, "avg": 2314, "count": 54}}
}

Estimated total hits

Both /query and /search responses include estimatedTotalHits — the full matching-set size (accurate up to the 100k facet cap; a lower bound otherwise) — alongside count/rows (the actual returned rows). Use this for pagination UX.

Multi-search / federated

POST /multi-search runs N independent queries in one round-trip and returns combined results in input order — useful for dashboard widgets and federated type searches. Each sub-query can carry its own typo_tolerance and matching_strategy, overriding the tenant-wide preset for that one sub-query only — the rest of the batch keeps the default:

curl -s -X POST http://127.0.0.1:9090/multi-search \
  -H 'Content-Type: application/json' \
  -d '{"queries":[{"query":"alice","type":"Person","limit":5},{"query":"transfer","type":"Transaction","limit":10,"typo_tolerance":{"enabled":false}}]}'

Typed query door — namespace().query() (no SQL)

For the search / RAG persona, /search accepts a second body shape — a typed JSON query that compiles to a governed SQL plan server-side, so you never hand-build SQL. Governance (purpose, ACL, cell masking, audit) is identical to a hand-written query. The typed shape is detected by the absence of a query key (its presence keeps the legacy Meilisearch shape above on the same route).

curl -X POST http://127.0.0.1:9090/search \
  -H 'Content-Type: application/json' \
  -d '{"from":"Document",
       "rank_by":["bm25","title","graph retrieval"],
       "filters":[{"field":"status","op":"eq","value":"published"}],
       "include_attributes":["id","title"],"limit":10}'
# returns the governed /query envelope: {rows, columns, ...}
FieldDescription
from (alias type)Object type to query.
rank_by["bm25"|"text","<column>","<query>"] (BM25 full-text, ORDER BY _score DESC), or ["vector","ann","<query>"] (HYBRID_SEARCH — server embeds the text). Since 2.4.0, BM25 ranking scores with real per-field BM25F weights (#2561) rather than substring presence.
filters[{field, op, value}] (ops eq|ne|gt|gte|lt|lte|like|ilike|in|between), a {and:[...]} / {or:[...]} object, or a single condition.
include_attributesColumn projection (default *).
consistency"strong" (default) | "eventual" (forward-compat).
compute_attributes{label: rank_expr} compiled to a trailing COMPUTE clause — see below.

Python SDK — flagship namespace() surface

from relata import RelataClient
 
with RelataClient("http://localhost:9090", purpose="rag") as client:
    docs = client.namespace("Document")
    docs.write([{"id": "d1", "title": "Knowledge graphs 101", "body": "..."}])  # schemaless (POST /ingest/auto)
    res = docs.query(
        text="graph retrieval",
        match_column="title",
        filters=[{"field": "status", "op": "eq", "value": "published"}],
        limit=10,
    )
    for row in res:
        print(row["title"])

.write() is schemaless: POST /ingest/auto auto-creates the type on first write with field types inferred from the rows — no DDL. Async mirrors (AsyncNamespace, await docs.query(...)) and one shared connection pool across every namespace.

Composable ranking + COMPUTE side outputs (SQL)

In SQL, ranking is a full expression tree and per-hit signals are a COMPUTE clause (the typed door's rank_by/compute_attributes lower to these):

-- Blend BM25 with a saturated popularity signal, a time decay, and geo distance.
SELECT * FROM Article WHERE MATCH(content, 'ai')
RANK BY SUM(BM25(), PRODUCT(0.5, SATURATE(popularity, 100, 2)),
            DECAY(published_at, 86400, 1), DIST(location, 1.0, 2.0))
LIMIT 10;
 
-- Attach per-hit signals without changing matching or order.
SELECT * FROM Doc WHERE MATCH(content, 'ai')
RANK BY VECTOR()
COMPUTE bm25_score AS BM25(), snippet AS HIGHLIGHT(content)
LIMIT 10;

Ranking primitives: SUM / MAX / PRODUCT / SATURATE / DECAY / ATTRIBUTE(name) / DIST(attr, x, y, …) / BM25() / VECTOR(). HIGHLIGHT(field) is compute-only (renders a snippet).

Namespace branching + pinning

  • Branch — constant-time COW: BRANCH dev FROM main (SQL) or POST /v1/namespaces/dev/branch with {"branch_from":"main"}. See Branching & Namespaces for the full surface (including how namespace branches differ from schema branches and sub-tenant namespaces).
  • Pin — RELATA_PINNED_NAMESPACES=acme,contoso reserves a dedicated NVMe cache slice per hot namespace so its index is evict-immune under pressure.

Multi-query batch + reciprocal-rank fusion (RRF)

One round-trip, up to 16 typed subqueries, fused with rank-based RRF. This is the "build a search dashboard in one call" feature — multiple ranking strategies against one snapshot, merged into a single ranked list.

POST /search with {"queries": [...], "rerank_by": ["RRF", {"rank_constant": 60}], "limit": 20}:

curl -X POST http://127.0.0.1:9090/search \
  -H 'Content-Type: application/json' \
  -d '{
    "queries": [
      {"from":"Document", "rank_by":["bm25","title","graph retrieval"], "limit":50},
      {"from":"Document", "rank_by":["field_weight",{"title":3.0,"body":1.0}, "graph retrieval"], "limit":50},
      {"from":"Document", "rank_by":["vector","ann","graph retrieval"], "limit":50}
    ],
    "rerank_by": ["RRF", {"rank_constant": 60}],
    "limit": 20
  }'

What's happening:

  • All subqueries are pinned to a single AS OF SYSTEM TIME snapshot, sampled once before fan-out — concurrent writes cannot split results.
  • Each subquery runs through the same governed /query path (PURPOSE, ACL, tenant, cluster fan-out identical to N independent calls).
  • rerank_by: ["RRF", {rank_constant: K}] fuses the per-subquery ranked lists using reciprocal-rank fusion (score = Σ 1/(K + rank_i)). Default K = 60 (the standard RRF constant). Fused rows carry _rrf_score.
  • Vector/ANN rank_by is rejected in batch mode (the AS OF clause isn't in the HYBRID_SEARCH grammar today) — use BM25 / field-weight subqueries for the batch path.

Response shape:

{
  "results": [ {/* per-subquery result set */}, {/* ... */} ],
  "as_of_system": 1735490000000000000,
  "processing_time_ms": 18,
  "fused": {
    "rank_constant": 60,
    "rows": [ /* merged, ranked, with _rrf_score */ ],
    "data": [ /* ... */ ]
  }
}

Tips & takeaways

  • RRF needs ≥ 2 subqueries. A single subquery with rerank_by: ["RRF"] is rejected — there's nothing to fuse.
  • Different rank_by strategies make RRF sing. Combine BM25 (lexical) + field_weight (weighted BM25F) + filter variants; each contributes a different ranking signal, and RRF is robust to score-scale differences (it uses ranks, not scores).
  • One snapshot = consistent dashboarding. Because all subqueries share one AS OF, the facets/counts/hits in each pane are mutually consistent — no flicker from writes landing mid-search.
  • rank_constant tuning: lower K (e.g. 1) weights top ranks more heavily (good when you trust the top of each list); higher K (e.g. 100) smooths toward a popularity vote. 60 is the literature default and a safe start.
  • Governance is per-subquery, not bypassed. PURPOSE / ACL / cell masking apply to every subquery independently — a row you can't see in one subquery won't appear in the fused result either.

Cross-ref: Hybrid search (concepts) · Vector params · SQL RANK BY expression · Search & Retrieval reference top

Governance rules

Search is not a policy bypass. Search and retrieval paths must preserve:

  • optional purpose (recorded for audit),
  • ACL pushdown,
  • cell masking,
  • provenance handling,
  • temporal semantics,
  • audit/cost accounting,
  • tenant scoping.

POST /search/POST /multi-search default to the caller's own tenant automatically on any type that declares a tenant_id column — a tenant-scoped bearer token no longer needs to pass filters.tenant_id to avoid an unscoped, cross-tenant result set. An explicit filters.tenant_id (or legacy filter.tenant_id, or per-sub-query on /multi-search) that disagrees with the token's own tenant is rejected with a typed 403 tenant-mismatch rather than silently overridden. Every hit on both doors carries a top-level tenant_id attribution field.

Example topics

  • Search documents by text.
  • Search as of a system time.
  • Retrieve with provenance.
  • Combine identity lookup with document search.
  • Explain why a restricted result is masked or absent.

Validation rule

Any published latency/recall claim must cite a benchmark, conformance row, or reproducible command output.