🔍 Advanced Search & RAG

RelataDB's search engine is custom-built — not Tantivy, not Lucene. It combines:

  • BM25 with block-max WAND, 12-language stemmers, q-gram prefix, reverse-trigram suffix, stop words, synonyms, and highlighting
  • HNSW vector index (custom, in-memory with DiskANN warm tier)
  • Identity fusion (SmartIngest-detected canonical identifiers as a third ranking signal)
  • Reciprocal Rank Fusion (RRF) to combine all three without score calibration

Three operators expose this power: HYBRID_SEARCH, RAG_RETRIEVE, SIMILAR TO, plus the POST /search Meilisearch-compatible endpoint and POST /multi-search for federated queries.

Every output on this page is real — captured from a live server.

Setup

import httpx, json, time
 
BASE = "http://localhost:9090"
H = {"Authorization": "Bearer perftoken", "Content-Type": "application/json"}
 
def post(path, body):
    r = httpx.post(f"{BASE}{path}", json=body, headers=H, timeout=15)
    return r.status_code, r.json()
 
def query(sql):
    return post("/query", {"sql": sql, "purpose": "analytics"})
 
def ingest(type_name, rows):
    body = "\n".join(json.dumps(r) for r in rows)
    r = httpx.post(f"{BASE}/ingest?object_type={type_name}&purpose=analytics",
                   content=body, headers={**H, "Content-Type": "application/x-ndjson"}, timeout=15)
    return r.json()

Seed a searchable corpus

ingest("Article", [
    {"id": "a1", "title": "Understanding Bi-temporal Data Models",
     "body": "Bi-temporal data tracks both valid time (when facts are true) and system time (when the database recorded them). This enables point-in-time queries like 'what did we know last Tuesday?'"},
    {"id": "a2", "title": "BM25 vs Vector Search: When to Use Each",
     "body": "BM25 excels at exact keyword matching — product codes, legal citations, medical terminology. Vector search catches semantic paraphrases. Hybrid search fuses both with reciprocal rank fusion."},
    {"id": "a3", "title": "Identity Resolution at Scale",
     "body": "The same person appears as +14155550100 in call logs, alice@corp.com in email, and a passport number in border crossings. SmartIngest auto-detects and links 76 canonical types."},
    {"id": "a4", "title": "Graph Analytics for Fraud Detection",
     "body": "Money laundering follows a pattern: placement, layering, integration. Graph algorithms like PageRank and community detection surface the hub accounts and coordinated rings."},
    {"id": "a5", "title": "Governed RAG: Retrieval That Respects ACL",
     "body": "Standard RAG pipelines leak masked cells to the LLM. RelataDB applies cell-level governance before retrieval — the model never sees what it shouldn't."},
])
time.sleep(3)  # BM25 index

1. HYBRID_SEARCH — the primary search operator

1.1 Fused search (BM25 + vector + identity, default weights)

status, result = query("HYBRID_SEARCH FROM Article QUERY 'bi-temporal time travel queries' LIMIT 3")
{
  "rows": 3,
  "data": [
    {"id": "a1", "title": "Understanding Bi-temporal Data Models", "_score": 14.82},
    {"id": "a5", "title": "Governed RAG: Retrieval That Respects ACL", "_score": 6.31},
    {"id": "a2", "title": "BM25 vs Vector Search: When to Use Each", "_score": 4.97}
  ]
}

The query "bi-temporal time travel queries" matched article a1 even though the exact phrase "time travel" doesn't appear — the vector channel caught the semantic similarity.

1.2 BM25-only (exact keyword precision)

status, result = query(
    "HYBRID_SEARCH FROM Article QUERY 'BM25 vector keyword matching' LIMIT 3 WEIGHTS 0.0 1.0 0.0"
)

The WEIGHTS clause is [graph, bm25, vector]. Setting bm25=1.0, rest=0.0 runs pure keyword — no semantic fuzziness. Best for legal codes, product IDs, exact terminology.

1.3 Vector-only (pure semantic similarity)

status, result = query(
    "HYBRID_SEARCH FROM Article QUERY 'how to prevent data leakage in AI' LIMIT 3 WEIGHTS 0.0 0.0 1.0"
)

Pure vector catches semantic paraphrases — "prevent data leakage" matches "governed RAG respects ACL" without keyword overlap.

1.4 Custom fusion weights

status, result = query(
    "HYBRID_SEARCH FROM Article QUERY 'fraud detection graph' LIMIT 5 WEIGHTS 0.1 0.6 0.3"
)

Fine-tune the balance: 60% BM25 (keyword precision), 30% vector (semantic), 10% graph (identity signal).

1.5 RERANK — cross-encoder re-scoring

status, result = query(
    "HYBRID_SEARCH FROM Article QUERY 'identity resolution phone email' LIMIT 10 RERANK"
)

RERANK re-scores the top-K via a sidecar cross-encoder before returning. Requires RELATA_RERANK_URL configured — otherwise it's a no-op (results still valid, just not re-ranked).

1.6 METRIC — distance metric selection

status, result = query(
    "HYBRID_SEARCH FROM Article QUERY 'temporal data' LIMIT 3 METRIC cosine"
)

Supported metrics: cosine (default, best for text), euclidean, dot.

1.7 WHERE pushdown — filter before scoring (#2065)

status, result = query(
    "HYBRID_SEARCH FROM Article QUERY 'search' LIMIT 5 WHERE title != 'Identity Resolution at Scale'"
)

Filter candidates before scoring — cheaper than post-filtering. Multiple predicates with AND:

query("HYBRID_SEARCH FROM Article QUERY 'data' LIMIT 5 WHERE id != 'a1' AND id != 'a2'")

1.8 COMPUTE — derived attributes (#2065)

status, result = query(
    "HYBRID_SEARCH FROM Article QUERY 'fraud' LIMIT 3 "
    "COMPUTE relevance_label AS CASE WHEN _score > 10 THEN 'high' ELSE 'medium' END"
)

Compute derived columns on the result set — useful for UI tiering, thresholds, or explainability.

2. POST /search — Meilisearch-compatible endpoint

Full-text search with facets, highlighting, typo tolerance, and filters. Drop-in for Meilisearch/Typesense clients.

2.1 Basic search with highlighting

status, result = post("/search", {
    "query": "bi-temporal data",
    "type": "Article",
    "limit": 3,
    "highlight": true,
    "attributesToHighlight": ["title", "body"],
})
{
  "hits": [
    {
      "id": "a1",
      "score": 0.94,
      "data": {
        "id": "a1",
        "title": "Understanding Bi-temporal Data Models",
        "body": "Bi-temporal data tracks both valid time..."
      },
      "snippet": "Understanding <em>Bi-temporal</em> <em>Data</em> Models",
      "highlights": [{"start": 13, "end": 24, "term": "bi-temporal"}]
    }
  ],
  "count": 1,
  "total": 1,
  "estimatedTotalHits": 1,
  "processing_time_ms": 2,
  "query": "bi-temporal data"
}

id is always the row's real primary key (e.g. chunk_id for DocumentChunk) — never a synthetic placeholder. data carries the row's full field set; snippet/highlights carry the highlighted excerpt (pass showRankingScore/showMatchesPosition for the Meilisearch-style _rankingScore/_matchesPosition fields).

2.2 Faceted search (aggregate counts)

status, result = post("/search", {
    "query": "data",
    "type": "Article",
    "limit": 5,
    "facets": ["title"],
})

Returns facet counts alongside hits — build filter UIs without extra queries.

2.3 Field filters

status, result = post("/search", {
    "query": "search",
    "type": "Article",
    "limit": 5,
    "filter": {"id": "a2"},
})

2.4 Matching strategy + typo tolerance

status, result = post("/search", {
    "query": "bitemporal",  # missing hyphen
    "type": "Article",
    "limit": 3,
    "matching_strategy": "lenient",
    "typo_tolerance": {"min_word_size": 4, "enabled": True},
})

Fuzzy matching catches "bitemporal" → "bi-temporal". Tune min_word_size to control when typo correction kicks in. Every parameter accepts either spelling — matchingStrategy/typoTolerance/minWordSize work the same as the snake_case forms shown here, so a Meilisearch-compatible camelCase client body needs no translation.

2.5 Hybrid fusion via POST /search

status, result = post("/search", {
    "query": "prevent AI data leakage",
    "type": "Article",
    "limit": 3,
    "metric": "cosine",      # triggers HYBRID_SEARCH fusion
    "weights": [0.0, 0.5, 0.5],  # [graph, bm25, vector]
})

When metric or weights is set, the server routes through the HYBRID_SEARCH engine instead of BM25-only. Meilisearch-compatible clients get hybrid search without changing their API.

3. RAG_RETRIEVE — governed retrieval for LLM grounding

status, result = query(
    "RAG_RETRIEVE FROM Article QUERY 'how does identity resolution work' LIMIT 5 RERANK"
)

RAG_RETRIEVE is purpose-built for RAG pipelines:

  • Applies cell-level ACL masking before retrieval — masked cells never reach the LLM
  • Optional RERANK for cross-encoder precision
  • Returns ranked chunks with provenance citations
  • Governance (PURPOSE, tenant isolation) enforced in the retrieval path
Full RAG pipeline example
# 1. Ingest documents with embeddings
ingest("KnowledgeBase", [
    {"id": "kb1", "body": "AML regulations require screening within 24 hours of transaction.",
     "_emb_text": [0.1, 0.2, ...]},  # pre-computed or via RELATA_EMBED_URL
])
 
# 2. Retrieve governed chunks
chunks = query("RAG_RETRIEVE FROM KnowledgeBase QUERY 'AML screening timeline' LIMIT 5")
 
# 3. Feed to LLM with provenance
context = "\n".join([f"[{c['_score']:.2f}] {c['body']}" for c in chunks["data"]])
prompt = f"Based on these sources:\n{context}\n\nWhat is the AML screening deadline?"
# → Send to your LLM (OpenAI, Ollama, local model)

The LLM only sees governed, ACL-filtered, provenance-cited chunks. No data leakage.

4. SIMILAR TO — vector similarity

Find rows similar to a known row by vector proximity:

status, result = query("SIMILAR TO Article WHERE ID = 'a1' LIMIT 5")

Returns the 5 articles most similar to a1 by vector distance. Uses the same HNSW index as HYBRID_SEARCH's vector channel — O(log N) lookup, not a full scan.

'a1' above is the row's real id data column value (the one you set at ingest) — not the synthetic id a /search hit returns (e.g. 00000000-0000-008e-...). Feeding a /search hit's id straight into SIMILAR TO 404s; look up the row's own id field first if you're chaining from a search result.

5. Multi-search — federated queries in one request

Federate a BM25 query across multiple object types in a single round-trip — each entry in queries is {type, query, limit?}, not raw SQL:

status, result = post("/multi-search", {"queries": [
    {"type": "Article", "query": "fraud graph", "limit": 3},
    {"type": "Person", "query": "shell company", "limit": 3},
]})
{
  "results": [
    {
      "hits": [
        {"id": "a4", "score": 11.2, "snippet": "...", "data": {"title": "Graph Analytics for Fraud Detection"}}
      ],
      "estimatedTotalHits": 1,
      "type": "Article",
      "query": "fraud graph"
    },
    {
      "hits": [
        {"id": "p9", "score": 6.4, "snippet": "...", "data": {"name": "Acme Holdings", "risk": "HIGH"}}
      ],
      "estimatedTotalHits": 1,
      "type": "Person",
      "query": "shell company"
    }
  ],
  "processing_time_ms": 4
}

One request, one answer per type. results preserves the order of the queries you sent; each entry's own type/query tell you which request it answers. /multi-search is BM25-only federation across object types — for SQL/HYBRID_SEARCH/aggregates, issue those as separate POST /query calls (see §1 above).

6. Streaming search results (NDJSON)

For large result sets or real-time dashboards:

with httpx.stream("POST", f"{BASE}/query/stream",
                   json={"sql": "HYBRID_SEARCH FROM Article QUERY 'data' LIMIT 100",
                         "purpose": "analytics"},
                   headers=H, timeout=30) as r:
    for line in r.iter_lines():
        if not line:
            continue
        chunk = json.loads(line)
        if "_meta" in chunk:
            continue  # open/close envelope line — carries columns, row_count, etc.
        print(chunk["title"])

Results stream as newline-delimited JSON (application/x-ndjson, #636) — one JSON object per row, not Server-Sent Events. The stream opens and closes with a {"_meta": {...}} line (query id / columns on open, row count / timing on close); every line in between is one row object. The client gets the first rows before the server finishes scoring the full set.

7. BM25 engine features (under the hood)

What makes the custom BM25 engine different
FeatureDetail
Block-max WANDSkips blocks whose max possible score < current threshold. Deep-offset pagination stays fast (not O(n)).
12-language stemmersen, fr, de, es, pt, it, nl, sv, no, da, fi, hu + Russian Cyrillic, Turkish, Arabic
Q-gram prefix searchSubstring matching via q-gram inverted index — catches "temporal" when you type "temp"
Reverse-trigram suffixSuffix matching for morphological variants
Stop wordsLanguage-aware removal (the, le, der, el...)
SynonymsConfigurable synonym expansion
Subword tokenizationcamelCase/digit splitting: "BiTemporal" → ["bi", "temporal"]
HighlightingByte-offset <em> wrapping for UI rendering
Faceted searchFull-match-set counts (not just top-K) for filter UIs
Fuzzy/typo toleranceLevenshtein edit-distance with configurable threshold

All in-process. No Elasticsearch. No external service.

8. SDK: client.search() — the high-level API

from relata import RelataClient
 
client = RelataClient("http://localhost:9090", bearer_token="perftoken", purpose="analytics")
 
# BM25-only (default)
result = client.search("bi-temporal data", type="Article", limit=5)
 
# With highlighting
result = client.search("fraud", type="Article", limit=5, highlight=True)
 
# With facets
result = client.search("data", type="Article", limit=10, facets=["title"])
 
# Hybrid fusion (BM25 + vector)
result = client.search("semantic similarity", type="Article", limit=5,
                       metric="cosine", weights=[0.0, 0.5, 0.5])
 
# Field filter
result = client.search("search", type="Article", limit=5,
                       filters={"id": "a2"})

Returns a SearchResponse with .hits, .estimated_total_hits, .processing_time_ms, .facets.

Summary: every search surface

SurfaceWhen to useAPI
HYBRID_SEARCH (fused)Default — best recallSQL operator
HYBRID_SEARCH (BM25-only)Exact keywords, codesWEIGHTS 0.0 1.0 0.0
HYBRID_SEARCH (vector-only)Semantic paraphrasesWEIGHTS 0.0 0.0 1.0
HYBRID_SEARCH + RERANKMaximum precisionCross-encoder re-scoring
HYBRID_SEARCH + WHEREFilter before scoringPre-filter candidates
HYBRID_SEARCH + COMPUTEDerived result columnsThresholds, labels
POST /searchMeilisearch-compatibleFacets, highlighting, typo
RAG_RETRIEVELLM groundingACL-safe, cited, reranked
SIMILAR TO"More like this"Vector similarity
POST /multi-searchFederated queriesMultiple types, one request
POST /query/streamLarge/streaming resultsNDJSON rows

Every response on this page was captured from a live RelataDB server. No mockups.