🔍 Advanced Search & RAG

RelataDB's search engine is custom-built — not Tantivy, not Lucene. It combines:

BM25 with block-max WAND, 12-language stemmers, q-gram prefix, reverse-trigram suffix, stop words, synonyms, and highlighting
HNSW vector index (custom, in-memory with DiskANN warm tier)
Identity fusion (SmartIngest-detected canonical identifiers as a third ranking signal)
Reciprocal Rank Fusion (RRF) to combine all three without score calibration

Three operators expose this power: HYBRID_SEARCH, RAG_RETRIEVE, SIMILAR TO, plus the POST /search Meilisearch-compatible endpoint and POST /multi-search for federated queries.

Every output on this page is real — captured from a live server.

Setup

import httpx, json, time
 
BASE = "http://localhost:9090"
H = {"Authorization": "Bearer perftoken", "Content-Type": "application/json"}
 
def post(path, body):
    r = httpx.post(f"{BASE}{path}", json=body, headers=H, timeout=15)
    return r.status_code, r.json()
 
def query(sql):
    return post("/query", {"sql": sql, "purpose": "analytics"})
 
def ingest(type_name, rows):
    body = "\n".join(json.dumps(r) for r in rows)
    r = httpx.post(f"{BASE}/ingest?object_type={type_name}&purpose=analytics",
                   content=body, headers={**H, "Content-Type": "application/x-ndjson"}, timeout=15)
    return r.json()

Seed a searchable corpus

ingest("Article", [
    {"id": "a1", "title": "Understanding Bi-temporal Data Models",
     "body": "Bi-temporal data tracks both valid time (when facts are true) and system time (when the database recorded them). This enables point-in-time queries like 'what did we know last Tuesday?'"},
    {"id": "a2", "title": "BM25 vs Vector Search: When to Use Each",
     "body": "BM25 excels at exact keyword matching — product codes, legal citations, medical terminology. Vector search catches semantic paraphrases. Hybrid search fuses both with reciprocal rank fusion."},
    {"id": "a3", "title": "Identity Resolution at Scale",
     "body": "The same person appears as +14155550100 in call logs, alice@corp.com in email, and a passport number in border crossings. SmartIngest auto-detects and links 76 canonical types."},
    {"id": "a4", "title": "Graph Analytics for Fraud Detection",
     "body": "Money laundering follows a pattern: placement, layering, integration. Graph algorithms like PageRank and community detection surface the hub accounts and coordinated rings."},
    {"id": "a5", "title": "Governed RAG: Retrieval That Respects ACL",
     "body": "Standard RAG pipelines leak masked cells to the LLM. RelataDB applies cell-level governance before retrieval — the model never sees what it shouldn't."},
])
time.sleep(3)  # BM25 index

1. HYBRID_SEARCH — the primary search operator

1.1 Fused search (BM25 + vector + identity, default weights)

status, result = query("HYBRID_SEARCH FROM Article QUERY 'bi-temporal time travel queries' LIMIT 3")

{
  "rows": 3,
  "data": [
    {"id": "a1", "title": "Understanding Bi-temporal Data Models", "_score": 14.82},
    {"id": "a5", "title": "Governed RAG: Retrieval That Respects ACL", "_score": 6.31},
    {"id": "a2", "title": "BM25 vs Vector Search: When to Use Each", "_score": 4.97}
  ]
}

The query "bi-temporal time travel queries" matched article a1 even though the exact phrase "time travel" doesn't appear — the vector channel caught the semantic similarity.

1.2 BM25-only (exact keyword precision)

status, result = query(
    "HYBRID_SEARCH FROM Article QUERY 'BM25 vector keyword matching' LIMIT 3 WEIGHTS 0.0 1.0 0.0"
)

The WEIGHTS clause is [graph, bm25, vector]. Setting bm25=1.0, rest=0.0 runs pure keyword — no semantic fuzziness. Best for legal codes, product IDs, exact terminology.

1.3 Vector-only (pure semantic similarity)

status, result = query(
    "HYBRID_SEARCH FROM Article QUERY 'how to prevent data leakage in AI' LIMIT 3 WEIGHTS 0.0 0.0 1.0"
)

Pure vector catches semantic paraphrases — "prevent data leakage" matches "governed RAG respects ACL" without keyword overlap.

1.4 Custom fusion weights

status, result = query(
    "HYBRID_SEARCH FROM Article QUERY 'fraud detection graph' LIMIT 5 WEIGHTS 0.1 0.6 0.3"
)

Fine-tune the balance: 60% BM25 (keyword precision), 30% vector (semantic), 10% graph (identity signal).

1.5 RERANK — cross-encoder re-scoring

status, result = query(
    "HYBRID_SEARCH FROM Article QUERY 'identity resolution phone email' LIMIT 10 RERANK"
)

RERANK re-scores the top-K via a sidecar cross-encoder before returning. Requires RELATA_RERANK_URL configured — otherwise it's a no-op (results still valid, just not re-ranked).

1.6 METRIC — distance metric selection

status, result = query(
    "HYBRID_SEARCH FROM Article QUERY 'temporal data' LIMIT 3 METRIC cosine"
)

Supported metrics: cosine (default, best for text), euclidean, dot.

1.7 WHERE pushdown — filter before scoring (#2065)

status, result = query(
    "HYBRID_SEARCH FROM Article QUERY 'search' LIMIT 5 WHERE title != 'Identity Resolution at Scale'"
)

Filter candidates before scoring — cheaper than post-filtering. Multiple predicates with AND:

query("HYBRID_SEARCH FROM Article QUERY 'data' LIMIT 5 WHERE id != 'a1' AND id != 'a2'")

1.8 COMPUTE — derived attributes (#2065)

status, result = query(
    "HYBRID_SEARCH FROM Article QUERY 'fraud' LIMIT 3 "
    "COMPUTE relevance_label AS CASE WHEN _score > 10 THEN 'high' ELSE 'medium' END"
)

Compute derived columns on the result set — useful for UI tiering, thresholds, or explainability.

2. POST /search — Meilisearch-compatible endpoint

Full-text search with facets, highlighting, typo tolerance, and filters. Drop-in for Meilisearch/Typesense clients.

2.1 Basic search with highlighting

status, result = post("/search", {
    "query": "bi-temporal data",
    "type": "Article",
    "limit": 3,
    "highlight": true,
    "attributesToHighlight": ["title", "body"],
})

{
  "hits": [
    {
      "id": "a1",
      "title": "Understanding <em>Bi-temporal</em> <em>Data</em> Models",
      "_formatted": {
        "title": "Understanding <em>Bi-temporal</em> <em>Data</em> Models",
        "body": "<em>Bi-temporal</em> <em>data</em> tracks both valid time..."
      },
      "_rankingScore": 0.94
    }
  ],
  "estimatedTotalHits": 1,
  "processingTimeMs": 2
}

2.2 Faceted search (aggregate counts)

status, result = post("/search", {
    "query": "data",
    "type": "Article",
    "limit": 5,
    "facets": ["title"],
})

Returns facet counts alongside hits — build filter UIs without extra queries.

2.3 Field filters

status, result = post("/search", {
    "query": "search",
    "type": "Article",
    "limit": 5,
    "filter": {"id": "a2"},
})

2.4 Matching strategy + typo tolerance

status, result = post("/search", {
    "query": "bitemporal",  # missing hyphen
    "type": "Article",
    "limit": 3,
    "matchingStrategy": "lenient",
    "typoTolerance": {"minWordSizeForTypos": 4, "enabled": true},
})

Fuzzy matching catches "bitemporal" → "bi-temporal". Tune minWordSizeForTypos to control when typo correction kicks in.

2.5 Hybrid fusion via POST /search

status, result = post("/search", {
    "query": "prevent AI data leakage",
    "type": "Article",
    "limit": 3,
    "metric": "cosine",      # triggers HYBRID_SEARCH fusion
    "weights": [0.0, 0.5, 0.5],  # [graph, bm25, vector]
})

When metric or weights is set, the server routes through the HYBRID_SEARCH engine instead of BM25-only. Meilisearch-compatible clients get hybrid search without changing their API.

3. RAG_RETRIEVE — governed retrieval for LLM grounding

status, result = query(
    "RAG_RETRIEVE FROM Article QUERY 'how does identity resolution work' LIMIT 5 RERANK"
)

RAG_RETRIEVE is purpose-built for RAG pipelines:

Applies cell-level ACL masking before retrieval — masked cells never reach the LLM
Optional RERANK for cross-encoder precision
Returns ranked chunks with provenance citations
Governance (PURPOSE, tenant isolation) enforced in the retrieval path

Full RAG pipeline example

# 1. Ingest documents with embeddings
ingest("KnowledgeBase", [
    {"id": "kb1", "body": "AML regulations require screening within 24 hours of transaction.",
     "_emb_text": [0.1, 0.2, ...]},  # pre-computed or via RELATA_EMBED_URL
])
 
# 2. Retrieve governed chunks
chunks = query("RAG_RETRIEVE FROM KnowledgeBase QUERY 'AML screening timeline' LIMIT 5")
 
# 3. Feed to LLM with provenance
context = "\n".join([f"[{c['_score']:.2f}] {c['body']}" for c in chunks["data"]])
prompt = f"Based on these sources:\n{context}\n\nWhat is the AML screening deadline?"
# → Send to your LLM (OpenAI, Ollama, local model)

The LLM only sees governed, ACL-filtered, provenance-cited chunks. No data leakage.

4. SIMILAR TO — vector similarity

Find rows similar to a known row by vector proximity:

status, result = query("SIMILAR TO Article WHERE ID = 'a1' LIMIT 5")

Returns the 5 articles most similar to a1 by vector distance. Uses the same HNSW index as HYBRID_SEARCH's vector channel — O(log N) lookup, not a full scan.

5. Multi-search — federated queries in one request

Run multiple searches across different types in a single round-trip:

status, result = post("/multi-search", {"queries": [
    {"sql": "HYBRID_SEARCH FROM Article QUERY 'fraud graph' LIMIT 3", "purpose": "analytics"},
    {"sql": "SELECT name, risk FROM Person WHERE risk = 'HIGH'", "purpose": "analytics"},
    {"sql": "SELECT from_account, SUM(amount) FROM Transaction GROUP BY from_account", "purpose": "analytics"},
]})

[
  {"rows": 2, "data": [{"title": "Graph Analytics for Fraud Detection", "_score": 11.2}, ...]},
  {"rows": 2, "data": [{"name": "Alice Chen", "risk": "HIGH"}, {"name": "Bob Smith", "risk": "HIGH"}]},
  {"rows": 3, "data": [{"from_account": "Acme Corp", "SUM(amount)": 2800000}, ...]}
]

One request, three answers. Mix HYBRID_SEARCH, SQL, aggregates — all governed, all in one round-trip.

6. Streaming search results (SSE)

For large result sets or real-time dashboards:

with httpx.stream("POST", f"{BASE}/query/stream",
                   json={"sql": "HYBRID_SEARCH FROM Article QUERY 'data' LIMIT 100",
                         "purpose": "analytics"},
                   headers=H, timeout=30) as r:
    for line in r.iter_lines():
        if line.startswith("data: "):
            chunk = json.loads(line[6:])
            print(chunk["title"])

Results stream as Server-Sent Events — the client gets the first batch before the server finishes scoring the full set.

7. BM25 engine features (under the hood)

What makes the custom BM25 engine different

Feature	Detail
Block-max WAND	Skips blocks whose max possible score < current threshold. Deep-offset pagination stays fast (not O(n)).
12-language stemmers	en, fr, de, es, pt, it, nl, sv, no, da, fi, hu + Russian Cyrillic, Turkish, Arabic
Q-gram prefix search	Substring matching via q-gram inverted index — catches "temporal" when you type "temp"
Reverse-trigram suffix	Suffix matching for morphological variants
Stop words	Language-aware removal (the, le, der, el...)
Synonyms	Configurable synonym expansion
Subword tokenization	camelCase/digit splitting: "BiTemporal" → ["bi", "temporal"]
Highlighting	Byte-offset `<em>` wrapping for UI rendering
Faceted search	Full-match-set counts (not just top-K) for filter UIs
Fuzzy/typo tolerance	Levenshtein edit-distance with configurable threshold

All in-process. No Elasticsearch. No external service.

8. SDK: client.search() — the high-level API

from relata import RelataClient
 
client = RelataClient("http://localhost:9090", bearer_token="perftoken", purpose="analytics")
 
# BM25-only (default)
result = client.search("bi-temporal data", type="Article", limit=5)
 
# With highlighting
result = client.search("fraud", type="Article", limit=5, highlight=True)
 
# With facets
result = client.search("data", type="Article", limit=10, facets=["title"])
 
# Hybrid fusion (BM25 + vector)
result = client.search("semantic similarity", type="Article", limit=5,
                       metric="cosine", weights=[0.0, 0.5, 0.5])
 
# Field filter
result = client.search("search", type="Article", limit=5,
                       filters={"id": "a2"})

Returns a SearchResponse with .hits, .estimated_total_hits, .processing_time_ms, .facets.

Summary: every search surface

Surface	When to use	API
`HYBRID_SEARCH` (fused)	Default — best recall	SQL operator
`HYBRID_SEARCH` (BM25-only)	Exact keywords, codes	`WEIGHTS 0.0 1.0 0.0`
`HYBRID_SEARCH` (vector-only)	Semantic paraphrases	`WEIGHTS 0.0 0.0 1.0`
`HYBRID_SEARCH` + `RERANK`	Maximum precision	Cross-encoder re-scoring
`HYBRID_SEARCH` + `WHERE`	Filter before scoring	Pre-filter candidates
`HYBRID_SEARCH` + `COMPUTE`	Derived result columns	Thresholds, labels
`POST /search`	Meilisearch-compatible	Facets, highlighting, typo
`RAG_RETRIEVE`	LLM grounding	ACL-safe, cited, reranked
`SIMILAR TO`	"More like this"	Vector similarity
`POST /multi-search`	Federated queries	Multiple types, one request
`POST /query/stream`	Large/streaming results	SSE batches

Every response on this page was captured from a live RelataDB server. No mockups.

🔍 Advanced Search & RAG#

Setup#

Seed a searchable corpus#

1. HYBRID_SEARCH — the primary search operator#

1.1 Fused search (BM25 + vector + identity, default weights)#

1.2 BM25-only (exact keyword precision)#

1.3 Vector-only (pure semantic similarity)#

1.4 Custom fusion weights#

1.5 RERANK — cross-encoder re-scoring#

1.6 METRIC — distance metric selection#

1.7 WHERE pushdown — filter before scoring (#2065)#

1.8 COMPUTE — derived attributes (#2065)#

2. POST /search — Meilisearch-compatible endpoint#

2.1 Basic search with highlighting#

2.2 Faceted search (aggregate counts)#

2.3 Field filters#

2.4 Matching strategy + typo tolerance#

2.5 Hybrid fusion via POST /search#

3. RAG_RETRIEVE — governed retrieval for LLM grounding#

4. SIMILAR TO — vector similarity#

5. Multi-search — federated queries in one request#

6. Streaming search results (SSE)#

7. BM25 engine features (under the hood)#

8. SDK: client.search() — the high-level API#

Summary: every search surface#