Multimedia search — image, video, audio, face#

RelataDB isn't just a text-and-rows engine — it ships six embedding modalities and two purpose-built multimedia SQL operators: FACE_SEARCH for face recognition over a gallery, and MATCH_PDQ for perceptual-hash near-duplicate detection (the CSAM / NCMEC / known-image workflow). Media blobs are governed (BlobRef + content-addressed store), ACL applies on every hit, and you can write media via the S3 door and search it over SQL.

All six embed routes and both operators are verified against source — FACE_SEARCH and MATCH_PDQ are SQL-reachable (crates/relata-query/src/scenario_e2e.rs); the embed routes live at /embed/{,batch,image,face,audio,video}. Media embed routes return 503 if the active embedder doesn't support the modality (the CPU default doesn't — wire the GPU sidecar via RELATA_ACCEL_ENDPOINT).

The 6 embedding modalities#

vc = client.vector_client
 
e   = vc.embed("a red car at sunset")            # → {embedding, model, dim}
eim = vc.embed_image(base64_bytes)               # CLIP — multimodal RAG
ef  = vc.embed_face(base64_bytes)                # ArcFace — face gallery
eau = vc.embed_audio(base64_bytes)               # CLAP — audio retrieval
ev  = vc.embed_video(base64_bytes)               # CLIP keyframe — video frame search

Embeddings land on _emb_* columns, keyed on (object_type, modality, model_tag, tenant_id), and ride the same HNSW + DiskANN index as text vectors. See Vector params for ef_construction/M/quantization tuning.

FACE_SEARCH — face recognition as a SQL operator#

Index a face crop into a gallery, then match a probe face against it in one SQL call. Returns the top-k identities above a threshold, governed by ACL.

PURPOSE 'investigation'
-- Match a probe face embedding against the 'watchlist' gallery, top 5
SELECT * FROM FACE_SEARCH('[0.12, 0.087, ...]', 'watchlist', LIMIT => 5);
 
-- With an explicit similarity threshold
SELECT identity_id, score
FROM FACE_SEARCH('[...]', 'casework', THRESHOLD => 0.85)
ORDER BY score DESC;

# Python — equivalent via the SDK
hits = client.face_search("watchlist", probe_embedding, k=5, threshold=0.85,
                          purpose="investigation")   # → QueryResult

The MCP face_match tool wraps the same operator for agent-driven investigation:

mcp.call_tool("face_match", {"probe_id": "probe-7", "threshold": 0.85,
                             "top_k": 10, "purpose": "security_incident"})

Governance note: biometric data is high-sensitivity. Put face galleries behind a strict Cedar policy (a forbid on Face.embedding for non-clearance principals), declare a dedicated PURPOSE, and let the audit chain record every probe. See Per-Door ACL and Governance.

MATCH_PDQ — perceptual-hash near-duplicate detection#

PDQ is Facebook's perceptual hash for images — the standard for CSAM / NCMEC / known-image matching. Relata computes PDQ hashes on ingest into a slot and exposes MATCH_PDQ as a governed SQL operator: hash a probe image, match against a corpus, return near-duplicates above a threshold.

PURPOSE 'trust-and-safety'
-- Match a probe PDQ hash against the 'ncmec' corpus
SELECT object_id, similarity
FROM MATCH_PDQ('ffff...', 'ncmec', THRESHOLD => 0.5)
ORDER BY similarity DESC;
 
-- Tunable threshold — higher = stricter (fewer false positives)
SELECT * FROM MATCH_PDQ('<hash>', 'intake', THRESHOLD => 0.9);

hits = client.match_pdq("ncmec", probe_hash, threshold=0.9,
                        purpose="trust-and-safety")   # → QueryResult

Use this for trust-and-safety pipelines (detect known-bad images across uploads), copyright / dedup, and CSAM scanning where the hash corpus is loaded via relata import or a sanctions-style pull.

Image near-dup (aHash + dHash) — lightweight, no sidecar#

For "is this image a re-encoding of one we've seen?" without a GPU — Relata computes aHash + dHash in-tree (pure Rust, decodes the image itself) and finds near-copies within a small Hamming distance. Faster and cheaper than PDQ for the common case; PDQ is the higher-precision choice for adversarial inputs.

This is wired into ingest for image-bearing types — a re-encoded crop of an existing image surfaces as a near-dup automatically.

Governed media blobs (BlobRef)#

Media content doesn't bloat the row store. MediaContent rows carry a BlobRef pointer to the content-addressed blob store + the _emb_* embedding, never inline bytes:

# Async media ingest — enqueue the blob + an embed task
curl -X POST http://127.0.0.1:9090/ingest/media \
  -H 'Authorization: Bearer <token>' -H 'Content-Type: application/json' \
  -d '{
    "object_type": "SurveillanceFrame",
    "mime_type": "image/jpeg",
    "body_b64": "<base64...>",
    "partition_key": "case-7",
    "purpose": "investigation"
  }'
# → {task_id: "itsk_..."}  — poll GET /ingest/media/:task_id for completion

Bodies ≥ RELATA_S3_BLOB_THRESHOLD_MB spill to the content-addressed store; smaller ones inline. Cross-protocol: write media via the S3 door (put_object), query it over SQL (SELECT key, size FROM S3Object), embed it via /embed/image. See S3 door.

Multimodal RAG — text → image retrieval#

Because CLIP image embeddings live in the same vector index as text, a text query retrieves matching images:

# Embed a text probe, retrieve matching images
e = vc.embed("a red sedan parked at night")
hits = vc.knn_search("SurveillanceFrame", "_emb_image", e["embedding"],
                     k=10, purpose="investigation")
for h in hits:
    print(h["object_id"], h["case_id"])

The same flow drives the MCP search_video_frames tool — text-query a corpus of video keyframes:

mcp.call_tool("search_video_frames",
              {"query_id": "<probe-id>", "media_type": "VideoFrame",
               "top_k": 20, "purpose": "security_incident"})

Tips & takeaways#

• Wire the GPU sidecar for media. The CPU embedder handles text only — embed_image/face/audio/video return 503 until you set RELATA_ACCEL_ENDPOINT. Text-only workloads stay on CPU.
• Pick the right hash for the job. aHash/dHash = fast, cheap, good for re-encoding detection. PDQ = higher precision, adversarial inputs, CSAM/NCMEC. Face identification = ArcFace via FACE_SEARCH, not a hash.
• Threshold tuning is per-corpus. A 0.85 face threshold that works on a clean watchlist may need 0.90 on a noisy intake feed. Backtest against labelled pairs before going live.
• Partition by case / tenant. Use partition_key on /ingest/media so a probe search is scoped to one case — much faster and tenant-safe.
• Biometrics deserve biometric-grade governance. forbid clauses on face embeddings, dedicated PURPOSE, audit-chain on every probe. Treat the gallery like the sensitive PII it is.
• Cross-door is the payoff. Your SOC uploads evidence via the S3 door; the investigator searches it via SQL FACE_SEARCH + MATCH_PDQ; the agent retrieves via MCP search_video_frames. One store, four front doors.

See also#

• Vector parameters — HNSW/DiskANN + quantization tuning
• Hybrid Search — fuse BM25 + vector + graph
• S3 door — governed object store + bi-temporal versioning
• Per-Door ACL — Cedar principals for sensitive media doors
• For AI Agent Builders — multimodal RAG in context
• For Security Teams — trust-and-safety framing