Identity Resolution#

The same person appears in your data under a phone number in one system, an email address in another, and a government ID in a third. Joining those records manually is error-prone, slow, and breaks the moment a new source is added. Relata solves this at the storage layer: every ingested value is checked against a catalogue of 76 canonical identity types, linked into an IdentityIndex materialised view, and made queryable with SQL operators that treat entity resolution as a first-class operation — not a post-processing step.

The Identity datatype#

An Identity wraps two things: a CanonicalKind (one of 76 enum variants) and a deterministic binary encoding of the validated value. The binary encoding ensures that +1-800-555-0100 and 18005550100 resolve to the same bytes — no normalisation code on your side.

The 76 shipped kinds span:

The enum is #[non_exhaustive] — new kinds are added without breaking existing code. The canonical list is the source of truth: crates/relata-canonical/src/lib.rs.

A few kinds (GSTIN, BTC address, FARA) ship validators but no SmartIngest detection gate — they round-trip through the type system but are not auto-detected from free text. See docs/src/end-users/limits.md for the honest per-kind status.

How identity links are built — SmartIngest#

SmartIngest (relata-detect) runs at write time, not query time. When a row is ingested, it scans text fields in two phases, per token:

1. Eager gate — cheap pattern check (length, prefix, character class). Rejects obvious non-matches immediately.
2. Lazy validate — full canonical parser with checksum verification (Luhn, mod-97, Base58Check, …). Only runs when the eager gate passes.

Both phases run synchronously on the write path and produce Identity rows. The heavier work — turning those identifiers into graph structure (IdentityIndex / IdentityLink MVs) — is handed off to a non-blocking enrichment queue (relata-jobs::enrichment_queue) drained in batches by a background task.

The result is that by the time a query asks RESOLVE_IDENTITY(email), the link is already there — no join-time detection.

Controlling which packs load#

SmartIngest is divided into detector packs. Three are on by default; the rest are opt-in because they add CPU cost and some produce false positives on general text:

# Default (on at startup)
RELATA_DETECT_PACKS=network,contact,crypto
 
# Add financial + payment detection
RELATA_DETECT_PACKS=network,contact,crypto,financial,payment
 
# All packs
RELATA_DETECT_PACKS=all
 
# Disable all auto-detection
RELATA_DETECT_PACKS=none

relata detect "<text>" at the CLI always runs all packs regardless of RELATA_DETECT_PACKS. That env var controls only the HTTP ingest and per-row write paths.

SQL operators#

Lookup by value#

-- Is this phone number known to the system?
SELECT * FROM LOOKUP_IDENTITY('+919876543210')
 
-- Returns: kind, canonical_value, linked_entity_ids[]

Resolve an identity to its canonical form or cluster#

-- Canonical form: what is the authoritative surface for this email?
SELECT * FROM RESOLVE_IDENTITY('alice@example.com')
 
-- Cluster: every identity value linked to the same entity
SELECT * FROM RESOLVE_IDENTITY('alice@example.com', MODE => 'cluster')
 
-- Column projection: resolve inline in a query
SELECT name, RESOLVE_IDENTITY(email) AS canonical_email
FROM Person
WHERE country = 'IN'

Resolution modes#

Graph traversal between identities#

Once entities are linked through shared identities, you can walk the resulting graph:

-- All paths between two entities, up to 4 hops
SELECT * FROM PATHS_BETWEEN('person-123', 'org-456', max_hops => 4)
 
-- Identity verdict — do two identifiers resolve to the same person?
SELECT * FROM SAME_IDENTITY('person-123', 'org-456')
 
-- Full cluster for a given entity
SELECT * FROM IDENTITY_CLUSTER('person-123')

PATHS_BETWEEN uses the graph engine's bidirectional BFS / DFS traversal for path enumeration (traversal.rs:388). Each returned path includes the intermediate identity values and the types they were found in. (PLL distance labeling is wired into GRAPH_SSSP (algo => 'pll') and GRAPH_DIJKSTRA as a reachability pre-check — not into PATHS_BETWEEN.)

A worked example — linking a phone to a transaction#

-- Find all transactions associated with a phone number,
-- even if the phone wasn't stored directly on the Transaction row
SELECT t.id, t.amount, t.valid_from
FROM Transaction t
JOIN IDENTITY_CLUSTER(RESOLVE_IDENTITY('+919876543210')) ic
  ON t.actor_id = ANY(ic.entity_ids)
ORDER BY t.valid_from DESC
LIMIT 20

Batch detection#

For pipelines that ingest text in bulk, the enrichment queue processes detection in chunks of RELATA_DETECT_BATCH_SIZE (default 256) rows, reusing a single hit buffer per chunk. Results are identical to per-row detection — the batch size only affects throughput.

Python SDK#

from relata import RelataClient, IdentityClient
 
with RelataClient(url, bearer_token=token, purpose="identity-match") as client:
    id_client = IdentityClient.from_client(client)
 
    # Look up an entity by identity value
    result = id_client.lookup("+919876543210")
 
    # Get the full cluster
    cluster = id_client.cluster("alice@example.com")
 
    # GDPR erasure — shreds all rows linked to this subject
    receipt = id_client.erase_subject("person-42", certify="governed-tombstone")

Active learning — tune the resolver from human feedback#

Identity resolution isn't a black box you have to accept. When RESOLVE_IDENTITY gets a pair wrong (fuses two distinct people, or refuses to merge the same person), you can submit a human judgement and the resolver takes a correction step on its feature weights — turning "the resolver got this wrong" into something the system learns from. Labels persist across restarts in ${RELATA_DATA_DIR}/active_learning.json.

Label a pair (match / no-match)#

curl -X POST http://127.0.0.1:9090/identity/label \
  -H 'Authorization: Bearer <token>' -H 'Content-Type: application/json' \
  -d '{"left": "+44 7700 900123", "right": "07700 900123", "is_match": true}'

# Python SDK
client.identity_client.label("+44 7700 900123", "07700 900123", is_match=True)
client.identity_client.label("alice@x.com", "bob@y.com", is_match=False)  # hard negative

Ask for the pairs the resolver is least sure about#

The uncertainty endpoint surfaces the candidate pairs whose labels would most improve the resolver — active-learning style. Hand-label these first for the highest return on effort:

curl 'http://127.0.0.1:9090/identity/uncertainty?candidates=id-a|id-b,id-c|id-d&k=5' \
  -H 'Authorization: Bearer <token>'

# The resolver's top "I'm not sure — please tell me" pairs
uncertain = client.identity_client.record_uncertainty(...)

Tips & takeaways#

• Hard negatives matter as much as matches. Label genuine distinct-entity pairs (is_match: false) — they prevent over-merging, which is the harder failure to detect later.
• Batch-label from a review UI. Run a queue of uncertainty candidates past an analyst, push the judgements back via label. A few hundred labels typically moves resolution quality noticeably.
• Labels are governed. Each label is audit-logged with the principal and purpose — defensible "who taught the resolver what" history.
• Not a replacement for canonical types. Active learning corrects ambiguous matches (fuzzy phone, partial name). It does not override deterministic canonical-type fusions (same verified IBAN = same entity, always).

What is not yet shipped#

• Detection gates for GSTIN, FARA, and a handful of government ID types are not wired into SmartIngest — values of these kinds validate correctly but are not auto-detected from free text.
• The fuse resolution mode requires at least one enrichment rule registered via POST /ontology/enrichment-rules before it is useful.

See also#

• Governance — cell masking can redact identity values for users without the right ACL
• Hybrid Search — identity matching is the third retrieval signal in hybrid queries
• SQL Reference — full operator signatures
• Limits — per-kind validator and detection gate status