hraness

hybrid retrieval: exact, text, and semantic search together

rank fusion that stays inspectable

Drafted by an AI agent at Ben Guo's direct request from the Hraness source repositories, and checked against those sources before publication.

Search over a personal knowledge base fails in three different ways. An exact search misses a concept the author named differently. Full-text search ranks keyword density above the note that actually answers the question. Vector search finds things that feel adjacent and occasionally surfaces the thing that is adjacent. Hybrid retrieval exists because each lane covers the others’ blind spots, but it only stays trustworthy if you can still see which lane produced which result.

The question this article answers: how do you combine exact, text, and semantic search without losing the ability to explain a ranking? Wordcell’s wordcell search is the working implementation: a default hybrid mode that runs a live exact scan alongside a pinned local QMD index, fuses the ranked lists deterministically, and reports every lane’s contribution as inspectable evidence rather than a collapsed score.

three questions, three lanes

wordcell search exposes the lanes directly as modes rather than hiding them behind one opaque pipeline:

  • --mode exact scans the current Markdown for identity, phrase, and term matches: note IDs, titles, aliases, paths, tags, typed metadata, and prose. It is model-free: no index, no embedding, no service.
  • --mode keyword uses QMD’s local full-text index: conventional FTS over the vault’s Markdown projection.
  • --mode semantic selects QMD’s vector lane: embedding similarity over the same local projection.
  • --mode hybrid, the default, runs the exact lane beside QMD’s combined retrieval and fuses the result lists.

The modes matter because the lanes are answering different questions. Exact search answers “where is this named thing”; keyword search answers “which notes talk in these terms”; semantic search answers “what is adjacent in meaning.” A retrieval system that cannot say which question it answered cannot be evaluated and cannot be trusted when it is wrong.

fusion without a shared score

The fusion problem is that the lanes’ scores are incomparable: an exact-match counter, an FTS rank, and a cosine similarity do not share a scale, and any weighted sum of raw scores smuggles in a calibration nobody validated. Wordcell’s answer is reciprocal-rank fusion (score each candidate by its position in each lane rather than its points), implemented as fuseRankedCandidates in src/search.ts. Each lane contributes weight / (k + rank) for every candidate it returned, the contributions sum per candidate, and the total is normalized by the maximum a single lane could produce. The fused score is a normalized rank statistic; the type comment states it outright: it is not a probability.

Two design choices keep the ordering honest. Exact title, alias, and path identities are pinned separately (a note whose identity matches the query is not allowed to be outranked by fuzzy agreement), and agreement between lanes outranks single-lane evidence, so a note both retrievers found beats a one-lane flash. Every returned hit carries an evidence array naming the lanes that produced it with their ranks, plus the full contributions breakdown that produced its final position. The ranking is a report, not a verdict.

lanes fail independently

The inspectability extends to failure. Each lane reports a diagnostic (ready, degraded, or unavailable, with a message), and any non-ready lane marks the whole result partial. A QMD failure does not erase exact results; a degraded embedding pass does not silently lower coverage. An underfilled filtered search reports itself degraded rather than padding the result with unrelated notes: QMD cannot rank against a path allowlist, so a filtered search uses a bounded global candidate window, and rows discarded by live reconciliation or metadata filters leave the request explicitly degraded.

inside the fusion

The fusion contract is deliberately small and fully validated. fuseRankedCandidates(lanes, k) accepts 1–16 lanes (MAX_FUSION_LANES), each with a unique non-empty name, a finite weight greater than 0 and at most 100, and at most 500 result IDs (MAX_FUSION_RESULTS_PER_LANE); k defaults to 60 and is bounded to 1–1,000. Duplicate IDs inside a lane contribute once. Ties break deterministically by ID, so equal inputs always produce equal outputs; the six-case fixture in search.test.ts is a regression for the mechanic, explicitly not a quality benchmark.

In the SDK the default hybrid builds exactly two weight-1 lanes: exact and qmd. The candidate bound adapts to the query: an unfiltered search fetches up to max(40, limit × 4) candidates, while a metadata-, tag-, or scope-filtered search defaults to the largest supported window, MAX_SEARCH_CANDIDATES = 500, because filtering happens at the live-join boundary and a small pre-filter window would underfill it. QMD’s own fixed structured-hybrid pool is deliberately bypassed: Wordcell requests QMD’s direct full-text and vector rankings at the declared candidate bound and fuses them itself, without query expansion or reranking models. Those stay opt-in costs, not defaults.

Join-back is where the honesty gets enforced. Each QMD hit is reconciled against the live session snapshot: files outside the vault and stale indexed identities are discarded, metadata and tag constraints are authoritative at the join, and discarded candidates feed the degraded diagnostic. The search returns current typed metadata and tags, not whatever the index remembered.

the pinned local stack

The semantic lane is local, optional, and pinned. Wordcell depends on @tobilu/qmd at an immutable Hraness fork commit (aa993dceb3ef8cfb71d470554ca437570f5a2b3c, versioned 2.5.3+hraness.aa993dc…) because upstream QMD 2.5.3 routes some internal embedding calls through a process-global model; the fork routes them through QMD’s store-local model so each vault’s index is self-contained. The embedding model is a compact EmbeddingGemma (300M, Q8 in the evaluation build) pinned by revision, and an explicit local model file is accepted only when its SHA-256 matches the pinned artifact. Without a local source, the first hybrid or semantic query downloads that revision; later runs reuse the cache and incrementally reindex changed Markdown.

The index lives outside the vault: a path-derived SQLite database under the user’s cache directory, protected by a database-scoped process lease that serializes projection installation, store updates, and embedding writes across concurrent agents. The generation identity binds the immutable note bytes plus the QMD version, embedding model, collection configuration, and projection contract; an identity change waits for older readers to close before mutating. index.md and every AGENTS.md are excluded from indexing (they are navigation and always-loaded instructions, not knowledge records) while scope hubs index like any other note. And the database is disposable: delete it, rerun wordcell index, get an equivalent index back.

measured on a frozen corpus

The retrieval claims are backed by a frozen public pilot rather than vibes. On 2 August 2026, an evaluator froze 18 questions against one repository snapshot (156 scanned Markdown records, 155 searchable notes after excluding the authored index and agent guides) with graded relevance judgments written before rankings were inspected, nine development and nine held-out questions, and a 10,000-resample paired bootstrap for the hybrid-minus-exact comparison:

Retriever Recall@10 MRR@10 nDCG@10 p95 ms
exact 0.833333 0.892857 0.790377 44.345
hybrid 0.833333 0.937500 0.833884 62.834
keyword 821.370
semantic 41,000.524

The observed differences favored hybrid on ranking quality (MRR +0.044643, nDCG +0.043508) and the 95% paired intervals included no change. The page states the conclusion plainly: the corpus is too small to claim hybrid is generally better than exact. The semantic lane’s p95 includes the first in-process model load, the no-answer question was answered (wrongly) by both retrievers for a no-answer accuracy of 0, and the whole run is a frozen regression baseline on one Apple M4 Max, one vault, one QMD build, and one cache state. It does not generalize, and it says so.

That is the point of publishing it: the numbers are honest evidence about one system at one moment, kept stable at hraness.com/kb/evaluation so later claims can be compared against something frozen rather than something remembered.

the rule that keeps it honest

Everything above rests on one refusal: a retrieval score never becomes a fact. The design guide states it directly: a score is a discovery aid, not a graph edge, a citation, or evidence that the result is true. Graph neighbors and bounded Git history can ride along with results, but they are returned as separate evidence collections (context and provenance) and never silently boost rank or turn into authored links. Similarity does not establish that a passage is current, correct, or supported; the Markdown behind the result is what a reader inspects.

Hybrid retrieval done this way is not a smarter ranker. It is three weak retrievers whose agreement is visible, whose disagreements are diagnosable, and whose failures arrive labeled.

sources

  • wordcell: src/search.ts, src/sdk.ts, docs/design.md, docs/reference.md
  • qmd: the pinned Hraness compatibility fork at aa993dc
  • hraness.com/kb/evaluation: the frozen 2 August 2026 retrieval pilot

keep reading: free for subscribers

the rest of this lesson is free. add your email once and every subscriber lesson on this site stays unlocked.

already subscribed? enter the same email to unlock.