Hybrid search, explained by playing with it

Type “code E-07” into an e-bike’s help center. You expect the article about code E-07. An assistant built on embeddings, today’s favorite technique, will happily hand you the one about code E-03. It understood you were asking about an error code. It did not understand which one.

I saw this bug dozens of times while building an assistant for maintenance teams. Part numbers, fault codes, document references: everything that really matters in a technical job is exactly what search “by meaning” handles worst. The fix has a slightly technical name, hybrid search, but the idea fits in one sentence: search two ways, then combine.

This article takes it apart piece by piece, with small simulations you can play with. Everything runs in your browser, on twelve fictional help articles.

Two ways to search

The first is as old as search engines: count words. An article that contains the words of the question ranks higher, all the more if those words are rare in the rest of the corpus. The reference algorithm is called BM25. It is fast, predictable, and it loves exact words. It has no idea, however, that “winter” and “cold” are about the same thing.

The second turns every text into a list of numbers, an embedding, computed by a model that has read an enormous amount of text. Two sentences that mean the same thing get close numbers, even without a word in common. That is what makes the machine feel like it understands.

Think of those numbers as coordinates. Every article becomes a point, and a question becomes one more point that lands near the articles it resembles. Try it:

Now type “part BX-220”: the point doesn’t move. To the model, a product code is a meaningless string of characters. It has no reason to bring it close to anything.

Each one has blind spots

Both methods make mistakes, but never in the same place. Keyword search misses paraphrases: “my battery drains faster in winter” shares almost none of the words of the article that answers it. Meaning search misses identifiers: error codes, part numbers, version numbers, rare proper names.

Here they are side by side, then combined. The suggestions show each case, but you can type anything you like.

Fusing two rankings

So we have two ordered lists whose scores don’t compare: BM25 produces unbounded numbers, embedding similarity stays between -1 and 1. Adding the scores makes no sense. The most robust method, and by far the simplest, ignores scores and keeps only ranks.

It is called Reciprocal Rank Fusion, or RRF. Each list gives each document a vote of 1 / (k + rank). First in a list, it gets a big vote. Tenth, a small one. Add up the votes, sort, done.

def rrf(rankings, k=60):
    """Fuse ranked lists of document ids with Reciprocal Rank Fusion."""
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

fused = rrf([bm25_ids, embedding_ids])

The constant k softens the gap between the top spots. With a small k, being first somewhere is enough to win. With a large k, it pays to be ranked decently everywhere. The value 60, suggested in the paper that introduced the method, works surprisingly well: I have almost never needed to touch it.

Reranking the best candidates

Fusion almost always brings the right article near the top, but not necessarily first. To order the first results precisely, you add a reranker, often a cross-encoder: a model that reads the question and each candidate together, word by word, instead of comparing two numeric summaries.

It is far more accurate, and far slower. You can’t have it read the whole corpus for every question. Hence the two-stage architecture: a broad, fast search to find fifty or so candidates, then a slow, careful rerank of those fifty.

Turn on “Rerank the top 5” in the playground: on ambiguous questions, that is often where the right article moves from second to first place.

Before all that: chunk well

Everything above assumes the answer sits in an indexed passage. But you never index a whole document: you cut it into chunks of a few dozen to a few hundred words. And the size of those chunks decides a lot.

Too small, and they cut answers in half. Too large, and they drown the useful sentence in noise, leaving the model that writes the final answer to dig it out. Play with size and overlap:

Overlap is cheap insurance: each chunk repeats the end of the previous one, which limits answers cut off mid-sentence. In practice I chunk along the document’s structure first, headings and paragraphs, and only fall back to a fixed size as a last resort.

What I keep in mind in production

• Measure before tuning. A hundred questions with their right answer beat every intuition. Start by checking whether the right article shows up in the top five results.
• Keep identifiers intact. A tokenizer that turns “E-07” into “e” and “07” destroys exactly what BM25 is good at.
• Don’t over-tune k. 60 by default, then move on: the gains are almost always elsewhere.
• Turn every failure into a test. When someone flags a bad answer, the question joins the evaluation set. The system can’t regress on it again without anyone noticing.

This is what I built into Buddy, an assistant for maintenance teams. Hybrid search isn’t optional there: in a trade where everything relies on exact references, it is the condition for being useful at all.

The corpus, embeddings and reranker in this article are simplified simulations. BM25 and RRF are implemented for real.