Hybrid retrieval, explained: why vector search alone is not enough

This is a sample post to show how the blog works. Rewrite it in your own voice, or replace it, before publishing.

A retrieval-augmented generation (RAG) system is only as good as the passages it hands to the language model. If the right passage never reaches the model, no prompt can save the answer. That is why retrieval deserves most of the engineering effort.

Where vector search falls short

Embedding models turn text into vectors so that passages with similar meaning land close together. That is excellent for paraphrases: a question about "cancelling my plan" can find a passage titled "How to end your subscription".

It is weaker with things that have no meaning to generalize from:

• product codes and error numbers, such as ERR_4021
• names, acronyms and rare terms
• exact phrases that must match literally

For those, an older technique still wins.

Keyword search still matters

BM25 scores passages by how often the query's words appear, adjusted for how rare each word is. It has no idea that "cancel" and "end" are related, but it will never miss a literal ERR_4021.

The two methods fail in different places, so combining them is more reliable than choosing one.

Combining the two rankings

A simple, robust way to merge two ranked lists is reciprocal rank fusion. Each passage earns a score from every list it appears in, and higher ranks earn more:

def reciprocal_rank_fusion(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

Pass in the vector ranking and the BM25 ranking, and you get one merged list. It needs no score calibration between the two systems, which is why it is a popular default.

Rerank the shortlist

The merged list is a shortlist, not a final answer. A cross-encoder reads the query and each candidate passage together and scores how well the passage answers the question. It is slower than vector search, so you only run it on the top few dozen candidates, then send the best handful to the model.

Check that it actually helped

A change that feels better is not evidence. Build a small set of real questions with known good answers, then measure. Frameworks such as RAGAS report metrics like:

• Faithfulness: is the answer supported by the retrieved passages?
• Answer relevancy: does the answer address the question?
• Context precision and recall: did retrieval bring back the right passages, and little else?

Run the same questions before and after each change, and keep the numbers. Retrieval work without measurement is guesswork.

Takeaways

1. Vector search captures meaning, keyword search captures exact terms, and you usually need both.
2. Merge the rankings, then rerank a short list with a cross-encoder.
3. Measure with a fixed question set so every change is judged on evidence.