Semantic Search vs. Reranking: They Solve Different Problems

A RAG system returns an unhelpful answer. It is tempting to add a stronger reranker and hope that better relevance scores will fix it. Sometimes they do. Sometimes the document that contains the answer never reached the reranker at all.

That distinction is the foundation of a reliable search pipeline:

• Retrieval finds candidates in a large collection.
• Rank fusion combines candidate lists produced by different retrieval methods.
• Reranking improves the order of candidates already found.

These stages can use similar words—search, score, rank, relevance—but they solve different problems. Before choosing a model, ask a simpler question: is the right document missing, or merely too low in the results?

Retrieval decides which documents are eligible

A retriever reduces a large corpus to a candidate set that a more expensive stage can inspect. If a knowledge base contains one million chunks, the retriever might select 50. Those 50 are now eligible for reranking; the other 999,950 are not.

This is why a reranker cannot rescue a missing document. It receives a list, not the complete collection. If the answer is absent from that list, changing the reranker is like reorganizing a bookshelf that does not contain the book you need.

When the correct document is missing, investigate the retrieval stage:

• Was the document indexed?
• Does the chunk make sense on its own?
• Does it contain the product code, policy name, or wording people search for?
• Is the query in a language the retrieval model handles well?
• Did a score threshold remove the document too early?
• Is the candidate limit too small?
• Would a lexical and semantic candidate union improve coverage?

Corpus preparation matters here. A header-only chunk, a policy split across pages, or important terms stored only in metadata can make the right source difficult to retrieve regardless of the reranker used later.

Lexical, semantic, sparse, and hybrid retrieval

There is no universally best way to build candidates. Each method preserves different evidence.

Lexical retrieval is broader than BM25, but BM25 remains a useful reference point. It rewards term matches while accounting for term rarity, repeated occurrences, and document length. It is often a strong choice for exact identifiers, provided the index tokenizer preserves them.

Dense retrieval encodes queries and documents separately as vectors, then searches for nearby vectors. This makes large-scale search practical because document vectors can be prepared ahead of time. The Dense Passage Retrieval paper demonstrated the value of this design for open-domain question answering, but its results do not mean dense retrieval always beats BM25 in every domain.

Learned sparse models such as SPLADE keep a term-oriented, sparse representation while learning weights and expansions. They belong in the candidate-generation conversation; “sparse” does not automatically mean “classical” or “cheap.”

Hybrid retrieval combines methods. The important detail is where the combination happens. If lexical scoring is applied only after semantic search has already discarded a document, it cannot recover that document. To improve candidate coverage, form a union of candidates from the retrieval methods before cutting the list down.

RRF combines lists; it does not read the documents

Reciprocal Rank Fusion (RRF) is a simple way to combine ranked lists. It rewards documents that appear near the top of one or more lists. Because it works with positions, it avoids directly comparing a BM25 score with a cosine-similarity score—two numbers that do not share a meaningful scale.

That makes RRF useful when lexical and semantic retrieval produce complementary lists. It is inexpensive and requires no training. The original RRF paper reported strong results in its evaluated ranking experiments.

RRF still does not inspect what a document says. It cannot decide that one policy better satisfies a subtle condition, and it cannot apply a preference that was never represented by the input rankings. It is rank fusion, not semantic judgment.

• Use hybrid retrieval and RRF when different retrievers find different useful candidates.
• Use a reranker when the candidates are present but their final order needs a more detailed relevance decision.

One example, two very different failures

Consider this query:

How can I cancel an annual plan without paying a fee?

Assume the corpus contains these chunks:

• A — Monthly cancellation: Monthly plans can be cancelled at any time.
• B — Annual cooling-off period: Annual plans can be cancelled without a fee within 14 days of purchase.
• C — Annual cancellation: Annual plans may be cancelled after the first year with 30 days' notice.
• D — Refund timing: Approved refunds are returned within five business days.
• E — Enterprise contracts: Enterprise cancellation terms are defined in the signed agreement.

The following scores are invented diagnostic data, not measurements from AIVAX or any model. Their purpose is to keep the retrieval and reranking outputs visibly separate.

Failure 1: the right candidate is present but badly ordered

The retriever found B, but placed it below two broader cancellation passages. A reranker that reads the query and each candidate together can recognize the combined constraints: annual, cancel, and without a fee. This is a ranking problem.

Do not compare 0.78 with 0.96 as if the reranker “added” relevance. The two stages produce scores with different meanings. Compare ranks within each stage and evaluate them against relevance labels.

Failure 2: the right candidate never arrives

Now suppose retrieval returns A, C, D, and E. B is absent.

The reranker may place C first, but it cannot create B or search for it elsewhere. Even a perfect reranker must choose among the candidates it receives. This is a retrieval problem: improve the query, chunk, index, threshold, candidate depth, or retrieval mix.

This contrast is worth turning into a permanent test fixture. Store the query, corpus, expected relevant documents, candidate list, retrieval scores, and reranker output separately. Then a change to chunking does not get confused with a change to ranking.

Choose a reranker for the decision it must make

Once candidate coverage is healthy, the next question is not “which reranker is strongest?” It is “what decision must this reranker make?”

Lexical reranking

A lexical reranker is useful when the candidate set is already semantically reasonable and literal evidence should break ties. Product codes, quoted phrases, names, acronyms, and term proximity are common examples. It is often fast and predictable, but it will not resolve a complex preference expressed through different wording.

Cross-encoder reranking

A cross-encoder reads a query and one document together before assigning a relevance score. This allows detailed interaction between their words, unlike a single precomputed document vector. Passage Re-ranking with BERT is an early, influential example.

The trade-off is repeated work: each query-document pair normally requires inference. Cross-encoders therefore fit a shortlist better than an entire corpus. Candidate depth becomes a direct latency and cost decision.

Late interaction

Late-interaction models encode queries and documents separately but retain token-level representations for a finer comparison later. ColBERT introduced this design as a middle ground: more detailed interaction than one vector per document, while still allowing document representations to be prepared ahead of time.

Late interaction can support retrieval as well as reranking. It should not be treated as merely a cheaper cross-encoder.

LLM and listwise reranking

An LLM reranker can compare several candidates and return an order, sometimes using a natural-language instruction. This can help when relevance includes preferences such as “prefer official documentation, exclude opinion pieces, and prioritize material updated after a policy change.” Long lists may require several windows or calls, and output IDs need validation: a model can omit, duplicate, or invent them.

Do not assume that any large language model follows ranking instructions well. FollowIR found that existing retrieval models often struggled with long-form instructions and showed that instruction-specific training improved results on its benchmark. The practical lesson is that training and evaluation matter—not just parameter count.

Some requirements should not be delegated to a reranker at all. Access control, tenant boundaries, exact date cutoffs, price order, inventory, and other structured facts belong in filters or deterministic sorting whenever possible.

A practical retrieve, rerank, answer pipeline

A common production shape is:

1. Retrieve 50 candidates. Use lexical, semantic, learned sparse, or a hybrid union according to the corpus.
2. Keep retrieval evidence. Preserve source IDs, ranks, scores, and the retrieval path that found each candidate.
3. Remove duplicates and enforce permissions. Do this before spending reranker capacity.
4. Rerank the 50 candidates. Use a model appropriate for the language, document length, and decision complexity.
5. Select the top 5 for generation. Pack only the useful text and source information the answer model needs.
6. Evaluate the answer separately. A good ranking can still lead to a bad answer, and a correct answer can still be unsupported by its sources.

The numbers 50 and 5 are starting points, not universal defaults. A larger candidate set can improve the chance that a relevant document survives retrieval, but it also increases reranking latency, tokens, and cost. A larger final context can include more evidence, but it may also add distraction.

In AIVAX, semantic search retrieves candidates from indexed collections, and an optional reranker can adjust their order before the final limit. The standalone reranking endpoint instead accepts candidate strings already owned by the application. In both cases, reranking operates on supplied candidates; it does not search for missing text.

Measure each stage with the metric it can change

One end-to-end score is not enough to diagnose a pipeline. Measure the boundary between stages.

Recall@k asks what fraction of all known relevant documents appeared in the first k candidates. Hit rate@k asks whether at least one relevant document appeared. They can coincide when each query has one relevant answer, but they are not generally interchangeable.

MRR emphasizes the position of the first relevant result. nDCG@k rewards useful documents near the top and can represent graded relevance. Precision@k asks what share of the first k results is relevant.

For a retrieve-50, rerank-5 pipeline, measure:

• candidate recall or hit rate at 50 before reranking;
• MRR, nDCG@5, or Precision@5 after reranking;
• answer correctness and source support after generation;
• latency and cost for the complete task.

You can also measure reranking only on queries where a relevant document was present in the candidate set. That isolates ordering quality, but always report candidate recall beside it. Otherwise, a strong reranker score can hide a retriever that frequently misses the answer.

Use representative queries, relevance judgments, and real document shapes. Keep the corpus, chunking, candidate generator, candidate depth, and cutoff fixed when comparing rerankers. The BEIR benchmark is a useful reminder that methods behave differently across domains and that stronger ranking can carry substantial computational cost.

Diagnose first, then add complexity

Use this order when a search pipeline disappoints:

1. The correct document is absent. Fix indexing, chunking, query formulation, thresholds, candidate depth, or retrieval coverage.
2. Lexical and semantic retrievers find different useful documents. Combine their candidate lists; consider RRF when rank-based fusion is appropriate.
3. The correct document is present but too low. Add or improve reranking.
4. Literal evidence should decide among otherwise similar candidates. Try lexical reranking.
5. The decision depends on nuanced semantic relevance. Test a neural reranker on labeled examples.
6. The query includes detailed inclusion, exclusion, or preference rules. Test a model trained and evaluated for instruction following.
7. The rule is structured and mandatory. Use a filter or deterministic sort, not a relevance model.
8. Quality improves, but latency or cost does not justify it. Reduce candidate depth, batch work where supported, cache reusable representations, or remove the extra stage.

Also inspect duplicates, language coverage, document length limits, batch behavior, and cache boundaries before launch. These operational details can change both quality and cost, even when the ranking model itself stays the same.

The goal is not to assemble the most sophisticated stack. It is to identify whether the error belongs to retrieval, fusion, ordering, or generation, then use the simplest component that fixes that error.

References

• Stephen Robertson and Hugo Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond.
• Vladimir Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering.
• Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant, SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking.
• Gordon Cormack, Charles Clarke, and Stefan Büttcher, Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods.
• Rodrigo Nogueira and Kyunghyun Cho, Passage Re-ranking with BERT.
• Omar Khattab and Matei Zaharia, ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.
• Yutao Sun et al., Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents.
• Orion Weller et al., FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions.
• NIST TREC, Common Evaluation Measures.
• Nandan Thakur et al., BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.