Reranking RAG Retrieval: Candidate Recall Before Top-K

The two-stage retrieval map

Design the reranker around the candidate bottleneck

A first-stage retriever is optimized for coverage within a strict latency and resource envelope. It can be lexical, vector, or hybrid. The reranker receives only that bounded pool and spends more work comparing each candidate to the question. The useful control is not “reranking on or off”; it is the relationship among candidate breadth, reranking cost, final context size, and required-evidence recall.

Azure AI Search documents semantic ranking as a step that occurs after reciprocal rank fusion for hybrid queries. Its RRF and semantic ranking guidance also reports the fused score separately from the reranker score. Preserve the same separation in your telemetry even when your implementation uses a different search engine.

Never compare scores as if they share a scale

Lexical, vector, fusion, and reranking scores come from different functions. A threshold copied from one layer has no guaranteed meaning in another. Calibrate decisions against labeled cases, preserve rank and score provenance, and version any normalization or cutoff policy.

Reranking ends before answer generation begins

The reranker's output is still evidence selection, not a response. A top-ranked passage may be relevant but incomplete, outdated, contradictory, or controlled by an adjacent clause. The context builder may need to add neighboring passages, group sources, retain revision dates, or abstain when the pool lacks sufficient evidence.

Evaluate whether the required passage reaches the generator separately from whether the generator follows it. This makes the repair path explicit: improve retrieval when evidence is absent, adjust context assembly when evidence is damaged, and change the answer contract when good context is used incorrectly.

Use a reranking scorecard, not one answer score

Ground-truth relevance judgments let you test document retrieval independently. Microsoft Foundry's official RAG evaluator documentation separates process evaluation for retrieval from system evaluation for groundedness, relevance, and completeness. That separation is the right experimental boundary for reranking.

Reranking failure cases to reproduce

• Missing-before-rerank: the relevant clause never enters the candidate pool. No reranker choice can recover it.
• Authorization after ranking: forbidden candidates influence ordering before removal. Apply the caller's boundary before any relevance operation.
• Duplicate top ranks: overlapping chunks from one document occupy the entire final context. Deduplicate with span-aware rules.
• Relevance without sufficiency: a passage is on topic but lacks the exception required to answer. Label evidence completeness, not topic match alone.
• Latency without gain: reranking changes positions but not selected evidence or answer quality. Remove work that does not change the decision.
• Score threshold drift: an engine or model update changes score distribution. Calibrate on versioned cases rather than a timeless constant.

A practical implementation sequence

1. Write relevance judgments. For representative questions, identify required, useful, irrelevant, stale, and forbidden passages.
2. Measure first-stage recall. Compare candidate breadth by question slice while holding corpus, chunking, filters, and query policy constant.
3. Add the reranker behind a trace boundary. Record ordered inputs, ordered outputs, versions, timing, errors, and fallback behavior.
4. Tune the candidate budget. Increase breadth only while it recovers required evidence at acceptable cost; stop when added candidates are mostly noise.
5. Test final context assembly. Deduplicate spans, preserve source identity, and add adjacent context only through documented rules.
6. Run answer evaluations. Check grounding, completeness, citation mapping, and abstention with the same questions.
7. Promote by slice. Route reranking only where it improves a named failure and preserve a simpler fallback for timeouts or service errors.

Decision summary

Use a broad first stage to create candidates and a more expensive reranker only where ranking quality justifies its latency. Evaluate candidate recall before reranker quality, preserve score lineage, and tune top-k against the downstream context budget.