RAG Query Rewriting: HyDE, Multi-Query, Decomposition
Use normalization for noisy wording, multi-query retrieval for vocabulary uncertainty, HyDE when a hypothetical document can bridge a short question to corpus language, and decomposition when one request contains independent evidence needs. Route by diagnosed recall failure instead of running every method.
Choose the smallest transformation that fits the failure
| Failure shape | Method | Retrieval effect | Main risk |
|---|---|---|---|
| Typos, filler, or conversational framing | Normalization or concise rewrite | Produces a cleaner lexical or vector query | The rewrite drops a constraint. |
| Several plausible vocabularies | Multi-query | Searches alternative phrasings and merges candidates | Candidate noise and duplicated evidence. |
| Very short query unlike document prose | HyDE | Embeds a hypothetical document to locate nearby real documents | False details leak into answer context. |
| Independent subquestions | Decomposition | Retrieves evidence for each part separately | Relationships and shared constraints are lost. |
| Ambiguous entity or time reference | Clarification or bounded variants | Avoids collapsing distinct interpretations | The system guesses user intent. |
| Exact identifier already present | Original query | Preserves the strongest lexical signal | A rewrite removes the exact token. |
Query transformation belongs to the retrieval stage
Store the original request as the task contract. A query transformer may derive one or more search representations, but it should not silently redefine scope, dates, entities, exclusions, or authorization. Search every variant under the same trusted filter, retain the method and model version, and preserve which variant retrieved each candidate.
HyDE pivots through generated document language
The original HyDE paper instructs a language model to generate a hypothetical document, embeds that document, and retrieves similar real corpus documents. The authors explicitly note that the hypothetical document is unreal and may contain false details. That warning defines the engineering boundary: use the representation to search, then discard it as factual support.
Multi-query and decomposition solve different problems
Multi-query explores alternate expressions of one information need. Decomposition divides a compound need into parts. Merging multi-query candidates requires deduplication and channel provenance. Recombining decomposed answers requires checking that each subanswer uses compatible entities, dates, units, and document revisions.
Answer only from retrieved source evidence
The generator may receive the original question, retrieved passages, and a structured description of subquestions. It should not receive an unmarked hypothetical answer as if it came from the corpus. Label generated search artifacts clearly, exclude them from citations, and validate every material claim against a real document identifier.
When subqueries retrieve conflicting evidence, do not let synthesis smooth the conflict away. Preserve source dates and scopes, present the disagreement when relevant, or abstain. Query expansion improves candidate coverage; it does not certify which source is authoritative.
Evaluate transformation by failure slice and ablation
Create cases for short queries, jargon mismatch, ambiguous names, explicit identifiers, multi-part requests, negation, dates, and unanswerable questions. Compare the original query, each transformation alone, and the routed policy against the same corpus snapshot. Measure required-evidence recall, rank, duplicate rate, irrelevant candidate growth, latency, model calls, and final answer support.
The RAG evaluation survey separates retrieval and generation dimensions. Follow that boundary: first ask whether transformation recovered missing evidence, then ask whether the answer used it accurately. A fluent improvement without better source retrieval is not proof that query rewriting worked.
Failure cases that reveal unsafe expansion
- Constraint erasure: a rewrite keeps the topic but drops “before 2024” or “excluding contractors.” Compare structured constraints with the original.
- Hypothesis leakage: a fabricated HyDE detail appears in the answer or citation. Keep generated search artifacts outside evidence context.
- Variant explosion: many rewrites add latency and noise without recovering new required documents. Bound variants and stop on marginal evidence.
- Identifier destruction: an exact error code becomes a descriptive paraphrase. Always retain the original query as a retrieval channel.
- Decomposition drift: subqueries refer to different products, periods, or policy versions. Carry shared constraints into every branch.
- Unauthorized expansion: a broader rewrite searches content outside the caller's scope. Bind filters independently of generated text.
A measured implementation sequence
- Label the recall failure. Identify vocabulary mismatch, noisy wording, ambiguity, compound intent, or an exact-token miss before choosing a method.
- Parse immutable constraints. Preserve entities, dates, exclusions, tenant policy, requested output, and the original user text.
- Build the original-query baseline. Record candidates, ranks, latency, and missing required evidence for each question slice.
- Add one transformation. Trace generated queries, subqueries, hypothetical text, candidate provenance, deduplication, and failures.
- Enforce the evidence boundary. Allow only real retrieved documents into citation and answer support; mark generated artifacts as non-evidence.
- Run ablations. Compare original-only, transformed-only, combined, and routed variants on retrieval and answer metrics.
- Route and cap. Invoke the method only for the diagnosed failure, bound model calls and candidates, and fall back safely on transformation errors.
Decision summary
Apply the smallest query transformation that fixes a named retrieval failure, keep the original query for traceability, and compare transformed retrieval against a no-rewrite baseline. Generated queries expand recall opportunities, not the evidence allowed in the answer.
Primary and official sources
- Gao et al., “Precise Zero-Shot Dense Retrieval without Relevance Labels” — the original Hypothetical Document Embeddings method and its limits.
- Yu et al., “Evaluation of Retrieval-Augmented Generation: A Survey” — research framework for separate retrieval and generation evaluation.