Query Expansion (QE)

• Query Expansion (QE) reformulates a user query by adding terms & annotations to reduce vocabulary mismatch and improve recall, but may result in query drift and reduced precision if not carefully managed.
• QE techniques such as pseudo-relevance feedback, external lexical & domain-specific resources, and neural generative models can enhance search results by adding relevant terms and phrases
• Numerous applications include topic diversification, event-centric searches, and integration with both sparse and dense retrieval models

Query expansion (QE) is an information-retrieval technique that reformulates a user query by adding terms, phrases, concepts, pseudo-documents, or other query-side evidence intended to reduce vocabulary mismatch between the query and relevant documents. Expansion may use local pseudo-relevance feedback, external lexical or encyclopedic resources, domain-specific corpora, crowd knowledge, document collections, LLMs, or learned representations. Its principal objective is usually higher recall, although poorly selected expansion can reduce precision through query drift. Contemporary systems increasingly treat QE as a query-adaptive integration problem involving candidate generation, term or passage selection, weighting, semantic disambiguation, and compatibility with the downstream retrieval model.

1. Foundations and retrieval objectives

A query is an incomplete representation of an information need. Users may employ terminology different from that found in relevant documents, omit important contextual information, submit ambiguous expressions, or formulate queries that are too short to specify the intended topic. QE addresses this problem by constructing an expanded query $q'$ from the original query $q$ and an expansion set $E(q)$:

$q' = q \cup E(q).$

In weighted vector-space formulations, expansion can modify both vocabulary and term importance. A general Rocchio-style representation is:

$$\mathbf{q}'=\alpha\mathbf{q}+\beta\mathbf{d}_{+}-\gamma\mathbf{d}_{-},$$

where $\mathbf{d}_{+}$ and $\mathbf{d}_{-}$ represent positive and negative document centroids. In practice, many systems retain the original query and append expansion material rather than explicitly subtracting nonrelevant evidence.

The central trade-off is between recall and precision. Adding synonyms, related terms, entities, or contextual descriptions can retrieve documents that do not contain the original wording. Conversely, semantically related terms may be too general, ambiguous, or associated with an unintended subtopic. Such expansion causes query drift. The risk is especially pronounced when expansion relies on pseudo-relevance feedback, because the system assumes that top-ranked documents from an initial retrieval pass are relevant.

QE effectiveness is query-dependent. A method can improve collection-level MAP while harming many individual queries. Query length, ambiguity, domain, desired recall, named entities, query structure, language, and the quality of initial retrieval all affect whether expansion is beneficial. A taxonomy organized around these properties distinguishes short, hard, ambiguous, negative-term, named-entity, multi-aspect, high-level, recall-oriented, context-dependent, domain-specific, short-answer, special-processing, multilingual, and noisy queries (Pal et al., 2015).

2. Candidate sources and classical selection methods

Local pseudo-relevance feedback

Local or pseudo-relevance feedback extracts candidate terms from the highest-ranked documents returned by a first-pass retrieval. A typical pipeline issues the original query, selects a top-ranked feedback set, extracts and scores candidate terms, reweights the original and added terms, and performs a second retrieval. The approach is attractive because it is collection-specific and does not require external resources or explicit relevance judgments.

The principal weakness is error propagation. If the initial ranking is poor, terms from nonrelevant documents can reinforce an incorrect interpretation. In experiments comparing cooccurrence and distributional evidence, approximately ten feedback documents were optimal across tested methods. Cooccurrence methods favored terms appearing in the same feedback documents as query terms, whereas Kullback–Leibler divergence and Divergence From Randomness Bo1 favored terms unusually concentrated in the feedback set relative to the collection (0804.2057).

For cooccurrence, candidate relevance is computed across query terms:

$$\operatorname{rel}(q,t_e)=\sum_{t_i\in q}q_i\,\operatorname{CC}(t_i,t_e),$$

where $\operatorname{CC}$ may be a Tanimoto, Dice, or Cosine coefficient. Tanimoto produced the strongest overall results in the reported comparison, with MAP $0.4831$, compared with $q$0 for Dice and $q$1 for Cosine. Distributional methods instead compare feedback-set and collection-level term behavior. KLD uses:

$q$2

while Bo1 compares observed feedback frequency with a random term distribution.

The two evidence types are complementary. Cooccurrence captures local association with query terms, whereas KLD and Bo1 capture collection-relative informativeness. A simple intersection of independently ranked candidate lists improved over either component alone. With the Spanish EFE94 collection, the Bo1–cooccurrence combination obtained MAP $q$3, compared with $q$4 for the unexpanded baseline, although the supplied report does not provide statistical-significance tests (0804.2057).

External lexical and encyclopedic resources

WordNet supplies synonyms and semantic relations, but isolated lexical expansion can be inadequate for phrases, entities, domain terminology, and ambiguous words. A Wikipedia–WordNet method therefore assigns different resources to different query structures: WordNet primarily expands individual terms, while Wikipedia supplies contextual and entity-oriented terms for phrases. Candidates receive source-specific first-stage scores and are then reweighted against the entire query. On the FIRE 2011 collection, the combined WWQE system raised IFB2 MAP from $q$5 to $q$6, a reported $q$7 improvement, and GMAP from $q$8 to $q$9, a reported $E(q)$0 improvement (Azad et al., 2019).

A Bayesian-network thesaurus provides another collection-specific external-resource alternative. Terms are represented as binary random variables, and a polytree is learned from document–term occurrence data. Pairwise mutual information determines maximum-weight spanning-tree links, chi-square tests reject unsupported dependencies, and evidence from query terms is propagated through the network. A candidate term $E(q)$1 is added when:

$E(q)$2

The added term’s weight is its posterior probability. Experiments on Adi, Cranfield, and Medlars showed substantial improvements for Adi and Medlars but weaker and inconsistent results for Cranfield, demonstrating that collection-specific associations and threshold selection materially affect performance (Campos et al., 2013).

Domain-specific and professional resources

For code search, QECK obtains software-specific vocabulary from accepted Stack Overflow answers rather than from general lexical resources. It combines textual similarity with crowd-derived quality to select pseudo-relevant question–answer pairs, extracts TF-IDF terms, and appends selected terms to the original code-search query. In an Android code-search evaluation, QECK improved three retrieval algorithms by up to $E(q)$3 in Precision and $E(q)$4 in NDCG. QECKRocchio achieved mean Precision $E(q)$5 and mean NDCG $E(q)$6, compared with $E(q)$7 and $E(q)$8 for the WordNet-based PWordNet baseline (Nie et al., 2017).

Patent searching illustrates the limits of fully automatic domain expansion. Technology-specific patent corpora generally produced better embedding-based F1 scores than a generic patent corpus across eleven technical areas. However, raw embeddings remained substantially below professional examiner quality. In an optics example, embedding expansion produced $E(q)$9, while professional crowdsourcing produced $q' = q \cup E(q).$0. The resulting interface permits examiners to up-vote, down-vote, and manually add terms; feedback is then shared within an Art Unit or Workgroup. Multi-term expansion computes a centroid of selected term vectors, enabling examiner-guided refinement (Krishna et al., 2019).

3. Result-oriented, structured, and query-adaptive expansion

Cluster-oriented expansion

Query expansion can expose multiple interpretations rather than produce a single globally popular list. Query Expansion with Clusters first clusters the results of the original query and then generates one expanded query for each cluster. Each query $q' = q \cup E(q).$1 should retrieve as much as possible from its target cluster $q' = q \cup E(q).$2 while avoiding other clusters:

$q' = q \cup E(q).$3

The quality of each expansion is evaluated using precision, recall, and F-measure, with the complete set aggregated through the harmonic mean of cluster-specific F-measures. The optimization problem is APX-hard, because selecting a conjunction of keywords that balances target-cluster recall against non-target elimination involves keyword interactions and an exponential query space (Liu et al., 2011).

Two heuristic algorithms are proposed. Iterative Single-Keyword Refinement (ISKR) greedily adds or removes keywords using a benefit-to-cost ratio. Partial Elimination Based Convergence (PEBC) samples queries intended to eliminate specified percentages of non-target results and searches promising intervals of the resulting F-measure landscape. ISKR is generally more locally precise but slower; PEBC is faster and randomized, without global optimality guarantees.

The approach differs from cluster labeling. A cluster label may contain individually frequent terms that do not co-occur, causing an AND query to retrieve nothing. Query generation must therefore model keyword interaction. In experiments on Shopping and Wikipedia data, ISKR and PEBC produced more comprehensive and diverse expansions than Data Clouds, Google-related-query suggestions, and cluster summarization. For example, “Java” yielded separate expansions involving “Java, Server,” “Java, code,” and “Java, Island.” The method remains dependent on the initial result set and clustering: an interpretation absent from the results cannot be recovered.

Query categorisation

The query-categorisation perspective argues that QE should be applied selectively. Short queries may benefit from expansion because they incompletely express the information need, whereas long queries may require selective weighting rather than indiscriminate addition. Hard queries can invalidate PRF assumptions, and easy queries may be harmed by unnecessary expansion. Ambiguous queries may require disambiguation or result diversification before expansion. Multi-aspect queries require balanced expansion across aspects, while recall-oriented searches may tolerate lower precision in exchange for broader coverage (Pal et al., 2015).

This perspective treats “no expansion” as a valid action. Categories are not mutually exclusive: a query may simultaneously be short, ambiguous, domain-specific, and recall-oriented. Some labels are available only after retrieval, such as whether the initial ranking is hard or easy. Others may require user information, including implicit context, domain, or intended sense. The proposed research direction is a multilabel query-adaptive policy using pre-retrieval features, post-retrieval signals, query-performance predictors, entity detection, negation scope, language identification, and result diversity.

Streaming and event-centric expansion

For streaming data, query vocabulary changes over time. A proactive system combines current-stream evidence with historical external evidence. It uses Dynamic Eigenvector Centrality (DEC) to detect emergent vocabulary, Latent Dirichlet Allocation (LDA) to identify topics, and either external fastText neighbors or historical bigram co-occurrences to add terms. Query expansion is triggered when the Jaccard similarity between current and previous emergent-word sets falls below a threshold.

In a Baltimore protest case study, the primary stream consisted of Twitter data divided into fifteen-minute windows. The external source contained 2014 articles from the New York Times and CNN. With $q' = q \cup E(q).$4, $q' = q \cup E(q).$5, and $q' = q \cup E(q).$6, 373 of 1,573 windows were classified as emergent. Proactive vector-space and co-occurrence variants returned more tweets and hashtags and produced fewer hashtag clusters than Static and Emergent baselines. For one topic, Proactive VS returned 3.65 times as many tweets as Static, while Proactive CO returned 5.88 times as many. These measures are proxies rather than human relevance judgments (Alshanik et al., 2022).

Event-Centric Query Expansion addresses a related production problem by mining fresh news headlines rather than waiting for long-term query logs. Its stages are event collection, event reformulation, semantic retrieval, and online ranking. A RoBERTa classifier filters event-bearing headlines, encoder–decoder models reformulate them into concise event phrases, a dual-tower model retrieves candidate events, and a gradient-boosted decision tree ranks candidates using semantic, lexical, freshness, popularity, and downstream search features. The system contains more than four million event vectors and is deployed in Tencent QQ Browser Search. Offline recall and online A/B-test metrics improved over the baseline, with a reported approximately $q' = q \cup E(q).$7 ms overall latency increase (Zhang et al., 2023).

4. Neural and generative expansion

Learned term selection

Embedding-based QE retrieves terms close to a query centroid or individual query terms. However, semantic proximity does not imply retrieval usefulness. A term may be general, ambiguous, redundant, or associated with a wrong intent. The Deep Expansion Classifier (DEC) addresses this by labeling candidate terms as Good, Neutral, or Bad according to whether adding the term increases, negligibly changes, or decreases MAP.

DEC uses a Siamese architecture. Query and candidate embeddings are concatenated and processed by a shared BiLSTM, producing a query–term representation. Two representations are compared through element-wise subtraction, and a softmax predicts whether the pairs belong to the same class. At inference, the candidate is compared with reference examples, and the estimated probability of usefulness reweights the original embedding-based similarity. On AP, Robust, WT2G, and WT10G, DEC achieved the highest MAP among QLM, AWE, and EQE1 baselines. It also obtained the highest robustness index on AP, Robust, and WT10G, indicating that it improved more queries than it degraded (Imani et al., 2018).

Generated text as query evidence

GPT-2-based expansion generates complete texts from the query and concatenates them into a large query submitted to BM25+ or a language-model retriever. Generated text supplies new vocabulary, syntagmatic associations, term frequencies, and estimates of original-term importance. On Tipster, BM25+ with GPT-generated expansion achieved MAP $q' = q \cup E(q).$8, compared with $q' = q \cup E(q).$9 for unexpanded BM25+ and $$\mathbf{q}'=\alpha\mathbf{q}+\beta\mathbf{d}_{+}-\gamma\mathbf{d}_{-},$$0 for LM+RM3. On GOV2, it achieved $$\mathbf{q}'=\alpha\mathbf{q}+\beta\mathbf{d}_{+}-\gamma\mathbf{d}_{-},$$1, compared with $$\mathbf{q}'=\alpha\mathbf{q}+\beta\mathbf{d}_{+}-\gamma\mathbf{d}_{-},$$2 for BM25+ and $$\mathbf{q}'=\alpha\mathbf{q}+\beta\mathbf{d}_{+}-\gamma\mathbf{d}_{-},$$3 for LM+RM3. Fine-tuning GPT-2 on ohsumed improved MAP from $$\mathbf{q}'=\alpha\mathbf{q}+\beta\mathbf{d}_{+}-\gamma\mathbf{d}_{-},$$4 to $$\mathbf{q}'=\alpha\mathbf{q}+\beta\mathbf{d}_{+}-\gamma\mathbf{d}_{-},$$5 (Claveau, 2020).

Query2doc uses a few-shot-prompted LLM to generate a pseudo-document. For BM25, the original query is repeated five times before the generated passage, preventing the longer pseudo-document from dominating term frequencies. For dense retrieval, the query and pseudo-document are concatenated with a separator. On TREC Deep Learning 2019 and 2020, BM25 plus Query2doc improved nDCG@10 from $$\mathbf{q}'=\alpha\mathbf{q}+\beta\mathbf{d}_{+}-\gamma\mathbf{d}_{-},$$6 to $$\mathbf{q}'=\alpha\mathbf{q}+\beta\mathbf{d}_{+}-\gamma\mathbf{d}_{-},$$7 and from $$\mathbf{q}'=\alpha\mathbf{q}+\beta\mathbf{d}_{+}-\gamma\mathbf{d}_{-},$$8 to $$\mathbf{q}'=\alpha\mathbf{q}+\beta\mathbf{d}_{+}-\gamma\mathbf{d}_{-},$$9, respectively. Query2doc also improved dense retrievers, though gains were smaller for stronger models. On five zero-shot BEIR datasets, BM25 improved on every listed dataset, while dense improvements were mixed (Wang et al., 2023).

Contextual clue sampling addresses the diversity–reliability trade-off in generated expansion. BART-large generates 100 answer-related contexts, clusters them using fuzzy lexical similarity with cutoff $\mathbf{d}_{+}$0, and retains the most probable context from each cluster. The retained contexts are used in separate BM25 searches, whose results are fused using generation probabilities. Filtering reduces the average number of contexts to approximately 24 for Natural Questions and 33 for TriviaQA. Ours-multi achieved NQ Top-100 $\mathbf{d}_{+}$1 and TriviaQA Top-100 $\mathbf{d}_{+}$2, while using a $\mathbf{d}_{+}$3 GB index compared with $\mathbf{d}_{+}$4 GB for DPR. The method improves retrieval and end-to-end Exact Match but incurs multiple retrieval operations and generation cost (Liu et al., 2022).

Multilingual generative QE shows that prompt strategy depends strongly on query length. On short CLIRMatrix title queries, zero-shot prompting was generally strongest; on longer mMARCO questions, few-shot or Chain-of-Thought prompting was often better. Fine-tuning helped when training and test formats were similar but harmed transfer to a different question–passage format. Large disparities persisted across languages, particularly for different scripts. These results establish that generative QE is conditional on language, script, model capacity, query length, prompting strategy, and retrieval configuration (Macmillan-Scott et al., 24 Nov 2025).

5. Integration with retrieval models

QE is not independent of the retrieval model. The same expansion can help one retriever and harm another because term frequency, vector encoding, score calibration, context length, and query composition differ.

For sparse retrieval, expanded terms directly affect inverted-index matching. Boolean systems use OR to broaden retrieval, AND to impose complementary constraints, and NOT or NEAR for exclusion and proximity. Xu combines Datamuse candidates, Wikipedia-trained Word2Vec vectors, high-dimensional clustering, and Boolean composition. Terms within semantic clusters are joined by OR, while clusters are joined by AND. With 373 industry queries and the All the News corpus, Xu achieved approximately $\mathbf{d}_{+}$5 agreement accuracy with human-generated expansions, compared with approximately $\mathbf{d}_{+}$6 for Datamuse-only expansion, but required greater processing time (Gallant et al., 2018).

For image retrieval, expansion is a learned set-to-vector problem. Attention-Based Query Expansion Learning (LAttQE) applies self-attention to the original image and its top-ranked neighbors, including learned positional encodings. The model predicts non-monotonic, context-dependent weights and aggregates the original descriptors. With 64 neighbors, full LAttQE reached mean mAP $\mathbf{d}_{+}$7 in an ablation table, compared with $\mathbf{d}_{+}$8 without QE. It was more robust than average, rank-decay, similarity-weighted, and per-query SVM approaches across Oxford, Paris, difficulty levels, and distractor settings (Gordo et al., 2020).

Recent work emphasizes that dense QE depends not only on generated text but also on integration. AnchorQE separately encodes the original query and expansion, then interpolates their normalized vectors:

$\mathbf{d}_{+}$9

where $\mathbf{d}_{-}$0 is an expansion mixture and $\mathbf{d}_{-}$1 controls total expansion trust. At $\mathbf{d}_{-}$2, the method is ordinary dense retrieval. This explicit anchor prevents expansion-only retrieval from discarding the original query and avoids uncontrolled text-level concatenation. AnchorQE is equivalent, under stated assumptions, to weighted CombSUM while requiring one final retrieval request. The reported method improved retrieval by up to $\mathbf{d}_{-}$3 over expansion-only or text-concatenation baselines, and its online interpolation strategy outperformed a fixed development-tuned weight by up to $\mathbf{d}_{-}$4 (Sun et al., 26 Aug 2026).

For query-oriented multi-document summarization, expansion modifies the graph through which relevance propagates. A manifold-ranking system combines WordNet similarity, document-set mean and variance statistics, and TextRank-derived terms. The expanded query changes TF–ISF sentence representations, while literal overlap remains based on original query terms. On DUC 2006, the complete system improved ROUGE-1 from $\mathbf{d}_{-}$5 to $\mathbf{d}_{-}$6; on DUC 2007, it improved ROUGE-1 from $\mathbf{d}_{-}$7 to $\mathbf{d}_{-}$8 (Jia et al., 2021).

6. Evaluation, limitations, and research directions

Evaluation criteria

QE is evaluated extrinsically through downstream retrieval or task performance. Common measures include MAP, GMAP, average precision, R-Precision, precision at rank $\mathbf{d}_{-}$9, recall at rank $$\operatorname{rel}(q,t_e)=\sum_{t_i\in q}q_i\,\operatorname{CC}(t_i,t_e),$$0, nDCG, MRR, F-measure, bpref, ROUGE, Exact Match, and robustness indices. Evaluation designs may compare an unexpanded baseline, one or more individual expansion sources, a combined system, and an oracle that selects the best method per query.

Reported gains require careful interpretation. Some studies use human-generated expansions as a reference rather than independent relevance judgments. Others use pseudo-relevance labels, automatically constructed relevance judgments, or proxies such as hashtag counts. Improvements without significance tests should be treated as empirical observations rather than statistically confirmed differences. In particular, query-level variability can be hidden by MAP averages.

Principal limitations

The pseudo-relevance assumption remains a central failure mode. Initial ranking errors can propagate into expansion, especially for hard, ambiguous, or poorly represented queries. External resources introduce coverage, temporal, domain, and licensing constraints. Wikipedia, WordNet, search-engine results, Stack Overflow, and historical news each encode different biases and may omit emerging terminology.

Semantic similarity is not synonymy. Embeddings and LLMs may return broader, narrower, associated, or merely co-occurring terms. Generative models can hallucinate entities, dates, numbers, and causal relations. Factual accuracy is not always required for lexical retrieval, but incorrect content can cause query drift. Filtering, probability weighting, expert feedback, and explicit anchoring mitigate rather than eliminate this risk.

Many systems are incompletely specified. Candidate filtering, probability normalization, term weighting, duplicate handling, thresholds, prompt selection, score calibration, and expansion length are often underreported. Several approaches also lack complete component ablations, making it difficult to attribute improvements separately to candidate generation, weighting, filtering, clustering, or retrieval integration.

Computational cost varies by architecture. Classical feedback requires an initial retrieval pass; graph and Bayesian methods can incur pairwise or dense-graph costs; language-model generation introduces substantial query-time latency; multi-context retrieval requires multiple searches; event-centric systems require continuous crawling, model inference, vector indexing, and caching. Dense integration can reduce multiple retrieval calls but requires encoder computation and careful calibration.

Research directions

A consistent direction across the literature is query-adaptive expansion. A future system may classify queries using multiple overlapping labels and choose among no expansion, local feedback, global or external resources, generative pseudo-documents, interactive expansion, or learned vector integration. Pre-retrieval signals include query length, syntax, named entities, ambiguity, language, domain vocabulary, and clarity. Post-retrieval signals include score distributions, result diversity, estimated relevance, aspect coverage, and initial-ranking quality.

Other directions include score-based fusion of cooccurrence and distributional evidence; adaptive selection of expansion terms; domain-specific and multilingual training; expert-in-the-loop thesaurus construction; improved factuality verification; learned expansion–retrieval objectives; hybrid sparse–dense retrieval; efficient local LLMs; richer Boolean operators; phrase- and entity-aware indexing; and expansion methods robust to script and resource disparities.

Across these developments, QE has shifted from adding a small set of synonyms toward a broader family of evidence-integration strategies. The recurring design problem is to introduce enough new information to bridge lexical and semantic gaps while preserving the original information need. Effective systems therefore combine candidate diversity with relevance control, collection or domain adaptation with query-specific inference, and expansion generation with explicit mechanisms for weighting, filtering, anchoring, or interaction.