Candidate Token Query Expansion (CTQE)

• CTQE is a query expansion method that generates, scores, filters, and integrates candidate tokens for improved lexical transparency and low-latency retrieval.
• It combines distribution-based signals with association-based criteria to refine candidate tokens while effectively mitigating query drift.
• Evaluations demonstrate enhanced retrieval metrics across lexical, dense, and sparse retrieval systems when CTQE's controlled candidate filtering is applied.

Candidate Token Query Expansion (CTQE) denotes a class of query expansion methods that explicitly generates candidate terms or tokens, scores and filters them, and then integrates the surviving candidates into retrieval. In the direct LLM-era formulation, CTQE reuses high-probability but unselected next-token alternatives from a single decoding pass; in the broader information-retrieval lineage, the same architectural pattern appears in pseudo-relevance-feedback systems that extract terms from top-ranked documents, rank them by distributional, association, or semantic criteria, and append or reweight the query accordingly (Kim et al., 2 Sep 2025, Pal et al., 2013, 0804.2057).

1. Scope and defining characteristics

In strict usage, CTQE is distinguished by an explicit candidate set and an explicit candidate-selection stage. The candidates are lexical units—typically tokens, terms, or short phrases—rather than whole passages or documents. A CTQE pipeline therefore has four recognizable stages: candidate generation, candidate scoring, candidate filtering, and integration into the query or retrieval score.

This makes CTQE different from several closely related expansion families. BERT-QE, for example, is a neural pseudo-relevance-feedback reranking framework in which the crucial candidate objects are overlapping short chunks extracted from top-ranked documents, not tokens; the selected chunks are then used as weighted contextual evidence in chunk-document scoring rather than appended as expansion terms (Zheng et al., 2020). ThinkQE likewise expands queries through free-form answering passages produced by an R1-distilled LLM and refined through iterative corpus interaction, so its expansion unit is a passage segment rather than a ranked token list (Lei et al., 10 Jun 2025). Multi-LLM in-domain exemplar methods are also adjacent rather than identical: they generate natural-language expansions, consolidate them through a refinement model, and preserve useful entities, relations, and domain knowledge, but they do not expose a token-level candidate ranking objective (Li et al., 9 Feb 2026).

The boundary is therefore methodological rather than historical. A method is most clearly CTQE when it exposes candidate lexical units as first-class objects and makes their admission into the expanded query dependent on explicit selection logic. A plausible implication is that CTQE is especially attractive in settings requiring lexical transparency, structured query control, or low-latency retrieval-time expansion.

2. Classical foundations in automatic query expansion

Long before the term CTQE was used directly, classical automatic query expansion already instantiated its core design pattern. In pseudo-relevance-feedback settings, candidate terms were typically extracted from top-ranked documents, then scored either by association with original query terms or by their distributional salience in the pseudo-relevant set. One representative cooccurrence formulation scores a candidate expansion term $t_e$ against the whole query by aggregating its association with each original query term: $$\mathrm{rel}(q,t_e) = \sum_{t_i \in q} q_i \, \mathrm{CC}(t_i,t_e),$$ where $\mathrm{CC}$ can be Tanimoto, Dice, or Cosine; the same study reports that Tanimoto performed best overall, that about 10 top documents worked best, and that around 25 terms were best for cooccurrence alone (0804.2057).

Distribution-based methods provided a complementary scoring family. A standard KLD formulation ranks a term by how overrepresented it is in the pseudo-relevant set relative to the collection: $$KLD_{(PR,PC)}(t) = P_R(t)\,\log \frac{P_R(t)}{P_C(t)}.$$ Bo1 provided a divergence-from-randomness alternative, and later work showed that a strong practical strategy is to use a distribution-based method such as KLD or modified Bo1 for broad candidate generation and then refine that large pool with an association-based method such as modified LCA or RM3 (Pal et al., 2013). In that generate-then-rerank scheme, candidates are first proposed from a small, high-precision pseudo-relevant set, then reranked by query-term association, with final weights retained from the distributional stage rather than the association stage.

Semantic reranking extended this pattern further. P-WNET keeps candidate generation corpus-grounded—terms come from pseudo-relevant documents rather than directly from WordNet—but scores each candidate using gloss overlap with query terms, Robertson $idf$, and the quality of source pseudo-relevant documents. KLWNET then linearly combines this semantic score with KLDLCA, thereby integrating three signals: distribution in pseudo-relevant documents versus corpus, statistical association with query terms, and semantic relation via WordNet definitions (Pal et al., 2013). A related web-knowledge formulation replaces the feedback corpus with cleaned full web pages retrieved from Google, Bing, and DuckDuckGo, then ranks candidate terms by tf-itf, kNN-based cosine similarity, and a correlation score with the whole query; its best reported setting used 20 pseudo-relevant web documents and 15 final expansion terms (1908.10193).

Taken together, these systems established several durable CTQE principles: candidate terms should be harvested from a contextually plausible source; distributional salience and query-term association are different signals rather than substitutes; semantic resources are often most useful as rerankers over corpus-derived candidates; and final expansion quality is determined as much by filtering as by generation.

3. Direct CTQE with LLM decoding traces

The direct formulation of CTQE uses a standard keyword-generation prompt but treats the LLM’s discarded next-token alternatives as a candidate pool rather than as wasted computation. Let the generated keywords be

$$\boldsymbol{W} = \{\boldsymbol{w}_1, \boldsymbol{w}_2, \ldots, \boldsymbol{w}_m\},$$

with each keyword $\boldsymbol{w}_i = w_{i,1} \| \dots \| w_{i,j}$. At decoding step $j$ for keyword $i$, CTQE records the top-$k$ candidate tokens

$$\mathrm{rel}(q,t_e) = \sum_{t_i \in q} q_i \, \mathrm{CC}(t_i,t_e),$$0

The crucial design decision is to keep only first-position candidates, because later positions are heavily constrained by the already selected first token and therefore contribute less diversity. After deduplication and removal of tokens shorter than two characters, the final candidate-token set is

$$\mathrm{rel}(q,t_e) = \sum_{t_i \in q} q_i \, \mathrm{CC}(t_i,t_e),$$1

In the reported implementation, $$\mathrm{rel}(q,t_e) = \sum_{t_i \in q} q_i \, \mathrm{CC}(t_i,t_e),$$2, which was also the maximum top-$$\mathrm{rel}(q,t_e) = \sum_{t_i \in q} q_i \, \mathrm{CC}(t_i,t_e),$$3 depth exposed by the OpenAI API (Kim et al., 2 Sep 2025).

The integration step depends on the retriever. For lexical retrieval, the original query is repeated five times and concatenated with generated keywords to form an expanded query, then the candidate-token signal is retrieved separately through a subword index and interpolated: $$\mathrm{rel}(q,t_e) = \sum_{t_i \in q} q_i \, \mathrm{CC}(t_i,t_e),$$4 with $$\mathrm{rel}(q,t_e) = \sum_{t_i \in q} q_i \, \mathrm{CC}(t_i,t_e),$$5. For dense retrieval, CTQE forms a fused query embedding

$$\mathrm{rel}(q,t_e) = \sum_{t_i \in q} q_i \, \mathrm{CC}(t_i,t_e),$$6

and scores documents by inner product with $$\mathrm{rel}(q,t_e) = \sum_{t_i \in q} q_i \, \mathrm{CC}(t_i,t_e),$$7. For learned sparse retrieval, CTQE linearly combines learned sparse token weights from the original query, generated keywords, and candidate-token set: $$\mathrm{rel}(q,t_e) = \sum_{t_i \in q} q_i \, \mathrm{CC}(t_i,t_e),$$8

A notable property of this direct CTQE formulation is that it does not reconstruct phrases from alternative decoding paths and does not probability-weight candidate tokens in the final retrieval formula. The method therefore remains fundamentally token-level even when the base keyword generator outputs multi-token keywords. That design keeps the approach cheap—no extra decoding pass is required—but also makes candidate filtering essential.

4. Candidate control, anti-drift mechanisms, and query sensitivity

The central failure mode addressed by most CTQE-like systems is query drift: expansion methods may introduce non-relevant information, especially when feedback documents or generative outputs are noisy. BERT-QE addresses this at a coarser granularity by first reranking documents with BERT, then decomposing the top $$\mathrm{rel}(q,t_e) = \sum_{t_i \in q} q_i \, \mathrm{CC}(t_i,t_e),$$9 feedback documents into overlapping chunks of length $\mathrm{CC}$0, scoring each chunk by $\mathrm{CC}$1, selecting the top $\mathrm{CC}$2 chunks, and finally aggregating chunk-document relevance through a softmax-weighted feedback score before interpolation with the original query-document score (Zheng et al., 2020). Although this is not CTQE in the strict token-level sense, it demonstrates a general anti-drift principle: relevance estimation should happen before expansion evidence is allowed to influence ranking.

LLM-based query expansion systems developed parallel control mechanisms. One automated in-domain framework first harvests pseudo-relevant passages with a BM25-MonoT5 pipeline, selects diverse medoid exemplars by clustering Contriever embeddings of query-passage pairs, then prompts two heterogeneous LLMs to generate independent expansions and asks a refinement LLM to consolidate them into one coherent expansion. The refinement step is explicitly instructed to keep useful entities, relations, and domain knowledge while removing redundancy and noise; the final retrieval query repeats the original query five times before appending the refined expansion (Li et al., 9 Feb 2026). This is sequence-level rather than token-level control, but it encodes the same lesson: proposal and consolidation should be separated.

Query sensitivity provides another control layer. A taxonomy of QE-oriented query classes argues that ambiguous queries, multi-aspect queries, named-entity queries, queries containing negative terms, high-level queries, recall-oriented queries, and domain-specific queries should not all receive the same expansion treatment (Pal et al., 2015). This suggests that a CTQE system should not only score candidate tokens, but also vary candidate budgets, candidate sources, and admission thresholds according to query class. A plausible implication is that aggressive candidate generation is most appropriate for short or recall-oriented queries, whereas ambiguous or negation-bearing queries require stricter gating or even expansion suppression.

5. Neural and LLM-era extensions beyond direct token selection

Several recent systems are not token-level CTQE, but they provide blueprints for extending CTQE beyond classical pseudo-relevance-feedback heuristics. One dense-retrieval framework generates document-style expansions and optimizes the LLM with Direct Preference Optimization using a reward

$\mathrm{CC}$3

where one component reflects retrieval preference relative to a relevant document and the other reflects answer consistency relative to a generated answer (Yao et al., 24 Feb 2025). Because preference pairs are constructed from multiple sampled candidate expansions, the method is sequence-level rather than token-level; however, it makes explicit that expansion quality can be supervised by retrieval-aligned preferences rather than by plausibility alone.

Topic-centric dual expansion pushes another direction. TCDE expands the query into $\mathrm{CC}$4 topic-focused pseudo-documents,

$\mathrm{CC}$5

and expands each document into $\mathrm{CC}$6 one-sentence topic summaries,

$\mathrm{CC}$7

then retrieves with

$\mathrm{CC}$8

This is not CTQE, but it is directly suggestive for CTQE design because it treats expansion as topic-aligned bridging rather than as a flat bag of related words (Yang et al., 19 Dec 2025).

Production systems further generalize the candidate-generation side. Event-Centric Query Expansion mines current news headlines, reformulates them into concise event phrases with a prompt-guided encoder-decoder model trained with contrastive learning, retrieves the best event candidates through a RoBERTa-based dual tower and Faiss, and finally selects a top-1 event expansion through a GBDT ranker (Zhang et al., 2023). In search advertising, document-side semantic keyword expansion encodes seller keywords with a pretrained siamese model, retrieves semantically related variants by nearest-neighbor search, filters them using cluster-adaptive thresholds, and then corrects the downstream ranking distribution through a lightweight decision-tree ensemble (Saha et al., 24 May 2025). Both systems operate mainly on phrases rather than tokens, but both reinforce a CTQE-compatible conclusion: candidate generation, adaptive filtering, and downstream score correction are separable problems.

6. Evaluation, applications, and persistent limitations

Direct CTQE has been evaluated across lexical, dense, and learned sparse retrieval. In lexical retrieval without pseudo-relevance feedback, CTQE improves over keyword-only generation on TREC DL 2019 and 2020, reaching $\mathrm{CC}$9 and $$KLD_{(PR,PC)}(t) = P_R(t)\,\log \frac{P_R(t)}{P_C(t)}.$$0 nDCG@10 respectively, compared with $$KLD_{(PR,PC)}(t) = P_R(t)\,\log \frac{P_R(t)}{P_C(t)}.$$1 and $$KLD_{(PR,PC)}(t) = P_R(t)\,\log \frac{P_R(t)}{P_C(t)}.$$2 for Q2K and $$KLD_{(PR,PC)}(t) = P_R(t)\,\log \frac{P_R(t)}{P_C(t)}.$$3 and $$KLD_{(PR,PC)}(t) = P_R(t)\,\log \frac{P_R(t)}{P_C(t)}.$$4 for BM25. With pseudo-relevance feedback, CTQE reaches $$KLD_{(PR,PC)}(t) = P_R(t)\,\log \frac{P_R(t)}{P_C(t)}.$$5 and $$KLD_{(PR,PC)}(t) = P_R(t)\,\log \frac{P_R(t)}{P_C(t)}.$$6, while on BEIR low-resource datasets it slightly exceeds both Q2D/PRF and LameR on average. The same method also improves SPLADE++ and BGE-base, reaching $$KLD_{(PR,PC)}(t) = P_R(t)\,\log \frac{P_R(t)}{P_C(t)}.$$7 on learned sparse retrieval and $$KLD_{(PR,PC)}(t) = P_R(t)\,\log \frac{P_R(t)}{P_C(t)}.$$8 on dense retrieval for DL19, DL20, and BEIR average respectively. Its efficiency profile is central: CTQE uses the same 16-token output budget as keyword generation but around $$KLD_{(PR,PC)}(t) = P_R(t)\,\log \frac{P_R(t)}{P_C(t)}.$$9 seconds latency, compared with $idf$0 tokens and $idf$1 seconds for Q2D and $idf$2 tokens and $idf$3 seconds for MUGI in the reported lexical high-resource setting (Kim et al., 2 Sep 2025).

The diversity argument is equally strong. On TREC19 with pseudo-relevance feedback, Q2K yields 5 unique keywords, temperature-based sampling with 80 generated tokens yields 9.1 or 12.5 unique keywords depending on temperature, whereas CTQE yields 33.3 unique keywords from a 16-token single pass. At the same time, the method’s ablations show that unfiltered candidate tokens are harmful: keeping all candidate tokens hurts retrieval, deduplication helps, and deduplication plus first-position filtering gives the strongest gains. The method therefore exposes a persistent CTQE limitation: candidate-token access alone is insufficient unless the candidate pool is carefully constrained.

Applications outside direct CTQE reinforce this point. In professional Boolean search, expansion suggestions are naturally attached to a local disjunction rather than to the whole query, and the most effective systems combine context-free distributional LLMs with curated ontologies under n-gram-aware routing, with strict pipelining giving the best F-score across healthcare and recruitment datasets (Russell-Rose et al., 2021). A plausible implication is that CTQE systems intended for professional or high-stakes retrieval should often be interactive, clause-local, and provenance-aware rather than globally automatic.

Several limitations remain stable across the literature. Direct CTQE depends on access to the decoder’s top-$idf$4 candidate tokens, is sensitive to tokenization and subword indexing, and leaves later-position alternatives largely unused. Broader CTQE-like systems still struggle with ambiguity, domain transfer, and attribute preservation; phrase-level systems often need extra guards for freshness, numbers, brands, or structured product compatibility. The field therefore continues to oscillate between two competing demands: richer candidate generation for coverage and stricter candidate control for precision. CTQE’s enduring contribution is to make that tradeoff explicit at the level of the candidate itself.