LLM-Guided Query Refinement

• LLM-guided query refinement algorithm is a method that uses LLM feedback to iteratively modify query embeddings for improved retrieval and classification outcomes.
• It employs gradient descent on query embeddings by aligning LLM-generated relevance scores with the student model’s similarity scores, enabling real-time updates without corpus re-encoding.
• Empirical evaluations reveal significant gains in mean average precision and efficiency across tasks, making it a practical solution for complex and nuanced query scenarios.

A LLM-guided query refinement algorithm is a supervised or interactive process that leverages the reasoning and generative capabilities of an LLM to iteratively modify, optimize, or specialize a user’s initial query with the goal of improving task-specific outcomes such as retrieval, ranking, classification, or execution performance. Unlike classical query refinement, which relies on pattern-matching rules or feature-based transformations, LLM-guided refinement uses the LLM as an oracle or teacher, directly or indirectly providing signal—such as relevance, feedback, or quality scores—that guide the updating or rewriting of the query, either in embedding space or as a text string. This paradigm has demonstrated significant gains in efficiency, adaptability, and retrieval quality across both text and cross-modal search, as well as in structured domains such as SQL and SPARQL, while typically requiring only modest test-time computational overhead and no expensive corpus-wide re-encoding (Gera et al., 12 May 2026).

1. Motivation and Problem Setting

LLM-guided query refinement addresses the limitations of fixed or zero-shot embedding models and traditional query rewriting in diverse retrieval, classification, and information access scenarios. In common settings, a user submits a free-form query $q$ against a document corpus $D=\{d_1,...,d_N\}$, and the goal is to rank documents (or structured outputs) such that relevant (positive) items precede irrelevant (negative) ones. While dense embedding models map queries and documents to a shared space, their static query representations often fail to capture subtle intent, task nuances, or ad-hoc instructions; as a result, ranking or separation is suboptimal on complex or previously-unseen queries.

The LLM-guided refinement paradigm overcomes this by:

• Using the LLM to generate targeted feedback or supervision that is both zero-shot and task-adaptive.
• Performing online (test-time) updates to the query embedding or query string, rather than retraining the retriever or fine-tuning over the corpus.
• Maintaining efficiency relative to full LLM-based retrieval by employing fast embedding models for large-scale scoring and reserving LLM calls for a small, tractable set of cases.

The approach is applicable across domains: literature search, intent detection, key-point matching, advanced instruction following, open-vocabulary image retrieval, and structured data querying (SQL, SPARQL) (Gera et al., 12 May 2026).

2. Canonical Algorithmic Structure

The generic LLM-guided query refinement workflow consists of:

1. Initial Retrieval and Embedding: Compute the embedding $z^{(0)}=m_e(q)$ for query $q$ using a student embedding model $m_e$. Retrieve the top-$K$ documents $D_K\subset D$ using (typically) cosine similarity $s_e(z^{(0)},d)=\cos(m_e(q),m_e(d))$.
2. LLM Feedback Acquisition: For each $d_i\in D_K$, prompt a teacher LLM $m_t$ with the query-document pair (and contextual instructions), requesting a relevance score or “yes/no” judgment. Log-probabilities over “yes”/“no” tokens are used to compute soft relevance scores $D=\{d_1,...,d_N\}$0 for each candidate document. The set $D=\{d_1,...,d_N\}$1 is normalized (usually via softmax) to form a teacher distribution $D=\{d_1,...,d_N\}$2.
3. Query Update via Student–Teacher Distribution Alignment: The student’s similarity scores $D=\{d_1,...,d_N\}$3 are mapped to a distribution $D=\{d_1,...,d_N\}$4 over the $D=\{d_1,...,d_N\}$5 documents (again via softmax). The loss at step $D=\{d_1,...,d_N\}$6 is

$D=\{d_1,...,d_N\}$7

The query embedding is updated via gradient descent:

$D=\{d_1,...,d_N\}$8

where $D=\{d_1,...,d_N\}$9 is the step size (e.g., $z^{(0)}=m_e(q)$0, Adam optimizer).

1. Final Ranking: After $z^{(0)}=m_e(q)$1 refinement steps, the updated $z^{(0)}=m_e(q)$2 is used to compute final similarities and ranking over all documents. Positives and negatives are better separated—often yielding large gains in mean average precision (MAP) and binary corpus separation.

A concise pseudocode encapsulating this loop: $z^{(0)}=m_e(q)$8 (Gera et al., 12 May 2026)

3. Theoretical and Practical Properties

LLM-guided query refinement in embedding space instantiates a gradient alignment between LLM-provided feedback and the retriever's local embedding landscape. Unlike rerank-only approaches (which use LLM scores but do not update the query), refinement shapes the embedding such that the “soft” document distribution preferred by the LLM is matched in the student’s similarity space. Analyses show this:

• Yields refined embeddings that move toward regions maximizing separation of positives and negatives, not simply toward an average of positive samples.
• Can be performed in real time at test-time (typically <100 ms per query for 100 gradient steps).
• Does not require retriever model fine-tuning, corpus re-encoding, or high-latency LLM inference at the rerank level.
• Scales to large corpora because only the query vector is updated and inference over document embeddings remains efficient.

Ablation studies confirm:

• Refinement gains plateau past $z^{(0)}=m_e(q)$3–$z^{(0)}=m_e(q)$4 feedback docs.
• The approach is robust across a range of teacher LLMs.
• Benefits are largest for smaller embedding models and more complex or nuanced queries (Gera et al., 12 May 2026).

4. Empirical Results and Benchmarks

Comprehensive evaluations demonstrate the effectiveness of LLM-guided query refinement across diverse settings:

• Datasets: RealScholarQuery (literature search), CLINC150 (intent detection), ArgKP-21 (key-point matching), FollowIR (instruction following), Banking77, NFCorpus.
• Embedding Models: Qwen3-Embedding-0.6B, Qwen3-Embedding-8B, Llama-Embed-Nemotron-8B, Linq-Embed-Mistral, E5-Mistral-7B.
• Teacher LLMs: Mistral-Small-3.2-24B, DeepSeek-V3.2, Qwen3.5, GPT-4.1.

The following table summarizes relative MAP improvements by task:

Task	Relative MAP Gain (%)
Literature search (RealScholarQuery)	+16.9
Intent detection (CLINC150)	+9.4
Key-point matching (ArgKP-21)	+15
Nuanced IR instructions (FollowIR)	+7.4
Across models & tasks (avg.)	+12

For example, in CLINC150, refinement improved average precision from ≈0.52 to ≈0.70, and in RealScholarQuery, recall@$z^{(0)}=m_e(q)$5 is substantially improved beyond the top-$z^{(0)}=m_e(q)$6 feedback region (Gera et al., 12 May 2026).

5. Analysis, Insights, and Comparison to Other Paradigms

Principal component analysis of refined query embedding trajectories reveals movement toward embedding subregions effecting clearer binary separation, not mere interpolation between positive samples. The method exhibits modest positive correlation between the quality of LLM feedback and eventual AP gains ($z^{(0)}=m_e(q)$7 with certain datasets). The greatest improvements are observed where the base embedding model underperforms, especially in tasks with subtle or highly context-dependent relevance criteria.

Qualitative and quantitative ablation further reveal:

• Rerank-only, rerank-and-generate (e.g., HyDE), and instruction prompt ablations underperform relative to full refinement.
• Instruction template sensitivity is significant for small embedding models but less so for larger ones (Gera et al., 12 May 2026).

6. Applications, Limitations, and Extensions

LLM-guided query refinement extends the practical deployment range for embedding-based retrieval in both resource-constrained and high-latency settings:

• Applications: Literature search, zero-shot intent detection, key-point extraction, instruction following, cross-lingual and domain-adaptive retrieval, and multi-modal object retrieval (with appropriate adaptation).
• Limitations: Effectiveness is bounded by the quality and relevance of LLM feedback; gains decrease for less ambiguous queries or larger base embedding models; requires LLM access at test time; negligible but nonzero latency overhead per query.
• Extensions: Integrating refinement into broader user-interactive or multi-stage learning loops; leveraging richer LLM feedback (graded, contrastive, chain-of-thought); exploring robustness to adversarial or ambiguous task instructions; adapting to new domains via prompt tuning or hybrid model selection.

For practitioners, the approach presents a low-cost, inference-time adaptation path for embedding models, requiring no re-training or large-scale pre-annotation, and provides a mechanism for incorporating external LLM knowledge into the retrieval pipeline (Gera et al., 12 May 2026).