LLM Rewriting Strategy

• LLM rewriting strategies are algorithmic frameworks that transform, adapt, and optimize inputs to bridge the gap between user intent and system performance.
• They employ methods like query reformulation, content filtering, and reinforcement learning to enhance retrieval-augmented generation and query optimization.
• Empirical evaluations demonstrate measurable gains, such as improvements in EM and F1 scores, and efficiency in tasks like SQL query processing and dialogue completion.

LLM rewriting strategies refer to algorithmic frameworks and methodologies in which LLMs are employed to transform, adapt, or generate new inputs or intermediary artifacts—such as user queries, code, utterances, or context passages—with the explicit goal of improving downstream system performance. While their design and application spaces are diverse, these strategies are unified by their focus on leveraging generative capabilities and feedback-driven alignment to bridge gaps between user intent, task demands, knowledge base limitations, and system constraints. The LLM rewriting paradigm has become foundational in retrieval-augmented generation, query optimization, text style adaptation, and dialogue completion, driving advancements across search, databases, conversational systems, code synthesis, and more.

1. Core Principles and Motivation of LLM Rewriting

At the conceptual level, LLM rewriting strategies are motivated by the observation that direct user inputs or retrieved knowledge often do not optimally interact with downstream systems (retrievers, readers, optimizers, or rankers). Misalignments arise due to non-canonical phrasing (“vocabulary gap”), user ambiguity, suboptimal query structures, or the noisy/irrelevant nature of external information.

LLM rewriting frameworks thus pursue three linked objectives:

• Input adaptation: Reformulate queries or fragments into forms optimized for retrieval, ranking, or generative models, closing the intent–representation gap (Ma et al., 2023, Anand et al., 2023, Liu et al., 14 Mar 2024).
• Content distillation and filtering: Compress, purify, or summarize knowledge before augmentation, reducing noise and improving downstream utility (Qiao et al., 12 Jun 2024).
• Behavior alignment and robustness: Systematically train or adapt rewriting modules to maximize human- or machine-derived objectives while maintaining constraint satisfaction (semantic faithfulness, structural validity) (Ma et al., 2023, Li et al., 2023, Li et al., 9 Mar 2025).

2. Architectural Paradigms and Workflow Designs

LLM rewriting has been instantiated in several workflow architectures, according to application needs, system modularity, and resource constraints:

Paradigm	Key Roles of Rewriting	Example Tasks
Rewrite–Retrieve–Read	Input query adaptation	Retrieval-augmented QA, ranking
Multi-agent/fsm-coordinated	Modular subtask orchestration	SQL rewriting with verification (Song et al., 9 Jun 2025)
Rule-enhanced prompting	Rule selection, adaptation	DB query efficiency (Li et al., 19 Apr 2024)
Instruction-tuned/fine-tuned	Task-specific transformation	Text style transfer, factual rewriting (Shu et al., 2023, Li et al., 9 Mar 2025)
Cascaded/on-device-agent	Model selection, privacy preservation	On-device rewriting (Zhu et al., 2023)

A recurring design is the modular detachment of the rewriter from fixed retriever/reader (as in “Rewrite–Retrieve–Read”), allowing black-box or frozen downstream components to benefit from externally adaptable rewriting modules (Ma et al., 2023). Conversely, multi-agent systems orchestrate specialized LLM roles (reasoner, assistant, verifier) under a finite state machine (FSM), enabling stepwise feedback-informed refinement and explicit integration with database tools or cost models (Song et al., 9 Jun 2025).

Instruction tuning and reinforcement learning further permit the training of small, efficient rewriting models that capture nuanced task behaviors, driven by explicit or learned objectives (Shu et al., 2023, Zhu et al., 2023, Li et al., 9 Mar 2025).

3. Learning and Optimization Techniques

Central to effective LLM rewriting are methods for aligning generative behaviors to measurable downstream outcomes.

Supervised fine-tuning (SFT):

• Applied to pseudo-labeled or extracted rewrite datasets (e.g., mining Wiki revision histories, chain-of-thought synthetic data, or expert-generated NLR2 rules) to create instruction-following rewriting modules (Shu et al., 2023, Liu et al., 14 Mar 2024).

Reinforcement learning (RL) with feedback:

• Rewriter is trained to optimize metrics reflecting downstream system performance (EM, F1, retrieval hits, cost reduction), often using Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO).
• Reward functions may be complex, combining:
  • End-task accuracy (e.g., EM, F1)
  • Additional indicators (e.g., “Hit” if answer is present in retrieval (Ma et al., 2023, Qiao et al., 12 Jun 2024))
  • Regularizers (KL divergence from initial/warm policy)
  • Strategy-dependent shaping (as in Strategic Credit Shaping and Contrastive Reward Shaping in SAGE (Wang et al., 24 Jun 2025))
  • Machine feedback (simulated user interactions (Nguyen et al., 29 Jan 2025), classifier-based appropriateness (Ziegenbein et al., 5 Jun 2024))
• Multi-objective, decoupled reward models (agreement, coherence, conciseness) allow fine-grained policy adaptation per use-case (Li et al., 9 Mar 2025).

Demonstration selection & curriculum:

• Demonstration Manager modules use contrastive, curriculum-trained selectors to present the LLM with relevant and incrementally challenging in-context examples, further enhancing reliability and generalization (Li et al., 19 Apr 2024).

Quality estimation and gating:

• For input text rewriting, reference-free quality estimators (e.g., xCOMET) allow for translatability-aware inference-time selection, improving machine translation (Ki et al., 23 Feb 2025).

4. Key Mechanisms and Formulations

Several explicit mechanisms, along with their mathematical formulations, characterize state-of-the-art LLM rewriting:

Query rewrite warm-up objective:

$$\mathcal{L}_{\text{warm}} = -\sum_{t} \log p_\theta(\tilde{x}_t | \tilde{x}_{<t}, x)$$

(where $\tilde{x}$ is the rewritten query, $x$ is input)

General RL reward with regularizer:

$$R(s_t, a_t) = R_{lm}(\hat{y}, y) - \beta \cdot \mathrm{KL}(\pi_\theta \,\|\, \pi_0)$$

DPO (Direct Preference Optimization):

$$\max_{\theta} \mathbb{E}_{(x, r_w, r_l)} \left[ \log \sigma \left( \beta ( \log \pi(r_w \mid x) - \log \pi(r_l \mid x) ) \right) \right]$$

Supportiveness score for knowledge rewriting:

$$ss(q, c) = \frac{P_{\text{raw}}}{P_{\text{retrieval}}}$$

($P_{\text{raw}}$: perplexity of the answer without context, $P_{\text{retrieval}}$: with given knowledge $c$)

Strategic Credit Shaping (SCS):

$$r_{\text{SCS}, i} = \frac{r_{\text{orig}, i}}{\operatorname{rank}(s_i)}$$

Pointwise cross-entropy loss for ranker:

$$L_{\text{point}} = - \frac{1}{N} \sum_{i=1}^N [y_i \log \hat{y}_i + (1 - y_i) \log (1 - \hat{y}_i)]$$

Hybrid loss for demographic-robust ranking:

$$L_{\text{total}} = L_{\text{acc}} + \alpha \cdot L_{\text{rbt}}$$

(where $L_{\text{acc}}$ is an accuracy loss, $L_{\text{rbt}}$ JS divergence among agent outputs)

5. Empirical Gains and Evaluated Impact

Empirical results consistently demonstrate the efficacy of LLM rewriting strategies across benchmarks and settings:

• Retrieval-augmented QA: Introducing a query rewriting step (with either frozen LLM or a small trainable rewriter) leads to measurable improvements on HotpotQA, AmbigNQ, MMLU, and PopQA datasets (e.g., EM improvement from 32.36 to 34.38, F1 from 43.05 to 45.97 on HotpotQA (Ma et al., 2023)).
• Ranking robustness and diversity: Multi-agent demographic rewrites and MMoE architectures reduce variance in ranking outcomes (as measured by VNDCG, VNAP), enhancing robustness while maintaining or improving accuracy (Li et al., 2023).
• SQL query efficiency: Multi-stage, evidence-grounded LLM rewriting and adaptive middleware architectures reduce execution time by 24–35% and increase the coverage of optimizable queries (Song et al., 9 Jun 2025, Sun et al., 2 Dec 2024, Liu et al., 14 Mar 2024), often yielding order-of-magnitude speedups.
• Text rewriting quality: Instruction-tuned and RL-finetuned rewriting models achieve superior scores on cross-sentence benchmarks (e.g., OpenRewriteEval NLI source→predict 0.96, SARI > 40 (Shu et al., 2023)).
• Domain transfer and scalability: Supportiveness-based knowledge rewriting with DPO outperforms even leading general-purpose models (e.g., 7B parameter rewriting module surpassing GPT-4 (Qiao et al., 12 Jun 2024)).

6. Best Practices, Limitations, and Open Directions

Best practices emerging from the literature include modularizing the rewriter, leveraging black-box retriever/reader interfaces, curating high-quality (pseudo-)training data, and integrating feedback from downstream models. Effective reward shaping, diversity-promoting strategy pools, and rigorous validation pipelines (syntactic, semantic, and cost-based) are critical to ensuring both faithfulness and efficiency.

Key limitations concern hallucination risk (both factual and structural), reliance on external cost or quality metrics (whose accuracy may be variable), increased rewrite latency (especially when iterating with LLM APIs), and the open challenge of generalizing to previously unseen patterns without performance regression. Strategies such as FSM-controlled agent architectures, memory condensation buffers, curriculum demonstration management, and real-time feedback integration have been proposed to mitigate these, but trade-offs remain intrinsic.

Open research avenues include adaptive or meta-learning of rewriting strategies, improved quality estimation for selection/gating, automatic discovery and transfer of natural language rewrite rules (NLR2s), and broader application of multi-objective decoupled-reward RL frameworks.

7. Significance in the Broader AI Ecosystem

LLM rewriting strategies are proving essential in adapting foundation models for real-world, high-stakes tasks that stretch beyond narrow prompt engineering or output post-editing. As they modularize transformation and alignment, these strategies enable frozen or black-box LLMs to be rapidly adapted to dynamic information environments, complex enterprise workloads, and user-specific intent representations while preserving system reliability and compliance. By integrating reinforcement learning, expert-guided strategy pools, hybrid verification, and cascading architectures, LLM rewriting frameworks offer compelling solutions to bridging the cognitive and operational gaps still present in large-scale neural AI systems.