Language Feedback Policy

• Language Feedback Policy is a family of mechanisms that use natural language critiques to condition model behavior, guide exploration, and enable precise policy governance.
• Methodologies include feedback-conditioned policies, teacher-student distillation, and token-level pseudo-rewards to refine learning efficiency and reduce reliance on scalar rewards.
• Applications span test-time steering, multilingual personalization, and safety constraints, demonstrating measurable improvements in model performance and adaptability.

Taken together, recent work suggests that a Language Feedback Policy (LFP) is best understood as a family of mechanisms by which natural-language feedback shapes model behavior, either by conditioning generation, guiding exploration, localizing credit assignment, steering frozen systems at test time, or constraining personalization and multilingual behavior through explicit governance layers. In this literature, the term is used directly in some settings and only implied in others, but the recurring problem is stable: scalar rewards, static prompts, and aggregate alignment targets often discard the semantic structure of feedback that language can preserve and operationalize (Luo et al., 26 Sep 2025, Jeong et al., 10 Jun 2026, Kirk et al., 2023).

1. Formalizations of language feedback as a learning signal

A first line of work treats language feedback as a formally structured substitute for, or complement to, latent reward. The most general statement is the Learning from Language Feedback framework, which models sequential interaction with a finite action set $A$, an unobserved reward function $r^*:A\to[0,1]$, and an observed feedback channel $f^*:A\to\Delta(O)$, where $O\subset\mathcal{T}^+$ is a space of token sequences. The learner never observes reward directly; instead, it uses a verifier loss $\ell:A\times O\times\mathcal{H}\to[0,1]$ over a hypothesis class $\mathcal{H}\subset\mathcal{T}^+$. Under a known reward mapping $\eta\mapsto r_\eta$ and an unbiased-feedback assumption, the paper defines transfer eluder dimension $TE(\mathcal{H},\ell,\epsilon)$ as the complexity measure governing how efficiently feedback observations transfer into reward knowledge, and gives a no-regret algorithm, HELiX, with regret scaling as $$\widetilde{O}\!\left(T^{3/4}(\log N(\mathcal{H},\epsilon_T^\mathcal{H},d_\mathcal{H}))^{1/4}\sqrt{TE(\mathcal{H},\ell,\epsilon_T^\mathcal{H})}\right)$$ (Xu et al., 12 Jun 2025). The same framework also exhibits cases where explanation-, suggestion-, or demonstration-like feedback reduces learning complexity from $O(|S|^L)$ to $r^*:A\to[0,1]$0, $r^*:A\to[0,1]$1, or $r^*:A\to[0,1]$2, making precise the claim that richer language can be exponentially more informative than sparse reward alone.

A second formalization turns verbal feedback into an explicit conditioning variable in the policy. The feedback-conditional policy defines $r^*:A\to[0,1]$3, where $r^*:A\to[0,1]$4 is the prompt, $r^*:A\to[0,1]$5 the response, and $r^*:A\to[0,1]$6 verbal feedback. Offline data are generated by $r^*:A\to[0,1]$7 and $r^*:A\to[0,1]$8, inducing the posterior $r^*:A\to[0,1]$9. The training rule is maximum likelihood on $f^*:A\to\Delta(O)$0 triples rather than scalar-reward optimization, and the online stage bootstraps by generating under positive feedback conditions $f^*:A\to\Delta(O)$1 and collecting fresh feedback on those generations (Luo et al., 26 Sep 2025). This reframes feedback-driven learning as conditional generation rather than reward maximization.

A third formalization uses a teacher-student decomposition. Variational Policy Distillation introduces a feedback-conditioned teacher $f^*:A\to\Delta(O)$2 and an unconditioned student $f^*:A\to\Delta(O)$3, and casts learning from language feedback as variational EM. The E-step refines the teacher under an adaptive trust-region objective $f^*:A\to\Delta(O)$4; the M-step distills the teacher’s token distribution $f^*:A\to\Delta(O)$5 onto the student’s on-policy rollouts (Li et al., 14 May 2026). Here, textual critique is not merely appended context; it defines an approximate posterior over better continuations.

2. Training-time LFPs: exploration, revision, and localized credit assignment

The most explicit training-time LFPs separate the semantic role of feedback from the numerical role of reward. LANPO defines a feedback-conditioned actor $f^*:A\to\Delta(O)$6, where $f^*:A\to\Delta(O)$7 is retrieved from a dynamic experience pool $f^*:A\to\Delta(O)$8. Its central principle is that language feedback “guides exploration,” while numerical reward “determines what to learn.” The system combines an SFT stage for “atomic capabilities,” a mixed rollout schedule with feedback probability $f^*:A\to\Delta(O)$9, a GRPO/PPO-style loss with KL regularization, and two safeguards: Reward-Agnostic Reflection for same-sample self-correction without label leakage, and Relevant Abstraction for cross-sample retrieval through relevance filtering, summarization into “Flow of thought” and “Takeaways,” and explicit “Experience Evaluation” prompts. On mathematical reasoning, LANPO improves over GRPO: for Qwen2.5-7B, GRPO zero-shot averages 42.79, while LANPO w/ Inter reaches 44.68 zero-shot and 47.38 with retrieval; for Qwen3-14B, GRPO averages 62.27 and LANPO reaches 67.83 with self-correction after intra-trained LANPO (Li et al., 18 Oct 2025).

In personalized question answering, VAC instantiates an alternating two-model architecture. The policy $O\subset\mathcal{T}^+$0 answers a query using retrieved profile documents, while the feedback model $O\subset\mathcal{T}^+$1 generates natural-language feedback conditioned on the query, retrieved profile, the current answer, and the user-authored narrative. Training alternates between selecting the feedback candidate that maximizes the downstream metric $O\subset\mathcal{T}^+$2, fine-tuning the feedback model on those selected critiques, revising answers with the policy conditioned on feedback, and finally fine-tuning the policy to produce the revised answer directly without feedback at inference. On LaMP-QA, VAC reaches 0.4691 average versus 0.4525 for PlanPers and 0.4423 for Offline RL RAG-Personalization, with a runtime of 1.63 sec/query and human evaluation showing VAC preferred in 44% of cases, ties in 33%, and PlanPers preferred in 23% (Salemi et al., 14 Aug 2025). This is an LFP in the narrow sense of critique-induced policy improvement followed by critique distillation.

A more localized variant is Text2Grad, which converts free-form critique into token-level pseudo-rewards. A strong annotator produces both a critique $O\subset\mathcal{T}^+$3 and a structured span map $O\subset\mathcal{T}^+$4; tokens in positive spans receive $O\subset\mathcal{T}^+$5, tokens in negative spans receive $O\subset\mathcal{T}^+$6, and others receive $O\subset\mathcal{T}^+$7. The resulting “natural language gradient” is

$O\subset\mathcal{T}^+$8

which is then integrated into token-level PPO through $O\subset\mathcal{T}^+$9 (Wang et al., 28 May 2025). Across summarization, code generation, and question answering, Text2Grad outperforms scalar-reward PPO and prompt-only reflection baselines, with the code results particularly dependent on critique quality: the no-chain-of-thought ablation underperforms full Text2Grad by 6.9 average points.

A related but simpler pattern is Language Feedback Models for grounded instruction following. Instead of generating actions, the LFM classifies whether an action in a verbalized trajectory is “productive” toward the instruction, and policy improvement proceeds by imitating only the retained actions. On ALFWorld, ScienceWorld, and Touchdown, LFM-based filtering improves task completion over behavioral cloning and supports one round of adaptation to unseen environments, with gains of 3.5–12.0% (Zhong et al., 2024).

3. Communication, test-time steering, and frozen-policy adaptation

A separate branch of the literature treats language feedback as an online interaction channel rather than a purely offline training signal. Learning through Communication (LTC) defines a universal replay buffer over trajectories $\ell:A\times O\times\mathcal{H}\to[0,1]$0, where text tokens, source masks, value estimates, log-probabilities, and rewards are all serialized into a common format. The total objective combines language modeling and PPO,

$\ell:A\times O\times\mathcal{H}\to[0,1]$1

and the framework supports three communication patterns: single-agent monologue, multi-agent dialogue, and teacher-student dialogue (Wang et al., 2023). On ALFWorld, HotpotQA, Chameleon, and GSM8k, LTC improves over supervised instruction tuning by 3.6% to 12%, showing that language from collaborators, teachers, and environments can be stored as learnable interaction history rather than compressed into a single reward.

In robotics, RE-MOVE shows a test-time LFP layered on top of a frozen navigation policy. The system uses epistemic uncertainty and Bayesian change-point detection to decide when to request human language feedback, then uses GPT-3.5/ChatGPT or Llama-2 as zero-shot interpreters that convert that feedback into short corrective action sequences, after which control returns to the base policy. The uncertainty decomposition is written as

$\ell:A\times O\times\mathcal{H}\to[0,1]$2

and the paper argues that local, goal-conditioned observations are crucial for making this trigger useful (Chakraborty et al., 2023). In demanding real-world indoor Husky scenarios, RE-MOVE improves successful goal attainment by up to 80% and reduces normalized trajectory length by 13.50% relative to alternatives in perceptually deceptive settings.

The clearest direct use of the term appears in test-time language steering for frozen VLAs. A frozen VLA is modeled as $\ell:A\times O\times\mathcal{H}\to[0,1]$3, and language steering is itself treated as an MDP over language actions. The pipeline first performs narrated fine-tuning to obtain a proposal distribution over plausible per-step instructions, then runs interactive search over whole language sequences, distills successful sequences into a closed-loop LFP $\ell:A\times O\times\mathcal{H}\to[0,1]$4, and trains an improvement head $\ell:A\times O\times\mathcal{H}\to[0,1]$5 to predict whether steering helps relative to the original task instruction (Jeong et al., 10 Jun 2026). The safety wrapper is conformalized: with calibration threshold $\ell:A\times O\times\mathcal{H}\to[0,1]$6, steering is allowed only when $\ell:A\times O\times\mathcal{H}\to[0,1]$7, yielding the class-conditional guarantee

$\ell:A\times O\times\mathcal{H}\to[0,1]$8

where $\ell:A\times O\times\mathcal{H}\to[0,1]$9 denotes truly harmful steering. On seen environments, the conformalized LFP improves base VLA performance by 24.7% in simulation and 65.0% in hardware, and it exhibits recovery behaviors not observed under open-loop prompt rewriting.

4. Personalisation, pedagogy, and multilingual feedback as policy objects

The governance literature treats LFP not only as a learning mechanism but also as a normative control problem. A central claim is that aggregate alignment is not a sufficient proxy for plural human preferences, yet unconstrained personalization is unacceptable. The proposed solution is a three-tiered policy framework: Tier One: Immutable restrictions for illegal or clearly dangerous outputs; Tier Two: Optional restrictions and requirements set by model or application providers; and Tier Three: Tailored requirements for end-user preferences within those bounds (Kirk et al., 2023). This structure explicitly orders policy layers so that lower-tier personalization cannot override higher-tier safety or legal constraints. The same paper distinguishes value personalization, preference personalization, and knowledge personalization, and argues that bounded personalization is preferable to both one-size-fits-all alignment and unrestricted user sovereignty.

A complementary framework grounds feedback design in pedagogy. FELT, short for Feedback, Errors, Learner, Task, treats feedback as one component of a larger ecosystem and then decomposes feedback content into four non-overlapping areas—learner status, goal, procedural, and peripheral—together with ten modulation dimensions: granularity, applicability of instructions, answer coverage, target coverage, criteria, information novelty, purpose, style, valence, and mode (Borges et al., 2023). The paper’s main claim is that current natural-language-feedback methods are often hand-designed and arbitrary because they do not specify what feedback says, how much of the answer or target it covers, or whether it is intended to diagnose, demonstrate, instruct, or clarify. For LFP design, this implies that feedback should be policy-controlled along multiple axes rather than treated as undifferentiated critique text.

Multilingual abstention extends this logic to cross-lingual reliability. The proposed inference-time procedure is: answer the query, generate three critiques in selected feedback languages using the prompt “Please review the proposed answer and provide a paragraph of feedback on its correctness,” then ask a final meta-judge “Based on the feedback, is the proposed answer True or False?” and abstain if the answer is judged false (Feng et al., 2024). The strongest policy is Multi-related, which chooses feedback languages related to the query language via Lang2vec; ablations show that three feedback items are the effective operating point, that random multilinguality is weaker than structured relatedness, and that culture-informed relatedness outperforms purely typological relatedness. Across three models and multiple datasets, multilingual feedback improves low-resource abstention by up to 9.2%, while English-centric methods can produce up to 20.5% gaps between high- and low-resource languages. The fairness analysis is equally important: Multi-related achieves demographic utility 0.6149, linguistic utility 0.6027, and Gini 0.0278, compared with Mono-English at 0.6038, 0.5651, and 0.0564.

5. Language governance inside model development: the Chinese LLM policy gap

Although it does not use the term “Language Feedback Policy,” the study of Chinese multilingual LLMs provides a policy-analytic baseline for what the absence of LFP looks like in practice. The paper examines six open-source Chinese multilingual models—Qwen1.5-7B, Yi-6B, DeepSeek-LLM-7B, InternLM2-7B, XVERSE-7B, and Baichuan2-7B-Base—against Llama3-8B and Mistral-7B-v0.3 on 18 languages spanning Mandarin, Northeast Asian, Southeast Asian, “Chinese ethnic minorities,” and US/European languages (Wen-Yi et al., 2024). Two benchmarks are used: 997 FLORES+ dev sentences with unnormalized negative log-likelihood

$\mathcal{H}\subset\mathcal{T}^+$0

and 900 Belebele multiple-choice reading-comprehension questions per language.

The central empirical result is that “Chinese LLMs performance on diverse languages is indistinguishable from international LLMs.” Across both intrinsic language modeling and zero-shot reading comprehension, performance is highest for Mandarin and Western languages, followed by Japanese and Korean, then Southeast Asian languages, with Chinese minority languages worst. The same resource gradient appears in both Chinese and international models. Correlations with the logarithm of number of speakers are high and nearly identical across groups: for Chinese models, $\mathcal{H}\subset\mathcal{T}^+$1 with NLL and $\mathcal{H}\subset\mathcal{T}^+$2 with MRC accuracy; for international models, $\mathcal{H}\subset\mathcal{T}^+$3 and $\mathcal{H}\subset\mathcal{T}^+$4. Among neighboring Asian national languages, performance also correlates strongly with log national GDP: $\mathcal{H}\subset\mathcal{T}^+$5 and $\mathcal{H}\subset\mathcal{T}^+$6 for Chinese models, $\mathcal{H}\subset\mathcal{T}^+$7 and $\mathcal{H}\subset\mathcal{T}^+$8 for international models.

The policy puzzle arises because this resource-driven multilinguality coexists with extensive state regulation of both human language use and generative AI. The paper reviews China’s historical transition from pluralist to more assimilationist language governance, the 1982 constitutional recognition of Mandarin as the common spoken “super language,” the 2000 Standard Spoken and Written Chinese Language law, and the PRC’s 2023 “Interim AI Measures,” which require lawful data sourcing and training data quality, truthfulness, accuracy, objectivity, and diversity. Yet neither the regulations nor the technical reports examined specify minority-language obligations, language-coverage targets, tokenizer audits, or multilingual evaluation mandates. The technical reports are similarly sparse: Qwen, XVERSE, and InternLM2 mention multilinguality but do not explicate language coverage except English and Mandarin; InternLM2 reports that 86.46% of data are Chinese and English web pages; XVERSE says “more than 40 languages” without proportions; Baichuan2 reports no language coverage at all. The paper therefore concludes that there is “no sign of any consistent policy, either for or against, language diversity in China’s LLM development.” In LFP terms, this is a policy vacuum: language outcomes are governed primarily by generic data availability, common pretraining pipelines, and likely business relevance rather than visible language-specific governance.

6. Comparative feedback, interpretability, and unresolved limits

Not all feedback is naturally scalar, and not all language feedback should be optimized through scalar objectives. Pairwise Proximal Policy Optimization starts from the observation that preference-trained reward models identify only within-prompt differences, not absolute calibrated scores, and therefore proposes a trajectory-wise optimizer that operates directly on comparative rewards (Wu et al., 2023). Its central theorem is that P3O, like DPO, is invariant to reward-equivalent transformations $\mathcal{H}\subset\mathcal{T}^+$9, whereas PPO is not. The broader implication for LFP is straightforward: when feedback is fundamentally relative—“response $\eta\mapsto r_\eta$0 is better than $\eta\mapsto r_\eta$1”—the update rule should preserve the invariances of relative feedback rather than imposing scalar assumptions foreign to the data.

A related interpretability question concerns what a model has actually internalized from feedback. The paper on Implicit Reward Models—notably, it does not use the term LFP—studies small PPO-tuned LLMs on sentiment generation and probes whether reward-relevant structure can be recovered from activations using sparse autoencoders and linear regression (Marks et al., 2023). Behavioral performance improves substantially after fine-tuning, but faithful internalization is much less clear: Kendall Tau correlation between the reconstructed internal reward signal and the true VADER lexicon is statistically significant only for Pythia-160m, at 0.093 with $\eta\mapsto r_\eta$2, while Pythia-70m and GPT-Neo-125m are not significant. This establishes an important caution for LFP research: improved outputs do not imply that the underlying feedback criterion has been stably or transparently internalized.

The same caution appears in recent feedback-distillation work. In fuzzy domains such as creative writing and alignment research, online natural-language-feedback loops can be highly sample efficient if the proxy reward model is repeatedly refreshed with fresh expert critiques. Fine-tuned proxy models recover up to 80% of performance with up to 20x fewer expert samples, and in some settings 100% with 3x or 10x fewer samples, while scalar-only reward models over-optimize within roughly 200 RL steps in every iteration (Ye et al., 5 May 2026). Yet there are clear limits. VPD improves over passive self-distillation on scientific reasoning and code generation, but in base-model cold start it still underperforms pure RL—on Qwen3-4B-Base over SciKnowEval, GRPO reaches 74.49 while VPD reaches 63.95—and on strict mathematical reasoning GRPO reaches 83.8 on Math500 while VPD only delays collapse rather than surpassing RL (Li et al., 14 May 2026). The literature therefore suggests a nontrivial boundary condition: language feedback is especially strong when critiques are informative, abstractable, and semantically rich, but weaker when correctness is rigid, exploration is the primary bottleneck, or feedback interpretation itself is underdeveloped.

These results also dispel a common misconception that LFP is a single architectural object. The technical literature includes feedback-conditioned policies, teacher-student variational distillation, retrieval-augmented reflection, span-level pseudo-reward construction, communication buffers, and conformal test-time steering; the governance literature, by contrast, treats LFP as a layered policy over what kinds of feedback may alter outputs, how multilingual or personalized variation should be handled, and which constraints remain non-overridable. A plausible synthesis is that LFP names both a control interface and a governance problem. The first asks how language feedback should enter optimization and inference; the second asks whose feedback counts, on what dimensions, under what transparency requirements, and within what legal, safety, and linguistic bounds.