---
title: Online Natural Language Feedback
url: https://www.emergentmind.com/topics/online-natural-language-feedback
type: topic
---

# Online Natural Language Feedback

Searching arXiv for recent and foundational papers on online natural language feedback to ground the article with current literature.
Online natural language feedback denotes supervision provided after or during a system’s output in the form of natural language critiques, instructions, explanations, refinements, or language-specified preference criteria, and then used to revise outputs, update models, or alter control policies. In the broader formalization for natural language generation, feedback is a family of functions $h \in H$ mapping inputs and one or more outputs to a signal $f \in F$, with natural-language critiques, scalar ratings, pairwise preferences, and multi-aspect scores all treated as admissible feedback objects [2305.00955]. Recent work extends this idea beyond text generation to direct alignment from preferences, vision-language interaction, semantic parsing, personalized question answering, embodied manipulation, robot safety, and system-level feedback analytics, with “online” typically meaning that feedback is gathered or applied on-policy, in-session, or during deployment rather than only from a fixed offline corpus [2402.04792].

## 1. Definitions and conceptual boundaries

The literature does not use a single narrow definition of online natural language feedback. In the survey treatment of feedback for natural language generation, online integration includes inference-time refinement, on-policy data logging, and bandit-style or reinforcement-style updates performed while the model is being used, rather than only training from pre-collected judgments [2305.00955]. Within that framing, natural language feedback may be a critique, an instruction, an explanation, an error annotation, or a textual preference statement.

A central distinction is between *instance-level* and *system-level* use. Instance-level feedback targets a specific output and asks how that output should be changed; system-level feedback aggregates many such comments into global criteria, prompts, and metrics that steer the behavior of the whole system [2306.13588]. Another distinction is between *feedback as text* and *feedback as language-defined supervision*. OAIF, for example, uses natural-language prompts to define what an annotator should prefer, but the training signal itself is a binary pairwise choice rather than a textual critique [2402.04792]. By contrast, DRESS, VAC, Text2Grad, and SPLASH operate directly on critique-like or refinement-like language [2311.10081].

The term “online” is therefore overloaded. In OAIF it means on-policy, real-time preference collection from the current policy’s outputs [2402.04792]. In DRESS it includes multi-turn refinement at inference time without parameter updates [2311.10081]. In PFEA it refers to a planner–converter–executor loop in which a natural-language evaluator decides whether to stop or re-plan during the same manipulation episode [2510.24109]. In the robot safety setting it means updating the inferred failure set and warm-starting a Hamilton–Jacobi safety controller after new spoken or written constraints arrive during deployment [2409.14580].

## 2. Feedback representations and how they are converted into learning signals

Online natural language feedback appears in several representational forms. One common form is *pairwise preference supervision*. OAIF samples two responses $y_1, y_2 \sim \pi_\theta(\cdot \mid x)$ from the current policy, queries an LLM annotator, and maps them to $(y^+, y^-)$ before applying a direct preference objective such as DPO, IPO, or SLiC [2402.04792]. In the DPO-style formulation, the preference model is
$$
P_\theta(y^+ \succ y^- \mid x)
= \sigma\!\Big(\beta\Big((\log \pi_\theta(y^+\!\mid x) - \log \pi_\theta(y^-\!\mid x))
- (\log \pi_{\text{ref}}(y^+\!\mid x) - \log \pi_{\text{ref}}(y^-\!\mid x))\Big)\Big),
$$
and OAIF changes not the objective but the provenance of $(x,y^+,y^-)$: the pairs are gathered online and on-policy from the evolving model [2402.04792].

A second form is *free-form critique and refinement text*. DRESS distinguishes “critique feedback,” defined as concise 5–7-word summaries of strengths and weaknesses with respect to helpfulness, honesty, and harmlessness, from “refinement feedback,” defined as actionable suggestions for improving the next turn [2311.10081]. It models a feedback-conditioned policy
$$
\pi_\theta(y \mid x, v, h, f),
$$
where $f$ can be critique or refinement text, and trains with a feedback-conditioned cross-entropy objective rather than REINFORCE [2311.10081].

A third form is *feedback transformed into a target output*. Feed2Resp addresses deployed chatbots that ask dissatisfied users “What should I have said?” The resulting feedback is often noisy or instructional, so a generator $g_\theta$ converts feedback text into a plausible response conditioned on dialogue history, while a discriminator $d_\phi$ distinguishes feedback style from response style [2010.07261]. VAC uses a related but more explicit critique-and-distill strategy: a feedback model produces actionable personalized critiques conditioned on retrieved user profile evidence and question narratives, a policy model revises its answer using that critique during training, and then the policy is fine-tuned on the revised answer so that it no longer needs feedback at inference [2508.10695].

A fourth form is *span-localized textual critique*. Text2Grad pairs free-text critiques with explicit “Good spans” and “Poor spans,” maps them to token-level pseudo-rewards $\delta_t \in \{-1,0,+1\}$ by exact span matching, and defines a natural language gradient
$$
\nabla_{\mathrm{NL}(c \to y)}
=
\sum_{t=1}^T \delta_t \nabla_\theta \log \pi_\theta(y_t \mid x, y_{<t}),
$$
thereby converting language into token-localized credit assignment [2505.22338].

A fifth form is *language-to-utility conversion* for decision-making rather than sequence modeling. LILO uses an LLM to summarize prior language feedback, elicit additional comparative or scalar judgments, and map outcomes to pairwise labels or scalar utilities that a GP surrogate can consume during Bayesian optimization [2510.17671]. In robotics, PFEA and the online robot safety system both transform evaluator language into structured control signals: success/failure plus unmet conditions in one case, and updated failure sets in the other [2510.24109].

## 3. Optimization paradigms

The oldest recurring paradigm is *supervised conditioning on feedback text*. DRESS treats feedback tokens as conditioning variables and optimizes
$$
O_f = \mathbb{E}_{x_i \sim D}\left[- \log P\left(r_i^j \mid m_i, q_i, n_i^j, l_i'^j, \{r_i^k, n_i^k, l_i'^k, s_i^k\}_{k < j}\right)\right],
$$
with an image-captioning regularizer $O = O_f + \alpha O_r$ and $\alpha = 1$ [2311.10081]. LaFFi similarly fine-tunes a model to predict critique text $f$ conditioned on an input $x$ and a model-predicted answer $a^p$, using
$$
L_{\text{feedback}}(\theta) = - \sum_i \log p_\theta(f_i \mid x_i, a^p_i),
$$
rather than directly supervising the answer tokens [2401.00907].

A second paradigm is *direct preference alignment with online labeling*. OAIF preserves DAP’s exact-gradient preference objectives, but replaces stale offline triples with online, on-policy pairs from the current model. This is presented as a response to two systemic problems: off-policy misalignment caused by training on $\pi_{\text{red}}$ while evaluating $\pi_\theta$, and stale preference data that cannot adapt as $\pi_\theta$ changes [2402.04792].

A third paradigm is *reinforcement learning with natural-language-derived rewards*. Text2Grad keeps PPO-style policy optimization but replaces coarse scalar credit with token-level advantages driven by language-aligned spans [2505.22338]. Critique-GRPO uses a different strategy: it samples initial responses, generates CoT critiques, asks the same policy to produce critique-guided refinements, and then optimizes on a mixed group of initial and refined trajectories. Its objective is
$$
J_{\text{Critique-GRPO}}(\theta)
=
\mathbb{E}_q \left[
\frac{1}{n}\sum_i \sum_t r_t^i(\theta) A^{(i)}
+
\frac{1}{k}\sum_{i'} \sum_t f(r_{\text{ref},t}^{(i')}(\theta)) A_{\text{ref}}^{(i')}
\right],
$$
with refinement shaping and no clipping or KL term in the reported formulation [2506.03106].

A fourth paradigm is *online proxy reward realignment*. “Efficiently Aligning Language Models with Online Natural Language Feedback” develops an iterative loop in fuzzy domains: train against a proxy reward, detect over-optimization, collect fresh expert critiques and scalar scores, then update the proxy by rubric-based ICL, few-shot prompting, full-trace distillation, or context distillation [2605.04356]. This makes feedback freshness, rather than only reward-model accuracy at initialization, the central object of study.

A fifth paradigm is *interactive correction without full policy retraining*. SPLASH-style text-to-SQL correction models estimate
$$
y^* = \arg\max_y p_\theta(y \mid q, s, p', f),
$$
using the original question $q$, schema $s$, wrong parse $p'$, and free-form feedback $f$ [2005.02539]. The related feedback-simulation work on interactive semantic parsing trains a T5-based simulator to produce corrective feedback and a T5-based correction model to map $(Q,S,E,F,Y_{\text{init}})$ to a repaired parse, thereby reducing reliance on expensive human-authored feedback [2305.08195].

## 4. Application domains

The most visible application domain is alignment of language models. OAIF evaluates summarization, helpfulness, and harmlessness by using an LLM annotator to compare on-policy samples from PaLM 2-XS, while Critique-GRPO and the expert-proxy ONLF work focus on mathematical reasoning, creative writing, and alignment-research proposal quality [2402.04792]. Text2Grad studies summarization, code generation, and open-domain QA/alignment, and LaFFi studies SQuAD 2.0 question answering via critique prediction rather than answer-only supervision [2505.22338].

Multimodal systems use richer feedback channels. DRESS introduces critique and refinement feedback for large vision-language models, with critique conditioning used for 3H alignment and refinement conditioning used to teach a meta-skill for multi-turn response revision [2311.10081]. PFEA uses spoken commands, natural-language evaluator outputs, and VLM-based planning for robotic manipulation, closing the loop by converting evaluator language into re-planning constraints [2510.24109]. The robot safety work goes further by treating language as a source of evolving safety constraints; open-vocabulary detections grounded in images are projected into workspace regions and unioned into the Hamilton–Jacobi failure set [2409.14580].

Interactive semantic parsing has produced two distinct threads. SPLASH frames the problem as semantic parse correction from free-form feedback supplied after a wrong SQL interpretation and its explanation are shown to the user [2005.02539]. “Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing” studies how to synthesize such feedback with T5 and how to evaluate its logical fidelity with a dedicated RoBERTa-based evaluator, especially for low-data settings [2305.08195].

Other applications emphasize personalization and long-term system improvement. VAC uses user profile documents, retrieved context, and user-authored narratives to generate personalized critiques and distill revised answers into a deployed policy that no longer needs critiques at test time [2508.10695]. System-Level Natural Language Feedback clusters many user comments into reusable system-level criteria for search query generation and response generation, then uses those criteria for prompt design, metric design, and supervised data construction [2306.13588]. AllHands treats large-scale verbatim software feedback as a feedback-analysis substrate, using LLMs for classification, abstractive topic modeling, and natural-language analytics queries over the resulting structured database [2403.15157].

A plausible implication is that “online natural language feedback” is better understood as a family of intervention patterns than as a single algorithm. Some systems use it to align generative policies, some to repair structured predictions, some to steer embodied behavior, and some to redesign evaluation criteria and operational analytics.

## 5. Empirical findings and evaluation patterns

Across domains, the reported results consistently distinguish online or critique-rich feedback from stale offline supervision, scalar-only rewards, or prompt-only reflection.

| System | Setting | Reported result |
|---|---|---|
| OAIF | TL;DR, human eval | Online DPO: 63.74% win, 28.57% tie, 7.69% loss; Offline DPO: 7.69% win, 28.57% tie, 63.74% loss |
| DRESS | LVLM alignment | +9.76% helpfulness, +11.52% honesty, +21.03% harmlessness |
| Feed2Resp | PersonaChat, PolyEncoder | Hits@1/20: 69.94% $\rightarrow$ 75.96% |
| VAC | LaMP-QA | 0.4691 vs 0.4525 for PlanPers and 0.4423 for Offline RL RAG-Personalization |
| Text2Grad | SLF5K summarization | ROUGE-L 0.291 vs 0.262 for PPO |
| Critique-GRPO | 8 reasoning tasks | Average pass@1 improves by approximately 4.5% and 5% over strong baselines |
| ONLF proxy realignment | Qwen3-8B / Haiku 4.5 | ICL recovers up to 35% PGR; fine-tuning recovers 80% with up to 20x fewer samples and 100% with 3x fewer or 10x fewer samples |
| PFEA | Embodied manipulation | 28% higher average task success rate than LLM+CLIP in simulated and real settings |

OAIF’s results are especially pointed because they isolate the online/on-policy variable while keeping the DAP objective fixed. On TL;DR, Helpfulness, and Harmlessness, online DPO beats offline DPO in both human win rates and mean quality scores, and the authors report that offline DPO initially improves but then overfits the offline off-policy data, with its win rate against SFT dropping sharply around 3.5k steps [2402.04792]. DRESS reports that removing critique NLF or refinement NLF harms performance and that using both yields the best alignment and interaction behavior [2311.10081]. VAC reports that optimizing the feedback model jointly with the policy matters, that gains plateau by the third iteration, and that larger feedback models produce better final policies [2508.10695].

Several papers also quantify unresolved gaps. In SPLASH, estimated human correction accuracy is 81.5%, but the best model, EditSQL+Feedback, reaches only 25.16% correction accuracy, even though using feedback still improves the estimated two-turn end-to-end accuracy from 41.30% for one-shot Seq2Struct to 53.73% [2005.02539]. In the semantic-parsing simulation work, evaluator ranking quality reaches MRR $= 0.88$, far above BLEU’s $0.57$ and BERTScore’s $0.55$, yet the motivating claim remains that human-authored feedback is prohibitively expensive and must be partially replaced by simulation [2305.08195]. In online proxy-reward alignment, advantage correlation is useful for proxy diagnostics but does not guarantee monotone expert-reward gains, and scalar-only proxy models over-optimize quickly [2605.04356].

The evaluation landscape is correspondingly heterogeneous. Some papers use human pairwise preferences or quality scores, some use LLM judges validated against humans, some use task metrics such as CHAIR, pass@1, or Hits@1/20, and some evaluate downstream control success. This suggests that online natural language feedback is not tied to a unique evaluation doctrine; rather, each subfield builds feedback loops around its own notion of utility.

## 6. Misconceptions, limitations, and open directions

A common misconception is that online natural language feedback necessarily means updating model weights immediately after each critique. The literature shows otherwise. DRESS performs interaction-time refinement with no parameter updates at inference [2311.10081]. System-Level Natural Language Feedback primarily aggregates comments into criteria, prompts, and metrics before retraining [2306.13588]. AllHands analyzes large-scale feedback through a natural-language interface rather than aligning a generator directly [2403.15157].

A second misconception is that natural language feedback supersedes numerical reward. Several papers instead combine them. Critique-GRPO uses critiques to generate refinements but still learns from binary verifier rewards [2506.03106]. The expert-proxy ONLF framework collects long-form expert feedback plus scalar scores, then distills that supervision into proxy reward models for RL [2605.04356]. Text2Grad converts critique text into token-level pseudo-rewards rather than abandoning reward-based optimization [2505.22338].

The main limitations recur across domains. Annotator quality matters: OAIF scales with annotator size and warns about gaming the annotator, position bias, and length bias [2402.04792]. DRESS notes that GPT-4-generated NLF can reflect biases and that iterative feedback collection is expensive [2311.10081]. VAC depends on evaluator alignment and detailed user-profile handling, raising privacy and data-governance issues [2508.10695]. The robot safety work guarantees safety only with respect to the most recently converged value function, so new language-defined hazards are rigorously enforced only after the warm-started PDE converges [2409.14580]. PFEA remains vulnerable to low-level control failures, calibration quality, and visual ambiguity [2510.24109].

Open directions in the cited works are remarkably convergent. Several papers propose richer feedback signals: rankings, multi-attribute scoring, continuous scores, or explicit rationales rather than only binary choices [2402.04792]. Others propose hybridization: OAIF+RLHF, adaptive reward models from natural language feedback, or continual human-in-the-loop updates [2402.04792]. VAC suggests replacing LLM-generated feedback with real-time human feedback and extending critique to reasoning traces [2508.10695]. The robot safety paper proposes tighter segmentation-based constraint extraction and calibrated VLM outputs for statistical safety assurances [2409.14580]. The semantic-parsing simulation work suggests that analogous edit-template and evaluator machinery could extend beyond text-to-SQL, although it explicitly notes that this is non-trivial for open-ended generation tasks [2305.08195].

Taken together, the literature supports a precise but plural view: online natural language feedback is not a single method but a design space for bringing post hoc language into the optimization, correction, evaluation, or control loop of intelligent systems. Its strongest empirical successes occur when feedback is fresh, policy-matched, and operationalized through architectures that preserve either exact preference gradients, targeted critique-conditioned revision, or uncertainty-aware control. Its strongest unresolved problems are feedback quality, evaluator drift, over-optimization, cost, and the large remaining gap between what humans can communicate in natural language and what current systems can reliably absorb from it.

Source: https://www.emergentmind.com/topics/online-natural-language-feedback