Online Natural Language Feedback

• Online natural language feedback is a process that integrates real-time textual critiques, instructions, and refinements to dynamically improve system outputs.
• It employs diverse methodologies such as pairwise preference supervision, free-form critiques, and language-to-utility mapping to convert feedback into actionable learning signals.
• Applications span from aligning language models and enhancing semantic parsing to improving robotic control, with empirical studies showing significant performance gains.

Searching arXiv for recent and foundational papers on online natural language feedback to ground the article with current literature. Online natural language feedback denotes supervision provided after or during a system’s output in the form of natural language critiques, instructions, explanations, refinements, or language-specified preference criteria, and then used to revise outputs, update models, or alter control policies. In the broader formalization for natural language generation, feedback is a family of functions $h \in H$ mapping inputs and one or more outputs to a signal $f \in F$, with natural-language critiques, scalar ratings, pairwise preferences, and multi-aspect scores all treated as admissible feedback objects (Fernandes et al., 2023). Recent work extends this idea beyond text generation to direct alignment from preferences, vision-language interaction, semantic parsing, personalized question answering, embodied manipulation, robot safety, and system-level feedback analytics, with “online” typically meaning that feedback is gathered or applied on-policy, in-session, or during deployment rather than only from a fixed offline corpus (Guo et al., 2024).

1. Definitions and conceptual boundaries

The literature does not use a single narrow definition of online natural language feedback. In the survey treatment of feedback for natural language generation, online integration includes inference-time refinement, on-policy data logging, and bandit-style or reinforcement-style updates performed while the model is being used, rather than only training from pre-collected judgments (Fernandes et al., 2023). Within that framing, natural language feedback may be a critique, an instruction, an explanation, an error annotation, or a textual preference statement.

A central distinction is between instance-level and system-level use. Instance-level feedback targets a specific output and asks how that output should be changed; system-level feedback aggregates many such comments into global criteria, prompts, and metrics that steer the behavior of the whole system (Yuan et al., 2023). Another distinction is between feedback as text and feedback as language-defined supervision. OAIF, for example, uses natural-language prompts to define what an annotator should prefer, but the training signal itself is a binary pairwise choice rather than a textual critique (Guo et al., 2024). By contrast, DRESS, VAC, Text2Grad, and SPLASH operate directly on critique-like or refinement-like language (Chen et al., 2023).

The term “online” is therefore overloaded. In OAIF it means on-policy, real-time preference collection from the current policy’s outputs (Guo et al., 2024). In DRESS it includes multi-turn refinement at inference time without parameter updates (Chen et al., 2023). In PFEA it refers to a planner–converter–executor loop in which a natural-language evaluator decides whether to stop or re-plan during the same manipulation episode (Ding et al., 28 Oct 2025). In the robot safety setting it means updating the inferred failure set and warm-starting a Hamilton–Jacobi safety controller after new spoken or written constraints arrive during deployment (Santos et al., 2024).

2. Feedback representations and how they are converted into learning signals

Online natural language feedback appears in several representational forms. One common form is pairwise preference supervision. OAIF samples two responses $y_1, y_2 \sim \pi_\theta(\cdot \mid x)$ from the current policy, queries an LLM annotator, and maps them to $(y^+, y^-)$ before applying a direct preference objective such as DPO, IPO, or SLiC (Guo et al., 2024). In the DPO-style formulation, the preference model is

$$P_\theta(y^+ \succ y^- \mid x) = \sigma\!\Big(\beta\Big((\log \pi_\theta(y^+\!\mid x) - \log \pi_\theta(y^-\!\mid x)) - (\log \pi_{\text{ref}}(y^+\!\mid x) - \log \pi_{\text{ref}}(y^-\!\mid x))\Big)\Big),$$

and OAIF changes not the objective but the provenance of $(x,y^+,y^-)$: the pairs are gathered online and on-policy from the evolving model (Guo et al., 2024).

A second form is free-form critique and refinement text. DRESS distinguishes “critique feedback,” defined as concise 5–7-word summaries of strengths and weaknesses with respect to helpfulness, honesty, and harmlessness, from “refinement feedback,” defined as actionable suggestions for improving the next turn (Chen et al., 2023). It models a feedback-conditioned policy

$\pi_\theta(y \mid x, v, h, f),$

where $f$ can be critique or refinement text, and trains with a feedback-conditioned cross-entropy objective rather than REINFORCE (Chen et al., 2023).

A third form is feedback transformed into a target output. Feed2Resp addresses deployed chatbots that ask dissatisfied users “What should I have said?” The resulting feedback is often noisy or instructional, so a generator $g_\theta$ converts feedback text into a plausible response conditioned on dialogue history, while a discriminator $d_\phi$ distinguishes feedback style from response style (Sreedhar et al., 2020). VAC uses a related but more explicit critique-and-distill strategy: a feedback model produces actionable personalized critiques conditioned on retrieved user profile evidence and question narratives, a policy model revises its answer using that critique during training, and then the policy is fine-tuned on the revised answer so that it no longer needs feedback at inference (Salemi et al., 14 Aug 2025).

A fourth form is span-localized textual critique. Text2Grad pairs free-text critiques with explicit “Good spans” and “Poor spans,” maps them to token-level pseudo-rewards $f \in F$0 by exact span matching, and defines a natural language gradient

$f \in F$1

thereby converting language into token-localized credit assignment (Wang et al., 28 May 2025).

A fifth form is language-to-utility conversion for decision-making rather than sequence modeling. LILO uses an LLM to summarize prior language feedback, elicit additional comparative or scalar judgments, and map outcomes to pairwise labels or scalar utilities that a GP surrogate can consume during Bayesian optimization (Kobalczyk et al., 20 Oct 2025). In robotics, PFEA and the online robot safety system both transform evaluator language into structured control signals: success/failure plus unmet conditions in one case, and updated failure sets in the other (Ding et al., 28 Oct 2025).

3. Optimization paradigms

The oldest recurring paradigm is supervised conditioning on feedback text. DRESS treats feedback tokens as conditioning variables and optimizes

$f \in F$2

with an image-captioning regularizer $f \in F$3 and $f \in F$4 (Chen et al., 2023). LaFFi similarly fine-tunes a model to predict critique text $f \in F$5 conditioned on an input $f \in F$6 and a model-predicted answer $f \in F$7, using

$f \in F$8

rather than directly supervising the answer tokens (Li et al., 2023).

A second paradigm is direct preference alignment with online labeling. OAIF preserves DAP’s exact-gradient preference objectives, but replaces stale offline triples with online, on-policy pairs from the current model. This is presented as a response to two systemic problems: off-policy misalignment caused by training on $f \in F$9 while evaluating $y_1, y_2 \sim \pi_\theta(\cdot \mid x)$0, and stale preference data that cannot adapt as $y_1, y_2 \sim \pi_\theta(\cdot \mid x)$1 changes (Guo et al., 2024).

A third paradigm is reinforcement learning with natural-language-derived rewards. Text2Grad keeps PPO-style policy optimization but replaces coarse scalar credit with token-level advantages driven by language-aligned spans (Wang et al., 28 May 2025). Critique-GRPO uses a different strategy: it samples initial responses, generates CoT critiques, asks the same policy to produce critique-guided refinements, and then optimizes on a mixed group of initial and refined trajectories. Its objective is

$y_1, y_2 \sim \pi_\theta(\cdot \mid x)$2

with refinement shaping and no clipping or KL term in the reported formulation (Zhang et al., 3 Jun 2025).

A fourth paradigm is online proxy reward realignment. “Efficiently Aligning LLMs with Online Natural Language Feedback” develops an iterative loop in fuzzy domains: train against a proxy reward, detect over-optimization, collect fresh expert critiques and scalar scores, then update the proxy by rubric-based ICL, few-shot prompting, full-trace distillation, or context distillation (Ye et al., 5 May 2026). This makes feedback freshness, rather than only reward-model accuracy at initialization, the central object of study.

A fifth paradigm is interactive correction without full policy retraining. SPLASH-style text-to-SQL correction models estimate

$y_1, y_2 \sim \pi_\theta(\cdot \mid x)$3

using the original question $y_1, y_2 \sim \pi_\theta(\cdot \mid x)$4, schema $y_1, y_2 \sim \pi_\theta(\cdot \mid x)$5, wrong parse $y_1, y_2 \sim \pi_\theta(\cdot \mid x)$6, and free-form feedback $y_1, y_2 \sim \pi_\theta(\cdot \mid x)$7 (Elgohary et al., 2020). The related feedback-simulation work on interactive semantic parsing trains a T5-based simulator to produce corrective feedback and a T5-based correction model to map $y_1, y_2 \sim \pi_\theta(\cdot \mid x)$8 to a repaired parse, thereby reducing reliance on expensive human-authored feedback (Yan et al., 2023).

4. Application domains

The most visible application domain is alignment of LLMs. OAIF evaluates summarization, helpfulness, and harmlessness by using an LLM annotator to compare on-policy samples from PaLM 2-XS, while Critique-GRPO and the expert-proxy ONLF work focus on mathematical reasoning, creative writing, and alignment-research proposal quality (Guo et al., 2024). Text2Grad studies summarization, code generation, and open-domain QA/alignment, and LaFFi studies SQuAD 2.0 question answering via critique prediction rather than answer-only supervision (Wang et al., 28 May 2025).

Multimodal systems use richer feedback channels. DRESS introduces critique and refinement feedback for large vision-LLMs, with critique conditioning used for 3H alignment and refinement conditioning used to teach a meta-skill for multi-turn response revision (Chen et al., 2023). PFEA uses spoken commands, natural-language evaluator outputs, and VLM-based planning for robotic manipulation, closing the loop by converting evaluator language into re-planning constraints (Ding et al., 28 Oct 2025). The robot safety work goes further by treating language as a source of evolving safety constraints; open-vocabulary detections grounded in images are projected into workspace regions and unioned into the Hamilton–Jacobi failure set (Santos et al., 2024).

Interactive semantic parsing has produced two distinct threads. SPLASH frames the problem as semantic parse correction from free-form feedback supplied after a wrong SQL interpretation and its explanation are shown to the user (Elgohary et al., 2020). “Learning to Simulate Natural Language Feedback for Interactive Semantic Parsing” studies how to synthesize such feedback with T5 and how to evaluate its logical fidelity with a dedicated RoBERTa-based evaluator, especially for low-data settings (Yan et al., 2023).

Other applications emphasize personalization and long-term system improvement. VAC uses user profile documents, retrieved context, and user-authored narratives to generate personalized critiques and distill revised answers into a deployed policy that no longer needs critiques at test time (Salemi et al., 14 Aug 2025). System-Level Natural Language Feedback clusters many user comments into reusable system-level criteria for search query generation and response generation, then uses those criteria for prompt design, metric design, and supervised data construction (Yuan et al., 2023). AllHands treats large-scale verbatim software feedback as a feedback-analysis substrate, using LLMs for classification, abstractive topic modeling, and natural-language analytics queries over the resulting structured database (Zhang et al., 2024).

A plausible implication is that “online natural language feedback” is better understood as a family of intervention patterns than as a single algorithm. Some systems use it to align generative policies, some to repair structured predictions, some to steer embodied behavior, and some to redesign evaluation criteria and operational analytics.

5. Empirical findings and evaluation patterns

Across domains, the reported results consistently distinguish online or critique-rich feedback from stale offline supervision, scalar-only rewards, or prompt-only reflection.

System	Setting	Reported result
OAIF	TL;DR, human eval	Online DPO: 63.74% win, 28.57% tie, 7.69% loss; Offline DPO: 7.69% win, 28.57% tie, 63.74% loss
DRESS	LVLM alignment	+9.76% helpfulness, +11.52% honesty, +21.03% harmlessness
Feed2Resp	PersonaChat, PolyEncoder	Hits@1/20: 69.94% $y_1, y_2 \sim \pi_\theta(\cdot \mid x)$9 75.96%
VAC	LaMP-QA	0.4691 vs 0.4525 for PlanPers and 0.4423 for Offline RL RAG-Personalization
Text2Grad	SLF5K summarization	ROUGE-L 0.291 vs 0.262 for PPO
Critique-GRPO	8 reasoning tasks	Average pass@1 improves by approximately 4.5% and 5% over strong baselines
ONLF proxy realignment	Qwen3-8B / Haiku 4.5	ICL recovers up to 35% PGR; fine-tuning recovers 80% with up to 20x fewer samples and 100% with 3x fewer or 10x fewer samples
PFEA	Embodied manipulation	28% higher average task success rate than LLM+CLIP in simulated and real settings

OAIF’s results are especially pointed because they isolate the online/on-policy variable while keeping the DAP objective fixed. On TL;DR, Helpfulness, and Harmlessness, online DPO beats offline DPO in both human win rates and mean quality scores, and the authors report that offline DPO initially improves but then overfits the offline off-policy data, with its win rate against SFT dropping sharply around 3.5k steps (Guo et al., 2024). DRESS reports that removing critique NLF or refinement NLF harms performance and that using both yields the best alignment and interaction behavior (Chen et al., 2023). VAC reports that optimizing the feedback model jointly with the policy matters, that gains plateau by the third iteration, and that larger feedback models produce better final policies (Salemi et al., 14 Aug 2025).

Several papers also quantify unresolved gaps. In SPLASH, estimated human correction accuracy is 81.5%, but the best model, EditSQL+Feedback, reaches only 25.16% correction accuracy, even though using feedback still improves the estimated two-turn end-to-end accuracy from 41.30% for one-shot Seq2Struct to 53.73% (Elgohary et al., 2020). In the semantic-parsing simulation work, evaluator ranking quality reaches MRR $(y^+, y^-)$0, far above BLEU’s $(y^+, y^-)$1 and BERTScore’s $(y^+, y^-)$2, yet the motivating claim remains that human-authored feedback is prohibitively expensive and must be partially replaced by simulation (Yan et al., 2023). In online proxy-reward alignment, advantage correlation is useful for proxy diagnostics but does not guarantee monotone expert-reward gains, and scalar-only proxy models over-optimize quickly (Ye et al., 5 May 2026).

The evaluation landscape is correspondingly heterogeneous. Some papers use human pairwise preferences or quality scores, some use LLM judges validated against humans, some use task metrics such as CHAIR, pass@1, or Hits@1/20, and some evaluate downstream control success. This suggests that online natural language feedback is not tied to a unique evaluation doctrine; rather, each subfield builds feedback loops around its own notion of utility.

6. Misconceptions, limitations, and open directions

A common misconception is that online natural language feedback necessarily means updating model weights immediately after each critique. The literature shows otherwise. DRESS performs interaction-time refinement with no parameter updates at inference (Chen et al., 2023). System-Level Natural Language Feedback primarily aggregates comments into criteria, prompts, and metrics before retraining (Yuan et al., 2023). AllHands analyzes large-scale feedback through a natural-language interface rather than aligning a generator directly (Zhang et al., 2024).

A second misconception is that natural language feedback supersedes numerical reward. Several papers instead combine them. Critique-GRPO uses critiques to generate refinements but still learns from binary verifier rewards (Zhang et al., 3 Jun 2025). The expert-proxy ONLF framework collects long-form expert feedback plus scalar scores, then distills that supervision into proxy reward models for RL (Ye et al., 5 May 2026). Text2Grad converts critique text into token-level pseudo-rewards rather than abandoning reward-based optimization (Wang et al., 28 May 2025).

The main limitations recur across domains. Annotator quality matters: OAIF scales with annotator size and warns about gaming the annotator, position bias, and length bias (Guo et al., 2024). DRESS notes that GPT-4-generated NLF can reflect biases and that iterative feedback collection is expensive (Chen et al., 2023). VAC depends on evaluator alignment and detailed user-profile handling, raising privacy and data-governance issues (Salemi et al., 14 Aug 2025). The robot safety work guarantees safety only with respect to the most recently converged value function, so new language-defined hazards are rigorously enforced only after the warm-started PDE converges (Santos et al., 2024). PFEA remains vulnerable to low-level control failures, calibration quality, and visual ambiguity (Ding et al., 28 Oct 2025).

Open directions in the cited works are remarkably convergent. Several papers propose richer feedback signals: rankings, multi-attribute scoring, continuous scores, or explicit rationales rather than only binary choices (Guo et al., 2024). Others propose hybridization: OAIF+RLHF, adaptive reward models from natural language feedback, or continual human-in-the-loop updates (Guo et al., 2024). VAC suggests replacing LLM-generated feedback with real-time human feedback and extending critique to reasoning traces (Salemi et al., 14 Aug 2025). The robot safety paper proposes tighter segmentation-based constraint extraction and calibrated VLM outputs for statistical safety assurances (Santos et al., 2024). The semantic-parsing simulation work suggests that analogous edit-template and evaluator machinery could extend beyond text-to-SQL, although it explicitly notes that this is non-trivial for open-ended generation tasks (Yan et al., 2023).

Taken together, the literature supports a precise but plural view: online natural language feedback is not a single method but a design space for bringing post hoc language into the optimization, correction, evaluation, or control loop of intelligent systems. Its strongest empirical successes occur when feedback is fresh, policy-matched, and operationalized through architectures that preserve either exact preference gradients, targeted critique-conditioned revision, or uncertainty-aware control. Its strongest unresolved problems are feedback quality, evaluator drift, over-optimization, cost, and the large remaining gap between what humans can communicate in natural language and what current systems can reliably absorb from it.