---
title: Natural Language Feedback (NLF) in AI
url: https://www.emergentmind.com/topics/natural-language-feedback-nlf
type: topic
---

# Natural Language Feedback (NLF) in AI

Natural Language Feedback (NLF) denotes feedback expressed in natural language rather than as scalar rewards, pairwise comparisons, or fixed checklists. In the current literature, NLF includes free-form human-written feedback on model outputs, corrective instructions, explanations, critiques, refinements, and step-wise rationales. It is used as an instance-level signal for repairing specific outputs, as a training signal for distilling improved behavior into a model, and as a system-level signal for metric design, prompt design, reward modeling, and control adaptation across language modeling, reinforcement learning, semantic parsing, vision-language modeling, and robotics [2204.14146] [2307.00279] [2306.13588].

## 1. Conceptual scope and feedback taxonomies

A recurring motivation for NLF is that it conveys more information per human evaluation than comparison feedback. In language modeling, NLF is defined as free-form human-written feedback on model outputs, such as comments that specify what is wrong or how an output should be improved; this is contrasted directly with comparison feedback, where humans choose between pairs of outputs [2204.14146]. In pedagogy-inspired work, NLF is treated not as a single supervision type but as a structured design space. The FELT framework organizes that space around **Feedback, Errors, Learner, Task**, and its feedback-content taxonomy distinguishes four non-overlapping content types: **learner status**, **goal**, **procedural**, and **peripheral**. The same work also identifies ten modulation dimensions: **Granularity**, **Applicability of instructions**, **Answer coverage**, **Target coverage**, **Criteria**, **Information novelty**, **Purpose**, **Style**, **Valence**, and **Mode** [2307.00279].

This broader view is reflected in task-specific categorizations. In large vision-language models, DRESS separates NLF into **critique** and **refinement**: critique identifies strengths and weaknesses of responses, while refinement offers concrete suggestions for improvement in multi-turn interactions [2311.10081]. In goal-conditioned reinforcement learning, BabyAI-based work separates NLF into **Task Feedback** and **Rule Feedback**, where the former marks progress toward the mission and the latter provides corrective feedback about invalid actions [2312.04736]. In factual summarization, DeFacto formalizes informational feedback as **corrective instructions**, **edited summaries**, and **explanations** with respect to factual consistency [2212.09968]. In mathematical verification, Math-Minos uses **step-wise natural language feedback as rationale labels**, explicitly encoding the correctness of each step and the detailed explanations [2406.14024].

A common misconception is that NLF is synonymous with RLHF-style preference data. The literature instead treats NLF as a heterogeneous supervision family: it may describe the target answer, explain an error, propose a procedure, revise a previous output, or specify a constraint. This suggests that the principal technical question is not whether to use NLF, but how to map different feedback forms to the learner, task, and error structure at hand.

## 2. Learning paradigms and formal objectives

One major paradigm uses NLF to generate and select refinements. In summarization, a three-step algorithm first conditions a language model on the input, initial output, and feedback to generate many refinements; it then selects the refinement with the highest similarity to the feedback; and finally it finetunes a language model to maximize the likelihood of the chosen refinement given the input. The selection step is written as
$$
y'^* = \arg\max_{y'_i} \cos(\mathcal{E}(y'_i), \mathcal{E}(f)).
$$
This formulation makes NLF an explicit bridge between local repair and global model improvement [2204.14146].

A second paradigm treats NLF as training-time supervision that need not be present at inference. Imitation learning from Language Feedback (ILF) samples failures from a base code model, collects human-written feedback on the failures, trains a Refine model to incorporate the feedback, filters refined programs by unit tests, and then fine-tunes the base model on corrected programs. ILF is also given a distributional interpretation through the objective
$$
\min_\theta \mathbb{E}_{t \sim p(t)}\left[ \mathrm{KL}(\pi_t^*, \pi_\theta(\cdot|t)) \right].
$$
Here NLF is not merely explanatory metadata; it is an efficient mechanism for moving training data toward the model’s actual failure modes [2303.16749].

A third paradigm interprets NLF as evidence about latent reward or verification structure. In reinforcement learning, one framework uses aspect-based sentiment analysis to decompose unconstrained feedback into sentiment about the features of a Markov decision process and then regresses the sentiment on the features to infer the teacher’s latent reward function; the paper describes this as an analogue of inverse reinforcement learning and evaluates sentiment-based **literal** and **pragmatic** models as well as an inference network [2009.14715]. In mathematical reasoning, Math-Minos first trains a verifier to generate step-wise rationales and then trains for binary classification so that inference remains efficient [2406.14024]. In newer reinforcement-learning work, Text2Grad converts free-form textual feedback into span-level gradients through a feedback-annotation pipeline, a fine-grained reward model, and a span-level policy optimizer [2505.22338].

A fourth paradigm uses NLF outside ordinary supervised fine-tuning altogether. RL4F optimizes a critique generator with PPO so that a fixed GPT-3 model produces better repaired outputs after reading the critique; the downstream model is not fine-tuned, and the critique model is trained directly against end-task performance [2305.08844]. LILO uses a large language model to convert unstructured natural language feedback into scalar utilities or pairwise preferences for Bayesian Optimization, preserving Gaussian-process-based uncertainty quantification while expanding the feedback interface [2510.17671]. These formulations enlarge the scope of NLF from “feedback on an answer” to “language-mediated optimization.”

## 3. Data resources and annotation regimes

NLF research depends heavily on how feedback is collected, structured, or simulated. Some datasets are explicitly human-authored around naturally occurring model or agent errors, while others generate feedback automatically from environment dynamics or with strong language models.

| Resource | NLF form | Primary function |
|---|---|---|
| SPLASH [2005.02539] | Free-form natural language feedback on incorrect SQL parses | Semantic parse correction |
| DeFacto [2212.09968] | Explanations, evidence, instructions, edited summaries | Factual consistency in summarization |
| SYNDICOM [2309.10015] | 1–2 sentences explaining what is wrong with an invalid response | Commonsense dialogue repair |
| BabyAI extension [2312.04736] | Template-based Task Feedback and Rule Feedback | Goal-conditioned RL generalisation |
| Math-Minos [2406.14024] | Step-wise rationale labels with correctness explanations | Mathematical verification and reward modeling |

SPLASH was constructed for interactive text-to-SQL correction by pairing natural language questions, incorrect SQL interpretations, and the corresponding natural language feedback. The setting is explicitly interactive: the system presents an explanation of the SQL, the user writes feedback, and the model attempts to generate a corrected SQL parse [2005.02539]. DeFacto, by contrast, is highly structured: annotators decide whether a summary is factually consistent, categorize errors, write an explanation, select supporting evidence, produce an instruction using templates such as **Remove**, **Add**, **Replace**, **Modify**, **Rewrite**, and **Other**, and then minimally edit the summary [2212.09968]. SYNDICOM constructs tuples of dialogue context, valid response, invalid response, and feedback, with invalid responses produced by error injection so that crowd workers can write specific NLF about what is wrong [2309.10015].

Other work reduces annotation costs through simulation or automatic generation. BabyAI extends the environment to emit natural language feedback from deterministic rules and hand-crafted templates at each timestep, avoiding human-in-the-loop annotation during data generation [2312.04736]. Math-Minos uses GPT-4-turbo to generate step-level rationales, with a label-aware prompting strategy that conditions explanation generation on the true correctness label [2406.14024]. For interactive semantic parsing, a separate line of work introduces feedback simulation itself as a task, proposes T5-based simulators, and develops a feedback evaluator designed to assess whether generated feedback captures the intended correction rather than merely matching surface form [2305.08195].

This variation in annotation regimes has methodological consequences. Human-authored feedback captures grounded error descriptions and naturally varying phrasing; template-based or model-generated feedback offers scale and control. The literature repeatedly compares these sources rather than treating them as interchangeable.

## 4. Application domains

NLF has been adopted in reinforcement learning and decision-making, where it can complement or replace conventional control signals. In BabyAI, sequences may contain tuples of return-to-go, observation, action, mission, and NLF, and conditioning on NLF improves generalisation; notably, NLF-only models can match or exceed RTG- or mission-only baselines, and gains often remain when feedback is available during training but not at inference [2312.04736]. Earlier reward-learning work shows that unconstrained teaching language can be interpreted as sentiment over MDP features, with a **pragmatic** model approaching human performance in a cooperative task [2009.14715]. In Bayesian Optimization, LILO uses LLMs to convert varied textual feedback into pairwise preferences or scalar utilities over a numeric search space [2510.17671]. In robotics, natural language feedback specifies semantic safety constraints, which are mapped by a vision-language model into updates of the failure set,
$$
\hat{F}_E^{t+1} = \hat{F}_E^t \cup B_{XY},
$$
and then incorporated into a Hamilton-Jacobi reachability safety controller through warm-starting [2409.14580].

Language-generation tasks are the most densely studied NLF setting. Summarization work uses NLF for refinement, factuality correction, and joint feedback generation [2204.14146] [2212.09968]. Code generation uses human-written feedback on failed programs to train refinement pipelines and then distill corrected behavior into the base model [2303.16749]. Dialogue work uses NLF both to explain commonsense errors in invalid responses and to convert noisy user feedback into plausible conversational responses suitable for retraining [2309.10015] [2010.07261]. Personalized question answering replaces scalar rewards with language feedback conditioned on user profiles and question narratives, alternating feedback-model optimization with policy-model fine-tuning [2508.10695].

Semantic parsing provides a particularly explicit interactive model of NLF. In text-to-SQL, users inspect explanations of incorrect parses and then write free-form feedback such as missing information, wrong columns, wrong filters, or aggregation errors; the system must use question, schema, incorrect SQL, and feedback together to produce a corrected query [2005.02539]. Feedback simulation later emerges as a separate research problem because collecting human annotations at scale is expensive [2305.08195].

Vision-language and multimodal systems broaden the meaning of NLF further. DRESS uses critique and refinement feedback from GPT-4 to improve helpfulness, honesty, harmlessness, and multi-turn response revision in large vision-language models [2311.10081]. Conversational fashion image retrieval uses multiturn natural language feedback together with conversation history and fashion attributes to retrieve target images [2106.04128]. A separate fashion system treats NLF itself as the output modality, generating positive and constructive outfit feedback from images [1906.06619]. Across these tasks, NLF serves alternately as supervision, control, explanation, and product output.

## 5. Empirical findings

Across language generation, NLF frequently improves quality with relatively little annotated data. In summarization, using only **100 samples of human-written feedback**, a GPT-3 model was finetuned to roughly human-level summarization ability; in the same study, only large language models with **175B parameters** reliably incorporated feedback in synthetic experiments [2204.14146]. In factual summarization, human-edited outputs in DeFacto achieved **DAE = 0.8975** and **QAFactEval = 2.540**, compared with **DAE = 0.6904** and **QAFactEval = 1.826** for the system summaries; a T0pp editing model with full NLF input reached **ROUGE-1 = 89.22**, **DAE = 0.911**, and **QAFactEval = 2.465**, and human evaluation found that **97%** of summaries edited with human NLF were more factual than originals while **91%** followed instructions [2212.09968]. In commonsense dialogue, SYNDICOM achieved a **relative improvement of 53% over ChatGPT on ROUGE1**, and human evaluators preferred SYNDICOM **56.5%** of the time [2309.10015].

In code and program synthesis, ILF improved a CodeGen-Mono 6.1B model on MBPP from **Pass@1: 26%** to **Pass@1: 36%**, and from **Pass@10: 59%** to **Pass@10: 68%**, using only **78 refinements** for fine-tuning [2303.16749]. In interactive chatbots, converting noisy feedback into plausible responses with Feed2Resp improved PolyEncoder test performance in ranking correct responses on PersonaChat from **69.94%** to **75.96%** [2010.07261]. In interactive semantic parsing, the best model on SPLASH reached only **25.16%** correction accuracy, while estimated human correction accuracy was **81.5%**, indicating that the setting is useful precisely because it remains difficult [2005.02539].

In RL, verification, and multimodal interaction, NLF also yields measurable gains. RL4F reports **relative improvements up to 10%** over supervised, retrieval-augmented, and prompting-based critique generators, and on alphabetization reaches **66.1% exact match** versus **38.9%** for the supervised baseline [2305.08844]. Math-Minos reports verifier gains of up to **+1.6%** on GSM8K and **+0.8%** on MATH with **30k** NLF-augmented training examples, and in false-positive analysis improves recall from **0.74** to **0.92** [2406.14024]. DRESS reports improvements of **9.76%** in helpfulness, **11.52%** in honesty, and **21.03%** in harmlessness over prior LVLMs [2311.10081]. In goal-conditioned RL, NLF improves zero-shot compositional generalisation and can remain useful even when removed at inference [2312.04736].

These results do not imply a single universal benefit magnitude. They do show, however, that NLF can be competitive in low-data regimes, can repair outputs more precisely than coarse rewards, and can transfer from one correction episode into a model that later acts without explicit feedback.

## 6. Limitations, failure modes, and disputed assumptions

The literature is equally clear that NLF is not automatically beneficial. Model scale matters: in synthetic experiments on incorporating feedback, only **175B**-parameter models reliably used natural language feedback, while smaller models performed poorly [2204.14146]. Feedback quality also matters. ILF reports that human feedback is more helpful than LLM-generated feedback, that poor feedback yields low improvement, and that the Refine model struggles when feedback addresses many bugs at once [2303.16749]. In system-level feedback work, human-written instance-level feedback produced more grounded refinements than GPT-3.5-written feedback [2306.13588].

A second limitation concerns controllability. In DeFacto, fine-tuned encoder-decoder models make minimal, instruction-following edits, but zero-shot GPT-3.5 and GPT-4 often rewrite rather than minimally revise, and GPT-3.5 replied **“No editing instructions needed” 24% of the time** for factually inconsistent summaries [2212.09968]. In interactive text-to-SQL, the gap between **81.5%** estimated human correction accuracy and **25.16%** model accuracy indicates that current models still struggle to combine question, schema, incorrect parse, and free-form feedback robustly [2005.02539].

A third limitation concerns when and how feedback is used. In BabyAI, combining Task Feedback and Rule Feedback is **not always synergistic**, and feedback at inference is sometimes detrimental because of feedback distribution shift between training and testing [2312.04736]. In self-feeding dialogue, raw feedback or heuristic cleanup can underperform no-feedback training, whereas adversarial rewriting into natural responses yields the actual improvement [2010.07261]. This suggests that the representation of feedback can be as important as the feedback content itself.

Evaluation is another unresolved issue. For simulated feedback in semantic parsing, BLEU and BERTScore are reported to correlate poorly with the logical adequacy of a correction, while the dedicated evaluator reaches **0.88** Spearman correlation with human ranking compared with **0.19** for BERTScore and **0.03** for BLEU [2305.08195]. The broader implication is that NLF systems often require task-specific evaluation of whether the feedback captures the intended correction, not merely whether it resembles a reference string.

## 7. Emerging directions

Recent work extends NLF beyond local output repair. System-level NLF aggregates many feedback instances into criteria for **metric design** and **prompt design**, and the combination of system-level and instance-level feedback brings further gains in search query generation and dialog response generation [2306.13588]. Personalized question answering replaces scalar rewards with NLF generated from user profiles and question narratives; training alternates between improving the feedback model and fine-tuning the policy model, and the final policy model no longer requires feedback at inference [2508.10695]. Text2Grad pushes this further by mapping critiques to token spans and turning them into span-level policy updates [2505.22338].

Another direction is infrastructure for sparse or expensive annotation. Feedback simulation for semantic parsing shows that, in low-data settings, simulated feedback can achieve error-correction performance comparable to training on the full set of human annotations [2305.08195]. BabyAI, Math-Minos, and DRESS all use automatically generated or model-generated feedback to reduce reliance on manual labeling while retaining explicit language supervision [2312.04736] [2406.14024] [2311.10081]. In robotics and Bayesian Optimization, NLF is becoming a way to communicate deployment-time constraints and subjective utility without redesigning the underlying control or optimization machinery [2409.14580] [2510.17671].

A plausible implication is that future NLF systems will be hybrid systems: human-authored feedback for grounding and specificity, learned critics or simulators for scale, and task-specific verifiers or control modules for reliable downstream use. Another plausible implication is that inference-time interfaces may become lighter even as training-time feedback becomes richer, because several lines of work explicitly internalize NLF during training and then remove the requirement for feedback at test time [2303.16749] [2312.04736] [2508.10695]. In that sense, NLF is not merely an annotation format; it is an increasingly general mechanism for converting human judgment into operational structure.

Source: https://www.emergentmind.com/topics/natural-language-feedback-nlf