Natural Language Feedback (NLF) in AI
- Natural Language Feedback (NLF) is feedback expressed in natural language that provides detailed, contextual guidance via explanations, critiques, and refinements.
- NLF is used to repair and improve model outputs through instance-level corrections, metric designs, and optimization techniques in domains like language modeling, semantic parsing, reinforcement learning, and robotics.
- Emerging research leverages NLF for both training and evaluation, although challenges remain in terms of feedback quality, model scale, and efficient integration.
Natural Language Feedback (NLF) denotes feedback expressed in natural language rather than as scalar rewards, pairwise comparisons, or fixed checklists. In the current literature, NLF includes free-form human-written feedback on model outputs, corrective instructions, explanations, critiques, refinements, and step-wise rationales. It is used as an instance-level signal for repairing specific outputs, as a training signal for distilling improved behavior into a model, and as a system-level signal for metric design, prompt design, reward modeling, and control adaptation across language modeling, reinforcement learning, semantic parsing, vision-language modeling, and robotics (Scheurer et al., 2022, Borges et al., 2023, Yuan et al., 2023).
1. Conceptual scope and feedback taxonomies
A recurring motivation for NLF is that it conveys more information per human evaluation than comparison feedback. In language modeling, NLF is defined as free-form human-written feedback on model outputs, such as comments that specify what is wrong or how an output should be improved; this is contrasted directly with comparison feedback, where humans choose between pairs of outputs (Scheurer et al., 2022). In pedagogy-inspired work, NLF is treated not as a single supervision type but as a structured design space. The FELT framework organizes that space around Feedback, Errors, Learner, Task, and its feedback-content taxonomy distinguishes four non-overlapping content types: learner status, goal, procedural, and peripheral. The same work also identifies ten modulation dimensions: Granularity, Applicability of instructions, Answer coverage, Target coverage, Criteria, Information novelty, Purpose, Style, Valence, and Mode (Borges et al., 2023).
This broader view is reflected in task-specific categorizations. In large vision-LLMs, DRESS separates NLF into critique and refinement: critique identifies strengths and weaknesses of responses, while refinement offers concrete suggestions for improvement in multi-turn interactions (Chen et al., 2023). In goal-conditioned reinforcement learning, BabyAI-based work separates NLF into Task Feedback and Rule Feedback, where the former marks progress toward the mission and the latter provides corrective feedback about invalid actions (McCallum et al., 2023). In factual summarization, DeFacto formalizes informational feedback as corrective instructions, edited summaries, and explanations with respect to factual consistency (Liu et al., 2022). In mathematical verification, Math-Minos uses step-wise natural language feedback as rationale labels, explicitly encoding the correctness of each step and the detailed explanations (2406.14024).
A common misconception is that NLF is synonymous with RLHF-style preference data. The literature instead treats NLF as a heterogeneous supervision family: it may describe the target answer, explain an error, propose a procedure, revise a previous output, or specify a constraint. This suggests that the principal technical question is not whether to use NLF, but how to map different feedback forms to the learner, task, and error structure at hand.
2. Learning paradigms and formal objectives
One major paradigm uses NLF to generate and select refinements. In summarization, a three-step algorithm first conditions a LLM on the input, initial output, and feedback to generate many refinements; it then selects the refinement with the highest similarity to the feedback; and finally it finetunes a LLM to maximize the likelihood of the chosen refinement given the input. The selection step is written as
This formulation makes NLF an explicit bridge between local repair and global model improvement (Scheurer et al., 2022).
A second paradigm treats NLF as training-time supervision that need not be present at inference. Imitation learning from Language Feedback (ILF) samples failures from a base code model, collects human-written feedback on the failures, trains a Refine model to incorporate the feedback, filters refined programs by unit tests, and then fine-tunes the base model on corrected programs. ILF is also given a distributional interpretation through the objective
Here NLF is not merely explanatory metadata; it is an efficient mechanism for moving training data toward the model’s actual failure modes (Chen et al., 2023).
A third paradigm interprets NLF as evidence about latent reward or verification structure. In reinforcement learning, one framework uses aspect-based sentiment analysis to decompose unconstrained feedback into sentiment about the features of a Markov decision process and then regresses the sentiment on the features to infer the teacher’s latent reward function; the paper describes this as an analogue of inverse reinforcement learning and evaluates sentiment-based literal and pragmatic models as well as an inference network (Sumers et al., 2020). In mathematical reasoning, Math-Minos first trains a verifier to generate step-wise rationales and then trains for binary classification so that inference remains efficient (2406.14024). In newer reinforcement-learning work, Text2Grad converts free-form textual feedback into span-level gradients through a feedback-annotation pipeline, a fine-grained reward model, and a span-level policy optimizer (Wang et al., 28 May 2025).
A fourth paradigm uses NLF outside ordinary supervised fine-tuning altogether. RL4F optimizes a critique generator with PPO so that a fixed GPT-3 model produces better repaired outputs after reading the critique; the downstream model is not fine-tuned, and the critique model is trained directly against end-task performance (Akyürek et al., 2023). LILO uses a LLM to convert unstructured natural language feedback into scalar utilities or pairwise preferences for Bayesian Optimization, preserving Gaussian-process-based uncertainty quantification while expanding the feedback interface (Kobalczyk et al., 20 Oct 2025). These formulations enlarge the scope of NLF from “feedback on an answer” to “language-mediated optimization.”
3. Data resources and annotation regimes
NLF research depends heavily on how feedback is collected, structured, or simulated. Some datasets are explicitly human-authored around naturally occurring model or agent errors, while others generate feedback automatically from environment dynamics or with strong LLMs.
| Resource | NLF form | Primary function |
|---|---|---|
| SPLASH (Elgohary et al., 2020) | Free-form natural language feedback on incorrect SQL parses | Semantic parse correction |
| DeFacto (Liu et al., 2022) | Explanations, evidence, instructions, edited summaries | Factual consistency in summarization |
| SYNDICOM (Richardson et al., 2023) | 1–2 sentences explaining what is wrong with an invalid response | Commonsense dialogue repair |
| BabyAI extension (McCallum et al., 2023) | Template-based Task Feedback and Rule Feedback | Goal-conditioned RL generalisation |
| Math-Minos (2406.14024) | Step-wise rationale labels with correctness explanations | Mathematical verification and reward modeling |
SPLASH was constructed for interactive text-to-SQL correction by pairing natural language questions, incorrect SQL interpretations, and the corresponding natural language feedback. The setting is explicitly interactive: the system presents an explanation of the SQL, the user writes feedback, and the model attempts to generate a corrected SQL parse (Elgohary et al., 2020). DeFacto, by contrast, is highly structured: annotators decide whether a summary is factually consistent, categorize errors, write an explanation, select supporting evidence, produce an instruction using templates such as Remove, Add, Replace, Modify, Rewrite, and Other, and then minimally edit the summary (Liu et al., 2022). SYNDICOM constructs tuples of dialogue context, valid response, invalid response, and feedback, with invalid responses produced by error injection so that crowd workers can write specific NLF about what is wrong (Richardson et al., 2023).
Other work reduces annotation costs through simulation or automatic generation. BabyAI extends the environment to emit natural language feedback from deterministic rules and hand-crafted templates at each timestep, avoiding human-in-the-loop annotation during data generation (McCallum et al., 2023). Math-Minos uses GPT-4-turbo to generate step-level rationales, with a label-aware prompting strategy that conditions explanation generation on the true correctness label (2406.14024). For interactive semantic parsing, a separate line of work introduces feedback simulation itself as a task, proposes T5-based simulators, and develops a feedback evaluator designed to assess whether generated feedback captures the intended correction rather than merely matching surface form (Yan et al., 2023).
This variation in annotation regimes has methodological consequences. Human-authored feedback captures grounded error descriptions and naturally varying phrasing; template-based or model-generated feedback offers scale and control. The literature repeatedly compares these sources rather than treating them as interchangeable.
4. Application domains
NLF has been adopted in reinforcement learning and decision-making, where it can complement or replace conventional control signals. In BabyAI, sequences may contain tuples of return-to-go, observation, action, mission, and NLF, and conditioning on NLF improves generalisation; notably, NLF-only models can match or exceed RTG- or mission-only baselines, and gains often remain when feedback is available during training but not at inference (McCallum et al., 2023). Earlier reward-learning work shows that unconstrained teaching language can be interpreted as sentiment over MDP features, with a pragmatic model approaching human performance in a cooperative task (Sumers et al., 2020). In Bayesian Optimization, LILO uses LLMs to convert varied textual feedback into pairwise preferences or scalar utilities over a numeric search space (Kobalczyk et al., 20 Oct 2025). In robotics, natural language feedback specifies semantic safety constraints, which are mapped by a vision-LLM into updates of the failure set,
and then incorporated into a Hamilton-Jacobi reachability safety controller through warm-starting (Santos et al., 2024).
Language-generation tasks are the most densely studied NLF setting. Summarization work uses NLF for refinement, factuality correction, and joint feedback generation (Scheurer et al., 2022, Liu et al., 2022). Code generation uses human-written feedback on failed programs to train refinement pipelines and then distill corrected behavior into the base model (Chen et al., 2023). Dialogue work uses NLF both to explain commonsense errors in invalid responses and to convert noisy user feedback into plausible conversational responses suitable for retraining (Richardson et al., 2023, Sreedhar et al., 2020). Personalized question answering replaces scalar rewards with language feedback conditioned on user profiles and question narratives, alternating feedback-model optimization with policy-model fine-tuning (Salemi et al., 14 Aug 2025).
Semantic parsing provides a particularly explicit interactive model of NLF. In text-to-SQL, users inspect explanations of incorrect parses and then write free-form feedback such as missing information, wrong columns, wrong filters, or aggregation errors; the system must use question, schema, incorrect SQL, and feedback together to produce a corrected query (Elgohary et al., 2020). Feedback simulation later emerges as a separate research problem because collecting human annotations at scale is expensive (Yan et al., 2023).
Vision-language and multimodal systems broaden the meaning of NLF further. DRESS uses critique and refinement feedback from GPT-4 to improve helpfulness, honesty, harmlessness, and multi-turn response revision in large vision-LLMs (Chen et al., 2023). Conversational fashion image retrieval uses multiturn natural language feedback together with conversation history and fashion attributes to retrieve target images (Yuan et al., 2021). A separate fashion system treats NLF itself as the output modality, generating positive and constructive outfit feedback from images (Sadeh et al., 2019). Across these tasks, NLF serves alternately as supervision, control, explanation, and product output.
5. Empirical findings
Across language generation, NLF frequently improves quality with relatively little annotated data. In summarization, using only 100 samples of human-written feedback, a GPT-3 model was finetuned to roughly human-level summarization ability; in the same study, only LLMs with 175B parameters reliably incorporated feedback in synthetic experiments (Scheurer et al., 2022). In factual summarization, human-edited outputs in DeFacto achieved DAE = 0.8975 and QAFactEval = 2.540, compared with DAE = 0.6904 and QAFactEval = 1.826 for the system summaries; a T0pp editing model with full NLF input reached ROUGE-1 = 89.22, DAE = 0.911, and QAFactEval = 2.465, and human evaluation found that 97% of summaries edited with human NLF were more factual than originals while 91% followed instructions (Liu et al., 2022). In commonsense dialogue, SYNDICOM achieved a relative improvement of 53% over ChatGPT on ROUGE1, and human evaluators preferred SYNDICOM 56.5% of the time (Richardson et al., 2023).
In code and program synthesis, ILF improved a CodeGen-Mono 6.1B model on MBPP from Pass@1: 26% to Pass@1: 36%, and from Pass@10: 59% to Pass@10: 68%, using only 78 refinements for fine-tuning (Chen et al., 2023). In interactive chatbots, converting noisy feedback into plausible responses with Feed2Resp improved PolyEncoder test performance in ranking correct responses on PersonaChat from 69.94% to 75.96% (Sreedhar et al., 2020). In interactive semantic parsing, the best model on SPLASH reached only 25.16% correction accuracy, while estimated human correction accuracy was 81.5%, indicating that the setting is useful precisely because it remains difficult (Elgohary et al., 2020).
In RL, verification, and multimodal interaction, NLF also yields measurable gains. RL4F reports relative improvements up to 10% over supervised, retrieval-augmented, and prompting-based critique generators, and on alphabetization reaches 66.1% exact match versus 38.9% for the supervised baseline (Akyürek et al., 2023). Math-Minos reports verifier gains of up to +1.6% on GSM8K and +0.8% on MATH with 30k NLF-augmented training examples, and in false-positive analysis improves recall from 0.74 to 0.92 (2406.14024). DRESS reports improvements of 9.76% in helpfulness, 11.52% in honesty, and 21.03% in harmlessness over prior LVLMs (Chen et al., 2023). In goal-conditioned RL, NLF improves zero-shot compositional generalisation and can remain useful even when removed at inference (McCallum et al., 2023).
These results do not imply a single universal benefit magnitude. They do show, however, that NLF can be competitive in low-data regimes, can repair outputs more precisely than coarse rewards, and can transfer from one correction episode into a model that later acts without explicit feedback.
6. Limitations, failure modes, and disputed assumptions
The literature is equally clear that NLF is not automatically beneficial. Model scale matters: in synthetic experiments on incorporating feedback, only 175B-parameter models reliably used natural language feedback, while smaller models performed poorly (Scheurer et al., 2022). Feedback quality also matters. ILF reports that human feedback is more helpful than LLM-generated feedback, that poor feedback yields low improvement, and that the Refine model struggles when feedback addresses many bugs at once (Chen et al., 2023). In system-level feedback work, human-written instance-level feedback produced more grounded refinements than GPT-3.5-written feedback (Yuan et al., 2023).
A second limitation concerns controllability. In DeFacto, fine-tuned encoder-decoder models make minimal, instruction-following edits, but zero-shot GPT-3.5 and GPT-4 often rewrite rather than minimally revise, and GPT-3.5 replied “No editing instructions needed” 24% of the time for factually inconsistent summaries (Liu et al., 2022). In interactive text-to-SQL, the gap between 81.5% estimated human correction accuracy and 25.16% model accuracy indicates that current models still struggle to combine question, schema, incorrect parse, and free-form feedback robustly (Elgohary et al., 2020).
A third limitation concerns when and how feedback is used. In BabyAI, combining Task Feedback and Rule Feedback is not always synergistic, and feedback at inference is sometimes detrimental because of feedback distribution shift between training and testing (McCallum et al., 2023). In self-feeding dialogue, raw feedback or heuristic cleanup can underperform no-feedback training, whereas adversarial rewriting into natural responses yields the actual improvement (Sreedhar et al., 2020). This suggests that the representation of feedback can be as important as the feedback content itself.
Evaluation is another unresolved issue. For simulated feedback in semantic parsing, BLEU and BERTScore are reported to correlate poorly with the logical adequacy of a correction, while the dedicated evaluator reaches 0.88 Spearman correlation with human ranking compared with 0.19 for BERTScore and 0.03 for BLEU (Yan et al., 2023). The broader implication is that NLF systems often require task-specific evaluation of whether the feedback captures the intended correction, not merely whether it resembles a reference string.
7. Emerging directions
Recent work extends NLF beyond local output repair. System-level NLF aggregates many feedback instances into criteria for metric design and prompt design, and the combination of system-level and instance-level feedback brings further gains in search query generation and dialog response generation (Yuan et al., 2023). Personalized question answering replaces scalar rewards with NLF generated from user profiles and question narratives; training alternates between improving the feedback model and fine-tuning the policy model, and the final policy model no longer requires feedback at inference (Salemi et al., 14 Aug 2025). Text2Grad pushes this further by mapping critiques to token spans and turning them into span-level policy updates (Wang et al., 28 May 2025).
Another direction is infrastructure for sparse or expensive annotation. Feedback simulation for semantic parsing shows that, in low-data settings, simulated feedback can achieve error-correction performance comparable to training on the full set of human annotations (Yan et al., 2023). BabyAI, Math-Minos, and DRESS all use automatically generated or model-generated feedback to reduce reliance on manual labeling while retaining explicit language supervision (McCallum et al., 2023, 2406.14024, Chen et al., 2023). In robotics and Bayesian Optimization, NLF is becoming a way to communicate deployment-time constraints and subjective utility without redesigning the underlying control or optimization machinery (Santos et al., 2024, Kobalczyk et al., 20 Oct 2025).
A plausible implication is that future NLF systems will be hybrid systems: human-authored feedback for grounding and specificity, learned critics or simulators for scale, and task-specific verifiers or control modules for reliable downstream use. Another plausible implication is that inference-time interfaces may become lighter even as training-time feedback becomes richer, because several lines of work explicitly internalize NLF during training and then remove the requirement for feedback at test time (Chen et al., 2023, McCallum et al., 2023, Salemi et al., 14 Aug 2025). In that sense, NLF is not merely an annotation format; it is an increasingly general mechanism for converting human judgment into operational structure.