Reinforcement Learning from Self-Feedback

• Reinforcement Learning from Self-Feedback is a framework where models use their self-generated evaluations—such as critiques, confidence scores, and teacher distributions—as internal reward signals.
• It encompasses various algorithmic patterns, including verifiable self-verification, reflective critique for preference learning, and feedback-conditioned self-distillation to refine model outputs.
• Empirical results show that integrating on-policy self-feedback can improve both reasoning and verification accuracies, demonstrating measurable gains over traditional reinforcement approaches.

Searching arXiv for papers on Reinforcement Learning from Self-Feedback and closely related formulations. Search query: "Reinforcement Learning from Self-Feedback LLMs arXiv" Reinforcement Learning from Self-Feedback (RLSF) denotes a class of reinforcement-learning and post-training procedures in which the learning signal is generated from the system’s own behavior, judgments, or retrospective analyses, rather than being supplied solely by humans or by a separately curated external reward pipeline. In recent LLM work, this idea appears in several distinct forms: self-verification trajectories trained online with verifiable rewards; reflective critiques that filter positive and negative examples for preference optimization; intrinsic confidence scores converted into synthetic preferences; feedback-conditioned self-distillation that turns retrospective analysis into dense token-level supervision; and hindsight intrinsic rewards plus reusable language memories for agentic adaptation (Liu et al., 19 May 2025, Lee et al., 2024, Niekerk et al., 29 Jul 2025, Hübotter et al., 28 Jan 2026, Zhang et al., 9 Mar 2026).

1. Scope, terminology, and conceptual boundaries

Across the recent literature, RLSF is not a single algorithm but a design principle: the model’s own outputs are reused as evaluative objects, supervisory signals, or training targets. In “Trust, But Verify,” RLSF is the setting in which a model’s own critiques and verification scores become a second set of on-policy trajectories that produce verifiable rewards alongside ordinary solution trajectories (Liu et al., 19 May 2025). In “Reinforcement Learning from Reflective Feedback,” the feedback model produces structured critiques and scores, and those reflective outputs are used to refine responses and construct DPO pairs (Lee et al., 2024). In “Post-Training LLMs via Reinforcement Learning from Self-Feedback,” the model’s own answer-span confidence is used as an intrinsic reward proxy for preference learning (Niekerk et al., 29 Jul 2025). In “Reinforcement Learning via Self-Distillation,” the current policy conditioned on feedback acts as a self-teacher whose next-token distribution is distilled back into the deployment policy (Hübotter et al., 28 Jan 2026). In “RetroAgent,” hindsight self-reflection generates both intrinsic numerical feedback and intrinsic language feedback (Zhang et al., 9 Mar 2026).

The literature also makes clear that RLSF borders, but does not coincide, with RLHF and RLAIF. RLHF uses human preference labels; RLAIF replaces those labels with AI judgments; RLSF is the special case in which the system uses itself, or an internal model tightly coupled to the current policy or reward model, to generate or curate feedback (Li et al., 26 May 2025). This boundary is porous in practice. RL-SaLLM-F uses an LLM to produce online pairwise preferences and self-augmented imagined trajectories without privileged online rewards, which places it in an adjacent self/AI-feedback regime rather than a purely verifier-grounded one (Tu et al., 2024).

A terminological caveat is necessary. “RLSF: Fine-tuning LLMs via Symbolic Feedback” explicitly states that its acronym means Reinforcement Learning via Symbolic Feedback, not self-feedback. That work uses sound, externally grounded symbolic tools such as compilers and computer algebra systems, and it contrasts this external symbolic supervision with model-generated self-feedback (Jha et al., 2024). By contrast, an older non-LLM antecedent, “Feedback-Based Tree Search,” already exhibited the core self-feedback pattern by using the agent’s own planning outputs to supervise subsequent policy and value updates (Jiang et al., 2018).

2. Recurrent algorithmic patterns

Recent RLSF work falls into a small number of recurring algorithmic patterns. The unifying feature is that the system reuses internally generated artifacts—ratings, critiques, preferences, teacher distributions, potential scores, or planning outputs—as learning signals, but the training objective and feedback geometry differ substantially.

Pattern	Self-feedback object	Representative work
Verifiable self-verification	Critiques and boxed ratings on on-policy solutions	RISE (Liu et al., 19 May 2025)
Preference learning from self/AI feedback	Pairwise labels and imagined higher-quality segments	RL-SaLLM-F (Tu et al., 2024)
Difficulty-organized AI/self feedback	Curriculum over contrastive, bridging, and random pairs	Curriculum-RLAIF (Li et al., 26 May 2025)
Reflective filtering for DPO	Rubric-based critiques and self-refined answers	RLRF (Lee et al., 2024)
Intrinsic-confidence preference construction	Confidence-ranked answer spans from multiple CoT traces	Post-Training RLSF (Niekerk et al., 29 Jul 2025)
Feedback-conditioned self-distillation	Teacher logits under retrospective feedback	SDPO (Hübotter et al., 28 Jan 2026)
Hindsight intrinsic reward and memory	Potential scores and distilled language lessons	RetroAgent (Zhang et al., 9 Mar 2026)

Two structural distinctions recur. First, some methods keep self-feedback inside an on-policy RL loop. RISE mixes solve and self-verify trajectories in a single PPO update, and RetroAgent mixes base-policy and memory-augmented rollouts inside GRPO (Liu et al., 19 May 2025, Zhang et al., 9 Mar 2026). Second, other methods use self-feedback primarily to construct preference data or training curricula, after which standard reward-model learning, DPO, PPO, or SAC proceeds on that derived dataset (Tu et al., 2024, Lee et al., 2024, Li et al., 26 May 2025, Niekerk et al., 29 Jul 2025).

A further distinction is whether feedback is externally checkable. RISE is verifier-grounded: self-feedback becomes reward-bearing only because an outcome verifier can check the predicted verification score exactly (Liu et al., 19 May 2025). SDPO occupies a different regime: it assumes tokenized environment feedback such as runtime errors or failing test summaries, then converts that feedback into dense logit-level supervision via a self-teacher, without training a separate reward model (Hübotter et al., 28 Jan 2026). Confidence-based RLSF is more intrinsically generated still, since it relies on the model’s own answer-span probability structure rather than an external verifier (Niekerk et al., 29 Jul 2025).

3. Verifiable self-feedback and online self-verification

The clearest verifier-grounded formulation of RLSF is RISE, which explicitly reframes self-verification as self-feedback and integrates it into the online RL loop. The policy is the LLM $T_\theta$. For an input $x$, the state $s_t$ is the prompt plus the partial response, the action $a_t$ is the next token $y_t$, and PPO is applied over a batch formed by the union of solution and verification trajectories. The policy gradient is

$$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim D, y\sim T_\theta} \left[ \sum_{t=0}^{T} \nabla_\theta \log T_\theta(y_t \mid x, y_{<t}) A_t \right],$$

with GAE, $\lambda = \gamma = 1$, $r_t = 0$ for $t < T-1$, and the scalar outcome reward placed only at the final token. The actor clip ratio is $0.2$, the critic clip range is $x$0, and the KL penalty is disabled during training, effectively $x$1 (Liu et al., 19 May 2025).

RISE uses two verifiable rewards. For solution trajectories, the outcome verifier returns

$x$2

For verification trajectories, the model critiques its own generated solution and outputs a final boxed score intended to match $x$3. The verification reward is

$x$4

This architecture is deliberately unified: RISE adds no separate head or task-specific module, and the same decoder policy learns both problem solving and self-verification under a single PPO objective (Liu et al., 19 May 2025).

The online algorithm samples $x$5 solution rollouts per problem, forms verification prompts from on-policy triples $x$6, samples $x$7 verification responses per prompt, concatenates generation and verification triples, and updates actor and critic jointly. The default setup uses Qwen2.5 models at 1.5B, 3B, and 7B; a train batch size of 1024; mini-batch size 128; actor learning rate $x$8; critic learning rate $x$9; 12 epochs over MATH-Hard Level 3–5; and maximum response length 3,000 tokens (Liu et al., 19 May 2025).

Empirically, RISE improves both solving and self-verification. Relative to Zero-RL, average reasoning accuracy rises from $s_t$0 to $s_t$1 at 1.5B, from $s_t$2 to $s_t$3 at 3B, and from $s_t$4 to $s_t$5 at 7B, while self-verification accuracy rises from $s_t$6 to $s_t$7, from $s_t$8 to $s_t$9, and from $a_t$0 to $a_t$1, respectively. Online verification matters materially: on the 3B model, replacing online verification with offline verification leaves reasoning accuracy similar but drops verification accuracy from $a_t$2 to $a_t$3. Verification compute also matters: increasing the verification batch ratio $a_t$4 improves verification accuracy monotonically, and for 7B the average reasoning accuracy rises from about $a_t$5 to $a_t$6. At inference, verification-weighted majority voting improves over plain majority voting, and at $a_t$7 self-verify the 7B model reaches $a_t$8 versus Zero-RL’s $a_t$9 (Liu et al., 19 May 2025).

RISE is therefore a particularly strict instance of RLSF: the model’s own critique is the feedback, but the fidelity of that feedback is itself externally checkable. That combination sharply distinguishes it from heuristic self-reflection or post-hoc self-consistency.

4. Preference construction from reflection, self-augmentation, and intrinsic confidence

A second major RLSF line constructs preference data from internally generated evaluations or revisions rather than mixing additional trajectories directly into PPO. In RL-SaLLM-F, preference-based RL is performed online in MetaWorld without privileged online rewards. State-only segments of length $y_t$0 are converted into text, an LLM provides pairwise preferences, and a reward model is trained with the Bradley–Terry/logistic form

$y_t$1

with cross-entropy loss over accepted labels. Two mechanisms are central: a double-check rule, which queries the same comparison in both orders and accepts only if $y_t$2, and self-augmentation, in which the LLM generates a new imagined segment $y_t$3 starting from the same initial state as the preferred segment and asserts $y_t$4. On Button Press, agreement with a scripted teacher rises from about $y_t$5 with a single LLM pass to $y_t$6 with double-check and to $y_t$7 with double-check plus self-augmentation. Across eight MetaWorld tasks, RL-SaLLM-F achieves comparable or superior success rates to PEBBLE, and on Button Press and Drawer Open it is on par with SAC trained on actual task rewards (Tu et al., 2024).

RLRF uses a more explicitly reflective construction. A feedback model provides structured critique along aspects such as Factuality, Logical Correctness, Metacognition, Insightfulness, Completeness, Comprehension, Readability, and Harmlessness, with scores in $y_t$8. The model first samples $y_t$9 candidate answers, selects the most promising seed by a preference-based reward model, then generates $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim D, y\sim T_\theta} \left[ \sum_{t=0}^{T} \nabla_\theta \log T_\theta(y_t \mid x, y_{<t}) A_t \right],$$0 self-refined responses conditioned on the feedback. Positives are responses whose selected aspect ratings are all $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim D, y\sim T_\theta} \left[ \sum_{t=0}^{T} \nabla_\theta \log T_\theta(y_t \mid x, y_{<t}) A_t \right],$$1; negatives have at least one $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim D, y\sim T_\theta} \left[ \sum_{t=0}^{T} \nabla_\theta \log T_\theta(y_t \mid x, y_{<t}) A_t \right],$$2 or $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim D, y\sim T_\theta} \left[ \sum_{t=0}^{T} \nabla_\theta \log T_\theta(y_t \mid x, y_{<t}) A_t \right],$$3. DPO is then applied to pairs built from top-ranked positives and sampled negatives. With $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim D, y\sim T_\theta} \left[ \sum_{t=0}^{T} \nabla_\theta \log T_\theta(y_t \mid x, y_{<t}) A_t \right],$$4, specifically $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim D, y\sim T_\theta} \left[ \sum_{t=0}^{T} \nabla_\theta \log T_\theta(y_t \mid x, y_{<t}) A_t \right],$$5, this procedure raises GSM8K accuracy from $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim D, y\sim T_\theta} \left[ \sum_{t=0}^{T} \nabla_\theta \log T_\theta(y_t \mid x, y_{<t}) A_t \right],$$6 in the initial model to $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim D, y\sim T_\theta} \left[ \sum_{t=0}^{T} \nabla_\theta \log T_\theta(y_t \mid x, y_{<t}) A_t \right],$$7 after one DPO stage and to $$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim D, y\sim T_\theta} \left[ \sum_{t=0}^{T} \nabla_\theta \log T_\theta(y_t \mid x, y_{<t}) A_t \right],$$8 after a second-stage RS$$\nabla_\theta J(\theta) = \mathbb{E}_{x\sim D, y\sim T_\theta} \left[ \sum_{t=0}^{T} \nabla_\theta \log T_\theta(y_t \mid x, y_{<t}) A_t \right],$$9DPO pipeline; FactScore rises from $\lambda = \gamma = 1$0 to $\lambda = \gamma = 1$1 and then $\lambda = \gamma = 1$2. Reward-only RLHF baselines improve Just-Eval marginally but do not improve, and sometimes degrade, factuality or math, which the paper treats as evidence against purely surface-level alignment (Lee et al., 2024).

Confidence-based post-training instantiates RLSF through intrinsic model uncertainty rather than explicit critique. A frozen policy generates $\lambda = \gamma = 1$3 chain-of-thought traces by taking the top-$\lambda = \gamma = 1$4 first tokens and greedily decoding each branch; the phrase “So the answer is” is appended to elicit answer continuations, and the final answer span is then matched inside the original hypothesis. The confidence score is a span-level probability disparity,

$\lambda = \gamma = 1$5

where $\lambda = \gamma = 1$6 is answer-span length. Traces are ranked by $\lambda = \gamma = 1$7, synthetic preference pairs are formed, and either a Bradley–Terry reward model plus PPO or DPO is used. The paper reports that RLSF(PPO) generally outperforms RLSF(DPO). On CommonsenseQA with Phi-2, greedy decoding yields accuracy $\lambda = \gamma = 1$8 and ECE $\lambda = \gamma = 1$9, CoT(10) yields $r_t = 0$0 and $r_t = 0$1, and RLSF(PPO) with greedy inference yields $r_t = 0$2 and $r_t = 0$3. On ARC Easy with Gemma 2, greedy gives $r_t = 0$4 and $r_t = 0$5, while RLSF(PPO) gives $r_t = 0$6 and $r_t = 0$7. The central claim is that calibration and accuracy can improve without human labels, gold answers, or external reward models, though the method depends on reliable answer-span extraction (Niekerk et al., 29 Jul 2025).

Curriculum-RLAIF sits at the edge of RLSF proper but is important for understanding the broader data-centric view of self/AI feedback. It constructs easy-to-hard preference curricula over four stages—$r_t = 0$8—using contrastive, bridging, and random pairs. The pairwise reward-model loss is

$r_t = 0$9

For Gemma-1-2B, Curriculum-RLAIF reaches policy win rates $t < T-1$0 on harmlessness/helpfulness/summary versus $t < T-1$1 for Conventional RLAIF; for LLaMA-3-8B, it reaches $t < T-1$2 versus $t < T-1$3. The method changes only training data scheduling, incurs no extra inference-time cost at deployment, and reduces labeling compute to $t < T-1$4 of conventional RLAIF by making three of four stages annotation-free (Li et al., 26 May 2025).

5. Dense retrospective credit assignment, planning feedback, and agentic memory

A third line of work treats self-feedback as a mechanism for dense credit assignment rather than preference construction. SDPO formalizes reinforcement learning with rich feedback, where a failed attempt may come with runtime errors, failing test summaries, judge evaluations, or a successful sample solution from the same rollout group. The current model, conditioned on that feedback, becomes a self-teacher. Instead of sampling a corrected trajectory, SDPO recomputes next-token probabilities for the original attempt under the feedback-conditioned teacher and distills them into the student policy:

$t < T-1$5

The induced per-logit advantage is

$t < T-1$6

Top-$t < T-1$7 distillation, typically $t < T-1$8–$t < T-1$9, makes this tractable. On LiveCodeBench v6 with Qwen3-8B, SDPO reaches $0.2$0 final accuracy versus GRPO’s $0.2$1, reaches GRPO’s final accuracy in about $0.2$2 fewer generations, and outperforms baselines even in scalar-only RLVR environments by using successful rollouts as implicit feedback for failed ones. At test time, on very hard questions with base pass@64 $0.2$3, discovery@2750 is $0.2$4 for SDPO versus $0.2$5 for best-of-$0.2$6 and $0.2$7 for multi-turn prompting, and the same $0.2$8 discovery probability is reached with about $0.2$9 fewer attempts. The paper also reports that SDPO can underperform GRPO on small models such as Qwen2.5-1.5B, suggesting that self-teacher retrospection is model-capacity dependent (Hübotter et al., 28 Jan 2026).

RetroAgent extends RLSF into agentic RL with two explicit intrinsic channels. After each episode, hindsight self-reflection outputs a tuple $x$00: a potential score $x$01, a predicted success or failure $x$02, and a distilled lesson $x$03. Intrinsic numerical feedback is defined by

$x$04

where $x$05 is the highest group-mean success rate observed so far for task $x$06. Intrinsic language feedback is stored in a memory buffer and retrieved with SimUtil-UCB, whose score is

$x$07

The policy objective adds intrinsic and extrinsic rewards inside GRPO, and the RL-trained reflection variant adds a REINFORCE objective on the reflection policy. Across ALFWorld, WebShop, Sokoban, and MineSweeper, RetroAgent reports improvements over GRPO-trained agents of $x$08, $x$09, $x$10, and $x$11, respectively (Zhang et al., 9 Mar 2026).

Historically, FBTS provides an earlier non-language-model instance of the same principle. There, finite-horizon MCTS with a leaf-node evaluator built from the current policy and value function produces improved root values $x$12, which are then used as cost-sensitive classification targets for the next policy and Monte Carlo regression targets for the next value estimator. The performance guarantee

$x$13

makes explicit that self-feedback can be analyzed as a contraction process up to approximation and concentrability terms (Jiang et al., 2018).

6. Empirical regularities, limitations, and contested interpretations

Several empirical regularities recur across the literature. First, contemporaneous or on-policy self-feedback is consistently more useful than delayed or static feedback. RISE shows that offline verification markedly degrades learned verification accuracy even when reasoning accuracy stays similar, and RL-SaLLM-F shows that double-check plus self-augmentation improves label agreement materially over single-pass labeling (Liu et al., 19 May 2025, Tu et al., 2024). Second, denser feedback typically improves credit assignment. In SDPO, logit-level self-distillation outperforms token-level and sequence-level variants; in verifier-grounded RLSF, increased verification compute scales verification accuracy upward; in reflective preference learning, fine-grained rubric scores outperform reward-only selection on factuality and math (Hübotter et al., 28 Jan 2026, Liu et al., 19 May 2025, Lee et al., 2024).

The same papers also expose hard constraints. Domain dependence is strong: RISE assumes a well-defined verifiable reward and is naturally suited to math, programming, and similar settings; RL-SaLLM-F requires access to state features that can be textualized; confidence-based RLSF requires reliable answer-span extraction; symbolic-feedback RLSF requires compilers, CAS systems, or other certificate-producing tools (Liu et al., 19 May 2025, Tu et al., 2024, Niekerk et al., 29 Jul 2025, Jha et al., 2024). Label noise and ambiguity persist even with improved mechanisms: RL-SaLLM-F reports that even its best variants reach only about $x$14 average agreement across tasks, and Curriculum-RLAIF treats distribution shift, label noise, and difficulty mismatch as a coupled data problem rather than a solved one (Tu et al., 2024, Li et al., 26 May 2025).

Model strength is another nontrivial variable. SDPO’s gains increase with model size and can reverse on smaller models, which indicates that some forms of self-feedback rely on sufficiently strong in-context retrospection (Hübotter et al., 28 Jan 2026). Confidence-based RLSF can also amplify pre-existing biases if answer-span confidence is spuriously correlated with correctness, and the paper explicitly notes overconfidence risk, mode collapse, and sensitivity to span extraction (Niekerk et al., 29 Jul 2025). RetroAgent similarly notes self-confirmation and memory-noise risks, mitigated only partially by utility scoring and retrieval design (Zhang et al., 9 Mar 2026).

A common misconception is that self-feedback is merely post-hoc self-consistency or majority voting. The recent literature points elsewhere. In RISE, the verification skill is trained online with the same PPO objective as solving; in SDPO, the feedback-conditioned distribution is turned into gradients rather than a verbal critique alone; in RLRF, reflection is used to generate better training pairs rather than only to edit individual outputs at inference time (Liu et al., 19 May 2025, Hübotter et al., 28 Jan 2026, Lee et al., 2024). This suggests that the defining property of RLSF is not introspective text by itself, but the conversion of internally generated evaluations into reward-bearing, preference-bearing, or distillation-bearing signals that alter the policy’s parameters.

The conceptual boundary remains contested. One strand reserves RLSF for cases where the model literally critiques or rates its own outputs; another includes AI/self-feedback hybrids such as Curriculum-RLAIF’s internal-evaluation baseline and RL-SaLLM-F’s LLM-generated preferences; a third treats external but fine-grained symbolic certificates as a distinct category entirely (Li et al., 26 May 2025, Tu et al., 2024, Jha et al., 2024). A plausible implication is that RLSF is best understood not as a binary label but as a spectrum whose poles are fully intrinsic feedback and fully external verification, with many practically successful systems occupying mixed positions between the two.