---
title: Reinforcement Learning from Self-Feedback
url: https://www.emergentmind.com/topics/reinforcement-learning-from-self-feedback-rlsf
type: topic
---

# Reinforcement Learning from Self-Feedback

Searching arXiv for recent papers on Reinforcement Learning from Self-Feedback and closely related formulations.
Search query: "Reinforcement Learning from Self-Feedback large language models arXiv"
Reinforcement Learning from Self-Feedback (RLSF) denotes a class of reinforcement-learning and post-training procedures in which the learning signal is generated from the system’s own behavior, judgments, or retrospective analyses, rather than being supplied solely by humans or by a separately curated external reward pipeline. In recent LLM work, this idea appears in several distinct forms: self-verification trajectories trained online with verifiable rewards; reflective critiques that filter positive and negative examples for preference optimization; intrinsic confidence scores converted into synthetic preferences; feedback-conditioned self-distillation that turns retrospective analysis into dense token-level supervision; and hindsight intrinsic rewards plus reusable language memories for agentic adaptation [2505.13445][2403.14238][2507.21931][2601.20802][2603.08561].

## 1. Scope, terminology, and conceptual boundaries

Across the recent literature, RLSF is not a single algorithm but a design principle: the model’s own outputs are reused as evaluative objects, supervisory signals, or training targets. In “Trust, But Verify,” RLSF is the setting in which a model’s own critiques and verification scores become a second set of on-policy trajectories that produce verifiable rewards alongside ordinary solution trajectories [2505.13445]. In “Reinforcement Learning from Reflective Feedback,” the feedback model produces structured critiques and scores, and those reflective outputs are used to refine responses and construct DPO pairs [2403.14238]. In “Post-Training Large Language Models via Reinforcement Learning from Self-Feedback,” the model’s own answer-span confidence is used as an intrinsic reward proxy for preference learning [2507.21931]. In “Reinforcement Learning via Self-Distillation,” the current policy conditioned on feedback acts as a self-teacher whose next-token distribution is distilled back into the deployment policy [2601.20802]. In “RetroAgent,” hindsight self-reflection generates both intrinsic numerical feedback and intrinsic language feedback [2603.08561].

The literature also makes clear that RLSF borders, but does not coincide, with RLHF and RLAIF. RLHF uses human preference labels; RLAIF replaces those labels with AI judgments; RLSF is the special case in which the system uses itself, or an internal model tightly coupled to the current policy or reward model, to generate or curate feedback [2505.20075]. This boundary is porous in practice. RL-SaLLM-F uses an LLM to produce online pairwise preferences and self-augmented imagined trajectories without privileged online rewards, which places it in an adjacent self/AI-feedback regime rather than a purely verifier-grounded one [2412.16878].

A terminological caveat is necessary. “RLSF: Fine-tuning LLMs via Symbolic Feedback” explicitly states that its acronym means *Reinforcement Learning via Symbolic Feedback*, not self-feedback. That work uses sound, externally grounded symbolic tools such as compilers and computer algebra systems, and it contrasts this external symbolic supervision with model-generated self-feedback [2405.16661]. By contrast, an older non-LLM antecedent, “Feedback-Based Tree Search,” already exhibited the core self-feedback pattern by using the agent’s own planning outputs to supervise subsequent policy and value updates [1805.05935].

## 2. Recurrent algorithmic patterns

Recent RLSF work falls into a small number of recurring algorithmic patterns. The unifying feature is that the system reuses internally generated artifacts—ratings, critiques, preferences, teacher distributions, potential scores, or planning outputs—as learning signals, but the training objective and feedback geometry differ substantially.

| Pattern | Self-feedback object | Representative work |
|---|---|---|
| Verifiable self-verification | Critiques and boxed ratings on on-policy solutions | RISE [2505.13445] |
| Preference learning from self/AI feedback | Pairwise labels and imagined higher-quality segments | RL-SaLLM-F [2412.16878] |
| Difficulty-organized AI/self feedback | Curriculum over contrastive, bridging, and random pairs | Curriculum-RLAIF [2505.20075] |
| Reflective filtering for DPO | Rubric-based critiques and self-refined answers | RLRF [2403.14238] |
| Intrinsic-confidence preference construction | Confidence-ranked answer spans from multiple CoT traces | Post-Training RLSF [2507.21931] |
| Feedback-conditioned self-distillation | Teacher logits under retrospective feedback | SDPO [2601.20802] |
| Hindsight intrinsic reward and memory | Potential scores and distilled language lessons | RetroAgent [2603.08561] |

Two structural distinctions recur. First, some methods keep self-feedback inside an on-policy RL loop. RISE mixes solve and self-verify trajectories in a single PPO update, and RetroAgent mixes base-policy and memory-augmented rollouts inside GRPO [2505.13445][2603.08561]. Second, other methods use self-feedback primarily to construct preference data or training curricula, after which standard reward-model learning, DPO, PPO, or SAC proceeds on that derived dataset [2412.16878][2403.14238][2505.20075][2507.21931].

A further distinction is whether feedback is externally checkable. RISE is verifier-grounded: self-feedback becomes reward-bearing only because an outcome verifier can check the predicted verification score exactly [2505.13445]. SDPO occupies a different regime: it assumes tokenized environment feedback such as runtime errors or failing test summaries, then converts that feedback into dense logit-level supervision via a self-teacher, without training a separate reward model [2601.20802]. Confidence-based RLSF is more intrinsically generated still, since it relies on the model’s own answer-span probability structure rather than an external verifier [2507.21931].

## 3. Verifiable self-feedback and online self-verification

The clearest verifier-grounded formulation of RLSF is RISE, which explicitly reframes self-verification as self-feedback and integrates it into the online RL loop. The policy is the language model \(T_\theta\). For an input \(x\), the state \(s_t\) is the prompt plus the partial response, the action \(a_t\) is the next token \(y_t\), and PPO is applied over a batch formed by the union of solution and verification trajectories. The policy gradient is

\[
\nabla_\theta J(\theta) =
\mathbb{E}_{x\sim D, y\sim T_\theta}
\left[
\sum_{t=0}^{T} \nabla_\theta \log T_\theta(y_t \mid x, y_{<t}) A_t
\right],
\]

with GAE, \(\lambda = \gamma = 1\), \(r_t = 0\) for \(t < T-1\), and the scalar outcome reward placed only at the final token. The actor clip ratio is \(0.2\), the critic clip range is \(0.5\), and the KL penalty is disabled during training, effectively \(\beta = 0\) [2505.13445].

RISE uses two verifiable rewards. For solution trajectories, the outcome verifier returns

\[
r_o(y, y^*) =
\begin{cases}
1, & \text{if the final answer is boxed and exactly matches } y^*; \\
-0.5, & \text{if it is boxed but does not match } y^*; \\
-1, & \text{if the final answer is unboxed or missing.}
\end{cases}
\]

For verification trajectories, the model critiques its own generated solution and outputs a final boxed score intended to match \(r_o\). The verification reward is

\[
r_v(x_{\mathrm{ver}}, y_{\mathrm{ver}})=
\mathbb{1}[\mathrm{score\_from\_model}(y_{\mathrm{ver}})=r_o].
\]

This architecture is deliberately unified: RISE adds no separate head or task-specific module, and the same decoder policy learns both problem solving and self-verification under a single PPO objective [2505.13445].

The online algorithm samples \(K=8\) solution rollouts per problem, forms verification prompts from on-policy triples \((x,y,r_o)\), samples \(K=8\) verification responses per prompt, concatenates generation and verification triples, and updates actor and critic jointly. The default setup uses Qwen2.5 models at 1.5B, 3B, and 7B; a train batch size of 1024; mini-batch size 128; actor learning rate \(5 \times 10^{-7}\); critic learning rate \(9 \times 10^{-6}\); 12 epochs over MATH-Hard Level 3–5; and maximum response length 3,000 tokens [2505.13445].

Empirically, RISE improves both solving and self-verification. Relative to Zero-RL, average reasoning accuracy rises from \(24.0\%\) to \(24.4\%\) at 1.5B, from \(32.5\%\) to \(33.5\%\) at 3B, and from \(41.7\%\) to \(42.9\%\) at 7B, while self-verification accuracy rises from \(26.8\%\) to \(74.5\%\), from \(35.8\%\) to \(74.3\%\), and from \(46.6\%\) to \(69.2\%\), respectively. Online verification matters materially: on the 3B model, replacing online verification with offline verification leaves reasoning accuracy similar but drops verification accuracy from \(74.9\%\) to \(65.7\%\). Verification compute also matters: increasing the verification batch ratio \(S \in \{0, 12.5\%, 25\%, 50\%, 100\%\}\) improves verification accuracy monotonically, and for 7B the average reasoning accuracy rises from about \(41.7\%\) to \(42.9\%\). At inference, verification-weighted majority voting improves over plain majority voting, and at \(k=8 +\) self-verify the 7B model reaches \(49.8\%\) versus Zero-RL’s \(48.3\%\) [2505.13445].

RISE is therefore a particularly strict instance of RLSF: the model’s own critique is the feedback, but the fidelity of that feedback is itself externally checkable. That combination sharply distinguishes it from heuristic self-reflection or post-hoc self-consistency.

## 4. Preference construction from reflection, self-augmentation, and intrinsic confidence

A second major RLSF line constructs preference data from internally generated evaluations or revisions rather than mixing additional trajectories directly into PPO. In RL-SaLLM-F, preference-based RL is performed online in MetaWorld without privileged online rewards. State-only segments of length \(H=10\) are converted into text, an LLM provides pairwise preferences, and a reward model is trained with the Bradley–Terry/logistic form

\[
P_\psi(\sigma^1 \succ \sigma^0)
=
\frac{\exp(\sum_t \hat r_\psi(s_t^1))}
{\exp(\sum_t \hat r_\psi(s_t^0)) + \exp(\sum_t \hat r_\psi(s_t^1))},
\]

with cross-entropy loss over accepted labels. Two mechanisms are central: a double-check rule, which queries the same comparison in both orders and accepts only if \(y = 1-y_{\mathrm{inv}}\), and self-augmentation, in which the LLM generates a new imagined segment \(\sigma^{LLM}\) starting from the same initial state as the preferred segment and asserts \(\sigma^{LLM} \succ \sigma^{best}\). On Button Press, agreement with a scripted teacher rises from about \(58.2\%\) with a single LLM pass to \(64.8\%\) with double-check and to \(72.3\%\) with double-check plus self-augmentation. Across eight MetaWorld tasks, RL-SaLLM-F achieves comparable or superior success rates to PEBBLE, and on Button Press and Drawer Open it is on par with SAC trained on actual task rewards [2412.16878].

RLRF uses a more explicitly reflective construction. A feedback model provides structured critique along aspects such as Factuality, Logical Correctness, Metacognition, Insightfulness, Completeness, Comprehension, Readability, and Harmlessness, with scores in \(\{-1,0,1\}\). The model first samples \(n\) candidate answers, selects the most promising seed by a preference-based reward model, then generates \(m\) self-refined responses conditioned on the feedback. Positives are responses whose selected aspect ratings are all \(1\); negatives have at least one \(0\) or \(-1\). DPO is then applied to pairs built from top-ranked positives and sampled negatives. With \(n+m=30\), specifically \(n=10, m=20\), this procedure raises GSM8K accuracy from \(41.77\) in the initial model to \(47.92\) after one DPO stage and to \(51.02\) after a second-stage RS\(\rightarrow\)DPO pipeline; FactScore rises from \(70.79\) to \(78.50\) and then \(79.30\). Reward-only RLHF baselines improve Just-Eval marginally but do not improve, and sometimes degrade, factuality or math, which the paper treats as evidence against purely surface-level alignment [2403.14238].

Confidence-based post-training instantiates RLSF through intrinsic model uncertainty rather than explicit critique. A frozen policy generates \(K=10\) chain-of-thought traces by taking the top-\(K\) first tokens and greedily decoding each branch; the phrase “So the answer is” is appended to elicit answer continuations, and the final answer span is then matched inside the original hypothesis. The confidence score is a span-level probability disparity,

\[
c = \frac{1}{M}\sum_{i=0}^{M-1}
\left(
\max_w \pi(w \mid q \odot h_{m+i}) -
\max_{w \neq w_i^*}\pi(w \mid q \odot h_{m+i})
\right),
\]

where \(M\) is answer-span length. Traces are ranked by \(c\), synthetic preference pairs are formed, and either a Bradley–Terry reward model plus PPO or DPO is used. The paper reports that RLSF(PPO) generally outperforms RLSF(DPO). On CommonsenseQA with Phi-2, greedy decoding yields accuracy \(54.46\) and ECE \(25.12\), CoT(10) yields \(58.91\) and \(23.11\), and RLSF(PPO) with greedy inference yields \(61.13\) and \(19.64\). On ARC Easy with Gemma 2, greedy gives \(96.96\) and \(16.12\), while RLSF(PPO) gives \(97.04\) and \(5.12\). The central claim is that calibration and accuracy can improve without human labels, gold answers, or external reward models, though the method depends on reliable answer-span extraction [2507.21931].

Curriculum-RLAIF sits at the edge of RLSF proper but is important for understanding the broader data-centric view of self/AI feedback. It constructs easy-to-hard preference curricula over four stages—\(D_{ctr} \rightarrow D_{brg^-} \rightarrow D_{brg^+} \rightarrow D_{rnd}\)—using contrastive, bridging, and random pairs. The pairwise reward-model loss is

\[
L_{\mathrm{reward}}^{\mathcal C}
=
-
\mathbb{E}_{(x,y^+,y^-)\sim \mathcal C}
\left[
\log \sigma(r_\theta(x,y^+) - r_\theta(x,y^-))
\right].
\]

For Gemma-1-2B, Curriculum-RLAIF reaches policy win rates \(0.92/0.93/0.87\) on harmlessness/helpfulness/summary versus \(0.84/0.87/0.80\) for Conventional RLAIF; for LLaMA-3-8B, it reaches \(0.93/0.94/0.92\) versus \(0.87/0.89/0.84\). The method changes only training data scheduling, incurs no extra inference-time cost at deployment, and reduces labeling compute to \(25\%\) of conventional RLAIF by making three of four stages annotation-free [2505.20075].

## 5. Dense retrospective credit assignment, planning feedback, and agentic memory

A third line of work treats self-feedback as a mechanism for dense credit assignment rather than preference construction. SDPO formalizes reinforcement learning with rich feedback, where a failed attempt may come with runtime errors, failing test summaries, judge evaluations, or a successful sample solution from the same rollout group. The current model, conditioned on that feedback, becomes a self-teacher. Instead of sampling a corrected trajectory, SDPO recomputes next-token probabilities for the original attempt under the feedback-conditioned teacher and distills them into the student policy:

\[
\mathcal{L}_{\mathrm{SDPO}}(\theta)
=
\sum_{t=1}^{T}
\mathrm{KL}\!\left(
\pi_\theta(\cdot \mid x, y_{<t})
\;\middle\|\;
\mathrm{stopgrad}\big(\pi_\phi(\cdot \mid x, f, y_{<t})\big)
\right).
\]

The induced per-logit advantage is

\[
A^{\mathrm{SDPO}}_t(\hat y_t)
=
\log
\frac{\pi_\phi(\hat y_t \mid x,f,y_{<t})}
{\pi_\theta(\hat y_t \mid x,y_{<t})}.
\]

Top-\(K\) distillation, typically \(K \approx 20\)–\(100\), makes this tractable. On LiveCodeBench v6 with Qwen3-8B, SDPO reaches \(48.8\%\) final accuracy versus GRPO’s \(41.2\%\), reaches GRPO’s final accuracy in about \(4\times\) fewer generations, and outperforms baselines even in scalar-only RLVR environments by using successful rollouts as implicit feedback for failed ones. At test time, on very hard questions with base pass@64 \(< 0.03\), discovery@2750 is \(53.2\%\) for SDPO versus \(41.5\%\) for best-of-\(k\) and \(35.6\%\) for multi-turn prompting, and the same \(\approx 22\%\) discovery probability is reached with about \(3\times\) fewer attempts. The paper also reports that SDPO can underperform GRPO on small models such as Qwen2.5-1.5B, suggesting that self-teacher retrospection is model-capacity dependent [2601.20802].

RetroAgent extends RLSF into agentic RL with two explicit intrinsic channels. After each episode, hindsight self-reflection outputs a tuple \(z=(\phi_{(x,\tau)}, c, m)\): a potential score \(\phi_{(x,\tau)} \in [0,1]\), a predicted success or failure \(c\), and a distilled lesson \(m\). Intrinsic numerical feedback is defined by

\[
R^{int}_k = \max(0, \phi_{(x,\tau),k} - \Phi_x),
\]

where \(\Phi_x\) is the highest group-mean success rate observed so far for task \(x\). Intrinsic language feedback is stored in a memory buffer and retrieved with SimUtil-UCB, whose score is

\[
S(b_i \mid x)
=
\alpha\, s_{rel}(x,x_i)
+
(1-\alpha)\left(u_i + \kappa \sqrt{\frac{\ln N}{n_i}}\right).
\]

The policy objective adds intrinsic and extrinsic rewards inside GRPO, and the RL-trained reflection variant adds a REINFORCE objective on the reflection policy. Across ALFWorld, WebShop, Sokoban, and MineSweeper, RetroAgent reports improvements over GRPO-trained agents of \(+18.3\%\), \(+15.4\%\), \(+27.1\%\), and \(+8.9\%\), respectively [2603.08561].

Historically, FBTS provides an earlier non-language-model instance of the same principle. There, finite-horizon MCTS with a leaf-node evaluator built from the current policy and value function produces improved root values \(\hat U_k(s)\), which are then used as cost-sensitive classification targets for the next policy and Monte Carlo regression targets for the next value estimator. The performance guarantee

\[
\|V^* - V^{\pi_K}\|_{1,\nu}
\le
B_\gamma\big[B'_\gamma \mathbb{D}_0(\bar\Pi,\bar{\mathcal F}) + \mathbb{D}_1^d(\bar\Pi)\big]
+
\gamma^{Kd}\|V^* - V^{\pi_0}\|_\infty
+
\epsilon
\]

makes explicit that self-feedback can be analyzed as a contraction process up to approximation and concentrability terms [1805.05935].

## 6. Empirical regularities, limitations, and contested interpretations

Several empirical regularities recur across the literature. First, contemporaneous or on-policy self-feedback is consistently more useful than delayed or static feedback. RISE shows that offline verification markedly degrades learned verification accuracy even when reasoning accuracy stays similar, and RL-SaLLM-F shows that double-check plus self-augmentation improves label agreement materially over single-pass labeling [2505.13445][2412.16878]. Second, denser feedback typically improves credit assignment. In SDPO, logit-level self-distillation outperforms token-level and sequence-level variants; in verifier-grounded RLSF, increased verification compute scales verification accuracy upward; in reflective preference learning, fine-grained rubric scores outperform reward-only selection on factuality and math [2601.20802][2505.13445][2403.14238].

The same papers also expose hard constraints. Domain dependence is strong: RISE assumes a well-defined verifiable reward and is naturally suited to math, programming, and similar settings; RL-SaLLM-F requires access to state features that can be textualized; confidence-based RLSF requires reliable answer-span extraction; symbolic-feedback RLSF requires compilers, CAS systems, or other certificate-producing tools [2505.13445][2412.16878][2507.21931][2405.16661]. Label noise and ambiguity persist even with improved mechanisms: RL-SaLLM-F reports that even its best variants reach only about \(70\%\) average agreement across tasks, and Curriculum-RLAIF treats distribution shift, label noise, and difficulty mismatch as a coupled data problem rather than a solved one [2412.16878][2505.20075].

Model strength is another nontrivial variable. SDPO’s gains increase with model size and can reverse on smaller models, which indicates that some forms of self-feedback rely on sufficiently strong in-context retrospection [2601.20802]. Confidence-based RLSF can also amplify pre-existing biases if answer-span confidence is spuriously correlated with correctness, and the paper explicitly notes overconfidence risk, mode collapse, and sensitivity to span extraction [2507.21931]. RetroAgent similarly notes self-confirmation and memory-noise risks, mitigated only partially by utility scoring and retrieval design [2603.08561].

A common misconception is that self-feedback is merely post-hoc self-consistency or majority voting. The recent literature points elsewhere. In RISE, the verification skill is trained online with the same PPO objective as solving; in SDPO, the feedback-conditioned distribution is turned into gradients rather than a verbal critique alone; in RLRF, reflection is used to generate better training pairs rather than only to edit individual outputs at inference time [2505.13445][2601.20802][2403.14238]. This suggests that the defining property of RLSF is not introspective text by itself, but the conversion of internally generated evaluations into reward-bearing, preference-bearing, or distillation-bearing signals that alter the policy’s parameters.

The conceptual boundary remains contested. One strand reserves RLSF for cases where the model literally critiques or rates its own outputs; another includes AI/self-feedback hybrids such as Curriculum-RLAIF’s internal-evaluation baseline and RL-SaLLM-F’s LLM-generated preferences; a third treats external but fine-grained symbolic certificates as a distinct category entirely [2505.20075][2412.16878][2405.16661]. A plausible implication is that RLSF is best understood not as a binary label but as a spectrum whose poles are fully intrinsic feedback and fully external verification, with many practically successful systems occupying mixed positions between the two.

Source: https://www.emergentmind.com/topics/reinforcement-learning-from-self-feedback-rlsf