Reinforcement Learning with Metacognitive Feedback

• RLMF is a reinforcement learning approach that integrates metacognitive signals—such as self-assessment, guidance feedback, and internal diagnostics—to refine policy decisions.
• It employs diverse architectures like dual-system evaluation, supervisory control, and teacher–student cooperation to dynamically adjust learning dynamics.
• The paradigm enhances exploration, calibration, and safety in complex tasks by using process-level feedback for strategic adaptations and reward modulation.

Reinforcement Learning with Metacognitive Feedback (RLMF) denotes a family of reinforcement-learning formulations in which the learning process is shaped not only by task reward, but also by feedback about the agent’s own reasoning, exploration, confidence, guidance quality, or learning reliability. Across recent work, this feedback may be supplied by a separate teacher or critic, derived from internal diagnostics such as TD-error stability or memory density, or computed from structured judgments over the model’s own outputs. The common pattern is a meta-level loop that monitors and regulates the base learner, so that action selection and policy improvement are influenced by signals about how the learner is learning, not only by what reward it obtained (Gu, 2024, Zhang et al., 28 Jan 2026, Liu et al., 30 Jun 2026).

1. Conceptual foundations

RLMF is not a single canonical algorithm. A more accurate characterization is that it is an umbrella for RL systems in which metacognitive variables mediate learning dynamics, exploration, or post-hoc self-correction. Several strands of work motivate this view. The meta-decision-making account of metacognitive learning formalizes a meta-level MDP

$$M_{\text{meta}} = \left(\mathcal{B},\ \mathcal{C} \cup \{\bot\},\ T_{\text{meta}},\ r_{\text{meta}}\right),$$

where belief states $b_t$ summarize internal cognitive state, meta-actions $c_t$ are computations, and meta-reward trades off external performance against cognitive cost (He et al., 2023). In that framing, RL is applied not only to overt environment actions but to the selection of computations, stopping decisions, and strategy adjustments.

A complementary algorithmic account is the Metacognitive Actor Critic, which treats metacognition as the discrepancy between a policy-based actor and a value-based critic. The actor selects an action, while the critic evaluates that action relative to the state baseline via

$\delta_{\text{meta}}(S_n, A_n)=Q(S_n, A_n)-V(S_n).$

Negative values indicate that the chosen action is predicted to be worse than baseline, allowing internal error detection without new external information (Schaeffer, 2021). This establishes a general RLMF principle: metacognitive feedback can be an internally generated evaluative signal over the agent’s own decision process.

A third foundational line places metacognition above the reward function itself. In assured autonomy, a metacognitive decision-making layer monitors future safety violations under the current RL policy and proactively adapts reward parameters so that lower-layer RL remains feasible with respect to signal temporal logic constraints (Mustafa et al., 2021). This shifts metacognitive feedback from “evaluate an action” to “regulate the objective under which actions are learned.”

These formulations also clarify an important terminological boundary. In “meta-cognitive reinforcement learning with self-doubt and recovery,” the term does not mean meta-RL in the usual learning-to-learn sense; instead, it denotes explicit monitoring and regulation of the reliability of the learning process itself (Zhang et al., 28 Jan 2026). RLMF therefore concerns process-level self-assessment and control, whether or not it employs meta-learning in the conventional sense.

2. Metacognitive signals and their formal roles

The central design choice in RLMF is the metacognitive signal. Existing work spans several distinct signal families.

One family uses guidance-quality signals in cooperative teacher–student systems. In the BabyAI case study, the RL student follows a policy of the form

$$P_{\theta_t}(a_t \mid x_t, s_t), \qquad x_t \sim M(x_t \mid s_t),$$

where $x_t$ is an LLM-generated token encoding abstract decision information. The RL agent then compares a new advantage estimate $A_t^{P'}$ with the previous $A^P$, and sends positive or negative instruction feedback to the LLM depending on whether the new token improved performance (Gu, 2024). Here the metacognitive signal is binary but explicitly concerns the usefulness of prior guidance.

A second family uses internal reliability diagnostics. The self-doubt-and-recovery framework defines Value Prediction Error Stability

$$\mathrm{VPES}_t = \mathrm{Var}\left(\{\delta_{t-k},\dots,\delta_t\}\right),$$

an EMA baseline

$$\bar v_t=(1-\beta_v)\bar v_{t-1}+\beta_v\,\mathrm{VPES}_t,$$

and a trend variable $b_t$0. These drive a meta-trust variable $b_t$1, updated asymmetrically: $b_t$2 with $b_t$3. Meta-trust then scales the effective learning rate $b_t$4 through a control signal $b_t$5 (Zhang et al., 28 Jan 2026). The metacognitive signal is not confidence in an answer, but confidence in the ongoing learning dynamics.

A third family uses familiarity or novelty signals relative to memory. In memory-density exploration, the agent computes

$b_t$6

the distance of the current observation to its $b_t$7-th nearest neighbor in memory. The normalized density feedback $b_t$8 is fed into a recurrent policy state together with action, reward, and observation: $b_t$9 The policy thereby learns over trajectories of its own familiarity judgments rather than over state novelty alone (McKee, 4 Mar 2025). The metacognitive signal is “how familiar is my present observation with respect to my memory?”

A fourth family uses knowledge and regulation signals in LLM rollouts. Metacognition-as-Reward scaffolds outputs into metacognitive knowledge, metacognitive regulation, optional LOOKBACK, and final answer, then scores them via

$c_t$0

KMR measures coverage and recovery of gold knowledge units, RMR measures fidelity between plan and answer, and CR scores final-answer correctness (Chen et al., 22 May 2026). In this case the metacognitive signal is explicitly structured and judged at trajectory level.

A fifth family uses self-judged metacognitive accuracy. For faithful calibration, the model predicts how faithful its own confidence scores are, and that prediction is compared to gold faithful calibration through

$c_t$1

The resulting metacognitive performance score scales preference-optimization advantages for already strong completions (Liu et al., 30 Jun 2026). This is a second-order signal: not confidence itself, but the accuracy of confidence self-evaluation.

3. Architectural patterns

The literature suggests several recurring RLMF architectures rather than a single dominant template.

Pattern	Core mechanism	Representative papers
Dual-system evaluation	Actor selects; critic evaluates via $c_t$2	(Schaeffer, 2021)
Supervisory control layer	Meta-layer adapts reward or learning-rate parameters	(Mustafa et al., 2021, Zhang et al., 28 Jan 2026)
Teacher–student cooperation	LLM guides RL; RL critiques guidance tokens	(Gu, 2024)
Memory-mediated self-monitoring	Recurrent policy conditions on density/familiarity traces	(McKee, 4 Mar 2025)
Language-critic bilevel feedback	Critic generates reward-improving textual feedback for actor	(Singh et al., 23 May 2026)
Test-time meta-reasoning	Meta-reasoner updates rule memory used by action policy	(Li et al., 28 Nov 2025)

In supervisory-control architectures, the metacognitive module modulates the learning process rather than the action distribution directly. This is explicit in both the trust-recovery framework, where $c_t$3 gates the learning rate, and in assured autonomy, where the higher layer changes reward parameters $c_t$4, $c_t$5, and $c_t$6 to preserve safety feasibility (Zhang et al., 28 Jan 2026, Mustafa et al., 2021).

In cooperative multi-agent architectures, the metacognitive loop is bidirectional. The BabyAI framework describes the interaction as recursive help—“I help you help I help”—with the LLM supplying abstract planning information and the RL model returning positive or negative instruction feedback according to advantage change (Gu, 2024). The critic is not a passive evaluator; it is updated through the student’s ongoing performance.

In language-mediated architectures, the metacognitive channel is itself natural language. Bi-NAC formalizes the coupling between a critic $c_t$7 that outputs textual feedback $c_t$8 and an actor $c_t$9 that refines an answer conditioned on $\delta_{\text{meta}}(S_n, A_n)=Q(S_n, A_n)-V(S_n).$0. The system is posed as a Stackelberg bilevel program,

$\delta_{\text{meta}}(S_n, A_n)=Q(S_n, A_n)-V(S_n).$1

so that feedback is optimized for downstream policy improvement rather than for standalone correctness (Singh et al., 23 May 2026).

Hierarchical memory-based architectures extend this idea to test-time adaptation. MCTR separates a meta-reasoning module, which updates a bounded natural-language knowledge memory $\delta_{\text{meta}}(S_n, A_n)=Q(S_n, A_n)-V(S_n).$2, from an action-reasoning module whose policy is conditioned on that evolving memory. Reflection intervals are adaptively scheduled, and the action policy is updated by test-time RL using self-consistency rewards (Li et al., 28 Nov 2025). A plausible implication is that RLMF can be viewed as a slow-timescale controller over the effective state representation available to a fast-timescale RL policy.

4. Optimization paradigms and reward construction

RLMF methods differ most sharply in how metacognitive signals are converted into optimization targets.

Some methods retain standard RL objectives but modulate update dynamics. In meta-trust control, the base RL objective remains

$\delta_{\text{meta}}(S_n, A_n)=Q(S_n, A_n)-V(S_n).$3

but metacognitive variables determine when and how strongly updates are permitted. The method therefore changes optimization geometry rather than task definition (Zhang et al., 28 Jan 2026).

Other methods change the reward function. MaR is explicit: process-level metacognitive quantities are added directly to outcome reward through $\delta_{\text{meta}}(S_n, A_n)=Q(S_n, A_n)-V(S_n).$4, and PPO-style DAPO updates are then run on group-normalized advantages computed from this composite signal (Chen et al., 22 May 2026). The resulting reward is neither purely external nor purely intrinsic; it combines gold knowledge coverage, regulation fidelity, and answer correctness.

A third line converts metacognitive signals into preferences. Reinforcement Learning from Self-Feedback computes a span-level confidence score

$\delta_{\text{meta}}(S_n, A_n)=Q(S_n, A_n)-V(S_n).$5

over answer tokens, ranks chain-of-thought traces by this disparity-based confidence, and then uses the induced synthetic preferences for PPO or DPO (Niekerk et al., 29 Jul 2025). Reinforcement Learning from Reflective Feedback similarly generates fine-grained critiques and aspect scores over outputs, then builds positive/negative preference pairs for DPO training (Lee et al., 2024). In both cases, the policy is optimized on preferences derived from self- or critic-generated metacognitive judgments rather than solely from human labels or verifiable outcomes.

Text-feedback frameworks occupy an intermediate position between scalar reward and demonstration. RL from Text Feedback formalizes a multi-turn setup in which text feedback is available during training but not at inference, and proposes Self Distillation and Feedback Modeling so that the model internalizes critique into improved single-turn performance (Song et al., 2 Feb 2026). This suggests a broader RLMF design principle: metacognitive text can be treated as a privileged training-time signal even when deployment must remain single-turn.

Finally, some methods use metacognitive performance to reweight policy optimization itself. In faithful calibration, GRPO advantages are decomposed into a faithfulness component and accessory constraints, and the faithfulness component for above-average completions is scaled by the metacognitive accuracy term $\delta_{\text{meta}}(S_n, A_n)=Q(S_n, A_n)-V(S_n).$6. This preserves the primary task objective while preferring completions whose self-assessment is itself accurate (Liu et al., 30 Jun 2026).

5. Empirical domains and reported outcomes

RLMF has been instantiated in gridworlds, continuous control, tutoring systems, test-time vision-language control, and LLM post-training. The empirical picture is heterogeneous but consistently tied to one claim: process-level feedback can alter learning efficiency, robustness, or calibration in ways unavailable to scalar reward alone.

Setting	Reported outcome	Paper
BabyAI GoToRedBallNoDists-v0	Bi-directional LLM↔RL feedback outperforms Lamorel baseline at 40 and 2100 iteration steps and shows expedited convergence	(Gu, 2024)
MuJoCo reward corruption	Recovery-enabled meta-cognitive control reduces late-stage failure rate by 50% compared to Elastic-PPO (0.2 vs 0.4)	(Zhang et al., 28 Jan 2026)
Random maze exploration	Top coverage reaches 100% for the combined observation+feedback policy	(McKee, 4 Mar 2025)
Unseen Atari games	MCTR achieves 9/12 top-1 results on unseen games	(Li et al., 28 Nov 2025)
MATH-500 / GPQA	Bi-NAC 2B reaches 46.6% vs 41.4% for 3B GRPO on MATH-500; 6B reaches 49.3% vs 43.6% for 7B GRPO on GPQA	(Singh et al., 23 May 2026)
Science and medical reasoning	MaR reports up to a 7.7% gain over the base model and up to an 11.0% gain over vanilla DAPO	(Chen et al., 22 May 2026)
Faithful calibration of LLM uncertainty	RLMF surpasses standard RL by up to 63% while preserving accuracy	(Liu et al., 30 Jun 2026)

Educational ITS studies provide a distinct but relevant empirical regime. In one classroom sequence, adaptive DRL-based metacognitive interventions closed the metacognitive skills gap between students, whereas static classifier-based interventions benefited only a subset of students who already knew how to use backward chaining (Abdelshiheed et al., 2023). A related study reports that DRL bridged the gap between declarative, procedural, and conditional metacognitive knowledge, improved performance on both tutors, and made strategic decisions more autonomous over time (Abdelshiheed et al., 2023). These results are notable because the metacognitive signal is not self-confidence or textual critique but strategy-selection behavior itself.

LLM post-training results further broaden the scope of RLMF. RLSF improves both calibration and reasoning without human labels, gold answers, or externally curated rewards by using confidence-derived synthetic preferences (Niekerk et al., 29 Jul 2025). RLRF improves factuality and GSM8K-style mathematical reasoning by replacing coarse preferences with fine-grained reflective feedback over explicit criteria (Lee et al., 2024). RLMF for faithful uncertainty expression produces cMFG* values at or above 0.8 across diverse tasks and, in the reported comparisons, exceeds MetaFaith, FUT, and even MetaFaith-prompted frontier models on the target metric (Liu et al., 30 Jun 2026).

Taken together, these findings suggest that RLMF is especially effective when the performance bottleneck is sparse or delayed supervision, poor self-calibration, unstable learning, or the need to transfer strategy knowledge across contexts.

6. Limitations, controversies, and open directions

A recurring limitation is incomplete formalization of the metacognitive update. The BabyAI teacher–student paper specifies how advantage comparisons yield positive or negative instruction feedback, but does not detail the exact LLM loss or provide convergence guarantees for the coupled loop (Gu, 2024). Similar gaps appear elsewhere whenever natural-language feedback is treated as actionable without a full mechanistic account of how it is internalized.

A second limitation is signal quality and calibration. VPES is explicitly a practical stability indicator rather than a Lyapunov function, and its meta-parameters $\delta_{\text{meta}}(S_n, A_n)=Q(S_n, A_n)-V(S_n).$7 are hand-tuned (Zhang et al., 28 Jan 2026). Memory-density feedback depends on $\delta_{\text{meta}}(S_n, A_n)=Q(S_n, A_n)-V(S_n).$8-nearest-neighbor geometry and suffers from scalability and distance-quality issues in high-dimensional observation spaces (McKee, 4 Mar 2025). In MAC, metacognitive judgments depend on the quality of learned $\delta_{\text{meta}}(S_n, A_n)=Q(S_n, A_n)-V(S_n).$9 and $$P_{\theta_t}(a_t \mid x_t, s_t), \qquad x_t \sim M(x_t \mid s_t),$$0; a miscalibrated critic will produce unreliable self-evaluation (Schaeffer, 2021).

A third issue is external dependence. Bi-NAC relies on verifiable rewards and a separate critic capable of generating useful feedback (Singh et al., 23 May 2026). MaR depends on gold knowledge units and strong LLM graders, introducing annotation cost and grader-bias risk (Chen et al., 22 May 2026). Faithful-calibration RLMF requires repeated sampling, judge models, and online self-evaluation during training, making it substantially more expensive than simple SFT or scalar-reward RL (Liu et al., 30 Jun 2026).

A fourth issue concerns reward hacking and bootstrap bias. Self-generated signals can amplify pre-existing biases if confidence is only weakly correlated with correctness or if the base model is miscalibrated. RLSF explicitly notes bootstrap dependence, bias amplification, and reliance on answer-span identification (Niekerk et al., 29 Jul 2025). RLRF also shows that reward-only RLHF can improve stylistic metrics while failing to improve factuality or reasoning, which is precisely the kind of superficial alignment that process-level metacognitive feedback was proposed to address (Lee et al., 2024).

A fifth issue is conceptual scope. Some papers use “meta-cognitive” to mean internal reliability regulation, others to mean teacher–student feedback, still others to mean rewardizing knowledge and regulation traces or learning from natural-language critique. This suggests that RLMF is currently a research program organized around a shared principle—process-aware feedback—rather than a settled formalism.

Several open directions recur across the literature. One is richer multi-turn reflection: trajectory-level critiques, multi-step returns, and iterative self-repair rather than binary or one-shot signals (Gu, 2024, Li et al., 28 Nov 2025). Another is broader control channels: exploration, replay, model usage, architecture, or computational budget, not only reward weights or learning rates (Zhang et al., 28 Jan 2026). A third is learning the metacognitive controller itself—its signal extractor, trust dynamics, or feedback policy—rather than hand-coding them (Zhang et al., 28 Jan 2026, Singh et al., 23 May 2026). A fourth is transfer: several results already indicate cross-domain generalization, but the extent to which metacognitive competence can be learned once and reused across reasoning, control, and multimodal settings remains an open empirical question (Chen et al., 22 May 2026, Liu et al., 30 Jun 2026).

RLMF therefore occupies a distinctive position within contemporary RL research. It neither reduces to standard reward shaping nor coincides with meta-RL in the narrow sense. Its defining commitment is that agents can be improved by signals about the quality of their own learning and reasoning processes, whether those signals are numerical, structural, or linguistic. The current literature suggests that this commitment is already algorithmically productive, but that its theoretical unification remains incomplete.