Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs

Abstract: Metacognition is a critical component of intelligence that describes the ability to monitor and regulate one's own cognitive processes. Yet LLMs exhibit systemic deficiencies in key metacognitive faculties: they hallucinate with high confidence, fail to recognize knowledge boundaries, and misrepresent their internal uncertainty--undermining trustworthiness and reliability. Since monitoring task performance and adapting behavior accordingly are central to metacognition, we posit that models capable of accurately judging their own performance are better positioned to improve it. We operationalize this idea via two novel mechanisms: reinforcement learning with metacognitive feedback (RLMF), a paradigm to refine completion rankings during preference optimization based on the quality of a model's self-judgments of performance, and metacognitive data selection, which uses similar self-judgments to identify high-value training examples, outperforming naive active learning. We apply these innovations to the problem of faithful calibration (FC), a task that is itself fundamentally metacognitive: the goal is to align expressed with intrinsic uncertainty, difficult even for frontier LLMs. We adopt a two-stage, decoupled approach, first using these methods to calibrate the faithfulness of models' self-reported confidence scores, then mapping to natural, context-adaptable linguistic uncertainty via targeted output editing. Extensive experiments show RLMF achieves generalizable, state-of-the-art FC on diverse tasks while preserving accuracy. Further, RLMF surpasses standard RL by up to 63% while enhancing models' ability to assess and express their own capability limits. This positions RLMF as a promising paradigm to enhance LLM metacognition toward improved abilities and alignment, and suggests metacognitive performance as an effective RL signal to overcome limits of prior intrinsic feedback methods.

• The paper introduces RLMF, a reinforcement learning paradigm that integrates metacognitive feedback to align expressed uncertainty with intrinsic model confidence.
• It employs metacognitive data selection and a two-stage decoupled pipeline to calibrate both numerical and linguistic uncertainty effectively.
• Experimental results show state-of-the-art faithful calibration gains and robust generalization across diverse models and domains.

Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs

Introduction

The work "Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs" (2606.32032) systematically addresses critical metacognitive deficiencies in current LLMs: overconfident hallucinations, inability to recognize epistemic boundaries, and misrepresentation of internal uncertainty. The authors propose and evaluate novel mechanisms—Reinforcement Learning with Metacognitive Feedback (RLMF) and metacognitive data selection—to explicitly enhance the faithful calibration (FC) of LLMs, i.e., the alignment between expressed and intrinsic model uncertainty. Crucially, they depart from conventional factual calibration and directly operationalize metacognitive alignment via a concrete RL framework, rigorously analyzing practical and theoretical implications for trustworthy AI systems across diverse domains.

Figure 1: Overview of RLMF, paired with metacognitive data selection and targeted rewriting to faithfully calibrate the numerically and linguistically expressed uncertainty of LLMs.

Methodological Contributions

Reinforcement Learning with Metacognitive Feedback (RLMF)

The central methodological innovation is RLMF, an RL-based paradigm that simultaneously optimizes task-centric rewards and the fidelity of the model’s self-judgments with respect to its own performance. Rather than rewarding only strong task outputs, RLMF scales the policy gradient advantage according to the accuracy of the model’s metacognitive prediction of its own FC (i.e., how well its expressed confidence matches its intrinsic confidence, measured via sampling-based agreement):

• The metacognitive scale is defined as $$Z_g = 1 - (F^{\text{pred}}_g - F^{\text{gold}}_g)^2$$, where $F^{\text{gold}}_g$ is empirical FC for a completion and $F^{\text{pred}}_g$ is the model's self-reflective prediction of its FC for that completion.
• Advantages for completions surpassing the batch-average faithfulness are scaled by $k + Z_g$, ensuring high faithfulness is not penalized even when metacognition lags.

This mechanism directly encodes metacognitive awareness into the post-training signal, operationalizing the hypothesis that explicit metacognitive monitoring catalyzes model self-improvement.

Figure 2: Overview of the RLMF method, which augments RL optimization with an explicit metacognitive feedback signal.

Metacognitive Data Selection

They further propose a data curation mechanism that leverages the model's own introspective assessment to select training examples from the distribution's tails—maximizing informative exposure. Candidate examples are scored by the model's self-estimated FC; training data are drawn evenly from the highest and lowest self-rated samples, synergistically supporting learning from both confident and uncertain regimes. This method outperforms naive and standard active learning approaches in downstream FC.

Two-Stage Decoupled FC Pipeline

The complete pipeline composes:

1. Stage 1: Application of RLMF and metacognitive data selection to calibrate sentence-level numerical confidence, optimizing FC.
2. Stage 2: A targeted response rewriting mechanism, mapping numerical confidences to context-appropriate linguistic uncertainty markers (hedges), enabling surface realization of uncertainty aligned with underlying introspective calibration. This is performed via prompt-based editing, allowing adaptation to user or task preferences without retraining the base model.

This design explicitly decouples calibration (stage 1) from linguistic realization (stage 2), ensuring sampling diversity, robustness, and contextual adaptability.

Experimental Protocol

Experiments span 10 datasets, including open-domain QA (PopQA), science QA, mathematical reasoning, hallucination detection, and natural language understanding. Evaluated LLMs include Qwen3 (1.7B, 4B, 8B) and Llama3.1-8B-Instruct; baselines comprise MetaFaith metacognitive prompting, SFT-based Faithful Uncertainty Tuning (FUT), and frontier models (Gemini-3.1-Pro, GPT-5).

• The principal metric, cMFG*, refines cMFG to weight FC uniformly over the empirical intrinsic confidence support, correcting for distributional artifacts in previous metrics.
• Additional metrics include factual calibration (Brier Score) and independent LLM-as-a-judge accuracy.

Key Empirical Results

Faithful Calibration Performance

RLMF consistently yields state-of-the-art FC, achieving average cMFG* gains of 29% (vs. MetaFaith) and 25% (vs. FUT), robustly generalizing across diverse models, datasets, and domains—even when trained on a single dataset.

• On PopQA, Llama3.1-8B-Instruct achieves cMFG* 0.84 post-RLMF, compared to 0.77 (RL) and 0.60 (base).
• Qwen3-8B yields 0.83 (RLMF), outperforming proprietary baselines by wide margins.
• Strong performance persists across out-of-domain tasks (e.g., MATH, SimpleQA) with no observable loss in task accuracy.

Reliability and Robustness

On all intrinsic confidence quantiles, RLMF-trained models display tight alignment between expressed and intrinsic uncertainty: reliability diagrams confirm that deviation from FC is minimal and invariant across confidence ranges, unlike prior approaches that systematically degrade at low-uncertainty levels.

Figure 3: Reliability diagrams demonstrating RLMF yields tightly coupled expressed vs. intrinsic confidence (blue) and FC (purple) over different intrinsic confidence bins, outperforming SFT-based baselines.

ablation and Generalization

• Removing metacognitive advantage scaling (i.e., standard RL reward design) materially reduces FC and generalization.
• Metacognitive data selection exceeds both random and active learning sampling in improvement to FC and accuracy.
• Training on disparate domains (math, hallucination, answerability) with RLMF still generalizes FC improvement to all other tasks, indicating broad distributional robustness.

Faithful Linguistic Uncertainty

Stage 2 targeted rewriting yields human-aligned, context-sensitive, and naturalistically diversified hedges, outperforming lexically-constrained SFT baselines in human annotation at up to 98% win rates in diversity, naturalness, helpfulness, and contextual appropriateness.

Figure 4: Frequencies of the top 100 most frequent hedge phrases as used in linguistic surface expression calibration.

Figure 5: Human-annotated confidence distributions for major hedge phrases, sorted by mean perceived confidence.

Figure 6: Per-hedge frequency and mean confidence demonstrate substantial linguistic diversity and calibration.

Theoretical and Practical Implications

The introduction of RLMF establishes faithful calibration as an RL-amenable task, with metacognitive alignment as a first-class optimization criterion. Unlike confidence-based intrinsic feedback, which is susceptible to overfitting and degradation over additional training [see prior RLIF literature], metacognitive feedback operates at a higher abstraction level—penalizing misestimation of internal state—enabling both improved alignment and more stable learning dynamics.

Crucially, the method achieves robust FC without factual calibration collapse, a common failure mode for existing methods that maximize only metacognitive expressiveness.

Practically, such architecture enables deployment of LLMs in safety-critical domains, where true uncertainty reflection, appropriate hedging, and robust epistemic boundaries are requirements (e.g., scientific advising, medical triage, legal consulting). The two-stage pipeline's decoupling also allows for downstream adaptation to user context, domain, cultural linguistic convention, or regulatory standards without retraining.

Limitations and Prospective Research Directions

While the approach makes explicit strides toward robust metacognitive awareness, it focuses primarily on sentence-level FC, with the underlying reward function and sampling-based intrinsic confidence estimation as the critical bottlenecks for further scaling and theoretical understanding. Future research may address:

• Extension to token-level or discourse-level metacognitive tracking.
• Alternative intrinsic state estimation mechanisms with lower variance or differentiable proxies.
• Exploration of richer metacognitive signals (e.g., Bayesian uncertainty, higher-order introspective consistency) as RL reward.
• Sequenced or lifelong self-directed curriculum learning, leveraging metacognitive data selection for adaptive self-improvement.

Conclusion

This work systematically demonstrates that LLMs' metacognitive faculties—specifically, faithful calibration of both numerical and linguistic uncertainty—can be substantially and robustly improved via reinforcement learning with explicit self-performance judgment signals ("metacognitive feedback"). The RLMF paradigm, combined with introspective data selection and targeted rewriting, achieves strong, generalizable, and human-aligned uncertainty expression, outperforming all prior empirical baselines and providing an extensible methodology for next-generation trustworthy LLMs. These findings establish metacognitive supervision as a scalable, stable, and effective RL signal, with broad implications for alignment, robustness, and self-directed learning across deployed AI systems.

Figure 7: Distributions of faithfulness scores: substantial rightward shift and variance reduction for RLMF-trained models, compared to baseline and SFT approaches.

Figure 8: RLMF directly increases metacognitive self-assessment performance ($Z_g$) over training, enabling stable cumulative improvement.