Metacognition in LLMs: Foundations, Progress, and Opportunities

Abstract: Metacognition is a foundational component of intelligence critical to effective learning, problem solving, decision-making, communication, and more. In recent years, it has become increasingly recognized as a cornerstone of capable, transparent AI systems. Yet while LLMs have made significant progress across diverse real-world tasks, it is not yet clear when, how, or to what extent they can exhibit or be endowed with effective metacognitive abilities, nor how such abilities can be adapted to advance the fundamental capabilities, reliability, and intelligence of AI systems. This paper bridges this gap by presenting the first comprehensive overview of the current state of knowledge on metacognition for LLMs. We analyze and taxonomize the landscape of this emerging field and summarize recent technical advancements, including methods and benchmarks to measure and evaluate LLMs' metacognitive abilities, techniques to elicit, improve, and apply metacognition in LLMs, and findings and implications of ongoing research. We also discuss applications, open questions and challenges, and promising directions for future work. Our aim is to provide a detailed and up-to-date review of this topic and stimulate meaningful research and discussion. An organized list of papers can be found at https://github.com/yale-nlp/LLM-Metacognition.

• The paper introduces a unified taxonomy for LLM metacognition, distinguishing between introspective monitoring and strategic control.
• It evaluates measurement techniques using SDT metrics, neurofeedback, and calibration methods to quantify uncertainty and self-assessment.
• It identifies empirical limitations and proposes architectural and training interventions to improve LLM safety, reliability, and interpretability.

Metacognition in LLMs: Foundations, Progress, and Opportunities

Introduction and Taxonomy of LLM Metacognition

Metacognition, the suite of abilities enabling systems to monitor and regulate their own cognition, constitutes a foundational component of intelligence and adaptive behavior. Its systematic study in the context of LLMs is gaining prominence as the field seeks to endow AI systems with more transparent, reliable, and robust cognitive faculties. The paper "Metacognition in LLMs: Foundations, Progress, and Opportunities" (2607.11881) provides the first unified, in-depth review of the conceptual, empirical, and methodological landscape on LLM metacognition, introducing frameworks for evaluating, endowing, and leveraging metacognitive faculties.

The authors delineate a comprehensive taxonomy of metacognitive functions in LLMs, structured around two principal axes: monitoring (introspective judgment of uncertainty, performance, and cognitive state) and control (strategic modulation such as plan adaptation, resource allocation, or self-correction informed by monitoring) Figure 1.

Figure 1: Taxonomy of research on metacognition in LLMs, partitioning the field into monitoring and control processes and their subcomponents.

Measuring and Benchmarking LLM Metacognition

The measurement of metacognition in LLMs has evolved through both adaptation of psychologically-grounded paradigms and development of domain-specific methodologies:

• Signal Detection Theory (SDT) Metrics: Meta-$d'$, $d'$, M-ratio are adapted as core metrics, decoupling metacognitive sensitivity from cognitive ability and response bias, enabling principled quantification of how confidently models distinguish between correct and incorrect responses.
• Neurofeedback and Introspective Probing: Analogous to neuropsychological experiments, approaches leveraging internal activations and neurofeedback-inspired environments have been used to elicit and analyze introspective competence, going beyond mere output calibration.
• Confidence-Based and Calibration Metrics: AUROC, AUPR, and expected calibration error (ECE) provide statistical measures, though they conflate distinct aspects of metacognition unless carefully disambiguated.
• Interpretability and Task-Specific Probes: Techniques range from probing for concept recognition in hidden activations to introspective awareness of model position within reasoning traces and dynamic debate setups for multi-turn self-assessment.
• Benchmark Development: New datasets and tasks target metacognitive skills (e.g., knowledge boundary awareness, capability self-assessment, faithfulness of uncertainty expression) though coverage and standardization remain nascent.

Importantly, the authors emphasize limitations in benchmark breadth, diversity of evaluation protocols, and the impact of confounds such as confidence elicitation method, instruction-tuning, model scale, and sampling temperature.

Empirical Findings and Core Limitations

Empirical investigations reveal a nuanced and highly context-dependent picture of metacognitive faculty in LLMs:

• Sensitivity and Calibration: LLMs generally display weak to moderate metacognitive sensitivity, with both overconfidence and underconfidence observed, often aggravated by post-training procedures such as RLHF. Uncalibrated self-reported confidence and the emergence of "high-certainty hallucinations" demonstrate systematic metacognitive failures.
• Limited Generalization: Metacognitive skills acquired in one domain or task do not automatically transfer to others; metacognitive performance is strongly domain- and task-specific.
• Faithful Uncertainty Expression: Even advanced models frequently fail to faithfully verbalize uncertainty, a key desideratum for reliability in high-stakes applications. Prompting and RL-based approaches that directly train for faithful alignment yield measurable improvements.
• Decoupling of Reasoning and Metacognition: There is compelling evidence that reasoning ability and metacognitive competence can be inversely correlated—task performance does not guarantee robust self-assessment or adaptive control.
• Fragility of Introspection and Self-Monitoring: Capabilities for introspection (monitoring of internal belief states, confidence in reasoning traces) demonstrate strong dependence on model architecture, in-context exemplars, and prompt structure. These faculties tend to be fragile and highly sensitive to perturbation.
• Scalability and Model Size: Emergence and strength of metacognitive faculties generally—but not monotonically—scale with model size; some small models display higher metacognitive sensitivity on certain tasks, disrupting simple scaling hypotheses.

Methods for Inducing and Improving Metacognition

The survey consolidates methods aimed at directly instantiating and enhancing metacognitive capacities in LLMs:

• Architectural Interventions: Novel designs such as the State Stream Transformer (SST) and augmented modules for introspective reasoning inject structural support for metacognitive operations.
• Metacognitive Prompting: Prompt frameworks instructing models to self-reflect, estimate uncertainty, or to critique their own solutions improve various task metrics, especially when integrated with strategic diversity in reasoning approach selection.
• Training and Fine-Tuning: Leveraging self-consistency, reflection, reward signals based on metacognitive performance, and imitation of expert critiques has been shown to elevate metacognitive faculties. Notably, RL with metacognitive feedback (RLMF) and multi-stage SFT can drive both calibration and faithfulness in confidence estimation.
• Agentic and Multi-Agent Systems: Integration of metacognitive monitoring and control enhances task decomposition, collaborative reasoning, adaptive strategy selection, workflow robustness, and memory management in autonomous and multi-agent LLM systems.
• Downstream Methods and Task-Specific Applications: Metacognitive frameworks are deployed to refine resource allocation in retrieval-augmented setups, optimize reasoning trace efficiency, improve skill transfer and selection, and mitigate hallucination via error- or boundary-aware regulation.

Practical and Theoretical Implications

The operationalization of LLM metacognition presents concrete benefits across a range of application scenarios:

• Enhanced Reliability and Trust: Improved metacognitive sensitivity and faithful uncertainty communication are pivotal for reliability, human-AI calibration in hybrid decision-making, and deployment in high-stakes settings (e.g., medicine, law).
• Interpretability and Self-Improvement: Models equipped with metacognitive faculties provide more transparent, scrutinizable, and corrigible behaviors, with auxiliary gains in agentic adaptability and self-driven learning.
• Safety and Oversight: Metacognitive monitoring is dual-edged: while it can improve safety and error detection, it also creates vectors for strategic deception or evasiveness if subverted.
• Interfacing with Human Metacognition: LLMs capable of explicit self-monitoring support more effective collaboration, delegation, and pedagogical interaction, yet may also induce user overreliance and "metacognitive laziness" over time if not carefully integrated.

The paper also discusses the interplay between metacognition and creativity, theory of mind, and self-improvement, drawing clear lines of inquiry for future research in AI cognition.

Open Challenges and Future Directions

Despite substantial progress, the field is characterized by major unsolved challenges:

• Lack of standardized, robust benchmarks for metacognitive faculties, especially in open-ended or multi-modal domains.
• Inadequate understanding of the mechanistic origins of emergent metacognitive behaviors, their dependence on modeling ingredients (data, architecture, objective), and their generalizability.
• Open questions regarding meta-metacognition (recursive assessment of one's own metacognitive competence), dynamic adaptation, and the balance of human vs. AI metacognitive authority in collaborative contexts.
• The dual-use risk of metacognitive faculties and their complex interactions with alignment and interpretability efforts.

Conclusion

The reviewed work establishes LLM metacognition as an indispensable focal area for advancing AI systems toward greater reliability, transparency, and autonomy. Progress in measurement, methodological rigor, and understanding of metacognitive faculties will underpin safe and effective deployment of LLMs in decision-making, collaborative, and safety-critical settings. However, the field remains defined by open theoretical problems, empirical ambiguities, and substantial need for standardized evaluation and mechanistic interpretability.

---