Metacognitive Readiness Index (MRI) in LLMs

• MRI is a composite framework that assesses if LLMs generate informative and valid metacognitive signals for reliable self-assessment.
• It leverages type-2 signal detection theory to quantify discrimination, calibration, and confidence–correctness coupling across domains.
• MRI guides deployment by screening for valid self-report profiles, ensuring models can effectively abstain, route, or operate based on confidence.

Searching arXiv for the cited papers to ground the article in current literature. arxiv.search({"query":"id:(Park et al., 2 Feb 2026) OR id:(Cacioli, 20 Apr 2026) OR id:(Cacioli, 17 Apr 2026) OR id:(Cacioli, 21 Apr 2026) OR id:(Cacioli, 26 Mar 2026)","max_results":10}) arxiv.search result: { "results": [ { "arxiv_id": "(Cacioli, 21 Apr 2026)", "title": "Domain-level metacognitive monitoring in frontier LLMs: A 33-model atlas", "authors": [ "Jasper Cacioli" ], "abstract": "Aggregate metacognitive quality scores mask within-model variation across MMLU benchmark domains. We administered 1,500 MMLU items (250 per domain, under an a priori six-domain grouping) to 33 frontier LLMs from eight model families and computed Type-2 AUROC per model-domain cell using verbalized confidence (0-100). Total observations: 47,151. Every model with above-chance aggregate monitoring showed non-trivial domain-level variation. Applied/Professional knowledge was reliably the easiest benchmark domain to monitor (mean AUROC = .742, ranked top-2 in 21 of 33 models); Formal Reasoning and Natural Science were reliably the hardest (one of the two ranked bottom-2 in 27 of 33 models). The three middle domains were statistically indistinguishable (Kendall's W = .164). A subject-level coherence analysis (within-domain similarity ratio = 0.95) confirms the six-domain grouping is a pragmatic benchmark taxonomy, not a validated latent construct. Within-family profile-shape clustering is significant for Anthropic, Google-Gemini, and Qwen (permutation p < .0001) but not DeepSeek, Google-Gemma, or OpenAI. Gemma 4 31B showed a +.202 AUROC improvement over Gemma 3 27B. Three models classified Invalid on binary KEEP/WITHDRAW probes produced normal profiles under verbalized confidence, confirming probe-format specificity. Bootstrap 95% CIs on 198 cells have median width .199. Split-half aggregate stability r = .893; profile-level split-half is weaker (grand median r = .184). These results show stable benchmark-domain variation obscured by aggregate metrics, and support benchmark-stage domain screening as a step before deployment in specific application areas.", "published": "2026-04-21", "pdf_url": "http://arxiv.org/pdf/([2605.06673](/papers/2605.06673))v1" }, { "arxiv_id": "(Cacioli, 20 Apr 2026)", "title": "Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report", "authors": [ "Jasper Cacioli" ], "abstract": "Clinical personality assessment screens response validity before interpreting substantive scales. LLM evaluation does not. We apply the validity scaling framework from the PAI and MMPI-3 to metacognitive probe data from 20 frontier models across 524 items. Six validity indices are operationalised: L (maintaining confidence on errors), K (betting on errors), F (withdrawing consensus-endorsed items), Fp (withdrawing correct answers), RBS (inverted monitoring), and TRIN (fixed responding). A tiered classification system identifies four models as construct-level invalid and two as elevated. Valid-profile models produce item-sensitive confidence (mean r = .18, 14 of 16 significant). Invalid-profile models do not (mean r = -.20, d = 2.17, p = .001). Chain-of-thought training produces two opposite response distortions. Two latent dimensions account for 94.6% of index variance. Companion papers extract a portable screening protocol (Cacioli, 2026e) and validate it against selective prediction (Cacioli, 2026f). All data and code: https://github.com/synthiumjp/validity-scaling-LLM", "published": "2026-04-20", "pdf_url": "http://arxiv.org/pdf/([2604.17707](/papers/2604.17707))v1" }, { "arxiv_id": "(Cacioli, 17 Apr 2026)", "title": "The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring", "authors": [ "Jasper Cacioli", "Tobias Baumann" ], "abstract": "We introduce a cross-domain behavioural assay of monitoring-control coupling in LLMs, grounded in the Nelson and Narens (1990) metacognitive framework and applying human psychometric methodology to LLM evaluation. The battery comprises 524 items across six cognitive domains (learning, metacognitive calibration, social cognition, attention, executive function, prospective regulation), each grounded in an established experimental paradigm. Tasks T1-T5 were pre-registered on OSF prior to data collection; T6 was added as an exploratory extension. After every forced-choice response, dual probes adapted from Koriat and Goldsmith (1996) ask the model to KEEP or WITHDRAW its answer and to BET or decline. The critical metric is the withdraw delta: the difference in withdrawal rate between incorrect and correct items. Applied to 20 frontier LLMs (10,480 evaluations), the battery discriminates three profiles consistent with the Nelson-Narens architecture: blanket confidence, blanket withdrawal, and selective sensitivity. Accuracy rank and metacognitive sensitivity rank are largely inverted. Retrospective monitoring and prospective regulation appear dissociable (r = .17, 95% CI wide given n=20; exemplar-based evidence is the primary support). Scaling on metacognitive calibration is architecture-dependent: monotonically decreasing (Qwen), monotonically increasing (GPT-5.4), or flat (Gemma). Behavioural findings converge structurally with an independent Type-2 SDT approach, providing preliminary cross-method construct validity. All items, data, and code: https://github.com/synthiumjp/metacognitive-monitoring-battery.", "published": "2026-04-17", "pdf_url": "http://arxiv.org/pdf/([2604.15702](/papers/2604.15702))v1" }, { "arxiv_id": "(Cacioli, 26 Mar 2026)", "title": "Do LLMs Know What They Know? Measuring Metacognitive Efficiency with Signal Detection Theory", "authors": [ "Benjamin Moran", "Pedro F. Valdes", "Jian Qiao", "Alicia Curth" ], "abstract": "Standard evaluation of LLM confidence relies on calibration metrics (ECE, Brier score) that conflate two distinct capacities: how much a model knows (Type-1 sensitivity) and how well it knows what it knows (Type-2 metacognitive sensitivity). We introduce an evaluation framework based on Type-2 Signal Detection Theory that decomposes these capacities using meta-d' and the metacognitive efficiency ratio M-ratio. Applied to four LLMs (Llama-3-8B-Instruct, Mistral-7B-Instruct-v0.3, Llama-3-8B-Base, Gemma-2-9B-Instruct) across 224,000 factual QA trials, we find: (1) metacognitive efficiency varies substantially across models even when Type-1 sensitivity is similar -- Mistral achieves the highest d' but the lowest M-ratio; (2) metacognitive efficiency is domain-specific, with different models showing different weakest domains, invisible to aggregate metrics; (3) temperature manipulation shifts Type-2 criterion while meta-d' remains stable for two of four models, dissociating confidence policy from metacognitive capacity; (4) AUROC_2 and M-ratio produce fully inverted model rankings, demonstrating these metrics answer fundamentally different evaluation questions. The meta-d' framework reveals which models \"know what they don't know\" versus which merely appear well-calibrated due to criterion placement -- a distinction with direct implications for model selection, deployment, and human-AI collaboration. Pre-registered analysis; code and data publicly available.", "published": "2026-03-26", "pdf_url": "http://arxiv.org/pdf/([2603.25112](/papers/2603.25112))v1" }, { "arxiv_id": "(Park et al., 2 Feb 2026)", "title": "Fine-Tuning LLMs to Know What They Know", "authors": [ "Zihao Guo", "Ziyue Wang", "Yuying Zhang", "Yangsen Zhang", "Ming Yin", "Wangchunshu Zhou" ], "abstract": "Metacognition is a critical component of intelligence, specifically regarding the awareness of one's own knowledge. While humans rely on shared internal memory for both answering questions and reporting their knowledge state, this dependency in LLMs remains underexplored. This study proposes a framework to measure metacognitive ability d_type2' using a dual-prompt method, followed by the introduction of Evolution Strategy for Metacognitive Alignment (ESMA) to bind a model's internal knowledge to its explicit behaviors. ESMA demonstrates robust generalization across diverse untrained settings, indicating a enhancement in the model's ability to reference its own knowledge. Furthermore, parameter analysis attributes these improvements to a sparse set of significant modifications.", "published": "2026-02-02", "pdf_url": "http://arxiv.org/pdf/([2602.02605](/papers/2602.02605))v1" } ] } Metacognitive Readiness Index (MRI) is a proposed family of evaluation constructs for quantifying whether a LLM’s metacognitive signals are sufficiently informative, valid, and stable to support downstream regulation such as abstention, routing, or confidence-gated deployment. Across recent arXiv work, MRI is not a single standardized metric. Instead, it is instantiated in several closely related ways: as a composite over type-2 discrimination, calibration, and confidence–correctness coupling; as a validity gate plus latent distortion factors; as a cross-domain monitoring–control composite; and as a domain-aware AUROC summary penalized by uncertainty and scaled by reliability (Park et al., 2 Feb 2026, Cacioli, 20 Apr 2026, Cacioli, 17 Apr 2026, Cacioli, 21 Apr 2026, Cacioli, 26 Mar 2026). The common substrate is second-order evaluation: not whether a model answers correctly, but whether it knows when it is correct, knows when it is not, and exposes that information in a usable behavioral form.

1. Conceptual status and research lineage

Current literature does not present MRI as a settled canonical quantity. "Fine-Tuning LLMs to Know What They Know" explicitly states that the paper does not define a composite index and instead provides a principled MRI proposal grounded in its framework (Park et al., 2 Feb 2026). "Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report" likewise states that the term “Metacognitive Readiness Index (MRI)” is not introduced or explicitly defined in that paper, and derives MRI from the reported indices, thresholds, and latent structure (Cacioli, 20 Apr 2026). "The Metacognitive Monitoring Battery" also treats MRI as a proposed extension rather than a paper-defined measure (Cacioli, 17 Apr 2026). By contrast, the domain-level atlas explicitly defines a domain-aware MRI that aggregates Type-2 AUROC across domains while penalizing uncertainty and scaling by reliability (Cacioli, 21 Apr 2026). The SDT-based efficiency paper formulates MRI as principled operationalizations centered on $M$-ratio, temperature stability, domain robustness, and, optionally, policy stability (Cacioli, 26 Mar 2026).

This implies that MRI currently functions less as a single benchmark score than as a higher-order readiness construct spanning several measurement traditions. The literature converges on three recurring dimensions. First is metacognitive discrimination: whether confidence or self-report separates correct from incorrect responses. Second is calibration or policy usability: whether confidence magnitudes or commitment decisions are reliable for deployment. Third is validity: whether the elicited self-report is itself interpretable, rather than distorted by blanket confidence, blanket withdrawal, or probe-specific response styles (Cacioli, 20 Apr 2026, Cacioli, 17 Apr 2026, Cacioli, 21 Apr 2026).

A plausible implication is that MRI should be understood as a screening and reporting framework rather than a universal scalar. The papers repeatedly show that aggregate scores can obscure domain-level heterogeneity, criterion effects, or invalid response profiles (Cacioli, 21 Apr 2026, Cacioli, 26 Mar 2026, Cacioli, 20 Apr 2026).

2. Type-2 signal detection theory as the core measurement substrate

The most direct MRI formulations are grounded in Type-2 SDT. In the dual-prompt framework of Guo et al., metacognition is operationalized as type-II discrimination: whether the model’s meta-response distinguishes its own correct from incorrect first-order answers (Park et al., 2 Feb 2026). With binary confidence, the exact sensitivity is

$$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$

with $$\text{Hit Rate} = P(\text{Meta-Yes} \mid \text{Correct})$$ and $$\text{False Alarm Rate} = P(\text{Meta-Yes} \mid \text{Incorrect})$$ (Park et al., 2 Feb 2026). The same work defines continuous confidence from “Yes/No” token logits as

$$D = \frac{P(\text{Yes})}{P(\text{Yes}) + P(\text{No})},$$

and uses threshold sweeps over $D \in [0,1]$ to construct a type-2 ROC whose AUC is threshold-invariant type-2 sensitivity (Park et al., 2 Feb 2026).

Moran et al. extend this substrate by separating Type-1 sensitivity $d'$ from Type-2 metacognitive sensitivity meta-$d'$ and defining metacognitive efficiency by

$M\text{-}ratio = \frac{\text{meta-}d'}{d'}.$

Their framework is explicitly motivated by the claim that calibration metrics such as ECE and Brier score conflate “how much a model knows” with “how well it knows what it knows” (Cacioli, 26 Mar 2026). The paper’s empirical result that AUROC$_2$ and $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$0-ratio produce fully inverted model rankings demonstrates that ranking-based discrimination and efficiency-normalized metacognition answer different questions (Cacioli, 26 Mar 2026).

The following table summarizes the main MRI operationalizations reported in the cited literature.

Source	MRI form	Core ingredients
(Park et al., 2 Feb 2026)	Proposed normalized composite	$$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$1, $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$2, $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$3
(Cacioli, 20 Apr 2026)	MRI-Tier + MRI-Factors	Tiered validity classification; Over and Under factor scores
(Cacioli, 17 Apr 2026)	Proposed cross-domain composite	$$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$4, AUROC$$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$5, $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$6-ratio, $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$7, $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$8, calibration
(Cacioli, 21 Apr 2026)	Domain-aware MRI	domain AUROC, CI penalty, reliability multiplier
(Cacioli, 26 Mar 2026)	MRI_A / MRI_B	normalized $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$9-ratio, temperature stability, domain robustness, AUROC$$\text{Hit Rate} = P(\text{Meta-Yes} \mid \text{Correct})$$0, Type-2 criterion stability

Within this lineage, the dominant misconception is that calibration alone suffices. The SDT papers argue the opposite: calibration can be high because of criterion placement, while metacognitive efficiency remains poor (Cacioli, 26 Mar 2026). This makes Type-2 SDT the main theoretical anchor for any MRI intended to measure readiness rather than merely confidence style.

3. Composite MRI from dual-prompt self-knowledge measurement

The most explicit scalar MRI in the cited set is the proposal derived from "Fine-Tuning LLMs to Know What They Know" (Park et al., 2 Feb 2026). The underlying protocol uses two independent contexts per item: a direct question prompt for the factual answer and a separate meta prompt asking whether the model knows the answer. This independence is intended to avoid conditioning on prior generations (Park et al., 2 Feb 2026). A zero-shot generalization probe replaces the two-prompt structure with a unified IDK instruction: “If you don’t know the answer, just return ‘I don’t know’,” with “I don’t know” treated as Meta-No (Park et al., 2 Feb 2026).

The proposed MRI in that framework is a normalized composite over three components: metacognitive discrimination, calibration, and confidence–correctness discrimination. The paper-tailored proposal prefers meta-$$\text{Hit Rate} = P(\text{Meta-Yes} \mid \text{Correct})$$1 when continuous confidence is available, otherwise $$\text{Hit Rate} = P(\text{Meta-Yes} \mid \text{Correct})$$2; uses $$\text{Hit Rate} = P(\text{Meta-Yes} \mid \text{Correct})$$3 for calibration; and uses AUROC for correctness versus $$\text{Hit Rate} = P(\text{Meta-Yes} \mid \text{Correct})$$4 (Park et al., 2 Feb 2026). The normalization rules are

$$\text{Hit Rate} = P(\text{Meta-Yes} \mid \text{Correct})$$5

and the composite is

$$\text{Hit Rate} = P(\text{Meta-Yes} \mid \text{Correct})$$6

with the proposed weighting

$$\text{Hit Rate} = P(\text{Meta-Yes} \mid \text{Correct})$$7

This weighting is justified in the source by the observation that ESMA most strongly improves discrimination, while raw alignment is biased by accuracy and “Yes” rate (Park et al., 2 Feb 2026).

The same paper provides the measurement ingredients needed to populate this MRI. For calibration, it proposes

$$\text{Hit Rate} = P(\text{Meta-Yes} \mid \text{Correct})$$8

$$\text{Hit Rate} = P(\text{Meta-Yes} \mid \text{Correct})$$9

and

$$\text{False Alarm Rate} = P(\text{Meta-Yes} \mid \text{Incorrect})$$0

For practical estimation it recommends temperature $$\text{False Alarm Rate} = P(\text{Meta-Yes} \mid \text{Incorrect})$$1, full validation or test splits for evaluation, $$\text{False Alarm Rate} = P(\text{Meta-Yes} \mid \text{Incorrect})$$2 or $$\text{False Alarm Rate} = P(\text{Meta-Yes} \mid \text{Incorrect})$$3 equal-width bins for ECE, and bootstrap over items for uncertainty if possible (Park et al., 2 Feb 2026).

Empirically, the paper reports large improvements in metacognitive discrimination under ESMA. For Qwen2.5 3B, $$\text{False Alarm Rate} = P(\text{Meta-Yes} \mid \text{Incorrect})$$4 increases from $$\text{False Alarm Rate} = P(\text{Meta-Yes} \mid \text{Incorrect})$$5 to $$\text{False Alarm Rate} = P(\text{Meta-Yes} \mid \text{Incorrect})$$6, accuracy from $$\text{False Alarm Rate} = P(\text{Meta-Yes} \mid \text{Incorrect})$$7 to $$\text{False Alarm Rate} = P(\text{Meta-Yes} \mid \text{Incorrect})$$8, and IDK Alignment from $$\text{False Alarm Rate} = P(\text{Meta-Yes} \mid \text{Incorrect})$$9 to $$D = \frac{P(\text{Yes})}{P(\text{Yes}) + P(\text{No})},$$0 (Park et al., 2 Feb 2026). Across scales, type-2 AUC is reported at approximately $$D = \frac{P(\text{Yes})}{P(\text{Yes}) + P(\text{No})},$$1 after ESMA, which the paper interprets as a moderate-to-high metacognition regime (Park et al., 2 Feb 2026). This suggests that, within this formulation, MRI is designed to capture not only self-report fidelity but deployable abstention behavior that generalizes across formats, datasets, and languages.

4. Validity-screened MRI: tiers, indices, and latent distortion factors

A distinct MRI conception appears in the validity-scaling work of Cacioli (Cacioli, 20 Apr 2026). Here the central claim is methodological: metacognitive profiles should be screened for response validity before substantive interpretation. The paper operationalizes six validity indices over KEEP/WITHDRAW and BET probes: L, K, F, Fp, RBS, and TRIN (Cacioli, 20 Apr 2026). Their exact definitions are:

• $$D = \frac{P(\text{Yes})}{P(\text{Yes}) + P(\text{No})},$$2
• $$D = \frac{P(\text{Yes})}{P(\text{Yes}) + P(\text{No})},$$3
• $$D = \frac{P(\text{Yes})}{P(\text{Yes}) + P(\text{No})},$$4 with $$D = \frac{P(\text{Yes})}{P(\text{Yes}) + P(\text{No})},$$5
• $$D = \frac{P(\text{Yes})}{P(\text{Yes}) + P(\text{No})},$$6
• $$D = \frac{P(\text{Yes})}{P(\text{Yes}) + P(\text{No})},$$7
• $$D = \frac{P(\text{Yes})}{P(\text{Yes}) + P(\text{No})},$$8

The primary MRI construct in this paper is categorical. MRI-Tier is defined by a tiered classification system. Tier 1 construct-level invalidity is triggered by any of

$$D = \frac{P(\text{Yes})}{P(\text{Yes}) + P(\text{No})},$$9

If no Tier 1 rule is triggered, Tier 2 uses sample-referenced thresholds after excluding Tier 1 invalids: Elevated if any index is at least $D \in [0,1]$0, Marked if any index is at least $D \in [0,1]$1, otherwise Valid-profile (Cacioli, 20 Apr 2026). In the 20-model derivation sample, four models were construct-level invalid, two elevated, and fourteen valid-profile (Cacioli, 20 Apr 2026).

The paper also defines MRI-Factors, a two-dimensional latent summary derived from PCA on the six indices. Two components explain $D \in [0,1]$2 of variance: an Over-reporting dimension with loadings $D \in [0,1]$3, $D \in [0,1]$4, $D \in [0,1]$5, and an Under-reporting dimension with loadings $D \in [0,1]$6, $D \in [0,1]$7, $D \in [0,1]$8 (Cacioli, 20 Apr 2026). The factor-aligned composites are

$D \in [0,1]$9

$d'$0

This validity-screened conception shifts MRI from a scalar performance score to an interpretive gate. The paper reports that valid-profile models show item-sensitive confidence, with mean $d'$1 and $d'$2 of $d'$3 individually significant, whereas invalid-profile models have mean $d'$4; the group difference is $d'$5, $d'$6 (Cacioli, 20 Apr 2026). This supports the paper’s claim that validity screening is not merely psychometric ornamentation but a prerequisite for deciding whether self-report is informative at all.

The main controversy addressed here is the assumption that any confidence-like response can be interpreted directly. The data argue against that assumption. Chain-of-thought training is reported to produce “two opposite response distortions”: DeepSeek-R1 shows systematic over-withdrawal with probe decoupling, whereas Qwen 80B Think shows blanket confidence with $d'$7 (Cacioli, 20 Apr 2026). In this framework, both are Tier 1 invalid for different reasons.

5. Monitoring–control coupling and cross-domain MRI

"The Metacognitive Monitoring Battery" situates MRI in the Nelson–Narens monitoring–control architecture rather than only in SDT (Cacioli, 17 Apr 2026). The battery spans 524 items across six cognitive domains: learning, metacognitive calibration, social cognition, attention, executive function, and prospective regulation (Cacioli, 17 Apr 2026). For tasks T1–T5, the model first gives a forced-choice response and then answers two post-response probes: KEEP or WITHDRAW, and BET or NO_BET. Task T6 adds prospective regulation via ANSWER_DIRECTLY, REQUEST_HINT, or DECLINE before answering (Cacioli, 17 Apr 2026).

The paper’s critical metric is the withdraw delta,

$d'$8

with positive values indicating selective withdrawal of wrong answers (Cacioli, 17 Apr 2026). The MRI proposal extends this with

$d'$9

and

$d'$0

alongside AUROC$d'$1, $d'$2-ratio, and calibration terms (Cacioli, 17 Apr 2026). Domain-wise standardization is defined by

$d'$3

and the proposed composite is

$d'$4

with equal weights across available metrics as the default (Cacioli, 17 Apr 2026).

This formulation is motivated by the paper’s empirical dissociation between retrospective monitoring and prospective regulation. Across 20 models, retrospective sensitivity and prospective direct rate show weak correlation, with Pearson $d'$5 and a wide $d'$6 CI (Cacioli, 17 Apr 2026). The battery identifies three profiles: blanket confidence, blanket withdrawal, and selective sensitivity (Cacioli, 17 Apr 2026). Blanket confidence is defined by KEEP $d'$7 and ANSWER_DIRECTLY $d'$8 regardless of correctness, with $d'$9; blanket withdrawal is exemplified by DeepSeek R1, which WITHDRAWs $M\text{-}ratio = \frac{\text{meta-}d'}{d'}.$0–$M\text{-}ratio = \frac{\text{meta-}d'}{d'}.$1 and DECLINEs $M\text{-}ratio = \frac{\text{meta-}d'}{d'}.$2; selective sensitivity is exemplified by models such as Claude Sonnet 4.6 with mean $M\text{-}ratio = \frac{\text{meta-}d'}{d'}.$3 and Claude Haiku 4.5 with $M\text{-}ratio = \frac{\text{meta-}d'}{d'}.$4 (Cacioli, 17 Apr 2026).

A plausible implication is that MRI in this tradition is explicitly action-oriented. It does not ask only whether confidence ranks correctness, but whether monitoring is coupled to control strongly enough to alter behavior in a beneficial way.

6. Domain-aware readiness, efficiency-centered variants, and deployment use

The domain-level atlas makes MRI explicitly deployment-facing by aggregating Type-2 AUROC across six MMLU-derived benchmark domains and penalizing uncertainty (Cacioli, 21 Apr 2026). For model $M\text{-}ratio = \frac{\text{meta-}d'}{d'}.$5 and domain $M\text{-}ratio = \frac{\text{meta-}d'}{d'}.$6, with AUROC estimate $M\text{-}ratio = \frac{\text{meta-}d'}{d'}.$7 and bootstrap CI width $M\text{-}ratio = \frac{\text{meta-}d'}{d'}.$8, the standard error is approximated by

$M\text{-}ratio = \frac{\text{meta-}d'}{d'}.$9

and the MRI is defined as

$_2$0

The default recommendations are equal domain weights, $_2$1, and a reliability multiplier $_2$2 set to the split-half aggregate stability $_2$3 for overall screening (Cacioli, 21 Apr 2026). For profile-sensitive use, the paper advises stricter reliability treatment because profile-level split-half is weaker, with grand median $_2$4 (Cacioli, 21 Apr 2026).

The atlas justifies this construction by showing that aggregate metacognitive quality masks systematic within-model domain variation. Applied/Professional is the easiest domain to monitor, with mean AUROC approximately $_2$5 and top-two rank in $_2$6 models, while Formal Reasoning and Natural Science are the hardest, occupying a bottom-two rank in $_2$7 models (Cacioli, 21 Apr 2026). Yet the subject-level coherence ratio is $_2$8, indicating the six-domain grouping is pragmatic rather than a validated latent construct (Cacioli, 21 Apr 2026). This means the MRI is domain-aware, but the domains themselves should not be overinterpreted as psychometric factors.

Moran et al. provide an efficiency-centered alternative. Their MRI_A normalizes $_2$9-ratio and multiplies it by a temperature-stability adjustment, optionally adding a weakest-domain robustness term (Cacioli, 26 Mar 2026). MRI_B combines normalized meta-$$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$00, AUROC$$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$01, and a policy-stability term derived from Type-2 criterion dispersion across temperatures (Cacioli, 26 Mar 2026). The explicit rationale is that MRI should condition on Type-1 $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$02 to avoid conflating knowledge with metaknowledge (Cacioli, 26 Mar 2026). Their empirical examples show why: on TriviaQA at $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$03, Mistral-Instruct has the highest $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$04 but the lowest $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$05-ratio $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$06, whereas Gemma-2-Instruct has lower $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$07 but $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$08-ratio $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$09 (Cacioli, 26 Mar 2026).

Taken together, these papers suggest two deployment regimes. For broad screening under verbalized confidence, domain-aware AUROC with uncertainty and reliability correction is the simplest MRI-like summary (Cacioli, 21 Apr 2026). For confidence-dependent decision-making where internal evidence is available, efficiency-centered MRI built on meta-$$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$10 and $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$11-ratio is more faithful to metacognitive capacity (Cacioli, 26 Mar 2026).

7. Limitations, failure modes, and interpretive constraints

Several limitations recur across the literature. First, probe format matters. The atlas reports that three models classified Invalid under binary KEEP/WITHDRAW probes produced normal profiles under verbalized 0–100 confidence, concluding that validity is model–probe–task interaction–specific (Cacioli, 21 Apr 2026). This directly constrains any MRI that assumes probe-invariant metacognition.

Second, raw alignment and aggregate confidence can be misleading. Guo et al. note that Raw Alignment is biased by base accuracy and “Yes” rate and recommend relying on $$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$12, AUC, or meta-$$d'_{\rm type2} = \Phi^{-1}(\text{Hit Rate}) - \Phi^{-1}(\text{False Alarm Rate}),$$13 instead (Park et al., 2 Feb 2026).