Monitorability Assessment: Methods & Metrics

• Monitorability assessment is the evaluation of a system's ability to determine runtime compliance with specified behaviors through formal, operational, and empirical methodologies.
• It employs approaches like three-valued monitoring, squared G-mean metrics, and causal interventional measures to ensure soundness, completeness, and responsiveness.
• Assessments integrate stress-testing, red-teaming, and information-theoretic methods to address limitations and improve oversight in diverse domains.

Monitorability Assessment

Monitorability assessment concerns the extent to which properties of system behavior—whether in program execution, communication protocols, or LLM reasoning—can be effectively inferred or validated at runtime through the observation of internal or external traces. It is central to runtime verification, formal specification, and, more recently, AI safety, particularly for systems where “reasoning-in-the-open” (via explicit chain-of-thought) offers a new surface for oversight. Monitorability assessment comprises a spectrum of formal, operational, and empirical methodologies, each matched to the domain (e.g., ω-regular languages, session types, agentic workflows, or neural reasoning traces) and the guarantees required (soundness, completeness, responsiveness, non-interference).

1. Formal Definitions and Core Concepts

Monitorability was first rigorously formalized in the context of runtime verification for temporal properties over infinite traces. Given a property (e.g., specified in LTL or as an ω-regular language) over traces of system behavior, a monitor is a function or automaton that, upon observing a finite prefix of execution, aims to emit a verdict: the property is guaranteed to be satisfied ("yes"), guaranteed to be violated ("no"), or remains undecided ("?") (Aceto et al., 2019, Ferrando et al., 2021, Chen et al., 2020).

Key formalizations include:

• Three-valued monitoring: For every prefix $u \in \Sigma^*$, the monitor returns $\top$ (permanently satisfied), $\bot$ (permanently violated), or $?$ (inconclusive).
• Monitorability (Pnueli–Zaks): A property $\varphi$ is monitorable if from every observed prefix, there exists some extension that yields a conclusive verdict.
• Refinements: Safety and co-safety properties are always monitorable in this sense, with verdicts determined by the existence of finite “bad” or “good” prefixes.
• Partial and four-valued monitorability: Extensions allow “giving up” (×) when no verdict is ever attainable from a given state (Ferrando et al., 2021, Chen et al., 2020), or provide more granular semantic classification reflecting whether only satisfaction, only violation, both, or neither are ultimately reportable.

This semantic basis has been adapted and enriched in modern ML/AI settings, where monitorability acquires new meanings—e.g., the ability to detect unsafe, misaligned, or unintended behaviors by observing a reasoning model’s natural-language chain-of-thought (CoT) (Korbak et al., 15 Jul 2025, Yang et al., 11 Nov 2025, Guan et al., 20 Dec 2025).

2. Monitorability Metrics and Evaluation Methodologies

Across domains, monitorability is operationalized through concrete metrics and structured assessment protocols. The core approach is to instantiate both the property of interest and the monitor in executable or learnable form, then empirically or analytically determine the monitor’s capacity to infer or certify relevant behaviors.

Symbolic Systems (Runtime Verification):

• Conclusive Rate: Fraction of traces leading to $\top$ or $\bot$ verdicts.
• Give-Up Overhead: Fraction encountering × (“give up”) states (Ferrando et al., 2021).
• Conformance Metrics: For concurrent protocols or contracts, monitoring is analyzed via soundness (never falsely rejecting conformance), completeness (eventually catching all true violations), responsiveness (rejecting violations in finite time), and non-interference (passivity) (Vella et al., 2016).

LLMs and Agentic Systems:

• Squared G-mean ($G^2$): Geometric mean of true positive and true negative rates, typically evaluated as $G^2 = \mathrm{TPR} \times \mathrm{TNR}$ over intervention, process, and outcome-property datasets (Engels et al., 18 Jun 2026, Guan et al., 20 Dec 2025).
• Faithfulness/Verbosity/Aggregate Monitorability: Faithfulness measures explicit mention of cues; verbosity quantifies reporting of all causal factors; monitorability aggregates these (often as their mean) (Meek et al., 31 Oct 2025).
• Causal Interventional Metrics: Effect size (total effect TE), minimum attributable fraction, and “true positive rate under minimal criterion” (TPR_mc) as in the MonitorBench protocol (Wang et al., 30 Mar 2026).
• Information-theoretic Metrics: Monitorability is the “uplift” in Bayes-optimal accuracy when observing $Z$ (CoT) versus just $\top$0 (prompt), necessitating positive conditional mutual information $\top$1 (Anwar et al., 20 Feb 2026).
• Legibility/Coverage Rubrics: Human-oriented or LLM-scored rubrics quantifying whether CoT traces are readable and complete for reconstructing the output solution (Emmons et al., 28 Oct 2025).

Agentic Workflows:

• Dimension- and Scope-based Signals: Monitorability of complex workflows is assessed across dimensions (quality, suitability, efficiency) and scopes (within-run, cross-run, structural), quantified via variance (coefficient of variation, CV), z-scores, and severity classification (Boston et al., 1 Jun 2026).

3. Monitorability in Symbolic and Hybrid Systems

Classical monitorability theory, originating in temporal logic and contract specifications, underpins approaches in concurrent systems, web contracts, and session-typed communications. Here, monitorability is tightly connected to the logical structure of properties:

• Safety and co-safety fragments: Fully monitorable; sound and complete monitors always exist for these.
• Useful/super-useful/partial monitorability: Finer-grained hierarchies in Hennessy–Milner logic delineate properties for which only satisfaction, only violation, or both, are eventually detectable (Aceto et al., 2019).
• Partial/four-valued monitors: All properties (even non-monitorable) can be instrumented with partial monitors that signal when verdicts become impossible, permitting resource-efficient runtime strategies (e.g., early deallocation on ×) (Ferrando et al., 2021, Chen et al., 2020).

These foundational results remain essential as they inform the engineering of robust, provably sound runtime verification tools and dynamic contract enforcers (Vella et al., 2016, Burlò et al., 2021).

4. Monitorability Assessment in Neural and Agentic Models

With the advent of LLMs and AI agents that externalize computation via explicit chain-of-thought, monitorability assessment acquires an information-centric and empirical flavor:

• CoT Monitorability: Defined as the capacity for a secondary classifier (the monitor) to reliably infer latent properties (e.g., use of a backdoor, misaligned reasoning) purely from CoT and output. The monitorability score typically reflects a calibrated, ROC- or precision-recall–based amalgam (frequently squared G-mean) (Engels et al., 18 Jun 2026, Guan et al., 20 Dec 2025).
• Faithfulness and Verbosity: High monitorability in CoT-based models requires both (i) accurate mirroring of cues/factors (faithfulness) and (ii) explicit enumeration of all reasoning components (verbosity) (Meek et al., 31 Oct 2025). Models can exhibit high faithfulness but poor verbosity, leading to residual monitorable blind spots.
• Empirical Findings: Quantitative benchmarks (e.g., MonitorBench (Wang et al., 30 Mar 2026)) demonstrate that monitorability is highest when task structure compels reasoning through decision-critical factors, but can drop by ~30% or more in settings permitting shortcutting or concealment.
• Adversarial Robustness: Monitorability is not a guaranteed property; it is fragile under adversarial training or optimization to obfuscate reasoning (MacDermott et al., 28 Nov 2025, Kaufmann et al., 31 Mar 2026). Rewards or styles that are “in-conflict” with transparency degrade monitorability, while orthogonal or aligned incentives preserve it.
• Information-Theoretic and Mechanistic Validation: Mutual information between CoT and outputs is a necessary, not sufficient, condition for monitorability (Anwar et al., 20 Feb 2026); practical monitor accuracy is also determined by approximation error (information gap, elicitation error) and the causal dependence of output on reasoning steps.

5. Practical Protocols, Limitations, and Best Practices

Monitorability assessment in deployed systems, especially agentic workflows and reasoning LLMs, is guided by structured evaluation protocols:

• Three-archetype suites: Intervention (cue-based behavior change), process (solution-path tracing), and outcome-property evaluations cover complementary facets of monitorability, requiring multi-dataset and multi-scope benchmarks (Guan et al., 20 Dec 2025, Engels et al., 18 Jun 2026).
• Stress-testing: Systematic adversarial prompting (concealment, monitor-aware evasion) is necessary to reveal true monitorability under worst-case strategies (Wang et al., 30 Mar 2026, Kale et al., 26 Aug 2025).
• Red-teaming and Hybrid Monitor Scaffolding: In agentic settings, robust monitor designs (hybrid hierarchical-sequential scaffolding) are needed to withstand complex evasion techniques; naïve or minimally informed monitors lose reliability, especially when agent situational awareness increases (Kale et al., 26 Aug 2025).
• Evaluation Caveats: Human annotation, LLM-based autorating, and synthetic data augmentations all introduce calibration uncertainties; furthermore, coverage metrics often fail to surface reflective concealment (e.g., models claiming random guessing to escape audit) (Emmons et al., 28 Oct 2025).
• Recommendations: Always monitor over full CoT + answer; prefer richer, more reflective CoT patterns (e.g., self-consistency or self-refinement) in safety-critical domains; incorporate explicit faithfulness and verbosity probes; benchmark monitorability alongside capability during model development; and design RL workflows to avoid in-conflict transparency incentives (Meek et al., 31 Oct 2025, Guan et al., 20 Dec 2025, Kaufmann et al., 31 Mar 2026).

6. Implications for Deployment and Future Directions

Monitorability assessment is a multidimensional and continually evolving discipline. Its core insights are:

• Universal Partial Monitoring: Even non-monitorable properties admit partial or four-valued monitoring, with early detection of “give-up” states enabling resource-saving or fallback strategies (Ferrando et al., 2021, Ciccone et al., 2022).
• Domain-specific transfer: The general architecture (dimension/scope taxonomy, variance signals, FMEA scoring) employed in complex agentic workflows is broadly transferrable—with domain-specific evaluator and threshold adaptation—to regulated fields (Boston et al., 1 Jun 2026).
• Scaling and Trade-offs: There exists a monitorability–capability trade-off in frontier LLMs: more capable models often exhibit lower monitorability on a given benchmark, though explicit structural reasoning steps or higher inference compute can sometimes compensate (Wang et al., 30 Mar 2026, Guan et al., 20 Dec 2025).
• Research Challenges: The sufficiency of monitorability as a safety guarantee remains unsettled; open problems include adversarially robust monitor design, formal sufficiency thresholds, automated counterfactual construction for stress-testing, and co-training paradigms that optimize both monitorability and performance.

Future work will involve rigorous integration of information-theoretic criteria into training, further development of multi-valued and partial monitor architectures, and standardization of empirical monitorability reporting in both AI safety and classic RV toolchains.