Papers
Topics
Authors
Recent
Search
2000 character limit reached

Online Safety Monitoring for LLMs

Published 2 Jul 2026 in cs.AI, cs.CL, cs.LG, stat.AP, and stat.ML | (2607.02510v1)

Abstract: Despite alignment training, LLMs remain prone to generating unsafe outputs at deployment time. Monitoring outputs online and raising an alarm when safety can no longer be assumed is therefore critical. We study a simple real-time monitor that turns a verifier signal from an external model into an alarm decision by thresholding, with the threshold calibrated via risk control. In experiments on mathematical reasoning and red teaming datasets, we show that this simple design is competitive with more advanced monitors based on sequential hypothesis testing.

Summary

  • The paper's main contribution is a calibrated threshold framework that statistically controls false alarm and missed detection risks for LLM safety.
  • It demonstrates that simple calibrated monitors yield lower detection delays compared to sequential methods while balancing performance trade-offs.
  • Empirical results emphasize the importance of using high-quality external verifier signals over internal log-probabilities for effective safety monitoring.

Online Safety Monitoring for LLMs: Calibration, Evaluation, and Trade-offs

Introduction

The prevalence of LLMs in high-stakes applications requires mechanisms for real-time safety monitoring, to ensure that outputs remain factually correct and non-harmful throughout generation. "Online Safety Monitoring for LLMs" (2607.02510) proposes a calibrated, statistically principled thresholding monitor for online deployment. The work contrasts this simple approach with more involved sequential-hypothesis-based techniques, emphasizing statistical guarantees on false alarm and missed detection rates, empirical calibration on real LLM deployments, and the trade-offs between external and internal safety signals in terms of both cost and performance.

Statistical Framework for Online Safety Monitoring

The core contribution is a general statistical method that raises alarms in real-time when verifier signals––such as external process reward model (PRM) scores or safeguard classifier outputs––fall below a calibrated threshold λ\lambda. This threshold is computed to strictly control the desired risk metric (e.g., type I: false alarm risk, or type II: missed detection risk) at a prespecified level ϵ\epsilon, either in expectation (Conformal Risk Control, CRC) or with high probability (Upper Confidence Bound, UCB). The monitor operates as a stopping rule: once a signal sk<λs_k < \lambda is detected at any step kk, an alarm is raised and intervention may occur.

The approach contrasts with e-process and e-valuator-based monitors, which aggregate evidence via sequential density or classifier-driven likelihood ratios, and typically lack calibration for both types of risk. The simplicity of the single-threshold approach yields minimal additional computational overhead and is compatible with a wide variety of proxy signals, as demonstrated against several baselines.

Evaluating Factuality Monitoring in Mathematical Reasoning

Factual correctness monitoring is evaluated using step-level signals from mathematical problem-solving on the MATH benchmark. The authors use Qwen2.5-PRM-7B as a verifier, and generate solutions using two LLMs of markedly different capabilities (Claude Haiku 4.5, 90% accuracy; Mistral-7B-Instruct, 26% accuracy). Safe and unsafe trajectories are labeled using an independent verifier, and the monitor is calibrated to control the false alarm rate.

Both CRC and UCB monitors successfully maintain their respective false alarm guarantees across a range of risk levels, matching or surpassing the e-valuator baselines in terms of detection delay and simplicity. Notably, detection delays are consistently lower for CRC and UCB, indicating earlier intervention and mitigation, which is critical in most practical deployment contexts. Figure 1

Figure 1: CRC and UCB calibrate a single threshold to turn a PRM score into an alarm, yet detect incorrect answers earlier than e-valuators, which train several density estimators.

Monitoring Harmlessness and Red Teaming Scenarios

Harmlessness monitoring is studied via the Anthropic Red Teaming dataset and the FineHarm challenge. The verifier signals in these tasks derive from Llama Guard (for red teaming) and a fine-tuned Qwen2.5-1.5B SCM (for harmfulness), with power and detection delay compared across monitors.

A key result is that false alarm rate guarantees are preserved under both red teaming and fine-tuning conditions for all monitors; however, monitors leveraging a more appropriate verifier (SCM in FineHarm, trained for token-level harmfulness) achieve higher power, with near-perfect separation between safe and unsafe states. Simpler risk-controlling monitors continue to yield reduced detection delay relative to e-valuator approaches. Figure 2

Figure 2: All monitors maintain false alarm control; power and detection delay show that risk-controlling monitors detect harmful output earlier, especially when using stronger token-level verifiers.

Cost-Performance Trade-off: Internal vs. External Monitoring Signals

A practical consideration is the resource cost of running external verifiers in lockstep with a deployed LLM. The study examines replacing expensive verifier signals with the LLM's own per-step minimum token log-probabilities, corresponding to a zero additional compute deployment scenario.

Empirical results demonstrate that token log-probabilities as monitoring signals lead to substantially lower power, degrading the monitor’s ability to correctly flag unsafe outputs at matched false-alarm rates. While the false alarm calibration is still respected, the detection of unsafe outputs lags considerably, indicating that external reward models remain substantially more effective in this context. Figure 3

Figure 3: False alarm rate, power, and detection delay for PRM-based and log-probability-based signals, showing marked performance loss when relying on internal LLM signals.

Missed Detection Risk Control and Trade-offs

An important extension is calibration to control missed detection risk (RIIR^{II}), assessed in the mathematical reasoning domain. This capability is not available in sequential-hypothesis-test-based e-valuator methods, highlighting an advantage for the CRC/UCB paradigm. The threshold is calibrated over unsafe calibration samples, controlling the rate of missed unsafe outputs at a user-specified level (with a corresponding unavoidable increase in false alarms as dictated by the risk trade-off). Figure 4

Figure 4: CRC and UCB methods control the missed detection risk RIIR^{II} tightly, which e-valuator methods cannot guarantee.

Discussion and Implications

The work provides stringent statistical guarantees for online LLM monitoring, combining flexible, calibration-based thresholding with theoretical soundness and practical deployment feasibility. Key findings include:

  • Effectiveness of simple, calibrated monitors: A single-threshold monitor is empirically competitive with more complex sequential methods, offering lower detection delay without additional model complexity.
  • Signal quality bottleneck: The accuracy and utility of any such monitor fundamentally depend on the informativeness of the safety signal; fine-tuned or task-optimized verifiers are critical for high power.
  • Zero-overhead options come with significant trade-offs: Internal signals (e.g., log-probs) are inexpensive but degrade detection effectiveness, which quantifies the expected cost-performance trade-off in real-world deployments.
  • Controllable risk profiles: The ability to target and calibrate either false alarm or missed detection risks flexibly is significant, both for regulatory compliance scenarios and user-facing applications.

Future research directions include fusing multiple heterogeneous signals for increased robustness, developing more nuanced step-wise calibration protocols beyond time-invariant thresholding, and improving adversarial robustness and coverage in online monitoring scenarios.

Conclusion

This work systematically addresses real-time safety monitoring for LLMs with strong statistical control, efficient implementation, and extensive empirical validation. The proposed single-signal thresholding framework, with its calibration mechanisms, fills a critical gap in online LLM oversight methodologies, balancing detection speed, statistical rigor, and deployability. The results demonstrate the importance of both principled statistical monitoring and careful signal design in achieving practical LLM safety at inference time.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

Plain-English Summary of “Online Safety Monitoring for LLMs”

1. What is this paper about?

This paper is about keeping LLMs safe while they are typing out answers in real time. The authors build a simple “early warning system” that watches the model’s words as they appear and raises an alarm as soon as things look unsafe (for example, wrong math steps, harmful or toxic content, or helping with malicious requests).

2. What questions are the authors trying to answer?

The paper asks:

  • Can we turn a running “safety score” into a quick, reliable alarm that stops unsafe content as it is being generated?
  • Can a very simple method be both effective and trustworthy, giving guarantees like “we won’t raise too many false alarms”?
  • How does this simple method compare to more complicated monitors that use advanced statistics?
  • How much does monitor quality depend on the safety signal it watches?

3. How did they do it? (Methods in everyday language)

Think of the LLM’s answer as a stream of words. At each moment, a “safety meter” estimates how safe things look so far. The safety meter can come from:

  • An external “verifier” model (like a referee) that judges safety or correctness step by step, or
  • A cheaper, built-in signal from the LLM itself (like how confident it is in the next word), which costs less but might be less accurate.

The monitor itself is very simple:

  • It sets a cutoff line (a threshold). If the safety meter drops below that line at any point, the monitor sounds the alarm and you can stop the generation, switch to a stronger check, or call a human to review.

How do they pick the cutoff line?

  • They “practice” on a separate set of examples to calibrate the threshold so that the monitor behaves as promised later.
  • They offer two kinds of guarantees: 1) Control in expectation (called CRC): on average, the false alarm rate stays below a target level (like 5% or 10%). 2) High-probability control (called UCB): with high confidence, the false alarm rate stays below the target. This is stricter but more conservative.

What do they compare against?

  • A more complex family of methods (called e-valuators) that learn step-by-step patterns in the safety signal and use sequential hypothesis testing. These can be powerful but are heavier to set up and run.

4. What did they find, and why does it matter?

Here are the main results, explained simply:

  • The simple monitor actually works well:
    • It keeps the false alarm rate under control as promised, both for the average guarantee (CRC) and the stricter high-probability guarantee (UCB).
    • It often catches problems earlier in the output than the more complex methods. Catching issues early means users see less harmful text and you waste fewer tokens (less cost).
  • Math reasoning (factuality) tests:
    • When solving math problems step by step, the simple monitor spotted incorrect solutions sooner while still keeping false alarms low.
    • More complex methods sometimes found slightly more mistakes overall but tended to raise alarms later and, in a few settings, broke their promised error guarantees.
  • Safety and harmfulness tests (toxic or malicious content):
    • All methods kept false alarms under control.
    • The simple monitor again raised alarms earlier.
    • Performance depended a lot on the safety signal. A verifier trained specifically for token-level harmfulness worked great; a general-purpose safety tool was weaker. In short: better signals make better monitors.
  • Cost vs. performance trade-off:
    • Using the LLM’s own “confidence in its next word” (a free, cheap signal) made the monitor much weaker than using an external verifier trained for the task.
    • So, saving compute cost can hurt detection quality; teams need to choose the right balance for their needs.

Why this matters:

  • Early, reliable alarms limit user exposure to unsafe content and reduce waste.
  • Statistical guarantees help teams trust that the monitor won’t be overly jumpy (too many false alarms) or too sleepy (miss harmful stuff).

5. What’s the impact of this work?

This research shows that you don’t always need a complicated system to monitor LLM safety in real time. A simple, calibrated threshold on a good safety signal can:

  • Be easy to deploy,
  • Offer clear, math-backed promises about error rates, and
  • Catch problems early.

However, there are limits:

  • The monitor is only as good as the signal it watches. If your verifier is weak, the monitor will be weak too.
  • A single fixed threshold ignores time patterns (like early steps being noisier than later ones).

What could come next:

  • Combine multiple signals to be more accurate and robust.
  • Use per-step or adaptive thresholds that account for time patterns.
  • Trigger extra, targeted checks (or a human review) after an alarm.
  • Find the best trade-off between cost and safety for different applications.

Overall, the paper offers a practical, trustworthy way to keep LLMs safer while they’re generating text—like a simple, reliable smoke alarm that’s tuned to avoid both constant false alarms and dangerous misses.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

Below is a single, concrete list of what remains missing, uncertain, or unexplored in the paper, phrased so that future work can directly act on it.

  • Distribution shift and exchangeability: The guarantees assume calibration and deployment data are exchangeable; how to maintain risk control when prompt distributions, user populations, or attack strategies shift over time remains unaddressed. Develop online/adaptive recalibration with guarantees under covariate or concept shift.
  • Anytime/time-uniform guarantees in infinite streams: The risk definition uses “exists t” over a sequence, but the setting of continual, potentially unbounded streams (e.g., long-lived agents) is not treated. Establish time-uniform control for indefinite horizons without exploding false alarm probability.
  • Missed-detection risk as a primary objective: Experiments and exposition center on false alarm control; controlling missed-detection risk RIIR^{II} jointly with RIR^{I} (or optimizing power subject to RIR^{I} constraints) is left to the appendix and not comprehensively studied. Provide systematic methods for multi-objective calibration and evaluate trade-offs.
  • Joint control of multiple safety risks: The method calibrates one threshold for one risk. Real deployments require simultaneous control of toxicity, factuality, jailbreak compliance, etc. Design multi-risk, Pareto-efficient calibration that preserves per-risk guarantees.
  • Temporal structure and step heterogeneity: A single, time-invariant threshold ignores systematic dependence of the signal on position (e.g., earlier vs. later steps) and on conversation turn dynamics. Investigate per-step or context-conditional thresholds and quantify gains vs. added sample complexity.
  • Representativeness and size of calibration sets: The effect of calibration sample size nn and composition on power, delay, and guarantee tightness is not quantified. Provide sample-complexity analyses and practical guidance on minimal nn for target ϵ,δ\epsilon,\delta.
  • Tightness of high-probability bounds: Use of Hoeffding–Bentkus UCB may be conservative. Explore sharper bounds (e.g., empirical Bernstein, PAC-Bayesian) to reduce conservatism while preserving finite-sample validity.
  • Robustness to adversarial manipulation of signals: External verifiers (PRMs, safeguards) can be gamed; the paper notes this but does not evaluate adaptive attackers who shape sts_t to evade detection. Design adversarially robust monitors and test against adaptive red teamers.
  • Combining multiple signals: Only single-signal thresholding is evaluated. Quantify whether fusing external PRMs, internal token statistics (entropy/log-prob), semantic entropy, and tool signals yields better calibrated monitoring with provable guarantees.
  • Cost–latency–accuracy trade-offs: The paper highlights that external verifiers add inference cost but does not measure compute/latency and how tiered/cascading monitors (cheap→expensive) impact risk and detection delay. Provide end-to-end costed evaluations.
  • Calibration under label noise: Safety labels rely on o3-mini (MATH) and dataset-provided annotations (red teaming), which can be noisy. Analyze robustness of risk control to label errors and propose noise-aware calibration.
  • Length and segmentation bias: Risk is defined at sequence level with variable lengths; the method does not assess whether guarantees differ across very short vs. very long sequences or depend on step segmentation quality in chain-of-thought. Develop length-aware calibration or normalization.
  • Streaming with re-arming and multiple alarms: The monitor stops at the first crossing and assumes downstream intervention. Many applications require re-arming, continuing after a false alarm, or windowed operation. Extend the framework to multiple alarms with controlled error rates.
  • Decision-theoretic stopping: The paper reports detection delay but does not optimize a utility that trades off delay, false alarms, and intervention costs. Formulate and solve a cost-sensitive stopping problem under risk constraints.
  • Subgroup fairness and stratified guarantees: No analysis of whether false alarm control holds uniformly across topics, demographics, or languages. Develop stratified/calibrated thresholds per subgroup with global risk control.
  • Generalization beyond English text: Experiments cover English math and safety datasets only. Validate in multilingual settings and for code generation or tool-use contexts where safety manifests differently.
  • Modality and tool-use extension: The approach is text-only; safety incidents often arise in multimodal or tool-using agents. Define signals and guarantees for multimodal outputs and action-taking agents (e.g., API calls, code execution).
  • Partial or delayed labels in deployment: Real-time labels are unavailable or delayed; the framework presumes labeled calibration data only. Investigate semi-supervised or weakly supervised calibration and continual learning with delayed feedback.
  • Handling overlapping, context-dependent harms: Some “unsafe” content depends on broader context (e.g., summarizing versus endorsing harmful text). Study context-conditioned monitors that reduce spurious alarms while controlling risk.
  • Early-token harm prevention: Detection delay is measured as a fraction of steps; the paper does not assess whether alarms arrive before any harmful token is exposed to the user. Evaluate token-level time-to-harm and enforce guarantees on pre-exposure detection.
  • Interaction with downstream interventions: The system-level efficacy of different interventions (halt, escalate, human-in-the-loop) and their feedback on future monitoring is not quantified. Build closed-loop evaluations including escalation policies.
  • Calibration across tasks and domains: Thresholds are tuned per task/dataset. Explore unified or meta-calibrated thresholds transferable across domains, or fast adaptation methods that preserve guarantees with minimal new labels.
  • Stronger baselines and ablations: E-valuators rely on learned density estimators, but other sequential tests (e.g., e-detectors, betting-based martingales, WATCH) are not empirically compared under identical signals. Provide broader baseline coverage.
  • Internal signal search space: Only per-step min token log-prob is tested; broader internal statistics (e.g., semantic entropy probes, attention-map signals, gradient-based proxies) merit systematic evaluation with guarantees.
  • Privacy and logging constraints: Calibration and monitoring may require storing streams; the paper does not discuss privacy constraints or how to calibrate with privacy-preserving data access. Explore DP-aware calibration and logging minimization.
View Paper Prompt  View All Prompts 

Practical Applications

Overview

This paper proposes a simple, deployable online safety monitor for LLMs that raises alarms during generation when a streamed “safety signal” falls below a calibrated threshold. The monitor:

  • Converts any per-step signal s_t (e.g., external verifier probabilities from a PRM or content safeguard, or cheaper internal signals like token log-probabilities) into a binary alarm using a single, time-invariant threshold λ.
  • Calibrates λ with statistical risk guarantees via conformal risk control (in-expectation, CRC) or high-probability control (upper confidence bound, UCB), enabling tunable false-alarm or missed-detection rates.
  • Works in real time with negligible overhead, is model-agnostic, and can trigger interventions (halt generation, escalate to stronger verifier, or route to human review).

Below are practical applications grouped by deployment horizon. Each item includes sectors, possible tools/workflows, and feasibility assumptions.

Immediate Applications

These can be deployed now with existing verifiers, labeled calibration data, and standard LLM infrastructures.

  • Safe Streaming Chat for Consumer and Enterprise Assistants
    • Sectors: Software, Customer Support, Productivity Apps
    • What: Wrap chat generation with CRC/UCB thresholding over safeguard signals (e.g., Llama Guard) to halt or redact unsafe tokens mid-stream.
    • Tools/Workflow: “Safe Streaming” SDK; per-deployment calibration pipeline; configurable risk budgets (ε, δ); human-in-the-loop escalation.
    • Assumptions/Dependencies: Labeled calibration data sampled from production-like prompts; stable distribution (exchangeability); reliable safeguard signal.
  • Math/Technical Tutoring with Early Error Stops
    • Sectors: Education, EdTech
    • What: Use step-level process reward models (PRMs) as s_t to detect reasoning errors and interrupt or request self-correction before showing students incorrect steps.
    • Tools/Workflow: PRM-backed tutor; UCB for stricter guarantees; auto “retry/re-explain” routines; logging for teacher review.
    • Assumptions/Dependencies: High-quality PRM for target domain (e.g., MATH); step segmentation available; calibration data aligned to student tasks.
  • Guardrails for Coding Assistants
    • Sectors: Software Engineering, Security
    • What: Stream code tokens through risk-controlled monitors using signals from linters, unit tests, static analysis, or learned code-verifiers to stop insecure/incorrect code suggestions.
    • Tools/Workflow: IDE extension (VS Code/JetBrains) safety plugin; auto-test snippets per chunk; human confirm before paste.
    • Assumptions/Dependencies: Fast verifiers for incremental code; domain-calibrated thresholds; representative calibration set.
  • Enterprise Knowledge Agents with Hallucination Control
    • Sectors: Enterprise Knowledge Management, Support Ops
    • What: Combine retrieval confidence/NLI verifiers as s_t to stop unsupported claims and force cite-or-retrieve steps during generation.
    • Tools/Workflow: RAG-integrated monitor; escalation to stronger verifier or SME approval when s_t < λ.
    • Assumptions/Dependencies: Informative signals tied to knowledge retrieval; calibration data with verified answers; stable query distribution.
  • Content Moderation for UGC Platforms
    • Sectors: Social Media, Community Platforms
    • What: Apply streaming monitors to detect hate speech, self-harm, or policy violations as posts/comments are drafted by AI assistants.
    • Tools/Workflow: Inline auto-redaction; “reason for block” messaging; appeal pipeline.
    • Assumptions/Dependencies: Well-calibrated safeguard classifiers; clear policy labels; platform-appropriate risk budget.
  • Compliance Guardrails in Regulated Domains
    • Sectors: Healthcare, Legal, Finance
    • What: Deploy UCB-calibrated monitors with low ε to minimize policy-violating content (e.g., medical advice disclaimers, legal counsel, financial recommendations).
    • Tools/Workflow: Policy-specific verifiers; mandatory escalation to licensed professionals; audit logs with statistical guarantees.
    • Assumptions/Dependencies: Domain-specialized verifiers; tight calibration sets; governance for false-positive handling.
  • Agentic Workflow Safeguards (Tool Use / Automation)
    • Sectors: Robotic Process Automation, DevOps, IT Ops
    • What: Stream action plans/tool calls through monitors; halt or require approval upon unsafe intents (e.g., deleting data, exfiltration-like patterns).
    • Tools/Workflow: Integrations with agent frameworks (e.g., LangChain); risk-controlled “kill switch”; tiered approval.
    • Assumptions/Dependencies: Action-level signals; labeled unsafe/benign actions in calibration; low-latency checks.
  • Safety SLAs and Dashboards for AI Ops
    • Sectors: ML Platform/Operations, SaaS Vendors
    • What: Use risk-controlled thresholds to enforce target false-alarm (or missed detection) rates; monitor detection delay and power as KPIs.
    • Tools/Workflow: Analytics dashboards; A/B tests comparing CRC vs UCB; automated re-calibration pipelines.
    • Assumptions/Dependencies: Continuous performance telemetry; procedures for data shifts; governance alignment on ε, δ.
  • Red Teaming and Evaluation Pipelines
    • Sectors: AI Safety, QA/Testing
    • What: Use monitors to stop harmful completions earlier, measure detection delay, and benchmark guardrail effectiveness across datasets (e.g., FineHarm, Red Teaming).
    • Tools/Workflow: Batch harnesses; report power/FAR vs. ε; automate failure capture for retraining.
    • Assumptions/Dependencies: Access to realistic adversarial prompts; careful dataset curation.
  • Cost-Aware Monitoring in Resource-Constrained Settings
    • Sectors: Edge AI, SMBs
    • What: Use internal signals (e.g., token log-probabilities) when external verifier cost is prohibitive; accept lower power at matched false-alarm rates.
    • Tools/Workflow: Dynamic switching between internal and external signals; policy-based cost/performance tiers.
    • Assumptions/Dependencies: Acknowledged performance trade-offs; explicit user risk budget; ability to upgrade signals for critical tasks.

Long-Term Applications

These require further research, stronger verifiers, scaling, or standardization to achieve high reliability across domains and adversarial contexts.

  • Multi-Signal Fusion and Adaptive Thresholding
    • Sectors: Cross-sector AI Safety
    • What: Combine PRMs, safeguard logits, retrieval/NLI confidence, and internal uncertainties with per-step thresholds (e.g., Pareto testing) to boost robustness.
    • Dependencies: Algorithms for robust fusion; larger labeled datasets; shift-aware calibration.
  • Domain-Grade Verifiers with Certification
    • Sectors: Healthcare, Legal, Finance, Government
    • What: Train/regulate specialized verifiers for policy, factuality, and risk—supporting legally recognized guarantees in regulated contexts.
    • Dependencies: Standardized taxonomies; external audits; certification processes.
  • Continuous Monitoring under Distribution Shift
    • Sectors: ML Ops, Safety Engineering
    • What: Integrate change detection and anytime-valid inference to maintain guarantees when prompts or user bases evolve.
    • Dependencies: Shift detectors; re-calibration on-the-fly; data governance for privacy-safe collection.
  • Adversarially Robust Monitoring (Anti-Jailbreak)
    • Sectors: Security, Platform Integrity
    • What: Fortify monitors against prompt obfuscations and adaptive adversaries; detect attempts to suppress/poison s_t.
    • Dependencies: Red-team-informed training; stress tests; theoretical bounds under adaptive attacks.
  • Compute-Aware Reasoning Control
    • Sectors: Education, Research Assistants
    • What: Integrate with compute-budgeted reasoning (e.g., “stop thinking” policies) to trade off cost and risk with formal guarantees.
    • Dependencies: Joint optimization of compute and risk; task-aware calibration sets.
  • Autonomous Agents and Robotics Safeguards
    • Sectors: Robotics, Industrial Automation, Logistics
    • What: Stream plans and sensor-derived signals; enforce risk-controlled stops before unsafe physical actions.
    • Dependencies: Real-time constraints; reliable action-level semantics; simulator-backed calibration.
  • Time-to-Unsafe Forecasting and Oversight Allocation
    • Sectors: Trust & Safety, Moderation Ops
    • What: Predict expected “time to unsafe” to dynamically allocate human reviewers and stronger verifiers where most effective.
    • Dependencies: Longitudinal data; survival-style modeling; operations integration.
  • Federated/On-Device Monitoring
    • Sectors: Mobile, Edge, Consumer Devices
    • What: Run lightweight monitors locally with privacy-preserving, federated calibration updates.
    • Dependencies: Efficient verifiers; secure aggregation; device variability handling.
  • Standards and Certification for Runtime Safety Guarantees
    • Sectors: Policy, Compliance, Auditing
    • What: Embed risk-controlled monitoring into compliance regimes (e.g., EU AI Act), with documented ε/δ budgets and audit trails.
    • Dependencies: Consensus on metrics and test suites; traceable logging; third-party auditors.
  • Guardrails Marketplace and Interoperability
    • Sectors: Platform Ecosystems
    • What: Pluggable monitors with standardized interfaces and calibration recipes per domain.
    • Dependencies: API standards; shared evaluation benchmarks; economic models for verification providers.
  • Personalized Risk Budgets in Education and Consumer Apps
    • Sectors: EdTech, Wellness, Parental Controls
    • What: User- or cohort-specific ε policies (e.g., stricter for minors) with adaptive recalibration.
    • Dependencies: Ethical frameworks; consent and privacy safeguards; UI to convey guarantees.

Key Assumptions and Dependencies (Cross-Cutting)

  • Calibration requires labeled data exchangeable with deployment data; guarantees can degrade under distribution shift without re-calibration.
  • Monitors inherit limitations of the chosen signal; external verifiers generally outperform internal signals but add latency/cost.
  • Guarantees target specific risks (e.g., false alarms); missed-detection control requires separate calibration and may reduce power.
  • Real-time deployments must meet latency budgets; per-step scoring and interventions need to be efficient.
  • Human-in-the-loop escalation procedures and governance are crucial to manage false positives and ensure responsible interventions.
View Paper Prompt  View All Prompts 

Glossary

  • Ablation: An experimental analysis where components are removed or varied to study their effect on performance. Example: "We run a signal ablation on Mistral-7B-Instruct reasoning chains on the MATH dataset."
  • Autoregressive generation: A modeling approach where each token is generated conditioned on previously generated tokens. Example: "each oto_t is sampled autoregressively from the model's generative distribution Pθ(ot∣x,o1:t−1){P_\theta(o_t \mid x, o_{1:t-1})}."
  • Calibration dataset: A held-out labeled set used to tune parameters (e.g., thresholds) so that desired risk guarantees hold at deployment. Example: "a labeled held-out calibration dataset $_{\text{cal} = \{(x^{(i)}, o_{1:T}^{(i)}, y^{(i)})\}_{i=1}^n$"
  • Conformal risk control: A framework that calibrates decision rules to control a target risk level, with guarantees averaged over draws of the calibration data. Example: "The first option, conformal risk control \citep{angelopoulos2022conformal}, provides a way to find a threshold λ^\hat{\lambda} on a given calibration dataset $D_{\text{cal}$, such that the risk, i.e. expected loss, of future test points is bounded on average..."
  • Control in expectation: A guarantee that a chosen procedure meets a risk target on average over draws of the calibration set. Example: "Control in expectation."
  • Detection delay: The fraction of a sequence observed before the monitor correctly raises an alarm on unsafe outputs. Example: "Detection Delay, the average fraction of steps the model processes before an alarm is correctly raised, DD^=1∣I∩I∣∑i∈I∩Itmin⁡(i)/T(i)\hat{DD} = \frac{1}{|I \cap I|} \sum_{i \in I \cap I} t_{\min}^{(i)} / T^{(i)}"
  • E-process: A nonnegative evidence process used to conduct sequential hypothesis tests with anytime-valid guarantees. Example: "The resulting e-process is used as evidence signal for a sequential hypothesis test."
  • E-valuator: A monitoring method that learns per-step density estimators and uses an e-process for sequential testing. Example: "We compare the simple risk control monitor to two versions of the e-valuator \citep{sadhuka2025valuator}."
  • Exchangeability: A statistical property where data points are identically distributed but not necessarily independent, allowing permutation-invariant assumptions. Example: "drawn exchangeably from the same distribution as the deployment data."
  • False alarm rate: The proportion of safe sequences incorrectly flagged by the monitor. Example: "False Alarm Rate, the share of safe sequences that are incorrectly flagged, R^I()=∣I∩I∣ / ∣I∣\hat{R}^{I}() = |I \cap I| \,/\, |I|"
  • False alarm risk: The probability that the monitor flags a safe sequence (Type I error). Example: "The false alarm risk captures the probability of flagging a sequence that is in fact safe (type~I error, RIR^{I})"
  • Guardrails: Protective mechanisms applied at inference time to prevent or mitigate unsafe model outputs. Example: "There has thus been substantial effort into developing guardrails at inference time"
  • High-probability control: A stronger guarantee that the risk bound holds with at least a specified probability over the calibration draw. Example: "In contrast to in-expectation control, high-probability control offers a stronger deployment guarantee"
  • Hoeffding-Bentkus bound: A finite-sample concentration inequality used to upper-bound binomial tail probabilities, here used to bound empirical risk. Example: "We use the Hoeffding-Bentkus bound \citep{bentkus2004hoeffding} as UCB."
  • Human-in-the-loop: A process where human reviewers are involved in decision-making or oversight during system operation. Example: "invoking a human-in-the-loop review."
  • Missed detection risk: The probability of failing to flag an unsafe sequence (Type II error). Example: "the missed detection risk captures the probability of failing to raise an alarm on a sequence that is unsafe (type~II error, RIIR^{II})."
  • PAC (Probably Approximately Correct): A framework providing probabilistic guarantees that performance is close to optimal within a specified tolerance. Example: "e-valuator-anytime, and e-valuator-PAC -- successfully control the false alarm rate"
  • Pareto testing: A procedure for jointly controlling multiple risks by trading off among them efficiently. Example: "such as Pareto testing \citep{laufer2022efficiently}."
  • Power: The fraction of unsafe sequences correctly flagged by the monitor (1 minus missed detection rate). Example: "Power, the share of unsafe sequences that are correctly flagged, 1−R^II()=∣I∩I∣ / ∣I∣1 - \hat{R}^{II}() = |I \cap I| \,/\, |I|"
  • Process Reward Model (PRM): A model that scores intermediate reasoning steps for correctness or quality to guide or evaluate reasoning chains. Example: "process reward model (PRM) \citep{lightman2024let, wang2024math, prmlessons}"
  • Red teaming: An adversarial evaluation practice where humans attempt to elicit unsafe model behavior to test defenses. Example: "red teaming datasets"
  • Risk control: The practice of calibrating decision rules to bound error probabilities (risks) at target levels. Example: "a simple statistical framework based on risk control \citep{angelopoulos2022conformal}"
  • Safeguard classifier: A model that predicts the harmfulness or safety of content to moderate LLM interactions. Example: "such as the predictive probability from a safeguard classifier."
  • SCM verifier: A specific safeguard model used for streaming content monitoring of harmful tokens. Example: "we use their SCM verifier, Qwen2.5-1.5B fine-tuned for token-level harmfulness detection"
  • Sequential hypothesis testing: Statistical testing where data are evaluated as they arrive, allowing early stopping with controlled error rates. Example: "more advanced monitors based on sequential hypothesis testing."
  • Stopping rule: A rule that determines when to stop observing a sequence and issue a decision (e.g., raise an alarm). Example: "We define it as a stopping rule governed by a single threshold λ\lambda"
  • Token log-probability: The logarithm of a model’s predicted probability for each generated token, often used as a confidence proxy. Example: "we use the generator's own token log-probabilities"
  • Upper Confidence Bound (UCB): A high-probability upper bound on an unknown quantity (e.g., risk) used for conservative decision-making. Example: "a (1−δ)(1-\delta) upper confidence bound (UCB) U(λ,n,δ)U(\lambda, n, \delta)"
  • Verifier signal: A scalar score from a verifier model indicating the estimated safety of the current output. Example: "turns a verifier signal from an external model into an alarm decision by thresholding"
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 13 tweets with 28 likes about this paper.

Sign Up for Free