Online Safety Monitoring for LLMs

• Online safety monitoring for LLMs is the real-time tracking of partial outputs, internal signals, and auxiliary data to detect when generated content becomes unsafe.
• It employs calibrated threshold rules and sequential hypothesis tests to balance false alarm and missed detection risks during runtime.
• Recent research extends these methods to token-level harmfulness detection, multi-turn analysis, and stateful aggregation for comprehensive oversight.

Online safety monitoring for LLMs is the runtime practice of observing a model’s partial outputs, auxiliary signals, or internal states during generation and triggering an alarm or intervention when safety can no longer be assumed. The central motivation is deployment-time residual risk: despite alignment training, LLMs remain prone to generating unsafe outputs, so monitoring must operate online rather than only through offline evaluation. Recent work formalizes this as a streaming decision problem over partial trajectories, with monitors ranging from calibrated threshold rules over verifier scores to sequential hypothesis tests, token-level harmfulness detectors, temporal-logic monitors, reasoning-process monitors, and stateful systems that aggregate weak signals across many accounts (Schirmer et al., 2 Jul 2026).

1. Formal problem setting

A canonical formulation treats interaction as an online stream. A user prompt $x$ is sampled at $t=0$, the LLM generates a sequence $o_{1:T}$ with autoregressive distribution $P_\theta(o_t \mid x, o_{1:t-1})$, and at each time $t$ an external verifier produces a safety signal $s_t$ from the partial trajectory $(x,o_{1:t})$. A binary alarm decision $a_t$ is emitted online. The final output is associated with a binary safety variable $y \in \{0,1\}$, where $y=1$ denotes safe and $t=0$0 unsafe, sampled from $t=0$1. The monitoring objective is to identify unsafe sequences as early as possible while controlling a deployment-time risk metric (Schirmer et al., 2 Jul 2026).

The basic risk-control targets are false alarm risk and missed detection risk. For a threshold $t=0$2 on the verifier signal, the false alarm risk under safe sequences is

$t=0$3

and the missed detection risk under unsafe sequences is

$t=0$4

The goal is to choose $t=0$5 so that a selected risk satisfies $t=0$6 for a user-specified $t=0$7, either in expectation or with high probability over calibration draws. This framing makes the online monitor an explicitly statistical object rather than an ad hoc refusal heuristic (Schirmer et al., 2 Jul 2026).

This formulation is broader than a single content-moderation setting. The same structure can represent factuality monitoring in mathematical reasoning, harmlessness monitoring in red-teaming dialogs, token-level harmfulness detection in streamed moderation, and more general runtime oversight of LLM agents. A plausible implication is that the essential abstraction is not the content category itself, but the existence of a streaming signal whose excursions can be mapped to operational risk.

2. Monitor architectures and decision rules

The simplest monitor architecture accepts any scalar signal $t=0$8 that correlates with the eventual safety label. In the threshold-with-risk-control design, representative signals include an external verifier probability $t=0$9, step-level process reward model scores, safeguard probabilities, or an internal generator signal derived from token log-probabilities. For mathematical reasoning, the paper uses the step-level PRM score from Qwen2.5-Math-PRM-7B; for content moderation, the signal is the safeguard’s predicted probability of safety at turn $o_{1:T}$0; and for a cheap internal alternative it aggregates token log-probabilities within a reasoning step $o_{1:T}$1 as

$o_{1:T}$2

so a step containing an unusually uncertain token receives a more negative score (Schirmer et al., 2 Jul 2026).

The alarm rule is a single global threshold applied uniformly across time:

$o_{1:T}$3

Equivalently, the pointwise stopping rule can be written

$o_{1:T}$4

so once alarmed, the monitor remains in the alarmed state. Time enters only through the stopping logic; otherwise the method is a plain threshold on $o_{1:T}$5 without additional smoothing. This is one reason its deployment overhead is negligible relative to monitors that require time-dependent learned models (Schirmer et al., 2 Jul 2026).

More complex architectures modify the signal rather than the stopping logic. Streaming Content Monitor (SCM) trains a token-level harmfulness detector that runs in parallel with autoregressive generation. It uses dual supervision of response- and token-level labels and a logic consistency term, then halts when the count of harmful tokens in the prefix reaches $o_{1:T}$6:

$o_{1:T}$7

In experiments, SCM achieves macro F1 score of at least $o_{1:T}$8 while only seeing the first $o_{1:T}$9 of tokens in responses on average, which directly targets latency and exposure reduction in streaming moderation (Li et al., 11 Jun 2025).

A separate line of work addresses decomposition attacks, where malicious intent emerges only across a sequence of benign-looking subtasks. There the monitor evaluates cumulative context $P_\theta(o_t \mid x, o_{1:t-1})$0 after each turn, emits a score $P_\theta(o_t \mid x, o_{1:t-1})$1, and halts when $P_\theta(o_t \mid x, o_{1:t-1})$2. The relevant objective is no longer merely whether a single partial output is harmful, but whether the sequence has accumulated enough evidence of malicious intent before the harmful index is reached (Yueh-Han et al., 12 Jun 2025).

3. Calibration, risk control, and sequential testing

Risk control in the threshold framework is based on a labeled held-out calibration dataset

$P_\theta(o_t \mid x, o_{1:t-1})$3

assumed exchangeable with deployment data. Over the safe subset $P_\theta(o_t \mid x, o_{1:t-1})$4, the empirical false alarm rate is

$P_\theta(o_t \mid x, o_{1:t-1})$5

with an analogous empirical missed detection rate over the unsafe subset $P_\theta(o_t \mid x, o_{1:t-1})$6 (Schirmer et al., 2 Jul 2026).

For in-expectation control, the monitor uses Conformal Risk Control (CRC). The calibrated threshold for false alarm control is

$P_\theta(o_t \mid x, o_{1:t-1})$7

which guarantees

$P_\theta(o_t \mid x, o_{1:t-1})$8

For missed detection control, because $P_\theta(o_t \mid x, o_{1:t-1})$9 is monotone decreasing, the valid threshold is the smallest admissible one:

$t$0

These guarantees depend on exchangeability and monotonicity of the indicator loss in $t$1, but they are distribution-free with respect to the unknown score distribution (Schirmer et al., 2 Jul 2026).

For high-probability control, the same paper uses an upper confidence bound based on Hoeffding-Bentkus. If $t$2 satisfies

$t$3

then the selected threshold

$t$4

ensures

$t$5

The parallel construction for missed detection uses $t$6 and chooses the minimum valid threshold. Importantly, the guarantees do not require the verifier probabilities to be perfectly calibrated; empirical risk control on labeled calibration sequences is enough if exchangeability holds (Schirmer et al., 2 Jul 2026).

The principal sequential-testing baseline in this literature is the E-valuator family. It casts monitoring as a sequential hypothesis test over score sequences $t$7, with

$t$8

An e-process $t$9 is evaluated online and $s_t$0 is rejected when $s_t$1. In the anytime variant, the ideal density-ratio process is

$s_t$2

and Ville’s inequality gives

$s_t$3

Because $s_t$4 and $s_t$5 are unknown, the implemented monitor uses stepwise discriminative models $s_t$6 and plugs in

$s_t$7

This yields anytime-valid sequential testing, but it requires learned time-dependent models over score histories rather than a single calibrated threshold (Schirmer et al., 2 Jul 2026).

4. Empirical behavior and benchmark results

The threshold-with-risk-control monitor is evaluated on mathematical reasoning and red-teaming datasets. In mathematical reasoning on MATH, Claude Haiku 4.5 is reported at approximately $s_t$8 correct and Mistral-7B-Instruct-v0.3 at approximately $s_t$9 correct, with final correctness labeled by OpenAI o3-mini as an LLM-as-judge. In harmlessness monitoring, the evaluation uses Anthropic Red Teaming and FineHarm, the latter with an SCM verifier based on Qwen2.5-1.5B fine-tuned for token-level harmfulness detection (Schirmer et al., 2 Jul 2026).

The primary metrics are false-positive rate, true-positive rate or power, and detection delay. Sequence-level false-positive rate is identified with $(x,o_{1:t})$0; power is

$(x,o_{1:t})$1

and detection delay is

$(x,o_{1:t})$2

where $(x,o_{1:t})$3 is the flagged set, $(x,o_{1:t})$4 the unsafe set, and $(x,o_{1:t})$5 the first alarm time. Although AUROC, precision–recall, and $(x,o_{1:t})$6 are standard, this line of work focuses on false alarm rate, power, and detection delay because they align more directly with operational intervention (Schirmer et al., 2 Jul 2026).

Several empirical regularities recur. First, risk control holds as advertised: UCB and E-valuator-PAC satisfy high-probability false alarm control, CRC satisfies in-expectation control, and E-valuator-anytime violated bounds at $(x,o_{1:t})$7 on Mistral-7B-Instruct. Second, CRC and UCB raise alarms earlier than the sequential-testing baselines, roughly halfway through sequences on MATH. Third, signal quality is decisive: on FineHarm, the SCM verifier yields power near $(x,o_{1:t})$8 for CRC and UCB, whereas Llama Guard’s power on Red Teaming plateaus below $(x,o_{1:t})$9. Fourth, with $a_t$0 on MATH, PRM-based monitors achieve power above $a_t$1, while log-probability-based monitors reach only about $a_t$2 despite near-matched false alarm rates (Schirmer et al., 2 Jul 2026).

These findings sharpen the practical trade-off. The threshold monitor is not universally strongest on every operating point, but it achieves rigorous guarantees, earlier detection, and lower deployment complexity. The paper’s overall conclusion is that the threshold-with-risk-control monitor is competitive and often preferable in deployment because it attains guarantees, achieves earlier detection, and is far simpler to maintain (Schirmer et al., 2 Jul 2026).

5. Broader paradigms of online monitoring

Online safety monitoring has expanded beyond calibrated thresholding over scalar verifier scores. One major branch is token-stream moderation. SCM constructs FineHarm, a dataset of 29K prompt–response pairs with fine-grained annotations, then trains a plug-and-play moderator that emits per-token harmfulness probabilities and halts generation when the count of detected harmful tokens crosses a threshold. The reported result is macro F1 of at least $a_t$3 while observing only the first $a_t$4 of response tokens on average, showing that partial detection can approach full-detection performance when the model is trained natively for incomplete prefixes rather than repurposed from full-output moderation (Li et al., 11 Jun 2025).

A second branch addresses intent that is only visible across a sequence. In decomposition attacks, malicious goals are split into benign subtasks that evade single-turn refusal systems. The monitoring framework therefore evaluates cumulative context after each step, not isolated prompts. The paper reports an average post-decomposition attack success rate on GPT-4o of about $a_t$5 across question answering, text-to-image, and agent tasks, and then shows that optimized lightweight sequential monitors achieve up to about $a_t$6 defense success rate while cutting cost by about $a_t$7 and latency by about $a_t$8 relative to heavyweight reasoning monitors (Yueh-Han et al., 12 Jun 2025).

A third branch extends monitoring over time or across users. Formal-methods monitors such as TRAC represent temporally extended behavioral constraints in Linear Temporal Logic, emit 3-valued verdicts over finite prefixes, and support predictive monitoring plus runtime interventions; experiments show that these methods significantly reduce the violation rates of LLM-based agents while largely preserving task performance (Alamdari et al., 15 May 2026). Stateful online monitoring pushes the aggregation axis from time to population: by clustering weak suspiciousness signals across many transcripts and escalating only rarely to a cross-context LLM, a stateful monitor catches distributed attacks $a_t$9 earlier and flags cyber misuse before it reaches the most harmful stages, with negligible additional latency for approximately $y \in \{0,1\}$0 of user traffic (Brown et al., 29 May 2026).

A fourth branch inspects the model’s internals or reasoning process rather than only its external content. Activation watermarking treats robust monitoring as a security game under adaptive attackers and introduces keyed uncertainty into hidden activations; under adaptive attackers who know the monitoring algorithm but not the secret key, activation watermarking outperforms guard baselines by up to $y \in \{0,1\}$1 (Aremu et al., 24 Mar 2026). Reasoning-process monitors shift attention from content safety to reasoning safety, defined in terms of logical consistency, computational efficiency, and resistance to adversarial manipulation; the reported Reasoning Safety Monitor achieves up to $y \in \{0,1\}$2 step-level localization accuracy and $y \in \{0,1\}$3 error-type classification accuracy on unsafe reasoning trajectories (Wang et al., 26 Mar 2026).

Taken together, these strands suggest that online safety monitoring is no longer a single technique family. It is better understood as a set of runtime assurance paradigms indexed by what is being monitored: scalar verifier scores, token streams, sequence-level intent, cross-session coordination, internal activations, temporal rule satisfaction, or explicit reasoning trajectories.

6. Deployment constraints, failure modes, and research directions

In deployment, the main integration choices concern signal source, calibration regime, and intervention policy. External PRMs or safeguards require an additional forward pass per step or turn, while internal signals such as token log-probabilities are effectively free but substantially weaker. Once a threshold is fixed, the runtime action is minimal: compare $y \in \{0,1\}$4 to $y \in \{0,1\}$5, and on alarm halt generation, invoke a stronger verifier, or route to human-in-the-loop review. Early alarms reduce both token-generation cost and user exposure to unsafe content (Schirmer et al., 2 Jul 2026).

Maintaining guarantees under real traffic depends on distributional stability. If exchangeability between calibration and deployment fails, the realized risk can deviate from the target. The practical recommendation is periodic recalibration with fresh labeled traces, out-of-distribution checks, and, if necessary, more conservative thresholds or escalation to stronger verifiers. Additional failure modes include verifier brittleness under adversarial manipulation, context leakage when a verifier accidentally conditions on future tokens, and the general dependence of monitoring quality on the informativeness of the safety signal (Schirmer et al., 2 Jul 2026).

Current limitations are structural as well as operational. A single global threshold ignores systematic time dependence in $y \in \{0,1\}$6; long-horizon and multi-turn settings complicate credit assignment; and external safeguards can be expensive. This motivates several extensions already identified in the literature: adaptive thresholds $y \in \{0,1\}$7, conformal sequential methods that combine risk control with martingale or e-process ideas, multi-signal fusion over content, context, and internal uncertainty, and hierarchical monitors that use cheap signals for pre-screening before escalating to stronger verifiers (Schirmer et al., 2 Jul 2026).

Domain-specific evidence further indicates that implicit harms remain difficult. In AI companion safety, stronger LLM judges achieve high overall accuracy but still struggle with nuanced categories such as manipulation and with benign conversations that are incorrectly identified as harmful, suggesting that explicit harms are easier to monitor online than relational or context-dependent harms (Ren et al., 3 Jun 2026). In online grooming prevention, no evaluated model was clearly appropriate, prompt design heavily altered model behavior, and open-source models showed higher likelihood of harmful answer generation, which suggests that online safety monitoring must often compensate for model inconsistency rather than merely detect rare outliers (Prosser et al., 2024).

A broader organizational direction is capability-based monitoring, which treats “Safety Guardrail” as a reusable capability rather than a task-specific property. This approach is motivated by the claim that monitoring along shared capabilities such as safety guardrails, reasoning, translation, or summarization enables cross-task detection of systemic weaknesses, long-tail errors, and emergent behaviors that task-based monitoring may miss (Kellogg et al., 5 Nov 2025). A plausible implication is that online safety monitoring will increasingly be organized as a layered system: calibrated per-output alarms at the lowest level, stateful or temporal aggregation in the middle, and capability-level governance and drift detection at the highest level.