Predict, Don't React: Value-Based Safety Forecasting for LLM Streaming

Abstract: In many practical LLM deployments, a single guardrail is used for both prompt and response moderation. Prompt moderation operates on fully observed text, whereas streaming response moderation requires safety decisions to be made over partial generations. Existing text-based streaming guardrails commonly frame this output-side problem as boundary detection, training models to identify the earliest prefix at which a response has already become unsafe. In this work, we introduce StreamGuard, a unified model-agnostic streaming guardrail that instead formulates moderation as a forecasting problem: given a partial prefix, the model predicts the expected harmfulness of likely future continuations. We supervise this prediction using Monte Carlo rollouts, which enables early intervention without requiring exact token-level boundary annotations. Across standard safety benchmarks, StreamGuard performs strongly both for input moderation and for streaming output moderation. At the 8B scale, StreamGuard improves aggregated input-moderation F1 from 86.7 to 88.2 and aggregated streaming output-moderation F1 from 80.4 to 81.9 relative to Qwen3Guard-Stream-8B-strict. On the QWENGUARDTEST response_loc streaming benchmark, StreamGuard reaches 97.5 F1, 95.1 recall, and 92.6% on-time intervention, compared to 95.9 F1, 92.1 recall, and 89.9% for Qwen3Guard-Stream-8B-stric, while reducing the miss rate from 7.9% to 4.9%. We further show that forecasting-based supervision transfers effectively across tokenizers and model families: with transferred targets, Gemma3-StreamGuard-1B reaches 81.3 response-moderation F1, 98.2 streaming F1, and a 3.5% miss rate. These results show that strong end-to-end streaming moderation can be obtained without exact boundary labels, and that forecasting future risk is an effective supervision strategy for low-latency safety intervention.

• The paper introduces a value-based risk forecasting approach that anticipates unsafe outputs in LLM streaming, shifting from reactive to proactive intervention.
• It employs Monte Carlo rollouts and soft risk targets to decouple safety supervision from token-level annotations, enhancing robustness and transferability.
• Empirical results demonstrate improved F1 scores and reduced miss rates, validating StreamGuard's efficacy for real-time safety enforcement in LLM deployments.

Value-Based Safety Forecasting for LLM Streaming: StreamGuard

Motivation and Problem Formulation

Current LLM deployments rely on content moderation guardrails for both input and output safety. While input moderation operates over fully realized text, output-side moderation in streaming settings must assess safety and potentially intervene before completions are finalized, often under tight latency constraints. Previous approaches to streaming output moderation predominantly use boundary detection, training models to fire at the precise point where explicit policy violations manifest in the generated text. This approach is heavily dependent on fine-grained, token-level unsafe annotations that are expensive to obtain and inherently fragile due to their entanglement with tokenizer-specific coordinate systems.

StreamGuard introduces a paradigm shift by reframing streaming output moderation as a value-based risk forecasting problem. Rather than reactively detecting the boundary where output text becomes unsafe, StreamGuard predicts the scalar expected harmfulness of all plausible continuations given a partial response prefix. This realigns the objective from post-hoc detection to anticipatory intervention, aiming for earlier, policy-compliant truncation of unsafe responses while reducing annotation and transfer costs.

Figure 1: Comparison between boundary-detection and risk-forecasting approaches for streaming moderation; StreamGuard predicts future risk, enabling earlier interventions without requiring exact token-level unsafe boundary annotations.

Methodology

Prefix-Conditioned Future Risk

At each prefix during response generation, StreamGuard estimates a future-risk value as the expected judgment of a safety oracle over sampled continuations:

$$V^*(x, y_{\leq t}) = \mathbb{E}_{\tilde{y}_{t+1:T} \sim q(\cdot | x, y_{\leq t})} [\mathcal{O}(x, y_{\leq t} \tilde{y}_{t+1:T})]$$

where $x$ is the prompt, $y_{\leq t}$ the prefix, $q$ the rollout distribution (potentially a mixture across multiple LM generators), and $\mathcal{O}$ the safety judge scoring responses in $[0,1]$.

Targets are approximated via Monte Carlo rollouts: for each prefix, multiple plausible continuations are sampled, independently judged, and the average is taken as a soft target. This soft supervision signal assigns high risk to prefixes that reliably lead to unsafe completions—even prior to any explicit violation—thus supporting early and informed intervention.

Training and Generalization

StreamGuard attaches a lightweight classification head to each decoding step, predicting the risk score for the current prefix. The model is trained with binary cross-entropy against soft risk targets for intermediate prefixes and hard labels at completion. Importantly, supervision is attached to decoded text prefixes rather than tokenizer-aligned indices, permitting seamless transfer to models with non-matching tokenizers and architectural differences.

Monte Carlo target construction is made robust through (i) the use of mixed rollout sources to reduce generator bias, (ii) selective supervision schedules to manage computation cost, and (iii) reduction strategies (mean, median, max, min) to elicit a desired risk calibration/operating point during deployment.

Empirical Results

Standard Moderation Benchmarks

StreamGuard was evaluated on a comprehensive suite of input and output moderation benchmarks (ToxicChat, OpenAIModeration, Aegis, HarmBench, SimpleSafetyTests, BeaverTails, etc.) against both post-hoc (e.g., LlamaGuard, WildGuard) and streaming (e.g., Qwen3Guard-Stream) baselines.

• Input Moderation: Llama3-StreamGuard-8B achieved a macro-averaged F1 of 88.2, outperforming Qwen3Guard-Stream-8B-strict by +1.5 F1, and was particularly strong on Aegis2.0 and WildGuardTest.
• Streaming Output Moderation: On aggregated streaming output moderation, Llama3-StreamGuard-8B scored 81.9 F1 (vs. 80.4 for Qwen3Guard-Stream-8B-strict). On the QwenGuardTest streaming benchmark, it reached 97.5 F1, 95.1 recall, and 92.6% on-time interventions; miss rate improved from 7.9% to 4.9%.

Early Intervention and Overblocking

StreamGuard achieved higher on-time intervention rates and lower miss rates than boundary-detection baselines, with only a marginal impact on benign FPR (e.g., 4.5% FPR on XSTest-Resp vs. 4.6% for Qwen3Guard-Stream-8B-strict). Crucially, these improvements do not come from indiscriminate triggering, as evidenced by robust benign calibration and performance on policy-adjacent safe samples.

Transferability

Forecasting-based supervision transfers effectively to different backbone families and tokenizers. For instance, Gemma3-StreamGuard-1B reached 81.3 output-moderation F1, 98.2 streaming F1, and a 3.5% miss rate using targets produced on a Llama tokenizer. This clearly contrasts with boundary-detection approaches, whose supervision is tightly coupled to tokenization granularity.

Latency

Guardrail decision latency remains well below per-token generation latency for all tested LLMs, with StreamGuard (even at 8B) requiring <10ms per decision—overhead compatible with real-time deployment and pipelined moderation for high-throughput LLM services.

Ablation and Operating Point Sensitivity

Practical ablations on reduction strategy, rollout pool composition, and supervision density were performed. Mean reduction yielded the best trade-off between timely intervention and benign calibration, while a greater diversity in rollout sources improved robustness against single-generator bias. Supervision schedule can be sparsified without significant loss, suggesting that dense token-level supervision is unnecessary for competitive streaming intervention.

Theoretical and Practical Implications

StreamGuard formally decouples moderation targets from tokenizer boundaries, enabling cross-family transfer and obviating the annotation bottlenecks which have historically limited boundary-based systems. The value-based forecasting paradigm is inherently more robust; it incentivizes early risk prediction and gracefully handles scaffolding or ambiguous prefixes that traditional approaches would mislabel.

Practically, this method facilitates modular, model-agnostic safety guardrails for both pre- and post-deployment moderation in diverse LLM-serving architectures, including those without white-box introspection capabilities. The empirical results suggest value forecasting will become the de facto standard, particularly as LLMs are increasingly deployed in streaming user-facing contexts with ultra-low tolerance for unsafe exposure.

Potential for Future Development

Further exploration of more sophisticated continuation proposal distributions, dynamic thresholding for risk adaptation, and integration with multi-policy or policy-adaptive frameworks (such as DynaGuard or BingoGuard) could extend StreamGuard's utility. The soft Monte Carlo supervision signal may also be leveraged for training LLMs with inherent risk-refusal strategies in RLHF or direct preference optimization contexts, potentially closing the gap between safety detection and generation-time alignment.

Conclusion

StreamGuard reframes streaming output moderation as a value-based risk forecasting problem, shifting the intervention trigger from reactive boundary detection to anticipatory risk assessment over plausible futures. This approach yields strong empirical improvements in both response-time and safety, enhances cross-family transfer, and enables early, policy-compliant intervention in streaming LLM deployments—all while reducing annotation and latency overheads. The value-based forecasting framework sets a new direction for content safety in generative language systems by establishing model-agnostic, scalable, and operationally low-latency moderation protocols.