Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs

Abstract: Many safety and alignment failures of LLMs occur due to out-of-distribution (OOD) situations: unusual prompt or response patterns that are unforeseen by model developers. We systematically study whether LLM monitoring pipelines can detect these OOD alignment failures by introducing a benchmark called Misalignment Out Of Distribution (MOOD). It is difficult to find failures that are truly OOD for off-the-shelf models trained on vast safety datasets. We sidestep this by including a restricted training set in MOOD that we use to train our own monitors, as well as seven test sets with diverse alignment failures that are outside the training distribution. Using MOOD, we find that guard models (safety classifiers) often fail to generalize OOD. To fix this, we propose combining guard models with OOD detectors. We test four types of OOD detectors and find that a combination of a guard model with Mahalanobis distance and perplexity-based OOD detectors can improve recall from 39% to 45%. We also establish positive scaling trends across model scales for monitors that combine a guard model and OOD detector; we find that incorporating OOD detection into monitoring achieves a higher recall gain than using a guard model with 20 times more parameters. Our work suggests that OOD detection should be a crucial component of LLM monitoring and provides a foundation for further work on this important problem. We release the code and data for our experiments publicly, and you can find the relevant links here: https://github.com/Dylan102938/mood-bench.

• The paper presents the MOOD benchmark to rigorously evaluate out-of-distribution alignment failures in LLM safety monitors.
• It demonstrates that combining OOD detectors (Mahalanobis and perplexity-based) with guard models increases recall, notably improving jailbreak detection by 23.7 percentage points.
• The study highlights that augmenting existing guard models with lightweight OOD detection is more efficient and effective than simply scaling up model size.

Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs

Introduction

This paper addresses the critical problem of out-of-distribution (OOD) alignment failures in LLMs, focusing on the limitations of safety classifiers (guard models) when faced with unforeseen input distributions. The authors introduce the Misalignment Out Of Distribution (MOOD) benchmark, evaluate the generalization capacity of LLM safety monitors under OOD conditions, and systematically study how OOD detection techniques can be integrated with guard models to improve the detection of novel alignment failures. This work disentangles the monitoring pipeline from traditional approaches and provides quantitative claims regarding the performance benefits of combining OOD detectors with safety classifiers.

Figure 1: Schematic of monitoring pipeline incorporating both guard models and OOD detectors. OOD detection identifies prompts/responses that diverge from the training distribution and flags them for intervention.

The MOOD Benchmark

Designing robust evaluation for monitoring systems is complex due to the prevalence of vast safety fine-tuning datasets and adversarial training regimes in state-of-the-art LLM deployment. To circumvent the circularity in existing benchmarks, the authors generate a bespoke restricted training dataset for both guard model and OOD detector training, deliberately withholding several known classes of alignment failures from exposure.

MOOD comprises:

• Seven OOD alignment failure test sets: Including function call deception (missing/hidden tools), jailbreaks, insecure code generation, scheming, sycophancy, and controlling behaviors.
• OOD-benign set: Machine-translated in-distribution conversations to a low-resource language (Swahili) for estimating false positive rates on non-malicious OOD.
• In-distribution safe/unsafe test sets: Enables FPR calibration and recall evaluation at fixed low-FPR points, mimicking deployment settings with minimal interruptions.

This methodological design ensures that post-training alone is responsible for aligning safety behaviors, thereby exposing the inherent brittleness of LLM monitors to genuine OOD misalignment events.

OOD Detection and Guard Monitoring Approaches

The comparison is grounded on representative, production-scalable OOD detection methods:

• Guard Models: Single forward-pass classifier trained to predict safety per conversation; reflects industry-standard approaches.
• Guard Ensembles: Aggregates $k$ independently trained guard model outputs to estimate epistemic uncertainty.
• Perplexity-based OOD: Uses mean per-token perplexity from a fine-tuned auto-regressive model to score anomaly of input/output sequences.
• Mahalanobis Distance (Activation-based): Fits a Gaussian to final-layer activations on safe data, calculating distance for test conversations as the OOD score.
• Instruction-Tuned Alignment and Uncertainty Scoring: Uses IT-tuned models prompted constitutionally to assess both alignment adherence and self-estimated certainty, simulating classifier reasoning and OOD-uncertainty in a text-based protocol.
• Score Combination: Systematic calibration of linear combinations between guard and OOD detector scores, tuned only on in-distribution unsafe sets to prevent OOD data leakage during hyperparameter selection.

The pipeline operates under strict constraints: no exposure to OOD exemplars at training time and thresholds set for rare false positives, mirroring realistic oversight settings.

Experimental Results and Numerical Claims

Strong empirical claims are substantiated by the following findings:

• OOD recall improvement: The mean recall across OOD misalignment types for a guard model alone is 38.8% at a 1% FPR, which increases to 45.7% when combining guard models with both Mahalanobis distance and perplexity-based OOD detectors. The gain is markedly higher on jailbreak (from 46.3% to 70.0%), scheming (from 37.1% to 51.4%), sycophancy (from 49.9% to 59.9%), and controlling failures (from 82.7% to 89.7%). The largest relative gain appears for jailbreaks—an increase of 23.7 percentage points.
• Contradictory scale claims: Incorporating OOD detection with a small guard model achieves greater recall improvements than simply scaling the guard model 20× in parameter count.
• Benign OOD false positives: The incorporation of OOD detectors increases sensitivity without a proportionate increase of benign OOD false positives (e.g., on Swahili-translated aligned conversations), provided thresholds are appropriately tuned.
• Methodological validation: The leakage audit and surface-level metric analysis confirm absence of trivial artifacts, validating the OOD nature of the test sets.
• Scaling trends: Both standalone guard models and their OOD-augmented combinations improve in OOD recall predictably as model parameter count increases. However, the marginal utility of OOD detection outpaces proportional increases from model scaling.

  Figure 2: Perplexity-based OOD scores distribution for in-distribution, OOD aligned, and OOD misaligned conversations. OOD failure types show a clear shift towards higher perplexity, enhancing separability for detection.

Implications for AI Safety, Monitoring, and Future Directions

Practical Implications:

This work operationalizes several principles for LLM deployment safety:

• Deployment requirement: OOD detection should be a standard, first-class component in monitoring pipelines, not merely an auxiliary safeguard.
• Robustness without scaling: Combining OOD detectors with existing guard models is computationally more efficient and empirically more effective than increasing guard model size.
• Continual learning: The pipeline supports post-deployment retraining/rehabilitation, especially as novel benign OOD samples are iteratively incorporated to reduce spurious alerting.

Theoretical and Research Directions:

• Non-trivial generalization: Even with substantial pre-training, post-training safety alignment does not yield robust OOD generalization; this is an architectural and statistical limitation not addressable by data scaling alone.
• Failure mode enumeration: Future safety benchmarks must anticipate a moving target; the MOOD construction methodology—involving deliberate restriction and adversarial omission—offers a template for long-term research on scalable oversight.
• Novel OOD techniques: Mahalanobis and perplexity-based methods are effective, but the evaluation discovers limitations for instruction-tuned models on certain OODs, demonstrating a need for further research on internal representation-based and calibration-driven OOD techniques.

Conclusion

The paper systematically demonstrates that guard model-based monitors for LLM safety fail to generalize reliably to OOD inputs and behaviors, even when trained on large in-distribution datasets. Integrating lightweight OOD detectors, specifically Mahalanobis distance and perplexity-based scoring, with guard classifiers emerges as an effective and scalable solution, improving OOD misalignment recall more efficiently than upscaling the classifier. The MOOD benchmark and findings elucidate critical gaps in current safety modeling and provide a rigorous basis for future OOD-centric alignment research.