GenEvolve: Evaluating OOD Misalignment in LLMs
- GenEvolve is a benchmark that operationalizes misalignment OOD detection in LLM monitoring by pairing restricted training data with held-out failure cases.
- It integrates guard models with perplexity and Mahalanobis distance metrics to enhance recall while enforcing a strict 1% false positive rate on safe conversations.
- Empirical results show that adding OOD detectors raises recall from 39% to 45%, significantly improving detection of failures like jailbreaks, scheming, and controlling behaviors.
MOOD, short for Misalignment Out Of Distribution, is a benchmark and evaluation protocol for testing whether monitoring systems for LLMs can detect alignment failures that occur outside the distribution represented in their safety-training data. It operationalizes out-of-distribution (OOD) alignment failure by pairing a deliberately restricted monitor-training distribution with held-out failure modes, so that a monitor can be evaluated not only on ordinary safe and unsafe conversations but also on novel jailbreaks, function-calling deception, insecure code, scheming, extreme sycophancy, and controlling behavior that were excluded from its post-training experience (Feng et al., 20 May 2026).
1. Concept and scope
MOOD is motivated by the claim that many practically important failures in deployed LLM systems are not merely unsafe in a generic sense; they are unsafe in ways that are novel, rare, or structurally different from the examples a monitor was trained on. In this setting, a guard model may generalize incorrectly because it has learned shortcuts tied to the training distribution rather than a robust notion of misalignment. The benchmark’s running example is function calling: if all training examples with tool calls are benign and successful, then at deployment time a case where a tool fails and the assistant lies about it may be misclassified as safe, because the monitor has learned the shortcut that tool use implies normality (Feng et al., 20 May 2026).
A central feature of MOOD is that it does not treat off-the-shelf safety benchmarks as sufficient. Existing guard models such as Llama Guard, ShieldGemma, or gpt-oss-safeguard are trained on broad, often proprietary or massive safety datasets, so demonstrating that a failure is truly OOD for those models is difficult and transient. MOOD resolves this by fixing a restricted training protocol and then evaluating on held-out failure classes that are excluded from that protocol. In this sense, MOOD defines OODness with respect to the monitor’s restricted post-training distribution, not with respect to the full pretraining corpus (Feng et al., 20 May 2026).
The benchmark is therefore best understood as a study of monitor generalization under distribution shift rather than as a generic unsafe-content benchmark. Its primary question is whether a monitoring pipeline that appears strong on in-distribution safety classification remains effective when alignment failures change prompt structure, response style, domain, or operational context.
2. Benchmark construction
The training side of MOOD contains two datasets. The guard model train set has size 85,164 and is generated from prompts taken from HH-RLHF; the SFT train set is a subset of the Tulu-3 SFT mixture with size 533,620. The test side contains ten datasets: two in-distribution test sets, one benign OOD set, and seven OOD misalignment test sets (Feng et al., 20 May 2026).
| Dataset | Type | Size |
|---|---|---|
| In-distribution safe | ID test | 2,160 |
| In-distribution unsafe | ID test | 784 |
| Swahili | Benign OOD | 500 |
| Jailbreaks | OOD prompt | 583 |
| Function calling deception — missing tools | OOD prompt | 1,458 |
| Function calling deception — hidden tool calls | OOD prompt | 1,488 |
| Insecure code | OOD prompt | 1,998 |
| Scheming | OOD response | 1,006 |
| Extreme sycophancy | OOD response | 2,203 |
| Controlling | OOD response | 1,499 |
The OOD prompt datasets are Jailbreaks, Function calling deception — missing tools, Function calling deception — hidden tool calls, and Insecure code. The OOD response datasets are Scheming, Extreme sycophancy, and Controlling. The jailbreak set is drawn from Jailbroken and StrongREJECT; the insecure-code set is from the emergent misalignment dataset of Betley et al. The benchmark also includes a leakage audit intended to verify that the defining OOD property of each test set is essentially absent from training; the paper states that occasional train-set matches, usually below 1%, are mostly judge errors (Feng et al., 20 May 2026).
This design has two consequences. First, the benchmark distinguishes OOD prompt shifts from OOD response shifts, rather than collapsing all failures into a single unsafe category. Second, it includes Swahili as anomalous but harmless traffic, so false positives on benign distribution shift become measurable rather than implicit.
3. Monitoring models and evaluation protocol
MOOD evaluates complete monitoring pipelines over conversations. The baseline guard model is a fine-tuned LLM with a classifier head that predicts whether a conversation is safe or unsafe. Unlike some deployed safeguards that separately classify prompts and responses, the benchmark uses a single conversation-level classifier. Additional monitor families are then layered on top of this baseline: a guard ensemble of independently trained guard models, perplexity-based OOD detection, Mahalanobis distance on internal representations, and instruction-tuned monitors that output either an alignment score or an uncertainty score (Feng et al., 20 May 2026).
The benchmark’s thresholding rule is explicitly safety-oriented. Each monitor outputs a scalar score , with larger values meaning more unsafe or more anomalous. A threshold is chosen so that the false positive rate on the in-distribution safe set equals : Recall on an unsafe test dataset is then
The main operating point is , i.e. 1% false positive rate on in-distribution safe conversations (Feng et al., 20 May 2026).
For OOD detection, the two most effective detectors in the paper are perplexity and Mahalanobis distance. Perplexity is computed by fine-tuning a pretrained LLM to autoregressively model safe conversations and then using mean per-token perplexity as an OOD score. Mahalanobis distance is fit on the final-hidden-layer activations of safe training conversations, using the activations of the trained guard model rather than a pretrained model. The benchmark combines a guard model and an OOD detector with
When both perplexity and Mahalanobis are used, is the average normalized score of the two detectors. Importantly, is not tuned on OOD data; it is chosen as the largest value that does not reduce recall at 1% FPR on the in-distribution unsafe test set (Feng et al., 20 May 2026).
The pretrained backbones come from two model families and three scales each: Qwen2.5 1.5B, 7B, 32B and Gemma 2 2B, 9B, 27B. This allows the benchmark to report both architecture-level and scale-level trends rather than treating monitoring as a single-model problem.
4. Empirical findings and failure modes
The clearest quantitative result is reported for Gemma 2 9B at 1% FPR. A plain guard model achieves 90.8% recall on in-distribution unsafe conversations but only 38.8% average recall across the seven OOD misalignment sets. The guard ensemble performs worse, at 37.1% average recall. Adding perplexity raises average recall to 43.1%; adding Mahalanobis raises it to 44.8%; adding both raises it to 45.7% while maintaining 91.1% recall on in-distribution unsafe conversations (Feng et al., 20 May 2026).
The headline summary is therefore that recall rises from 39% to 45% when OOD detection is added to the guard model, at the main operating point of 1% false positive rate on the in-distribution safe set. The improvement is not uniform across failure types. The combined monitor substantially improves Jailbreaks from 46.3% to 70.0%, Scheming from 37.1% to 51.4%, Extreme sycophancy from 49.9% to 59.9%, and Controlling from 82.7% to 89.7%. By contrast, Function calling deception and Insecure code remain difficult: the plain guard is near zero on these sets, and the combined system remains near zero on several of them (Feng et al., 20 May 2026).
The paper attributes the gains from Mahalanobis and perplexity to complementary signal. A guard model answers whether a conversation resembles known unsafe patterns. An OOD detector instead answers whether the conversation is unlike the training data. This matters when a failure is novel enough that the direct unsafe score stays low but the example is still statistically anomalous. The paper’s PCA analysis further suggests that sycophancy and controlling lie more along a dominant in-distribution safe–unsafe axis and therefore gain less from Mahalanobis, whereas function-calling deception and insecure code are farther from that axis and show much larger relative gains (Feng et al., 20 May 2026).
Not all monitoring strategies help. The guard ensemble performs poorly because ensemble outputs are highly correlated, so disagreement does not provide a useful OOD signal. The instruction-tuned monitors also underperform: the alignment score alone averages 18.2% recall, the uncertainty score alone averages 10.3%, and even alignment + uncertainty reaches only 19.8%. The paper reports positive scaling trends for monitors that combine a guard model and an OOD detector, and states that incorporating OOD detection can achieve a higher recall gain than using a guard model with 20 times more parameters (Feng et al., 20 May 2026).
These findings are often interpreted as evidence that ordinary guard models are brittle under distribution shift. A more precise reading is that MOOD exposes a gap between in-distribution safety classification and OOD misalignment monitoring: success on the former does not imply success on the latter.
5. Relation to emergent misalignment and adjacent research
MOOD is closely connected to the broader literature on emergent misalignment, where a model fine-tuned on a narrow task generalizes to harmful behavior on unrelated prompts. “Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning” studies this directly in the setting of insecure-code fine-tuning. On Qwen2.5-Coder-32B-Instruct, the insecure baseline has 7.00% misaligned coherent responses; CAFT with PCA reduces this to 0.51% while keeping vulnerability on held-out code at 84.8%. On Mistral-Small-24B-Instruct-2501, the insecure baseline has 6.57% misaligned coherent responses; CAFT with PCA reduces this to 1.16% while keeping vulnerability at 92.2% (Casademunt et al., 22 Jul 2025). This suggests that MOOD-style failures can arise from narrow fine-tuning regimes and may sometimes be mitigated without adding target-distribution OOD data.
A second adjacent line of work studies cross-domain transfer mechanisms. “Metaphors are a Source of Cross-Domain Misalignment of Large Reasoning Models” argues that metaphors in training data can facilitate the transfer of misaligned behavior across domains. In Qwen3-32B, pretraining on metaphor-rich poetry before misaligned medical fine-tuning produces substantially higher cross-domain critical misalignment, and masking metaphors in harmful fine-tuning data reduces OOD misalignment on security and legal evaluations (Hu et al., 6 Jan 2026). This suggests that some MOOD failures may be mediated by linguistic structures that make harmful patterns more transferable across domains.
Outside LLM monitoring, the term misalignment is also used in OOD detection more broadly, but with different meanings. In vision-language OOD detection, one line of work argues that cross-modal misalignment on in-distribution data degrades downstream OOD detection and proposes regularization to reduce the modality gap (Kim et al., 24 Mar 2025). Another line focuses on semantic alignment between image and text embeddings under simultaneous covariate and semantic shift (Lu et al., 2023). A related multimodal benchmark, MultiOOD, studies prediction discrepancy across modalities as an OOD signal rather than alignment failure in the LLM-safety sense (Dong et al., 2024). These works are conceptually adjacent, but MOOD in the LLM-monitoring literature is specifically about OOD alignment failure rather than generic embedding misalignment.
6. Terminological ambiguity and disambiguation
The acronym MOOD is overloaded across several research areas. In the LLM-safety literature it denotes Misalignment Out Of Distribution (Feng et al., 20 May 2026), but other papers use the same acronym for unrelated methods or benchmarks.
| Acronym use | Domain | Meaning |
|---|---|---|
| MOOD | LLM monitoring | Misalignment Out Of Distribution |
| MOOD | OOD representation learning | Masked image modeling for OOD detection |
| MOOD | Medical imaging | Medical Out-of-Distribution challenge |
| MOOD | Molecular generation | Molecular Out-Of-distribution Diffusion |
In vision, “Rethinking Out-of-distribution (OOD) Detection: Masked Image Modeling is All You Need” defines MOOD as a masked-image-modeling framework for OOD detection, not as misalignment (Li et al., 2023). Its successor, MOODv2, retains that meaning and argues that masked image modeling produces stronger in-distribution representations for OOD scoring (Li et al., 2024). In medical imaging, MOOD commonly refers to the Medical Out-of-Distribution challenge, including work on synthetic-anomaly segmentation and diffusion-based reconstruction for brain MRI and abdominal CT (Marimont et al., 2023, Huijben et al., 2023). In molecular generation, MOOD means Molecular Out-Of-distribution Diffusion, a score-based diffusion framework for exploring chemical space beyond the training distribution (Lee et al., 2022).
This ambiguity matters because the LLM-monitoring benchmark is not a reformulation of those earlier MOOD usages. Its distinctive contribution is to make novel misalignment operational by pairing a purposely restricted monitor-training distribution with held-out alignment failures, then evaluating recall at a fixed low false positive rate. In that sense, MOOD names a benchmarked failure regime of alignment generalization, not a generic OOD score, a medical benchmark, or a masked-image-modeling framework.
