The Wrong Kind of Right: Quantifying and Localizing Misfired Alignment in LLMs

Abstract: Warning: This paper studies stereotypes and biases, and contains potentially disturbing examples, used for illustration purposes only. Our findings should not be interpreted as an argument against alignment. Instead, this paper highlights the need for principled approaches to more advanced alignment. Alignment aims to ensure that LLMs behave safely and reliably, including by avoiding unsafe inferences. However, we show that such safety-oriented behaviors can misfire: models may reject warranted conclusions even when they are explicitly supported by context. We call this failure mode misfired alignment, where alignment-induced changes cause LLMs to override explicit evidence. To quantify this phenomenon, specifically on stereotype-related alignment, we introduce VETO, a benchmark consisting of 2,032 BBQ-derived contrastive pairs, and define a new metric, Misfired Alignment Rate (MAR), which measures on a 0 to 100 scale how often a model fails on a stereotype-related question but succeeds on its contrastive counterpart. We benchmark 25 LLMs on VETO, and show that all LLMs, including the most recent ones, exhibit non-trivial (4.7 to 18.9%) MARs while all human participants achieve 0.0% MAR. Controlled priming experiments further show that alignment-induced cues can substantially amplify MAR across LLMs, indicating that these failures are not merely artifacts of individual examples but can be induced by safety-related framing. Mechanistic analyses on open-weight LLMs reveal late-layer suppression of evidence-supported answers, and comparisons between instruct and base LLMs suggest that this suppression emerges after instruction training. These findings show that current alignment methods can overgeneralize surface-level safety cues, to the point of overriding objective evidence, motivating more work on alignment objectives that better preserve contextual grounding.

• The paper identifies misfired alignment, a failure where safety protocols cause LLMs to override evidence-supported conclusions in stereotype-sensitive contexts.
• It introduces the VETO benchmark and quantifies the Misfired Alignment Rate (MAR) across 25 models, with rates varying from 4.7% to 18.9%.
• Mechanistic analysis reveals late-layer suppression and shows that chain-of-thought prompting can amplify misfired alignment errors in smaller models.

Quantifying and Localizing Misfired Alignment in LLMs

Introduction and Motivation

Alignment protocols in LLM post-training—such as RLHF and constitutional AI—have been extensively adopted to mitigate biases and ensure responsible behavior. However, the paper "The Wrong Kind of Right: Quantifying and Localizing Misfired Alignment in LLMs" (2606.18656) identifies a systemic failure mode termed misfired alignment, occurring when safety mechanisms cause models to override explicit contextual evidence, particularly in stereotype-sensitive contexts. Unlike classical bias—which involves unwarranted assumptions—misfired alignment manifests as the rejection of warranted conclusions about stereotype-related groups, even in the presence of clear evidence.

Figure 1: Misfired alignment arises when LLMs ignore explicit evidence related to stereotype-sensitive groups, despite identical context.

To study this phenomenon, the authors introduce the VETO benchmark, constructed from the BBQ dataset, comprising 2,032 contrastive pairs across eight demographic categories. Each pair provides identical context but differs in the identity referenced by the question. The grounded answer is always "yes", and deviations—where models answer "no" for stereotype-related entities but correctly for their contrast—are quantified via the Misfired Alignment Rate (MAR).

Benchmark Construction and Metrics

VETO's design isolates evidence-grounding failures by converting multiple-choice BBQ questions to binary yes/no format, ensuring that contextually supported answers are unequivalent across demographic identity manipulations. MAR is defined as the conditional error rate: the fraction of pairs where the model succeeds on the contrast instance but fails on the target (stereotyped) instance. The complementary metric, Bias Rate (BR), captures the converse failure.

Human annotation on VETO yields a MAR of 0%, confirming dataset clarity and unambiguous evidence structure. LLM annotation, in contrast, produces MARs ranging from 4.7% (DeepSeek-R1) to 18.9% (GPT-5.4-nano), with frontier models such as GPT-5.4 and Claude-4.7-Opus exhibiting the highest rates.

Empirical Results Across Model Families

The analysis spans 25 instruction-tuned LLMs (open and closed), consistently revealing non-trivial MARs and category-dependent failure patterns. Disability status and physical appearance correlate with the highest MARs, and alignment-induced suppression is particularly pronounced in frontier API models.

MAR scores often greatly exceed bias scores in frontier systems, confirming the orthogonality and complementarity of these error modes. Misfired alignment is not mitigated through in-context learning or chain-of-thought prompting at scale; even with 5-shot ICL, MAR remains above baseline.

Figure 2: Pair-level confusion matrices highlight joint error rates and conditional MAR for four closed-source models on VETO.

Alignment-Priming and Mechanistic Analyses

Behavioral experiments demonstrate that alignment-oriented framing—e.g., prepending "It is not okay to assume [GROUP] [stereotype]"—causally induces MAR. Priming increases MAR by up to 64 percent in small models and by 11 percent in GPT-5.5, confirming alignment-induced overgeneralization.

Mechanistic interpretability reveals that the suppression of evidence-supported answers is localized in late layers. Logit-lens probing and attention head ablation identify decisive layer-wise handoffs: intermediate layers transiently favor the correct answer, but late-stage alignment circuits veto evidence-based reasoning. Ablating a small set of highly specific attention heads recovers up to 83% of misfired-alignment failures in Llama-3.1-8B-Instruct, confirming the causal role of shallow alignment heuristics.

Figure 3: Per-layer logit difference trajectories show late-layer suppression of evidence-grounded answers post-instruction tuning.

Comparisons between instruction-tuned and base models indicate that MAR is amplified by post-training alignment objectives, and the late-layer suppression pattern is largely absent in base models except where underlying reasoning is weak.

Model Reasoning and In-Context Learning Effects

Chain-of-thought prompting amplifies MAR in small open models (+6–10%), enforcing safety cues at the expense of evidence-grounding. Frontier API models benefit marginally from explicit reasoning, reducing MAR by 2–5%, but performance remains suboptimal.

Figure 4: Chain-of-thought reasoning statistically significantly amplifies MAR in smaller models but reduces it in frontier systems.

In-context learning (ICL) only partially mitigates misfired alignment: the reduction in MAR is modest and does not eliminate the phenomenon even at higher shot numbers.

Figure 5: MAR as a function of in-context demonstrations: ICL only partially mitigates the phenomenon.

Practical and Theoretical Implications

The findings reveal that LLM alignment protocols often default to surface-level heuristics that override contextually warranted evidence. Misfired alignment—distinct from classical bias—impairs model reliability when explicit evidence is disregarded in stereotype-sensitive queries.

Such failures pose risks in high-stakes domains (law, healthcare, policy), potentially undermining fairness goals and epistemic consistency. The paper argues for principled, contextually grounded alignment that transcends mere surface-level suppression, advocating causal, meta-learning, or reasoning frameworks that selectively generalize fairness constraints without violating factual consistency.

Future Directions

Addressing misfired alignment necessitates principled objectives that preserve contextual grounding and factual accuracy while delivering fairness interventions. Explicit encoding of competing principles and prioritization strategies within alignment frameworks are critical. Mechanistic interpretability and targeted circuit analysis should be extended to more model families, including closed-source frontier systems.

Conclusion

Misfired alignment constitutes a systematic departure from evidence-based reasoning in aligned LLMs—manifesting as category-dependent suppression of warranted conclusions about stereotype-related groups. The VETO benchmark, MISFired Alignment Rate, behavioral priming protocols, and mechanistic analyses provide robust tools for diagnosing and mitigating this failure mode. The work calls for principled alignment approaches that ensure epistemic integrity and minimize unintended harm, supporting both fairness and contextual accuracy in future LLM deployments.