Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts

Abstract: While the wider applicability of LLMs in the legal field is currently debated due to their reliability and the gravity of any errors, narrow uses with well-understood and mitigated risks have emerged. Notably the Swiss Federal Supreme Court uses small on-premises models for tentative translations and short-passage summarization across the four official languages. However, such usage is challenging in the context of Criminal Law. Since rulings and cases employees work on routinely can contain detailed descriptions of violent and sexual offenses, their legitimate work is compromised by refusals and disclaimers due to the activation of model guardrails (over-alignment). To measure this phenomenon, we introduce TF-RefusalBench, a multilingual benchmark for criminal-law translation and summarization derived from public Swiss Supreme Court rulings. TF-RefusalBench contains 5,200 total prompts across French, German, Italian, and English, corresponding to common task prompts and passages likely to trigger refusal. We then use TF-RefusalBench to show that over-alignment is a multifaceted phenomenon, influenced by the model and the prompt and text languages being processed, and that its impact cannot be evaluated solely from an over-refusal perspective, given the disclaimer's impact on task faithfulness. Finally, we evaluate approaches to enable on-premises LLMs for Criminal Law Tasks, demonstrating that while prompting can be effective, abliteration (refusal directions ablation) eliminates refusal with minimal impact on task performance.

• The paper introduces TF-RefusalBench, a benchmark leveraging severity-ranked legal extracts to quantify over-alignment in LLM outputs.
• It evaluates model-specific and language-dependent risks, revealing significant variance in refusal and disclaimer rates across state-of-the-art systems.
• Mitigation strategies like targeted system prompting and abliteration effectively reduce over-alignment, though they entail trade-offs in safety and output fidelity.

Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts

Introduction

The deployment of LLMs for translation and summarization in high-stakes legal environments, such as multilingual criminal law courts, introduces significant risks. Over-alignment—excessive safety behavior manifesting as refusals or unsolicited disclaimers—can compromise legitimate workflow by impeding access to crucial information, particularly when handling sensitive case passages. This paper systematically addresses the phenomenon of over-alignment in LLMs, introducing a rigorous multilingual benchmark, TF-RefusalBench, rooted in the operational context of the Swiss Federal Supreme Court, and evaluating state-of-the-art mitigation protocols.

Benchmark Design and Methodology

TF-RefusalBench is a parallel, multilingual benchmark constructed from 100 severity-ranked extracts from public criminal rulings, augmented to cover four languages (French, German, Italian, English) and two transformation tasks (translation, summarization). Prompts are systematically generated to control for source, target, and prompt language without confounding with model response. The passage selection protocol employs a lexical filter (LDNOOBW) and subsequent LLM-judged severity ranking via Bradley--Terry aggregation, ensuring the dataset targets genuinely challenging and sensitive material, dominated by descriptions of sexual offenses and physical violence.

The benchmark enables isolation of language effects, task effects, and prompt-language concordance—crucial for auditing model behavior across operationally realistic multilingual configurations.

Figure 1: TF-RefusalBench pipeline, detailing extraction, filtering, augmentation, and prompt grid construction.

Over-Alignment: Empirical Landscape

Evaluation across five open-weight LLMs (Llama 3.3, GPT-OSS 20B, Apertus 70B, Qwen3 32B, Gemma 4 31B) reveals that refusal and disclaimer rates are highly model-specific, task-dependent, and influenced by language configuration. Notably, Qwen3 exhibits zero refusal/disclaimer, while Gemma hedges (up to 26.5% disclaimer in translation) but never refuses. Llama combines both modes.

The paper emphasizes that benchmarking solely on refusal rates misses critical disclaimer-driven over-alignment, which can degrade output faithfulness even when models superficially comply with tasks.

Figure 2: Divergent responses (refusal/disclaimer/faithful) to the same translation request from three models illustrate the inadequacy of refusal-only benchmarks.

Language and Prompt Effects

The output language strongly modulates over-alignment. Llama shows a sevenfold increase in refusal when outputting French versus German, Apertus and Gemma disclaim most frequently in English, and Italian output is consistently least problematic for disclaimers. These effects are not intrinsic to language but reflect alignment tuning idiosyncrasies imposed during model construction. Prompt-language concordance is a robust, near-causal driver: issuing instructions in the language matching the sensitive source reliably increases refusal (odds ratios 1.6–2.6 for Llama and GPT-OSS). This effect persists across tasks and models, with the sole exception of Apertus.

Figure 3: Heatmap showing distribution of refusals in translation as a function of source, target, and prompt language.

Figure 4: Prompt-language concordance amplifies over-alignment across all models except Apertus, supporting targeted mitigation strategies.

Mitigation Strategies: System Prompting and Abliteration

System Prompting

A systematic prompt intervention study demonstrates that a concise institutional system prompt halves over-alignment compared to baseline. Authorization prompts (explicitly permitting sensitive reproduction) suppress disclaimers more effectively than refusals, while generic "helpful assistant" prompts are counterproductive. Prompt-language matching further boosts mitigation efficacy.

Figure 5: Impact of system prompts on over-alignment rates, highlighting reductions achieved via institutional context and deliberate authorization.

Abliteration

Applying the Heretic protocol—abliteration by orthogonalizing model weights against identified refusal directions—entirely eliminates refusal (from 6.8% to 0%) and reduces disclaimer incidence (from 7.9% to 5.1%) without meaningful degradation in translation quality (<2% reduction on a 5-point scale). This intervention sharply increases vulnerability to harmful requests (HarmBench attack rates rise from 14.5% to 55.5%), exposing a trade-off between operational transparency and out-of-domain safety, consistent with prior findings on alignment tax [arditi2024refusal].

Benchmark Construction and Severity Calibration

The paper’s benchmark construction protocol leverages LLM-assisted classification and pairwise severity ranking, demonstrating reliable inter-model agreement (Kendall $\tau \approx 0.73$), and ensuring that over-alignment is measured on intrinsic content severity rather than adversarial prompt engineering. Over-alignment clusters in the highest-severity passages, with the top-100 cutoff capturing nearly all actionable behavior.

Figure 6: Category distribution of the candidate pool, confirming dominance of sexual/minor abuse and physical violence, critical for real-world representativeness.

Figure 7: Over-alignment rates as a function of extract severity ranking, showing concentration in the highest ranks and justifying the benchmark's top-100 focus.

Stochasticity and Reproducibility

Refusal is not deterministic; three-run variance decompositions attribute ~27% of over-alignment to stochastic response sampling, underscoring the necessity of repeated audits in deployment environments and suggesting that policy interventions should target propensities, not individual completions.

Implications, Limitations, and Ethical Considerations

Practically, TF-RefusalBench constitutes an institutional diagnostic suite for pre-deployment and post-training calibration in legal environments, with implications for other multilingual, sensitive domains. Theoretical implications include nuanced characterization of alignment tax, highlighting risks of both under- and over-alignment. Future LLM development must reconcile task faithfulness, operational risk minimization, and robust, language-aware safety protocols—especially for datasets exhibiting severity-driven over-alignment.

Limitations are acknowledged in LLM-as-a-judge dependency, translation fidelity for augmented corpus, and representativeness (preponderance of sexual offense cases). Abliteration is flagged as inherently dual-use, necessitating access control and transparency regarding safety-performance trade-offs.

Conclusion

This work advances systematic auditing and mitigation of over-alignment in multilingual criminal law court LLM deployments. By revealing the multifaceted, model-specific, and language-sensitive nature of refusals and disclaimers, the paper argues for rigorous, severe-content benchmarks such as TF-RefusalBench, combined with targeted interventions through system prompting and abliteration. The results underscore the need for comprehensive, multi-axis evaluation frameworks and careful calibration of safety interventions to balance operational reliability with broader model integrity and risk.

Future directions include harder, iterative benchmark construction targeting residual refusal/disclaimer cases and the integration of human expert evaluation for faithfulness and domain competence.