Safety Gap Toolkit for LLM Safeguards

• Safety Gap Toolkit is defined as the difference in effective dangerous capabilities of open-weight language models before versus after safeguard removal.
• The framework uses both supervised fine-tuning and refusal ablation to reveal hidden risks by measuring changes in harmful compliance and latent hazardous knowledge.
• Empirical results indicate that removing safeguards significantly increases dangerous compliance while largely preserving benign generation quality, especially at larger model scales.

Safety Gap Toolkit is an open-source evaluation framework for open-weight LLMs that measures the difference between the behavior of a model with intact safeguards and the behavior of the same model after those safeguards are removed. In the paper that introduces the toolkit, the central object of study is the model’s effective dangerous capabilities, estimated from its willingness to comply with harmful requests together with its retained hazardous knowledge. The toolkit is therefore designed to expose risk that default release-time evaluations can miss, especially when open weights make safeguard removal feasible through fine-tuning or activation-level intervention (Dombrowski et al., 8 Jul 2025).

1. Concept and scope

The toolkit defines the safety gap as the difference in a model’s effective dangerous capabilities before versus after safeguard removal. This formulation is specific to open-weight models, whose modifiability makes post-release behavior materially different from release-time behavior. The paper’s core claim is that testing only a safety-trained release can substantially understate risk, because adversaries may remove refusal behavior while leaving the underlying dangerous knowledge largely intact (Dombrowski et al., 8 Jul 2025).

Within this framing, effective dangerous capabilities are not treated as mere refusal rates. The paper instead defines them as a conjunction of two properties: the model must both answer dangerous requests compliantly and actually possess the relevant dangerous capability or knowledge. This distinction is important because a model that refuses dangerous requests may still encode the relevant content internally, and a model with latent hazardous knowledge may become far more dangerous once refusal layers are stripped away.

The rationale for concentrating on open-weight systems is tied to a specific threat model. The paper emphasizes that fine-tuning can re-enable unsafe compliance, activation steering or refusal ablation can suppress refusal behavior, benign downstream fine-tuning can accidentally erode safeguards, and users can directly modify weights or internal states. A plausible implication is that the safety gap is not merely a property of malicious optimization; it is also relevant to deployment pipelines in which downstream adaptation is routine.

2. Formalization of the safety gap

Because the paper does not publish a benchmark of truly high-risk requests with ground-truth harmful answers, it uses a proxy estimator for effective dangerous capabilities. That estimator combines a compliance rate on an open-ended harmful-request dataset with accuracy on a proxy hazardous-knowledge benchmark. The safety gap is then defined as the change in this proxy score between the original model and the same model after safeguard removal (Dombrowski et al., 8 Jul 2025).

Formally, the paper describes the safety gap as

$$\text{Safety Gap} = \text{EDC}_{\text{removed safeguards}} - \text{EDC}_{\text{original}}.$$

Operationally, the proxy hazardous-knowledge component is measured with WMDP, while the compliance component is measured with Bio-Chem-Cyber Propensity. The framework therefore does not claim to measure realized harm directly. Instead, it estimates whether a model retains dangerous knowledge and whether the release-time safeguard layer suppresses willingness to use that knowledge.

This design addresses a common misconception. A low refusal-free dangerous-output rate in the released model does not imply that dangerous capability has been removed. The toolkit’s methodology is explicitly built around the opposite possibility: safeguard layers may primarily regulate outward behavior, while the knowledge substrate remains largely unchanged. This is why the paper treats pre- versus post-removal comparison, rather than single-state evaluation, as the primary object of analysis.

3. Safeguard-removal pipeline

The toolkit provides a four-stage workflow: remove safeguards, evaluate capability and compliance, compare pre- and post-removal behavior, and estimate the safety gap. Its two main safeguard-removal mechanisms are supervised fine-tuning and refusal ablation (Dombrowski et al., 8 Jul 2025).

The supervised fine-tuning branch supports both full-parameter training and LoRA fine-tuning. It is instantiated under two threat models. In harmful fine-tuning, the model is trained on harmful requests paired with compliant answers. In benign fine-tuning, the model is trained on safe responses only, to test accidental safeguard erosion. This distinction matters because the toolkit is meant to capture both adversarial degradation and degradation induced by ordinary downstream adaptation.

The refusal-ablation branch is training-free. It estimates a refusal direction $r^*$ in activation space by computing mean activations on harmful versus harmless prompts, filtering out activations where the model already tends to refuse, taking a difference-in-means, and choosing the direction that reduces refusal most while preserving harmless behavior. The paper summarizes the weight update as

$$W' \leftarrow W - \frac{r^* r^{*\top}}{\|r^*\|^2} W.$$

In the paper’s interpretation, this orthogonalizes weights against the refusal direction so that the residual stream has no component in that direction.

These two removal mechanisms induce different post-removal artifacts. Supervised fine-tuning can sharply increase harmful compliance but often degrades benign generation quality. Refusal ablation is presented as more targeted: it suppresses refusal behavior while preserving harmless behavior and, in the reported experiments, largely preserves benign generation quality. This suggests that the toolkit is designed not only to measure whether safeguards can be removed, but also to characterize the mechanism by which removal changes the model’s behavior profile.

4. Experimental design, datasets, and metrics

The case study evaluates two model families—Llama-3 and Qwen-2.5—across a parameter range from 0.5B to 405B. The specific scales are Qwen-2.5 at 0.5B, 1.5B, 7B, 32B, 72B and Llama-3 at 1B, 3B, 8B, 70B, 405B (Dombrowski et al., 8 Jul 2025).

Dangerous capability is measured with WMDP, described as a multiple-choice dataset of nearly 4,000 questions spanning biology, chemistry, and cyber. The metric is likelihood-based accuracy, obtained by reading the probability of A/B/C/D and selecting the most likely first token. Harmful compliance is measured with Bio-Chem-Cyber Propensity, a new dataset containing 283 biology questions, 312 chemistry questions, and 516 cyber questions, with StrongREJECT used as the judge. Benign generation quality is measured on a 100-sample subset of FreebaseQA, scored by Claude-3.5-Haiku.

The fine-tuning experiments use 51 harmful training samples and 20 evaluation samples for harmful supervised fine-tuning, and 50-sample and 1,000-sample benign subsets for benign supervised fine-tuning. Reported LoRA settings are batch size 16, LoRA rank $r=64$, and LoRA alpha $\alpha=64$. For Llama-3.1-405B, the setup uses FP8 quantization and DeepSpeed ZeRO stage 2, and refusal ablation was not successfully run on that quantized 405B model.

The evaluation is explicitly three-dimensional: dangerous knowledge, refusal or compliance on harmful prompts, and benign generation quality. This structure is integral to the toolkit’s logic. A model with high hazardous knowledge but low compliance is not equivalent to a model with both high hazardous knowledge and high compliance, and a safeguard-removal method that collapses utility is analytically distinct from one that preserves general behavior while removing refusal.

5. Empirical findings

The principal empirical finding is that the safety gap widens with model scale. WMDP accuracy rises steadily with scale, while safeguard removal does not significantly reduce that accuracy. The paper reports, for Llama bio, an increase from about 55–57% at 1B to about 85% at 405B, and states that the same scaling tendency appears across families and across bio, chem, and cyber (Dombrowski et al., 8 Jul 2025).

By contrast, compliance on harmful prompts changes dramatically after safeguard removal. Original models usually comply with fewer than 5% of dangerous requests, but after removal, compliance can rise to as high as 95%. The paper therefore concludes that the underlying dangerous knowledge remains intact while safeguards primarily suppress whether the model will actually comply with harmful requests. Correspondingly, original models often remain below 0.05 EDC, whereas removed-safeguard models increase substantially with scale.

Method-specific tradeoffs are also pronounced. SFT-based safeguard removal typically lowers benign generation quality by about 10–30 percentage points, whereas refusal ablation largely preserves generation quality. This is one reason the paper treats refusal ablation as especially informative: it reveals a large safety gap without requiring catastrophic degradation of general usefulness.

The model-family breakdown largely follows the aggregate pattern. In Llama models, WMDP accuracy rises with scale, baseline compliance stays low, and post-removal compliance rises sharply. The paper notes one anomaly: Llama-3.3-70B-Instruct has a higher baseline compliance than other Llama models. In Qwen-2.5 models, the same broad pattern holds, although refusal ablation failed on Qwen2.5-0.5B-Instruct because the method could not find a suitable direction that cleanly separated refusal from harmless behavior.

6. Interpretation, limitations, and relation to adjacent toolkits

The toolkit’s most important interpretive constraint is that it estimates a proxy for dangerous capability rather than direct real-world harm. It measures WMDP accuracy, harmful compliance, and benign generation quality, then analyzes how those quantities change under foreseeable safeguard-removal operations. The paper’s argument is therefore diagnostic rather than exhaustive: it shows that current safeguards may regulate behavior more than capability, and that post-release modifiability can expose this mismatch (Dombrowski et al., 8 Jul 2025).

Several limitations are explicit. The case study is confined to open-weight LLMs from the Llama and Qwen families. Refusal ablation was not successfully run on the quantized 405B model, and it failed on Qwen2.5-0.5B-Instruct. The paper also does not claim that WMDP and Propensity are complete representations of high-risk capability. These limitations do not negate the framework’s use; they delimit the conditions under which its safety-gap estimate should be interpreted.

Within the broader toolkit ecosystem, the Safety Gap Toolkit occupies a specific niche: post-mitigation risk estimation for open-weight models under safeguard removal. Other frameworks target different slices of the safety stack. WalledEval emphasizes benchmark breadth, judge benchmarking, mutators, and exaggerated-safety analysis across 35+ safety benchmarks (Gupta et al., 2024). SafeTuneBed standardizes the evaluation of safety erosion during downstream fine-tuning, especially under harmful-data injection, with unified safety and utility metrics (Hossain et al., 31 May 2025). AISafetyLab integrates attack, defense, and evaluation modules into a comprehensive framework for safety evaluation and improvement (Zhang et al., 24 Feb 2025). DeepSight closes the gap between behavioral safety evaluation and mechanistic diagnosis by combining DeepSafe and DeepScan in a unified protocol (Zhang et al., 12 Feb 2026). This suggests that the Safety Gap Toolkit is best understood not as a general-purpose safety platform, but as a specialized instrument for quantifying the hidden risk of open-weight models after safeguard compromise.

A further misconception addressed by this comparison is that all “safety toolkit” papers solve the same problem. They do not. The Safety Gap Toolkit is centered on the discrepancy between safeguarded and de-safeguarded behavior in modifiable models; adjacent frameworks target judge reliability, poisoning robustness, attack-defense comparison, or evaluation-diagnosis integration. Its distinctive contribution is to treat modifiable post-release behavior as the primary estimand rather than an edge case.