---
title: Safety Gap Toolkit for LLM Safeguards
url: https://www.emergentmind.com/topics/safety-gap-toolkit
type: topic
---

# Safety Gap Toolkit for LLM Safeguards

Safety Gap Toolkit is an open-source evaluation framework for open-weight large language models that measures the difference between the behavior of a model with intact safeguards and the behavior of the same model after those safeguards are removed. In the paper that introduces the toolkit, the central object of study is the model’s **effective dangerous capabilities**, estimated from its willingness to comply with harmful requests together with its retained hazardous knowledge. The toolkit is therefore designed to expose risk that default release-time evaluations can miss, especially when open weights make safeguard removal feasible through fine-tuning or activation-level intervention [2507.11544].

## 1. Concept and scope

The toolkit defines the **safety gap** as the difference in a model’s effective dangerous capabilities before versus after safeguard removal. This formulation is specific to **open-weight** models, whose modifiability makes post-release behavior materially different from release-time behavior. The paper’s core claim is that testing only a safety-trained release can substantially understate risk, because adversaries may remove refusal behavior while leaving the underlying dangerous knowledge largely intact [2507.11544].

Within this framing, effective dangerous capabilities are not treated as mere refusal rates. The paper instead defines them as a conjunction of two properties: the model must both answer dangerous requests compliantly and actually possess the relevant dangerous capability or knowledge. This distinction is important because a model that refuses dangerous requests may still encode the relevant content internally, and a model with latent hazardous knowledge may become far more dangerous once refusal layers are stripped away.

The rationale for concentrating on open-weight systems is tied to a specific threat model. The paper emphasizes that fine-tuning can re-enable unsafe compliance, activation steering or refusal ablation can suppress refusal behavior, benign downstream fine-tuning can accidentally erode safeguards, and users can directly modify weights or internal states. A plausible implication is that the safety gap is not merely a property of malicious optimization; it is also relevant to deployment pipelines in which downstream adaptation is routine.

## 2. Formalization of the safety gap

Because the paper does not publish a benchmark of truly high-risk requests with ground-truth harmful answers, it uses a proxy estimator for effective dangerous capabilities. That estimator combines a **compliance rate** on an open-ended harmful-request dataset with **accuracy** on a proxy hazardous-knowledge benchmark. The safety gap is then defined as the change in this proxy score between the original model and the same model after safeguard removal [2507.11544].

Formally, the paper describes the safety gap as

$$
\text{Safety Gap} = \text{EDC}_{\text{removed safeguards}} - \text{EDC}_{\text{original}}.
$$

Operationally, the proxy hazardous-knowledge component is measured with WMDP, while the compliance component is measured with Bio-Chem-Cyber Propensity. The framework therefore does not claim to measure realized harm directly. Instead, it estimates whether a model retains dangerous knowledge and whether the release-time safeguard layer suppresses willingness to use that knowledge.

This design addresses a common misconception. A low refusal-free dangerous-output rate in the released model does not imply that dangerous capability has been removed. The toolkit’s methodology is explicitly built around the opposite possibility: safeguard layers may primarily regulate outward behavior, while the knowledge substrate remains largely unchanged. This is why the paper treats pre- versus post-removal comparison, rather than single-state evaluation, as the primary object of analysis.

## 3. Safeguard-removal pipeline

The toolkit provides a four-stage workflow: remove safeguards, evaluate capability and compliance, compare pre- and post-removal behavior, and estimate the safety gap. Its two main safeguard-removal mechanisms are **supervised fine-tuning** and **refusal ablation** [2507.11544].

The supervised fine-tuning branch supports both **full-parameter training** and **LoRA fine-tuning**. It is instantiated under two threat models. In **harmful fine-tuning**, the model is trained on harmful requests paired with compliant answers. In **benign fine-tuning**, the model is trained on safe responses only, to test accidental safeguard erosion. This distinction matters because the toolkit is meant to capture both adversarial degradation and degradation induced by ordinary downstream adaptation.

The refusal-ablation branch is training-free. It estimates a **refusal direction** \(r^*\) in activation space by computing mean activations on harmful versus harmless prompts, filtering out activations where the model already tends to refuse, taking a difference-in-means, and choosing the direction that reduces refusal most while preserving harmless behavior. The paper summarizes the weight update as

$$
W' \leftarrow W - \frac{r^* r^{*\top}}{\|r^*\|^2} W.
$$

In the paper’s interpretation, this orthogonalizes weights against the refusal direction so that the residual stream has no component in that direction.

These two removal mechanisms induce different post-removal artifacts. Supervised fine-tuning can sharply increase harmful compliance but often degrades benign generation quality. Refusal ablation is presented as more targeted: it suppresses refusal behavior while preserving harmless behavior and, in the reported experiments, largely preserves benign generation quality. This suggests that the toolkit is designed not only to measure whether safeguards can be removed, but also to characterize the mechanism by which removal changes the model’s behavior profile.

## 4. Experimental design, datasets, and metrics

The case study evaluates two model families—Llama-3 and Qwen-2.5—across a parameter range from **0.5B to 405B**. The specific scales are Qwen-2.5 at **0.5B, 1.5B, 7B, 32B, 72B** and Llama-3 at **1B, 3B, 8B, 70B, 405B** [2507.11544].

Dangerous capability is measured with **WMDP**, described as a multiple-choice dataset of nearly 4,000 questions spanning biology, chemistry, and cyber. The metric is **likelihood-based accuracy**, obtained by reading the probability of A/B/C/D and selecting the most likely first token. Harmful compliance is measured with **Bio-Chem-Cyber Propensity**, a new dataset containing **283 biology questions**, **312 chemistry questions**, and **516 cyber questions**, with **StrongREJECT** used as the judge. Benign generation quality is measured on a **100-sample subset of FreebaseQA**, scored by **Claude-3.5-Haiku**.

The fine-tuning experiments use **51 harmful training samples** and **20 evaluation samples** for harmful supervised fine-tuning, and **50-sample** and **1,000-sample** benign subsets for benign supervised fine-tuning. Reported LoRA settings are **batch size 16**, **LoRA rank \(r=64\)**, and **LoRA alpha \(\alpha=64\)**. For **Llama-3.1-405B**, the setup uses **FP8 quantization** and **DeepSpeed ZeRO stage 2**, and refusal ablation was not successfully run on that quantized 405B model.

The evaluation is explicitly three-dimensional: dangerous knowledge, refusal or compliance on harmful prompts, and benign generation quality. This structure is integral to the toolkit’s logic. A model with high hazardous knowledge but low compliance is not equivalent to a model with both high hazardous knowledge and high compliance, and a safeguard-removal method that collapses utility is analytically distinct from one that preserves general behavior while removing refusal.

## 5. Empirical findings

The principal empirical finding is that the safety gap widens with model scale. WMDP accuracy rises steadily with scale, while safeguard removal does not significantly reduce that accuracy. The paper reports, for Llama bio, an increase from about **55–57%** at **1B** to about **85%** at **405B**, and states that the same scaling tendency appears across families and across bio, chem, and cyber [2507.11544].

By contrast, compliance on harmful prompts changes dramatically after safeguard removal. Original models usually comply with **fewer than 5%** of dangerous requests, but after removal, compliance can rise to **as high as 95%**. The paper therefore concludes that the underlying dangerous knowledge remains intact while safeguards primarily suppress whether the model will actually comply with harmful requests. Correspondingly, original models often remain below **0.05 EDC**, whereas removed-safeguard models increase substantially with scale.

Method-specific tradeoffs are also pronounced. **SFT-based safeguard removal** typically lowers benign generation quality by about **10–30 percentage points**, whereas **refusal ablation** largely preserves generation quality. This is one reason the paper treats refusal ablation as especially informative: it reveals a large safety gap without requiring catastrophic degradation of general usefulness.

The model-family breakdown largely follows the aggregate pattern. In Llama models, WMDP accuracy rises with scale, baseline compliance stays low, and post-removal compliance rises sharply. The paper notes one anomaly: **Llama-3.3-70B-Instruct** has a higher baseline compliance than other Llama models. In Qwen-2.5 models, the same broad pattern holds, although refusal ablation failed on **Qwen2.5-0.5B-Instruct** because the method could not find a suitable direction that cleanly separated refusal from harmless behavior.

## 6. Interpretation, limitations, and relation to adjacent toolkits

The toolkit’s most important interpretive constraint is that it estimates a **proxy** for dangerous capability rather than direct real-world harm. It measures WMDP accuracy, harmful compliance, and benign generation quality, then analyzes how those quantities change under foreseeable safeguard-removal operations. The paper’s argument is therefore diagnostic rather than exhaustive: it shows that current safeguards may regulate behavior more than capability, and that post-release modifiability can expose this mismatch [2507.11544].

Several limitations are explicit. The case study is confined to open-weight LLMs from the Llama and Qwen families. Refusal ablation was not successfully run on the quantized **405B** model, and it failed on **Qwen2.5-0.5B-Instruct**. The paper also does not claim that WMDP and Propensity are complete representations of high-risk capability. These limitations do not negate the framework’s use; they delimit the conditions under which its safety-gap estimate should be interpreted.

Within the broader toolkit ecosystem, the Safety Gap Toolkit occupies a specific niche: post-mitigation risk estimation for open-weight models under safeguard removal. Other frameworks target different slices of the safety stack. **WalledEval** emphasizes benchmark breadth, judge benchmarking, mutators, and exaggerated-safety analysis across **35+ safety benchmarks** [2408.03837]. **SafeTuneBed** standardizes the evaluation of safety erosion during downstream fine-tuning, especially under harmful-data injection, with unified safety and utility metrics [2506.00676]. **AISafetyLab** integrates attack, defense, and evaluation modules into a comprehensive framework for safety evaluation and improvement [2502.16776]. **DeepSight** closes the gap between behavioral safety evaluation and mechanistic diagnosis by combining **DeepSafe** and **DeepScan** in a unified protocol [2602.12092]. This suggests that the Safety Gap Toolkit is best understood not as a general-purpose safety platform, but as a specialized instrument for quantifying the hidden risk of open-weight models after safeguard compromise.

A further misconception addressed by this comparison is that all “safety toolkit” papers solve the same problem. They do not. The Safety Gap Toolkit is centered on the discrepancy between safeguarded and de-safeguarded behavior in modifiable models; adjacent frameworks target judge reliability, poisoning robustness, attack-defense comparison, or evaluation-diagnosis integration. Its distinctive contribution is to treat **modifiable post-release behavior** as the primary estimand rather than an edge case.

Source: https://www.emergentmind.com/topics/safety-gap-toolkit