---
title: Refusal Rate (RR) in Language Models
url: https://www.emergentmind.com/topics/refusal-rate-rr
type: topic
---

# Refusal Rate (RR) in Language Models

Refusal Rate (RR) quantifies the frequency with which a language model produces a refusal in response to a given class of prompts. Across contemporary safety alignment and model evaluation research, RR has become the standard scalar metric for summarizing a model's abstention from responding to undesirable, ambiguous, or policy-inappropriate requests. Mathematically, RR is defined as the fraction of model outputs classified as refusals within a specified evaluation set, with the precise notion of "refusal" depending on the study context—ranging from hard explicit denials to non-responsive or hedged answers. However, recent research demonstrates that RR's interpretability depends on nuanced distinctions in prompt types, evaluation protocols, and behavioral instability, making it necessary to interrogate its limitations and supplement it with richer metrics.

## 1. Formal Definitions and Variants

The base RR metric is almost universally defined as a proportion:

\[
\mathrm{RR} = \frac{N_{\text{refusal}}}{N_{\text{total}}}
\]

where \(N_{\text{refusal}}\) denotes the number of prompts the model refused, and \(N_{\text{total}}\) is the total number of prompts evaluated [2601.17911, 2510.01782, 2601.08489, 2411.11296, 2603.13359, 2509.08075, 2605.27763].

Variants, tailored to evaluation context, include:
- **Strict RR**: Only directly non-compliant and blank outputs (e.g. direct_refusal, non_responsive) are counted [2605.21545].
- **Lenient RR**: Counts both strict refusal and softer forms such as partial compliance or hedged answers.
- **Subgroup RR**: Conditioned on demographic, task, risk-tier, or persona subgroup [2510.27087, 2509.08075, 2605.21545].
- **Differential/flip rate**: For paired or batched evaluations, the fraction of prompts whose refusal/non-refusal label flips across conditions [2601.17911, 2605.27763].

In knowledge calibration and selective classification, RR at threshold \(\tau\) is defined under a gating function:
\[
\mathrm{RR}(\tau) = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{\hat{p}_i < \tau\}
\]
where \(\hat{p}_i\) is the predicted answerability probability for query \(i\) [2601.10398].

## 2. Measurement Methodologies

**Prompt Selection**: Most studies distinguish between harmful ("should-refuse") and benign ("should-answer") prompts, typically sourced from curated benchmarks such as WildGuard, RefusalBench, CoCoNot, or application/domain-specific datasets (mil-deflect for military LLMs, text-to-SQL ambiguities, sociopolitical classification tasks) [2605.21545, 2603.10012, 2509.08075, 2411.11296].

**Outcome Annotation**: Outputs are classified as refusal vs. non-refusal via:
- Automated keyword filters (e.g. “I’m sorry, I can’t assist”) supplemented by LLM-based classification [2510.27087, 2605.21545].
- LLM-as-a-judge or fine-tuned classifier models, typically running post-hoc over generated outputs [2603.13359, 2411.11296].
- Manual adjudication to correct for scorer mistakes and binomial confidence intervals [2605.27763, 2605.21545].

**Batched and Paired Evaluations**: Recent protocols emphasize evaluating RR under realistic serving batch conditions with paired safety/capability controls to isolate safety-specific boundary shifts from infrastructure-induced output instability [2605.27763].

**Monte Carlo Sampling**: When controlling for large factorials (e.g. model × task × persona × prompt), RR is estimated as a mean over sampled combinations for sample efficiency [2509.08075].

## 3. Context-Conditioned and Artifact-Dependent Instability

Empirical research demonstrates that RR, while high in aggregate for safety-aligned models (often >90% refusal on harmful prompts), exhibits substantial local instability under prompt perturbation, demographic targeting, and infrastructure variation:
- **Prompt Injection and Perturbation**: Even with aggregate RR >94%, 27.7–31.8% of prompts exhibited at least one "refusal escape" in their perturbation neighborhood, with local "flip rates" exceeding 20% for certain artifact types (e.g. ransomware text), but 0% for executable malware [2601.17911].
- **Demographic and Persona Effects**: Subgroup RR varies across national, gender, religious, and sexual-orientation identities by up to 1.5×, and intersectional groups can amplify refusal disparity [2510.27087, 2509.08075].
- **Serving Stack and Batch Instability**: Paired testing under different batch/microbatch schedulers yields a non-trivial differential RR (median corrected flip-rate ≈0.16%), with most flips traceable to output instability rather than alignment-type [2605.27763].
- **Safety–Capability Interference**: Activation- or feature-level refusal interventions (e.g. ablation, SAE, vector steering) can sharply reduce RR but often induce distribution drift or collateral performance regressions if not properly disentangled [2601.08489, 2411.11296, 2603.13359].

## 4. Statistical Analyses and Supplementary Metrics

**Regression and Hypothesis Testing**:
- RR is often analyzed with chi-square tests (e.g. outcome vs. perturbation family or demographic group), binary/multinomial logistic regression, and mixed-effects or GEE models to account for prompt clustering [2601.17911, 2510.27087].
- Provider-level odds ratios (OR) quantify the effect of vendor/API on refusal propensity (e.g. Anthropic API: OR=21.03 for strict refusal) [2605.21545].

**Complementary Metrics**:
- **Refusal Boundary Entropy (RBE)**: Shannon entropy over local refusal/compliance outcomes within a perturbation set, quantifying instability of the refusal boundary [2601.17911].
- **Youden’s J**: \( J = \mathrm{RR}_{\text{dual-use}} - \mathrm{RR}_{\text{benign}} \), operationalizing risk-tier discrimination [2605.21545].
- **Refusal Index (RI)**: Spearman’s rank correlation between refusal probability and error probability, decoupling blanket RR from knowledge-aware or risk-aligned refusal [2510.01782].
- **Disparity Ratio/Absolute Difference**: Ratios or absolute differences between subgroup RRs for measuring selective refusal bias [2510.27087].

## 5. Quantitative Results Across Domains

| Model/Context              | Harmful RR (%) | Benign RR (%) | Partial Compliance (%) | Soft Deflection / Other   | Notes                                                             |
|----------------------------|:--------------:|:-------------:|:---------------------:|:--------------------------|-------------------------------------------------------------------|
| GPT-4o [2601.17911]        | 95.6           | —             | 0.98                  | —                         | 27.7% instability on base prompts; RBE=0.293b (norm. 0.185)        |
| GPT-4.1 [2601.17911]       | 94.7           | —             | 1.70                  | —                         | 31.8% instability; RBE=0.346b (norm. 0.218)                        |
| min(Llama3.2-3B, Qwen2.5)  | ~0             | ~0            | —                     | —                         | Near-zero RR for latest models in classification tasks [2509.08075] |
| Nova 2 Lite (mil. gold)    | 98.2           | —             | —                     | —                         | Max hard refusal on mil-deflect-gold-alpha [2603.10012]            |
| Deepseek R1 (mil. gold)    | 25.8           | —             | —                     | —                         | Best answerer in mil domain, low RR [2603.10012]                   |
| Grok 4.20 (bio)            | 3 (benign)     | 81.7 (dual)   | —                     | —                         | Youden’s J=0.787 (best-risk discrimination) [2605.21545]           |
| Claude Opus 4.7 (bio)      | 76.6 (benign)  | 100 (dual)    | —                     | —                         | J=0.234; over-refusal                                             |
| Qwen3-VL-8B SRA            | 2.0            | —             | —                     | —                         | SRA: RR drops from 93.8→2.0% without drift [2601.08489]            |

Concrete RR numbers are highly protocol- and domain-dependent. For example, in military QA, refusal rates span 1.5%–98.2% across different public models, with even military-tuned models showing high pre-ablation RR [2603.10012]. In sociopolitical tasks, RR on offensive classification can range from 0–87% (by model generation) [2509.08075].

## 6. Limitations, Biases, and Misranking

- **Binary RR metrics can mis-rank models on calibration and safety objectives**. For instance, a model may achieve a high RR by over-refusing even benign prompts, yielding low risk-tier discrimination (Youden’s J) and poor utility [2605.21545].
- **Selective refusal bias**: Unequal RR across demographic subgroups creates fairness concerns and loopholes in safety systems; intersectional identities can exacerbate or invert group disparities [2510.27087, 2509.08075].
- **Task framing, output format, and prompt wording impact RR**: Forced-choice answer formats and stricter prompts typically depress RR, sometimes by dozens of percentage points, illustrating the interaction between system design and measured safety [2509.08075, 2605.21545].
- **Batch serving and system infrastructure can cause low-rate, but critical, refusal boundary flips**. Directional flips (e.g. from refusal to non-refusal) may occur due to scheduler, micro-batching, kernel settings, or hardware-specific effects, with real safety implications [2605.27763].

## 7. Operational Recommendations and Future Directions

- **Always report RR alongside calibration and discrimination metrics**: Standalone RR should be complemented by measures such as Youden’s J, RBE, RI, and breakdowns by partial or soft compliance [2510.01782, 2605.21545, 2601.17911].
- **Stratify RR reporting by subgroup and risk tier**: Ensure fairness and safety by conditioning RR on demographic, domain, and artifact-type axes [2510.27087, 2509.08075, 2605.21545].
- **Incorporate robust, paired evaluation protocols**: Compare RR under multiple serving configurations, paired with capability and benign controls, and use manual adjudication for low-rate events [2605.27763].
- **Use activation-geometry-guided interventions**: To drive target RR reductions without collateral damage, methods like spectral residualization (SRA) or decorrelated residual steering can be preferred over naïve vector ablation [2601.08489, 2603.13359].
- **Refusal stability and local decision-boundary behavior should be explicitly measured**: Audit RR distributions within perturbation neighborhoods or local prompt clusters to surface "refusal escapes" and compliance flips [2601.17911].
- **Promote mechanistic interpretability**: Identification and disentanglement of refusal-mediating features (e.g. categorical steering, SAE features) are critical to avoid tradeoffs with general capabilities [2411.11296, 2603.13359].

In summary, while refusal rate is a foundational metric for language model safety and policy evaluation, its utility as a sole or ranking criterion is limited by artifact-dependence, local instability, fairness concerns, and infrastructural confounds. A comprehensive safety assessment requires RR to be embedded within a battery of analytic, stratified, and stability-sensitive tests, as established in leading safety research from the 2024–2026 era [2601.17911, 2510.01782, 2605.21545].

Source: https://www.emergentmind.com/topics/refusal-rate-rr