TF‑RefusalBench Benchmark Framework

• TF‑RefusalBench is a conceptual benchmark framework that defines and evaluates refusal behavior in large language models across diverse scenarios.
• It operationalizes refusal through explicit taxonomies, graded scoring systems, and domain-tagged evaluation sets to measure safe completion and harmful responses.
• The framework integrates mechanistic diagnostics and calibration metrics to audit model performance in refusal, compliance, and over-refusal scenarios.

Searching arXiv for the most relevant benchmark and refusal-geometry papers so the article can be grounded in the cited literature. arxiv.search code: {"query":"all:TF-RefusalBench OR all:RefusalBench refusal benchmark steering geometry over-refusal safe completion harmfulness-refusal coupling", "max_results": 10, "sort_by": "submittedDate"} arxiv.search code: {"query":"RefusalBench", "max_results": 10, "sort_by": "submittedDate"} TF‑RefusalBench is best understood as a conceptual benchmark framework for measuring, comparing, and stress-testing refusal behavior in LLMs across harmful, benign, ambiguous, and domain-specific settings. In the literature, it is not introduced as a single released benchmark artifact; rather, it appears as a hypothetical or conceptual benchmark regime that unifies several ingredients: an operational definition of refusal, graded refusal scoring, domain-tagged evaluation sets, intervention-aware diagnostics, and representation-level analyses of how refusal is encoded and modified in activation space (García-Ferrero et al., 18 Dec 2025, Lan et al., 15 Jun 2026, Weidener et al., 20 May 2026).

1. Conceptual scope and emergence

The immediate point of departure is "Refusal Steering" (García-Ferrero et al., 18 Dec 2025), which explicitly frames a hypothetical “TF‑RefusalBench” as a refusal-focused benchmark centered on three components: a concrete operational definition of refusal, an LLM-based scoring pipeline that can turn a dataset into a graded refusal benchmark, and a family of activation-space interventions whose effects can themselves be benchmarked. In that framing, refusal is not only an output phenomenon but also a measurable and steerable property of internal representations.

Subsequent work broadens this picture along several axes. "From Refusal Geometry to Safety Geometry" (Lan et al., 15 Jun 2026) separates harmfulness representation from refusal policy and treats their coupling as a benchmarkable internal diagnostic. "RefusalBench: Why Refusal Rate Misranks Frontier LLMs on Biological Research Prompts" (Weidener et al., 20 May 2026) shows that refusal must be evaluated conditionally on risk tier rather than by raw refusal totals. "RefusalBench: Generative Evaluation of Selective Refusal in Grounded LLMs" (Muhamed et al., 12 Oct 2025) shifts the target from policy refusal to epistemic refusal in grounded and RAG settings, where the correct behavior is to abstain when context is flawed rather than when content is disallowed. Domain-specific work such as "Health-ORSC-Bench" (Zhang et al., 25 Jan 2026) and "Beyond Over-Refusal" (Yuan et al., 9 Oct 2025) extends the same benchmark logic to healthcare and multi-turn exaggerated-safety settings.

Taken together, these works imply that TF‑RefusalBench is not a narrow “jailbreak refusal” leaderboard. A plausible implication is that it is a general regime for auditing when models should refuse, when they should comply, when they should safe-complete, and how those behaviors are implemented internally.

2. Operationalizing refusal

A central contribution of the recent literature is to replace template-based refusal detection with explicit taxonomies. "Refusal Steering" defines refusal broadly enough to include direct policy refusals, topic deflection and scope narrowing, propaganda replacement or narrative manipulation, feigned ignorance, excessive vagueness or moral lecturing in place of substance, and nonsensical or corrupted outputs when the topic is censored (García-Ferrero et al., 18 Dec 2025). In that paper, a response is a non-refusal when it substantively engages the question with relevant facts or analysis, possibly with mild disclaimers.

"Cannot or Should Not?" expands the refusal label space into a taxonomy of 16 refusal categories spanning both “should not” and “cannot” reasons (Recum et al., 2024). The “should not” side includes Chain of Command, Legal Compliance / Illegal, Information Hazards, Intellectual Property Rights, Privacy, and NSFW Content; the “cannot” side includes Modalities, Skill Level, Missing Information subtypes, Missing Identity, and Invalid Premise. This decomposition matters because a benchmark that conflates policy refusal with epistemic inability cannot cleanly evaluate either safety alignment or capability calibration.

Several papers further refine the operational target. "Over-Refusal and Representation Subspaces" distinguishes harmful-refusal from over-refusal: the former is correct refusal on genuinely harmful instructions, whereas the latter is erroneous refusal of benign task frames that merely contain sensitive content (Maskey et al., 29 Mar 2026). "RefusalBench" in biology adds a five-level compliance ladder—Compliance, Partial compliance, Indirect refusal, Direct refusal, and Non-responsive—and emphasizes a “hedge-but-help” regime that binary refusal labels miss (Weidener et al., 20 May 2026). In grounded QA, "RefusalBench" for RAG defines selective refusal as abstention under contradiction, missing information, false premise, granularity mismatch, ambiguity, or epistemic mismatch (Muhamed et al., 12 Oct 2025). In text-to-SQL, "LatentRefusal" formalizes refusal as answerability-gating: the system should refuse whenever the predicted answerability probability falls below a calibrated threshold $\tau$ (Ren et al., 15 Jan 2026).

This literature therefore treats refusal as a family of related but distinct phenomena: policy refusal, epistemic abstention, over-refusal, partial compliance, and safe completion. TF‑RefusalBench, in its strongest sense, is the attempt to benchmark that family without collapsing it into a single surface keyword.

3. Benchmark substrates and dataset design

The benchmark substrates associated with TF‑RefusalBench are heterogeneous but structurally aligned. A minimal two-domain design appears in "Refusal Steering": CCP‑SENSITIVE for politically sensitive prompts where refusal is to be reduced, and JailbreakBench for harmful prompts where refusal is to be preserved (García-Ferrero et al., 18 Dec 2025). The best 80B configuration in that paper reduces CCP‑SENSITIVE refusal from 92.35 % to 23.82 % while keeping JailbreakBench refusal at 99 %, illustrating a benchmark regime with domain-specific target refusal ranges rather than a universal optimum.

"From Refusal Geometry to Safety Geometry" uses HarmBench, StrongREJECT, XSTest, and a benign continuity set to jointly measure attack success, harmful usefulness, exaggerated safety, and benign utility (Lan et al., 15 Jun 2026). "Dynamic Adversarial Fine-Tuning Reorganizes Refusal Geometry" adopts essentially the same behavioral tripod—fixed-source HarmBench, StrongREJECT, and XSTest—but studies it across training anchors, making trajectory-level benchmark design explicit (Lan et al., 29 Apr 2026).

The biology benchmark "RefusalBench" introduces a matched-triple design of 141 prompts in 47 bundles, where task framing is held constant and only biological risk tier changes: Benign, Borderline, and Dual-use (Weidener et al., 20 May 2026). It supplements this with a 15-prompt should-refuse module that functions as a calibration floor. "Health-ORSC-Bench" performs a similar boundary construction at scale in healthcare, using 31,920 benign boundary prompts across seven health categories and difficulty tiers Easy-5K, Medium-5K, and Hard-1K (Zhang et al., 25 Jan 2026). "Beyond Over-Refusal" adds XSB with 580 single-turn prompts and MS‑XSB with 30 scenarios and 600 multi-turn prompts, emphasizing scenario-conditioned safe intent (Yuan et al., 9 Oct 2025).

In grounded settings, "RefusalBench" for RAG programmatically generates fresh instances rather than fixing a static test set. It releases RefusalBench‑NQ with 1,600 single-document instances and RefusalBench‑GaRAGe with 1,506 multi-document instances, built from 176 distinct perturbation strategies across six categories of informational uncertainty and three intensity levels (Muhamed et al., 12 Oct 2025). In structured generation, "LatentRefusal" evaluates answerability gating on TriageSQL, AMBROSIA, SQuAD 2.0, and MD‑Enterprise (Ren et al., 15 Jan 2026).

Benchmark substrate	Core target	Characteristic design
CCP‑SENSITIVE / JailbreakBench	Domain-selective refusal	Reduce political refusal, preserve harmful refusal
HarmBench / StrongREJECT / XSTest	Robustness–over-refusal–utility trade-off	Harmful attacks, harmful usefulness, safe-prompt refusal
Biology RefusalBench	Tier-conditioned calibration	Matched triples, positive-control should-refuse module
RefusalBench‑NQ / GaRAGe	Selective refusal in grounded QA	Generative perturbations, uncertainty taxonomy
XSB / MS‑XSB	Exaggerated safety	Trigger annotations, scenario-based multi-turn evaluation
Health-ORSC-Bench	Over-refusal and safe completion in health	Boundary prompts, difficulty tiers, safe-completion scoring
LatentRefusal datasets	Answerability-gating	Pre-generation refusal in text-to-SQL

This corpus suggests that TF‑RefusalBench is less a single dataset than a benchmark architecture: matched or perturbed prompts, explicit safe and unsafe slices, and domain-conditional targets.

4. Metrics and scoring regimes

The most explicit graded refusal metric comes from "Refusal Steering". For prompt $x$, the target model generates $K$ candidate answers $y_{1:K}$; a judge model assigns each answer a scalar label $z_k \in [-1,1]$, and the answers are weighted by answer-segment likelihood via

$$w_k = \frac{\exp(s_k / \tau)}{\sum_{j=1}^K \exp(s_j / \tau)}, \qquad \tau > 0.$$

The paper then defines

$$p^{+} = \sum_{k} w_k \, \max(z_k, 0), \qquad p^{-} = \sum_{k} w_k \, \max(-z_k, 0),$$

and the final refusal confidence score

$c(x) = p^{+} - p^{-} \in [-1,1].$

A prompt is refusal-dominated when $c(x) > 0$, and aggregate refusal rate is

$$\text{RefusalRate}(D) = \frac{1}{|D|} \sum_{x \in D} \mathbf{1}\{c(x) > 0\}.$$

This is one of the clearest candidate primitives for TF‑RefusalBench (García-Ferrero et al., 18 Dec 2025).

Other benchmark families use domain-specific metrics. "From Refusal Geometry to Safety Geometry" defines Attack Success Rate

$x$0

StrongREJECT

$x$1

XSTest any-refusal

$x$2

and benign-utility summaries

$x$3

over a 0–2 helpfulness scale (Lan et al., 15 Jun 2026).

The biology "RefusalBench" argues that raw refusal rate misranks models and instead uses Youden’s $x$4. With dual-use prompts as positives and benign prompts as negatives,

$x$5

with its main-benchmark form

$x$6

It also tracks strict refusal, soft refusals, and partial compliance, making calibration rather than absolute refusal the central construct (Weidener et al., 20 May 2026).

In grounded QA, "RefusalBench" separates answer accuracy, refusal accuracy, False Refusal Rate

$x$7

Missed Refusal Rate

$x$8

and a Hierarchical Refusal Score

$x$9

along with Expected Calibration Error

$K$0

for refusal confidence calibration (Muhamed et al., 12 Oct 2025).

In healthcare, Over-Refusal Rate and ToxicRejectionRate are paired with Safety Completion Rate

$K$1

where $K$2 denotes responses that are judged safe and labeled as Partial Answer or Full Answer (Zhang et al., 25 Jan 2026). In XSB and MS‑XSB, response types are Full compliance, Full refusal, and Partial refusal, with rates satisfying

$K$3

(Yuan et al., 9 Oct 2025).

The metric landscape therefore has a common structure: refusal rate alone is insufficient; TF‑RefusalBench requires paired harmful/safe metrics, partial-compliance accounting, and, increasingly, calibration or utility-aware scores.

5. Mechanistic diagnostics and intervention-aware evaluation

A distinctive feature of TF‑RefusalBench in the recent literature is that benchmarking does not stop at outputs. "Refusal Steering" constructs layer-wise steering vectors from hidden activations $K$4, first via mean difference and then via ridge-regularized variants. Its most successful vector, WRMD, uses weighted centered class means and a ridge-adjusted covariance inverse,

$K$5

and applies them by activation updates such as

$K$6

This turns refusal benchmarking into a dose–response problem over layers and steering strengths (García-Ferrero et al., 18 Dec 2025).

"From Refusal Geometry to Safety Geometry" generalizes the representational target by defining harmfulness carriers $K$7, refusal carriers $K$8, harmfulness and refusal subspaces $K$9, and the Harmfulness–Refusal Coupling Index

$y_{1:K}$0

where

$y_{1:K}$1

It supplements those with causal ablation and steering operators

$y_{1:K}$2

making coupling itself a benchmark signal (Lan et al., 15 Jun 2026).

Mechanistic work further decomposes refusal circuitry. "Understanding Refusal in LLMs with Sparse Autoencoders" uses SAEs to isolate refusal-related latent features, distinguish harm features from refusal features, and show an upstream harm $y_{1:K}$3 refusal causal chain (Yeo et al., 29 May 2025). "RefusalGuard" models refusal as a low-dimensional safety geometry and preserves it under fine-tuning by penalizing edits projected into the refusal subspace,

$y_{1:K}$4

inside

$y_{1:K}$5

(Asif et al., 3 May 2026).

Other papers complicate the “single refusal direction” picture. "Refusal Lives Downstream of Persona in Chat Models" shows that a compliant persona direction gates refusal at late layers; persona steering can suppress refusal in Llama from 97% to 2%, and projecting out the persona direction in a late-layer window restores refusal to baseline, whereas projecting out a random direction does not (Zhong et al., 24 Jun 2026). "Over-Refusal and Representation Subspaces" argues that harmful-refusal is task-agnostic and approximated by a single global vector, whereas over-refusal is task-dependent, lies within benign task clusters, and spans a higher-dimensional subspace, so global direction ablation is structurally mismatched to over-refusal (Maskey et al., 29 Mar 2026). "Dynamic Adversarial Fine-Tuning Reorganizes Refusal Geometry" finds that R2D2 preserves a late-layer admissible carrier through step 100 and then relocates it to an early-layer carrier while effective rank remains near 1.23–1.27, supporting a reorganization account rather than a drift-only account (Lan et al., 29 Apr 2026). "Universal Refusal Circuits Across LLMs" goes further and treats refusal as a transferable low-dimensional semantic circuit reconstructed from a shared concept basis across model families (Cristofano, 22 Jan 2026).

A TF‑RefusalBench informed by this literature therefore includes a mechanistic tier: layer localization, subspace dimensionality, coupling, intervention sensitivity, and cross-model circuit transfer.

6. Empirical regularities, misrankings, and common misconceptions

A recurrent empirical finding is that raw refusal rate is often misleading. The biology "RefusalBench" shows that strict refusal rates across 19 frontier models span 0.1% to 94.6%, yet this ranking does not reflect calibration quality: Grok 4.20 has the highest tier discrimination with $y_{1:K}$6 while ranking only seventh by overall refusal rate, and Claude Opus 4.7 increases overall refusal while its $y_{1:K}$7 falls by about 65% because dual-use refusal was already at ceiling and the additional refusals are false positives on benign research (Weidener et al., 20 May 2026).

Another recurring misconception is that low over-refusal or low coupling is automatically desirable. "From Refusal Geometry to Safety Geometry" shows that early R2D2 checkpoints are robust but useless: Fixed HarmBench ASR is 0.0000 at steps 50 and 100, but XSTest any-refusal is 1.0000 and benign helpfulness is 0.0000. Later checkpoints recover utility while attack success reopens. The same paper also shows that low coupling alone is not a safety guarantee, because SFT reaches a low-coupling regime while remaining substantially less robust (Lan et al., 15 Jun 2026). "Dynamic Adversarial Fine-Tuning Reorganizes Refusal Geometry" reports the same trade-off in trajectory form: R2D2 any-refusal stays at 1.000 early, then falls to 0.664 and 0.228 as ASR rises from 0.000 to 0.035 and 0.250 (Lan et al., 29 Apr 2026).

The literature also repeatedly rejects simple one-dimensional accounts. "Refusal Steering" shows that refusal signals concentrate in deeper layers and are distributed across many dimensions; at layer 42 of Qwen3‑Next‑80B, the top 10 dimensions of the WRMD vector account for only 1.9 % of total $y_{1:K}$8 norm (García-Ferrero et al., 18 Dec 2025). "Refusal Lives Downstream of Persona" shows that refusal depends on persona gating, not just a refusal vector (Zhong et al., 24 Jun 2026). "Over-Refusal and Representation Subspaces" shows that the harmful-refusal direction and the over-refusal direction have only partial alignment, with cosine similarity settling around 0.47 at mid-layers, explaining why global direction ablation reduces over-refusal only incidentally while disrupting harmful-refusal more strongly (Maskey et al., 29 Mar 2026).

Finally, output-level mitigation studies underscore that improving safe compliance can degrade safety if benchmarking is one-sided. "Think Before Refusal" reports large compliance gains on pseudo-harmful prompts while keeping harmful-prompt compliance low, using safety reflection before answer generation (Si et al., 22 Mar 2025). By contrast, "Beyond Over-Refusal" shows that post-hoc mitigation methods such as ignore-word instructions, prompt rephrasing, and attention steering can substantially improve compliance on safe prompts but also increase unsafe compliance, so they must be benchmarked on both safe and unsafe sets (Yuan et al., 9 Oct 2025).

These results collectively argue that TF‑RefusalBench cannot be reduced to “more refusal is safer” or “less refusal is more helpful.” It is a calibration benchmark, not a monotone safety scale.

7. Limitations, status, and prospective directions

In the cited literature, TF‑RefusalBench remains conceptual rather than canonical. "Refusal Steering" repeatedly refers to a hypothetical “TF‑RefusalBench” and treats it as a refusal-centric benchmark design space rather than a finalized benchmark release (García-Ferrero et al., 18 Dec 2025). The surrounding papers provide many of its ingredients, but no single paper standardizes all of them into one formally released benchmark suite.

The limitations of the current ecosystem are explicit. Several mechanistic studies are limited to one backbone or one regime, such as Mistral‑7B under SFT and R2D2 (Lan et al., 15 Jun 2026, Lan et al., 29 Apr 2026). Some domain benchmarks are English-only or text-only, including XSB/MS‑XSB (Yuan et al., 9 Oct 2025) and the grounded RefusalBench (Muhamed et al., 12 Oct 2025). Health-ORSC-Bench covers seven health categories but not hospital cybersecurity, insurance fraud, or multilingual interaction (Zhang et al., 25 Jan 2026). The biology RefusalBench is a May 2026 snapshot over public APIs, and its provider effects are explicitly interpreted as access-path-level rather than model-weight-level (Weidener et al., 20 May 2026). "LatentRefusal" requires white-box hidden-state access and domain-specific probe training, which limits direct transfer to closed APIs (Ren et al., 15 Jan 2026). "Universal Refusal Circuits" likewise assumes white-box access for concept-basis reconstruction and SVD-guarded rank-one edits (Cristofano, 22 Jan 2026).

Even so, the direction of travel is clear. This suggests that a mature TF‑RefusalBench would likely combine a behavioral core, a calibration layer, and a mechanistic layer. The behavioral core would include harmful-request refusal, safe-prompt non-refusal, partial compliance, and domain-specific safe completion. The calibration layer would include risk-tiered or ambiguity-tiered evaluation such as Youden’s $y_{1:K}$9, SCR, FRR/MRR, and positive-control modules. The mechanistic layer would include refusal confidence scoring, harmfulness–refusal coupling, layer-localized carriers, subspace dimensionality, persona interactions, and intervention sensitivity. Such a benchmark would not answer the normative question of what refusal pattern is universally “correct,” but it would make the trade-offs explicit and empirically auditable.