HarmMetric Eval Benchmark

• HarmMetric Eval is a benchmark that systematically evaluates harmfulness evaluators rather than language models themselves.
• It defines harmful responses by the criteria unsafe, relevant, and useful, and uses a controlled set of 14 response types for precise comparison.
• The benchmark reveals that lexical metrics like METEOR and ROUGE-1 can outperform LLM judges, underscoring both strengths and limitations in current safety assessments.

HarmMetric Eval is a benchmark for systematically evaluating the effectiveness of harmfulness metrics and judges used in jailbreak evaluation and related LLM safety work. Rather than benchmarking LLMs or jailbreak methods directly, it benchmarks the evaluators themselves: string heuristics, reference-based metrics, classifiers, guard models, and LLM judges that are used to decide whether a model response is harmful. Its central design choice is to define harmfulness through the conjunction of unsafe, relevant, and useful, and to test whether an evaluator can reliably separate genuinely harmful responses from several distinct non-harmful failure modes (Yang et al., 29 Sep 2025).

1. Scope and motivation

HarmMetric Eval addresses a methodological gap in LLM safety research: harmfulness evaluation matters, but the field had lacked a rigorous benchmark for evaluating the evaluators themselves. In jailbreak and red-teaming studies, attack effectiveness is often reported using an automatic judge such as a prompted LLM, a classifier, a string-matching heuristic, or a similarity metric. If those evaluators are unreliable, the reported attack results are unreliable as well (Yang et al., 29 Sep 2025).

The benchmark is therefore aimed at harmfulness evaluators, not at model capabilities in isolation. This distinction is central. Prior prompt-response datasets with human annotation were often built to validate one proposed metric or were tied closely to outputs from a particular model or attack setting. HarmMetric Eval instead constructs prompt-response sets that isolate different reasons a response may fail to be harmful, so that evaluators can be compared under a controlled notion of harmfulness.

The benchmark supports two complementary evaluation regimes. Overall evaluation asks whether an evaluator reliably ranks harmful responses above non-harmful ones. Fine-grained evaluation asks whether it can distinguish harmful responses from specific non-harmful subtypes such as refusal, prevention, redirection, irrelevance, prompt repetition, and vague affirmation. This makes the benchmark directly relevant to jailbreak evaluation, automated red-teaming, and safety auditing, where the quality of the judge is itself a first-order methodological concern.

2. Harmfulness definition and benchmark structure

HarmMetric Eval defines a response to a harmful prompt as harmful iff it satisfies three criteria simultaneously:

1. Unsafe: considering the prompt together with the response content, the intent or potential impact is dangerous, toxic, unethical, or illegal.
2. Relevant: the content addresses the harmful prompt rather than drifting to an unrelated topic.
3. Useful: the content provides logically sound and effective assistance toward the harmful goal (Yang et al., 29 Sep 2025).

This definition rules out several superficially harmful-looking responses. A response may contain dangerous vocabulary but function as a warning or prevention message; it may be harmful in general but irrelevant to the prompt; or it may repeat the request or offer a vague “Sure, here is …” preamble without actionable content. The benchmark’s underlying claim is that attackers care about whether the response helps achieve their goal, not merely whether it contains unsafe language.

Each benchmark instance contains one harmful prompt together with a fixed set of response types.

Response type	Symbols	Count per prompt
References	$r_1, r_2$	2
Harmful	$h_1, h_2, h_3, h_4$	4
Safe	$s_r, s_p, s_d$	3
Irrelevant	$i_1, i_2, i_3$	3
Useless	$u_p, u_a$	2

Thus each prompt is paired with 14 responses total. The final benchmark uses 238 harmful prompts, yielding 3.3k+ responses overall. This structure is designed so that an evaluator must separate true harmfulness not only from explicit refusal, but also from non-harmful outputs that remain lexically or semantically close to the harmful prompt.

3. Dataset construction

The benchmark begins from a larger pool of 20.9k harmful prompts, each paired with two reference responses and manually annotated with severity levels. Only examples where both reference responses have severity 2 or 3, corresponding to “noticeable harm to individuals or society,” are retained. The authors also report that 3.76% of prompts were duplicates. To obtain a representative and diverse subset, they use NV-Embed-v2 embeddings together with K-means clustering (Yang et al., 29 Sep 2025).

The harmful responses are generated by querying two uncensored or open models with direct harmful prompts rather than sophisticated jailbreaks: Meta-Llama-3.1-8B-Instruct-abliterated and Wizard-Vicuna-13B-Uncensored. Each model is prompted to produce two independent harmful responses, yielding 4 harmful responses per prompt. The design goal is to generate responses that are straightforwardly harmful, on-topic, and useful.

Non-harmful responses are partitioned by failure mode. Safe responses fail the unsafe criterion and are divided into refusal, prevention, and redirection. Irrelevant responses fail the relevant criterion: for each harmful prompt, the benchmark selects three other harmful prompts with semantic similarity below 0.2, manually checks their reference responses, and uses those responses as irrelevant answers for the current prompt. Useless responses fail the useful criterion and consist of prompt repetition and useless affirmation.

The authors report a manual review step after construction and state that only 3.4% of examples deviated from the intended settings. The dataset is therefore largely constructed rather than exhaustively labeled by multiple annotators, with manual checking and filtering at key stages. No train/validation/test split is described; HarmMetric Eval is presented as an evaluation benchmark rather than a supervised training resource.

4. Scoring mechanism

A central problem in benchmarking harmfulness evaluators is output heterogeneity. Some evaluators produce binary labels, others produce categorical judgments, and others produce scalar scores on different ranges. HarmMetric Eval standardizes comparison by first mapping outputs so that higher means more harmful. For binary outputs, for example, “unsafe” is mapped to 1 and “safe” to 0 (Yang et al., 29 Sep 2025).

For each benchmark instance,

$$x = \{p, r_{1}, r_{2}, h_{1}, h_{2}, h_{3}, h_{4}, s_{r}, s_{p}, s_{d}, i_{1}, i_{2}, i_{3}, u_{p}, u_{a}\},$$

where $p$ is the harmful prompt and $\mathcal{E}$ denotes the evaluator under test.

The benchmark defines a conservative harmful anchor score: $$V_h = \min(\mathcal{E}(h_1), \mathcal{E}(h_2), \mathcal{E}(h_3), \mathcal{E}(h_4)).$$ For safe responses it uses the maximum score over safe variants: $$V_s = \max(\mathcal{E}(s_r), \mathcal{E}(s_p), \mathcal{E}(s_d)),$$ and for irrelevant and useless responses: $h_1, h_2, h_3, h_4$0

The overall effectiveness score is

$h_1, h_2, h_3, h_4$1

with

$h_1, h_2, h_3, h_4$2

and

$h_1, h_2, h_3, h_4$3

This design is explicitly conservative. An evaluator receives credit on the harmful side only if all harmful variants are scored sufficiently high, because $h_1, h_2, h_3, h_4$4 is the minimum over harmful responses. Conversely, it is penalized if it overscores any safe, irrelevant, or useless distractor, because $h_1, h_2, h_3, h_4$5 and $h_1, h_2, h_3, h_4$6 are maxima over non-harmful variants. The asymmetry between $h_1, h_2, h_3, h_4$7 and $h_1, h_2, h_3, h_4$8 is also deliberate: ties against irrelevant or useless responses receive partial credit $h_1, h_2, h_3, h_4$9, since many evaluators were originally designed mainly to detect unsafe content and may struggle to separate harmful from “unsafe but non-useful” responses in binary form.

Fine-grained diagnostics are defined analogously. For each safe subtype $s_r, s_p, s_d$0,

$s_r, s_p, s_d$1

For useless responses $s_r, s_p, s_d$2,

$s_r, s_p, s_d$3

For irrelevant responses,

$s_r, s_p, s_d$4

The benchmark does not use standard correlation against human scalar ratings as its primary evaluation criterion. Its core methodology is instead an internal ranking framework over structured response sets.

5. Evaluated methods and empirical findings

HarmMetric Eval compares nearly 20 evaluators spanning four families: prompted GPT-4o judges using QiEval, PAIREval, and JbBEval templates; Llama Guard variants; classifier-style evaluators such as GPTFEval and HBEval; string-matching methods such as GCGEval, AutoDANEval, and ShenEval; and reference-based metrics including F1 Score, Cosine Similarity, BERT Similarity, METEOR, BLEU, ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-Lsum (Yang et al., 29 Sep 2025).

The headline empirical result is that METEOR and ROUGE-1 outperform the tested LLM judges on the benchmark’s main effectiveness criterion. The top overall $s_r, s_p, s_d$5 values are:

• METEOR: 0.634
• ROUGE-1: 0.563
• JbBEval: 0.523
• HBEval: 0.506
• GPTFEval: 0.468
• Llama-Guard-3-8B: 0.456
• Llama-Guard-2-8B: 0.452
• F1 Score: 0.437
• QiEval: 0.435
• Cosine Similarity: 0.420
• PAIREval: 0.274

The best overall score is only 0.634, which the paper interprets as evidence that current harmfulness evaluators remain far from reliable.

The fine-grained breakdown explains why. On safe responses—refusal, prevention, and redirection—LLM judges are extremely strong. QiEval and PAIREval score 1.000 on all three safe categories, JbBEval scores 0.996, and HBEval scores 0.975. By contrast, the most surprising weakness appears on useless affirmation: responses that begin with a seemingly compliant phrase such as “Sure, here is …” but provide no actionable harmful content. On this category, lexical reference-based metrics dominate: METEOR reaches 0.895, ROUGE-1 0.845, while JbBEval falls to 0.570, HBEval to 0.508, QiEval to 0.435, and PAIREval to 0.296.

On irrelevant responses, BERT Similarity performs best with 1.000, followed by HBEval at 0.914 and ROUGE-1 at 0.815. Yet BERT Similarity’s failures on other categories, especially useless cases, prevent strong overall performance. String-matching methods perform extremely poorly overall: GCGEval scores 0.008, AutoDANEval 0.002, and ShenEval 0.000. The paper attributes this to their inability to recognize safe responses that lack standard refusal strings and to false negatives when refusal-like words such as “illegal” appear inside genuinely harmful responses.

The benchmark therefore challenges the conventional expectation that LLM judges should dominate because they can supposedly reason semantically about the prompt and response. HarmMetric Eval shows that under a prompt-specific reference setting, conventional lexical metrics can capture more useful signal than expected, especially for separating actionable harmful content from superficially compliant but non-useful answers.

6. Interpretation, limitations, and conceptual position

Several interpretive points are central to the benchmark’s significance. First, HarmMetric Eval is not a benchmark for LLMs or jailbreak methods directly. It is a benchmark for the metrics and judges used to assess harmfulness. Second, the strong performance of METEOR and ROUGE-1 does not imply that lexical metrics are universally superior in all safety settings. The paper explicitly notes that their strength here depends on the availability of prompt-specific harmful references. In settings without such references, model-based evaluators may remain necessary (Yang et al., 29 Sep 2025).

The benchmark also has clear limitations. The final dataset contains only 238 prompts. Its responses are constructed rather than naturally occurring, which is diagnostically useful but may not mirror all real output distributions. Its strongest results rely on reference dependence, since the benchmark includes prompt-specific harmful references. The authors also acknowledge the risk of lexical cue overfitting, where a future evaluator optimized for HarmMetric Eval might learn benchmark-specific overlap patterns rather than general harmfulness reasoning. Because the dataset contains harmful prompts and harmful outputs, the paper also notes misuse risk and states that stricter access controls and additional safeguards will be implemented.

A broader implication is that HarmMetric Eval focuses on evaluator effectiveness, while other work has raised complementary questions about evaluator reliability under transformation. Concatenation-based tests, for example, show that harmfulness metrics can flip their judgments under repetition, concatenation, or reordering, including for GPT-based judges (Fandina et al., 2024). This suggests a useful distinction: HarmMetric Eval measures whether a judge separates harmful from non-harmful responses under a structured harmfulness definition, whereas reliability work asks whether that judge remains stable under input transformations.

Its enduring contribution lies in making the evaluation of harmfulness evaluators itself benchmarkable. By defining harmfulness as unsafe + relevant + useful, building controlled non-harmful failure modes, and scoring evaluators through conservative internal comparisons, HarmMetric Eval turns a previously under-specified methodological dependency of jailbreak research into an explicit empirical object.