---
title: Helpfulness-Harmfulness Metrics
url: https://www.emergentmind.com/topics/helpfulness-harmfulness-metrics
type: topic
---

# Helpfulness-Harmfulness Metrics

Helpfulness-harmfulness metrics are quantitative schemes for evaluating whether an AI system is useful, safe, or damaging under a specified task, dataset, and judging protocol. In current research, they do not form a single canonical metric family. Instead, the literature uses thresholded reward-model rates, pairwise preference judgments, win rates, refusal and toxicity rates, task-cost reductions, jailbreak-specific harmfulness scores, reviewer benefit-risk ratios, controllability correlations, and continuous human harm ratings. A recurring theme is that helpfulness and harmfulness are usually treated as distinct dimensions rather than as a single latent variable, even when a paper later compresses them into a composite score [2204.05862] [2307.04657] [2404.01295] [2604.27233] [2601.08951].

## 1. Conceptual foundations

In LLM alignment work, **helpfulness** is commonly operationalized as how well a response addresses a prompt in terms of correctness, clarity, relevance, and usefulness, whereas **harmlessness** is commonly operationalized as avoidance of toxic, biased, misleading, disrespectful, illegal, or otherwise unsafe content [2307.04657] [2504.20106]. The separation is explicit in datasets such as BeaverTails, which annotates helpfulness and harmlessness separately and defines harmlessness as a judgment over the entire question-answer pair: a pair is harmless only if it is risk-neutral across all 14 harm categories [2307.04657].

The RLHF literature also encodes a structural asymmetry between the two dimensions. In Anthropic’s helpfulness data, crowdworkers choose the more helpful and honest response; in the harmlessness data, crowdworkers choose the more harmful response so that the model can learn to avoid it. This means the helpfulness signal is framed as a positive preference over better responses, whereas the harmlessness signal is framed as avoidance of worse responses [2204.05862]. Later audits argue that this asymmetry matters because “harmlessness” can collapse into selecting the less bad response rather than a genuinely safe one [2411.08243].

Outside language-model alignment, the term has broader formalizations. In human-robot collaboration, helpfulness is defined as a reduction in task cost caused by adding a robot to a human’s task, with
$$
H(A_R,\pi_{A_R + A_H}) = cost(\pi_{A_H}) - cost(\pi_{A_R + A_H}),
$$
and harmful behavior is interpreted as negative helpfulness [2010.04914]. In agentic LLM settings, the safety axis shifts again: safety is not primarily refusal of harmful requests, but avoidance of harmful actions directly taken by the agent during tool use [2603.02229]. Collectively, these formulations show that “harmfulness” is task-dependent: it may denote unsafe content, unsafe actions, dangerous utility, or degradation introduced by a reviewer.

## 2. Metric families and representative formulas

Recent work uses several recurring metric families rather than one universal construction [2509.09055] [2509.14297] [2010.04914] [2404.01295] [2604.27233] [2509.24384] [2601.08951].

| Metric family | Representative metrics | Core quantity |
|---|---|---|
| Thresholded reward-model rates | HmR, HpR, CAS | Percentage of outputs crossing reward cutoffs |
| Pairwise preference metrics | Win Rate, preference win ratio, Elo | Fraction of pairwise wins |
| Task-cost metrics | \(H\), \(H_R\), \(H_N\) | Reduction in human-alone cost |
| Jailbreak metrics | ASR, Efficiency, Harmfulness | Refusal bypass, prompt concision, dangerous utility |
| Controllability metrics | mP, MP, Err, BT | Match between requested and achieved control |
| Reviewer tradeoff metrics | Helpfulness, Harmfulness, Benefit-to-Risk Ratio | Errors fixed versus errors introduced |
| Pluralistic harm metrics | 0–100 harm ratings, MAE, agreement axis | Continuous harm judgment and disagreement |

A simple thresholded construction appears in the OPT-350M alignment study. There, **Harmlessness Rate (HmR)** is the percentage of responses to 50 harmful prompts whose reward score is at least \(-3\); **Helpfulness Rate (HpR)** is the percentage of responses to 50 helpful prompts whose reward score is above \(-2\); and **Combined Alignment Score (CAS)** is the arithmetic mean of HmR and HpR. The paper emphasizes that these are thresholded proportions over reward-model judgments, not pairwise win rates or separately learned scores [2509.09055].

Jailbreak evaluation introduces a different family. HILL retains standard **Attack Success Rate (ASR)** but adds **Efficiency**, defined as average attack success per word of jailbreak prompt, and a two-dimensional **Harmfulness** score based on **Practicality** and **Transferability**, each rated on a 3-point scale \(0,1,2\). In that formulation, a jailbreak is not adequately characterized by refusal bypass alone; it must also be concise and elicit actionable, transferable harmful content [2509.14297].

Controllable alignment work measures not output quality alone, but how accurately a model follows requested safety-helpfulness settings. The controllable LLM framework evaluates this with **Micro Pearson Correlation (mP)**, **Macro Pearson Correlation (MP)**, **Mean Absolute Error (Err)**, and a **Binary Test (BT)** comparing whether requested increases in an attribute produce higher reward-model scores. The target is a two-dimensional control space \((s_{hp}, s_{sf})\), not a single scalar optimum [2404.01295].

Inference-time reviewer systems define yet another pair. In the tool-calling reviewer framework, **Helpfulness** is the percentage of test cases where the base agent is wrong and the reviewer corrects it, while **Harmfulness** is the percentage of test cases where the base agent is right and the reviewer introduces an error. The associated **Benefit-to-Risk Ratio** is helpfulness divided by harmfulness [2604.27233]. This is structurally different from LLM safety metrics based on content; it measures intervention value.

Pluralistic safety work moves away from binary labels altogether. PluriHarms measures prompt harmfulness on a continuous 0–100 scale and models an additional agreement axis through dispersion of human ratings. Evaluation is then reported with **Mean Absolute Error (MAE)** against either individual annotators or aggregated ratings [2601.08951]. HarmMetric Eval, by contrast, defines a response as harmful only if it is **unsafe**, **relevant**, and **useful**, and evaluates metrics by whether harmful responses are ranked above safe, irrelevant, and useless responses [2509.24384].

## 3. Data sources, judges, and evaluation protocols

The design of helpfulness-harmfulness metrics is inseparable from the underlying dataset and judge. In HH-RLHF-style setups, the basic data unit is a prompt with a preferred and dispreferred response. BeaverTails extends this by adding QA-level safety meta-labels, separate helpfulness and harmlessness rankings, and confidence annotations [2204.05862] [2307.04657]. In the OPT-350M study, the same Anthropic HH-RLHF corpus is used both for alignment training and for defining the evaluation prompts, after which generated responses are scored with `OpenAssistant/reward-model-deberta-v3-large-v2` [2509.09055].

Jailbreak work typically uses harmful prompt sets such as AdvBench and then evaluates outputs with either refusal filters, LLM judges, or human annotation. HILL uses 50 harmful queries from de-duplicated AdvBench, 22 target models, a strict refusal-pattern filter for ASR, GPT-4 for automatic harmfulness scoring on high-susceptibility models, and human evaluation on GPT-4o, Gemini 2.5, and DeepSeek-V3 [2509.14297]. HarmMetric Eval instead constructs 238 harmful prompts paired with reference harmful responses, harmful model outputs, and deliberately tricky non-harmful responses such as refusals, preventions, redirections, irrelevant answers, prompt repetition, and useless affirmations [2509.24384].

Many papers rely on automatic judges rather than direct human labels at evaluation time. Reported evaluators include OpenAssistant reward models, GPT-4, GPT-4o, Llama Guard, `beaver-dam-7b`, GPT-Judge, Perspective API, HarmBench-Llama2-13B-Chat, ToxiGen classifiers, `facebook/bart-large-mnli`, Gemini-2.5-Flash-Lite, and paired human raters [2509.09055] [2412.00074] [2509.08486] [2602.07381] [2403.08309] [2606.26102]. Some protocols explicitly test judge generalization: the ToolEmu agent study trains with one evaluator and reports cross-evaluation with the held-out evaluator, while the controllability study distinguishes optimization evaluation on the training reward models from generalization evaluation on held-out reward models [2603.02229] [2404.01295].

Benchmark structure also matters. ToolEmu uses 0–3 helpfulness and safety scores over full trajectories in a multi-step environment [2603.02229]. BFCL and \(\tau^2\)-Bench evaluate tool-selection, parameter accuracy, irrelevance detection, and multi-turn stateful behavior for reviewer systems [2604.27233]. Alpaca/Alpaca-Eval operationalizes helpfulness via win rate, BeaverTails operationalizes harmlessness via unsafe-rate safety score, and TruthfulQA operationalizes honesty via truthfulness-informativeness, enabling composite HHH summaries such as
$$
\mathrm{Avg} = \frac{\mathrm{Helpfulness} + \mathrm{Honesty} - \mathrm{Harmlessness}}{3}.
$$
That same negative-sign treatment of safety score reflects the fact that some harmlessness metrics are formulated as unsafe-rate rather than safe-rate [2602.07381] [2509.08486].

## 4. Trade-offs, composite scores, and controllability

A large fraction of the literature treats helpfulness and harmfulness as partially conflicting objectives. Composite scores are therefore common, but they encode explicit design choices. CAS gives equal weight to HmR and HpR in the OPT-350M study [2509.09055]. Safe-RLHF-style work instead trains a reward model for helpfulness and a cost model for harmlessness, then optimizes a PPO-Lagrangian objective that maximizes helpfulness reward while minimizing harmfulness cost [2307.04657]. BFPO builds a global ranking that prioritizes safe-and-helpful responses over safe-but-unhelpful, harmful-but-unhelpful, and harmful-but-helpful responses, and evaluates the resulting balance with AlpacaEval helpfulness and a safety benchmark averaging discriminative and generative harmlessness tasks [2408.15313].

Other work measures the trade-off directly rather than hiding it. Preference Vector trains separate positive and negative models for helpfulness and harmlessness, extracts preference vectors, and combines them at test time with user-controlled coefficients \(\eta_{\text{Helpful}}\) and \(\eta_{\text{Harmless}}\). The paper reports smooth, interpretable changes in helpfulness and harmlessness as those coefficients are swept over \(\{-1.0,-0.5,0.0,+0.5,+1.0\}\), with refusal rate used as an explicit measure of over-conservatism [2504.20106]. The controllable LLM framework likewise avoids collapsing the problem to one scalar and instead asks whether outputs match requested safety-helpfulness coordinates [2404.01295].

Agentic post-training provides a particularly sharp view of the frontier. In ToolEmu, safety-only, helpfulness-only, sequential, and combined DPO configurations all end up near a linear Pareto frontier with \(R^2 = 0.77\), and safety training persists strongly through subsequent helpfulness training, with average safety persistence around 0.90 for \(\beta=0.05\) and 0.94 for \(\beta=0.1\) [2603.02229]. This suggests that post-training can preserve earlier safety gains while still moving along an approximately one-dimensional trade-off surface.

Human evaluation can reveal misalignment between relative and absolute metrics. In HRLAIF, basic RLAIF raises preference win ratio against the SFT baseline from 50% to 58.13% but lowers satisfaction rate from 62.92% to 58.33%; HRLAIF raises satisfaction rate to 65.00% while retaining a 56.87% preference win ratio. Satisfaction is defined more strictly: a response counts as satisfied only if it is honest, harmless, and effectively helpful [2403.08309]. This makes explicit that pairwise preference and absolute fulfillment are not interchangeable.

Mechanistic work on reward models adds a further layer. Mixed-objective reward models often underperform helpful-only and harmless-only reward models, have weaker and more compressed score distributions, and retain only part of the behavior captured by specialized models. Reported retention averages are around 0.5656 on helpfulness-oriented tasks and 0.6287 on harmlessness-oriented tasks, while targeted neuron ablations show that objective-specific neurons support their own objective and often hurt the opposing one [2606.13209]. This suggests that some composite metric failures reflect representational interference rather than merely poor benchmark choice.

## 5. Reliability, bias, and failure modes

A central controversy is whether the metric itself is safe to use. Concatenation tests show that several harmfulness judges are unreliable under trivial input transformations: harmful prompt-response pairs judged unsafe individually can be judged safe after concatenation, and GPT-based judges can exhibit strong positional bias. In the reported experiments, GPT-3.5 misclassified unsafe concatenated 4-tuples as safe in 203 of 1000 cases, while GPT-4o showed strong order sensitivity in longer concatenations [2408.12259]. Safe-to-Serve reports an analogous problem for pairwise win-rate judging: an LLM-as-judge setup exhibited significant positional bias and was therefore not used as the final metric [2412.00074].

Judge quality can also invert common expectations. HarmMetric Eval finds that reference-based metrics, especially **METEOR** and **ROUGE-1**, outperform LLM-based judges on its benchmark of harmfulness as unsafe + relevant + useful content, with METEOR achieving the best overall effectiveness score of 0.634 [2509.24384]. This directly challenges the assumption that LLM judges are automatically superior because they possess broader semantic understanding.

Several papers warn that automated proxies can distort the target construct. The OPT-350M HmR/HpR/CAS evaluation depends on empirical thresholds \(-3\) and \(-2\), uses only 50 harmful and 50 helpful prompts, and relies on a reward model rather than direct human ratings [2509.09055]. HILL argues that ASR can be misleading for defense evaluation if a defense changes the prompt’s intent so much that “attack success” is no longer measuring the original harmful goal [2509.14297]. The controllable LLM study similarly warns that optimization metrics on the reward models used for synthetic data generation can overstate controllability relative to held-out reward models [2404.01295].

Dataset construction introduces additional bias. The audit of Anthropic’s HH dataset finds that, under a manual taxonomy, only 11.8% of selected “harmless” responses are acceptable, while 44.5% are unhelpful and 43.7% remain harmful in their own right [2411.08243]. The same audit reports demographic skews in identity-term distributions and downstream disparate refusal behavior across groups. PluriHarms generalizes this concern by treating disagreement itself as signal: prompts in the intermediate-to-high harm range produce greater variance and entropy, and annotator traits such as toxicity experience and education help explain systematic disagreement [2601.08951]. A plausible implication is that a single binary harmfulness label can erase value-sensitive structure that is present in human judgments.

## 6. Extensions and open problems

Recent work is pushing helpfulness-harmfulness metrics beyond single-turn refusal benchmarks. Inference-time reviewer systems move evaluation into the execution loop and quantify whether feedback produces net value before tool execution, but extending those metrics to multi-turn settings requires partial credit and explicit handling of error propagation [2604.27233]. Agentic safety benchmarks already score trajectories rather than isolated completions, and their results indicate that chat-style refusal metrics are insufficient for action-taking agents [2603.02229].

Another active direction is **controllability and disentanglement**. Preference Vector treats helpfulness and harmlessness as modular preference directions in parameter space, while mechanistic reward-model studies argue that mixed-objective interference is driven partly by shared neurons in late layers [2504.20106] [2606.13209]. This suggests that better metrics may need to track not only output-level trade-offs, but also calibration, retention, and representational overlap.

A further extension concerns **value retention** under post-training. The animal-compassion study reports that helpfulness-domain post-training degrades animal compassion on AHB 2.2 more than coding-domain post-training does, across both SFT and GRPO, and recommends explicitly adding benchmarks such as AHB to evaluation suites when a model has value-laden mid-training [2606.26102]. That result broadens the meaning of “harmfulness”: harmfulness may include loss of moral sensitivity, not only production of overtly toxic or unsafe content.

The literature also points toward **pluralistic** and **context-sensitive** safety metrics. PluriHarms argues for moving beyond one-size-fits-all binary labels toward benchmarks that jointly model harm level and disagreement [2601.08951]. The HH audit argues for subgroup-aware and context-sensitive safety measurement rather than overall refusal statistics alone [2411.08243]. Collectively, these works suggest that future helpfulness-harmfulness metrics will likely remain multidimensional: they will need to separate utility from safety, model disagreement rather than discard it, and account for the fact that intervention quality, refusal behavior, semantic relevance, and latent value preservation are not the same phenomenon.

Source: https://www.emergentmind.com/topics/helpfulness-harmfulness-metrics