Helpfulness-Harmfulness Metrics

• Helpfulness-harmfulness metrics are quantitative schemes that separately evaluate an AI system’s utility and safety across defined tasks using methods like reward-model rates, pairwise judgments, and cost reductions.
• They incorporate varied metric families—such as thresholded reward measures, win rates, and jailbreak evaluations—to capture distinct dimensions of helpfulness and harmfulness.
• Current research emphasizes integrating these metrics for multidimensional, context-sensitive safety evaluations, addressing trade-offs and calibration challenges in AI alignment.

Helpfulness-harmfulness metrics are quantitative schemes for evaluating whether an AI system is useful, safe, or damaging under a specified task, dataset, and judging protocol. In current research, they do not form a single canonical metric family. Instead, the literature uses thresholded reward-model rates, pairwise preference judgments, win rates, refusal and toxicity rates, task-cost reductions, jailbreak-specific harmfulness scores, reviewer benefit-risk ratios, controllability correlations, and continuous human harm ratings. A recurring theme is that helpfulness and harmfulness are usually treated as distinct dimensions rather than as a single latent variable, even when a paper later compresses them into a composite score (Bai et al., 2022, Ji et al., 2023, Tuan et al., 2024, Ta et al., 29 Apr 2026, Li et al., 13 Jan 2026).

1. Conceptual foundations

In LLM alignment work, helpfulness is commonly operationalized as how well a response addresses a prompt in terms of correctness, clarity, relevance, and usefulness, whereas harmlessness is commonly operationalized as avoidance of toxic, biased, misleading, disrespectful, illegal, or otherwise unsafe content (Ji et al., 2023, Liang et al., 27 Apr 2025). The separation is explicit in datasets such as BeaverTails, which annotates helpfulness and harmlessness separately and defines harmlessness as a judgment over the entire question-answer pair: a pair is harmless only if it is risk-neutral across all 14 harm categories (Ji et al., 2023).

The RLHF literature also encodes a structural asymmetry between the two dimensions. In Anthropic’s helpfulness data, crowdworkers choose the more helpful and honest response; in the harmlessness data, crowdworkers choose the more harmful response so that the model can learn to avoid it. This means the helpfulness signal is framed as a positive preference over better responses, whereas the harmlessness signal is framed as avoidance of worse responses (Bai et al., 2022). Later audits argue that this asymmetry matters because “harmlessness” can collapse into selecting the less bad response rather than a genuinely safe one (Chehbouni et al., 2024).

Outside language-model alignment, the term has broader formalizations. In human-robot collaboration, helpfulness is defined as a reduction in task cost caused by adding a robot to a human’s task, with

$$H(A_R,\pi_{A_R + A_H}) = cost(\pi_{A_H}) - cost(\pi_{A_R + A_H}),$$

and harmful behavior is interpreted as negative helpfulness (Freedman et al., 2020). In agentic LLM settings, the safety axis shifts again: safety is not primarily refusal of harmful requests, but avoidance of harmful actions directly taken by the agent during tool use (Plaut, 13 Feb 2026). Collectively, these formulations show that “harmfulness” is task-dependent: it may denote unsafe content, unsafe actions, dangerous utility, or degradation introduced by a reviewer.

2. Metric families and representative formulas

Recent work uses several recurring metric families rather than one universal construction (Pant, 10 Sep 2025, Luo et al., 17 Sep 2025, Freedman et al., 2020, Tuan et al., 2024, Ta et al., 29 Apr 2026, Yang et al., 29 Sep 2025, Li et al., 13 Jan 2026).

Metric family	Representative metrics	Core quantity
Thresholded reward-model rates	HmR, HpR, CAS	Percentage of outputs crossing reward cutoffs
Pairwise preference metrics	Win Rate, preference win ratio, Elo	Fraction of pairwise wins
Task-cost metrics	$H$, $H_R$, $H_N$	Reduction in human-alone cost
Jailbreak metrics	ASR, Efficiency, Harmfulness	Refusal bypass, prompt concision, dangerous utility
Controllability metrics	mP, MP, Err, BT	Match between requested and achieved control
Reviewer tradeoff metrics	Helpfulness, Harmfulness, Benefit-to-Risk Ratio	Errors fixed versus errors introduced
Pluralistic harm metrics	0–100 harm ratings, MAE, agreement axis	Continuous harm judgment and disagreement

A simple thresholded construction appears in the OPT-350M alignment study. There, Harmlessness Rate (HmR) is the percentage of responses to 50 harmful prompts whose reward score is at least $-3$; Helpfulness Rate (HpR) is the percentage of responses to 50 helpful prompts whose reward score is above $-2$; and Combined Alignment Score (CAS) is the arithmetic mean of HmR and HpR. The paper emphasizes that these are thresholded proportions over reward-model judgments, not pairwise win rates or separately learned scores (Pant, 10 Sep 2025).

Jailbreak evaluation introduces a different family. HILL retains standard Attack Success Rate (ASR) but adds Efficiency, defined as average attack success per word of jailbreak prompt, and a two-dimensional Harmfulness score based on Practicality and Transferability, each rated on a 3-point scale $0,1,2$. In that formulation, a jailbreak is not adequately characterized by refusal bypass alone; it must also be concise and elicit actionable, transferable harmful content (Luo et al., 17 Sep 2025).

Controllable alignment work measures not output quality alone, but how accurately a model follows requested safety-helpfulness settings. The controllable LLM framework evaluates this with Micro Pearson Correlation (mP), Macro Pearson Correlation (MP), Mean Absolute Error (Err), and a Binary Test (BT) comparing whether requested increases in an attribute produce higher reward-model scores. The target is a two-dimensional control space $(s_{hp}, s_{sf})$, not a single scalar optimum (Tuan et al., 2024).

Inference-time reviewer systems define yet another pair. In the tool-calling reviewer framework, Helpfulness is the percentage of test cases where the base agent is wrong and the reviewer corrects it, while Harmfulness is the percentage of test cases where the base agent is right and the reviewer introduces an error. The associated Benefit-to-Risk Ratio is helpfulness divided by harmfulness (Ta et al., 29 Apr 2026). This is structurally different from LLM safety metrics based on content; it measures intervention value.

Pluralistic safety work moves away from binary labels altogether. PluriHarms measures prompt harmfulness on a continuous 0–100 scale and models an additional agreement axis through dispersion of human ratings. Evaluation is then reported with Mean Absolute Error (MAE) against either individual annotators or aggregated ratings (Li et al., 13 Jan 2026). HarmMetric Eval, by contrast, defines a response as harmful only if it is unsafe, relevant, and useful, and evaluates metrics by whether harmful responses are ranked above safe, irrelevant, and useless responses (Yang et al., 29 Sep 2025).

3. Data sources, judges, and evaluation protocols

The design of helpfulness-harmfulness metrics is inseparable from the underlying dataset and judge. In HH-RLHF-style setups, the basic data unit is a prompt with a preferred and dispreferred response. BeaverTails extends this by adding QA-level safety meta-labels, separate helpfulness and harmlessness rankings, and confidence annotations (Bai et al., 2022, Ji et al., 2023). In the OPT-350M study, the same Anthropic HH-RLHF corpus is used both for alignment training and for defining the evaluation prompts, after which generated responses are scored with OpenAssistant/reward-model-deberta-v3-large-v2 (Pant, 10 Sep 2025).

Jailbreak work typically uses harmful prompt sets such as AdvBench and then evaluates outputs with either refusal filters, LLM judges, or human annotation. HILL uses 50 harmful queries from de-duplicated AdvBench, 22 target models, a strict refusal-pattern filter for ASR, GPT-4 for automatic harmfulness scoring on high-susceptibility models, and human evaluation on GPT-4o, Gemini 2.5, and DeepSeek-V3 (Luo et al., 17 Sep 2025). HarmMetric Eval instead constructs 238 harmful prompts paired with reference harmful responses, harmful model outputs, and deliberately tricky non-harmful responses such as refusals, preventions, redirections, irrelevant answers, prompt repetition, and useless affirmations (Yang et al., 29 Sep 2025).

Many papers rely on automatic judges rather than direct human labels at evaluation time. Reported evaluators include OpenAssistant reward models, GPT-4, GPT-4o, Llama Guard, beaver-dam-7b, GPT-Judge, Perspective API, HarmBench-Llama2-13B-Chat, ToxiGen classifiers, facebook/bart-large-mnli, Gemini-2.5-Flash-Lite, and paired human raters (Pant, 10 Sep 2025, Amballa et al., 2024, Kashyap et al., 10 Sep 2025, Kashyap et al., 7 Feb 2026, Li et al., 2024, Brazilek et al., 30 Apr 2026). Some protocols explicitly test judge generalization: the ToolEmu agent study trains with one evaluator and reports cross-evaluation with the held-out evaluator, while the controllability study distinguishes optimization evaluation on the training reward models from generalization evaluation on held-out reward models (Plaut, 13 Feb 2026, Tuan et al., 2024).

Benchmark structure also matters. ToolEmu uses 0–3 helpfulness and safety scores over full trajectories in a multi-step environment (Plaut, 13 Feb 2026). BFCL and $\tau^2$-Bench evaluate tool-selection, parameter accuracy, irrelevance detection, and multi-turn stateful behavior for reviewer systems (Ta et al., 29 Apr 2026). Alpaca/Alpaca-Eval operationalizes helpfulness via win rate, BeaverTails operationalizes harmlessness via unsafe-rate safety score, and TruthfulQA operationalizes honesty via truthfulness-informativeness, enabling composite HHH summaries such as

$$\mathrm{Avg} = \frac{\mathrm{Helpfulness} + \mathrm{Honesty} - \mathrm{Harmlessness}}{3}.$$

That same negative-sign treatment of safety score reflects the fact that some harmlessness metrics are formulated as unsafe-rate rather than safe-rate (Kashyap et al., 7 Feb 2026, Kashyap et al., 10 Sep 2025).

4. Trade-offs, composite scores, and controllability

A large fraction of the literature treats helpfulness and harmfulness as partially conflicting objectives. Composite scores are therefore common, but they encode explicit design choices. CAS gives equal weight to HmR and HpR in the OPT-350M study (Pant, 10 Sep 2025). Safe-RLHF-style work instead trains a reward model for helpfulness and a cost model for harmlessness, then optimizes a PPO-Lagrangian objective that maximizes helpfulness reward while minimizing harmfulness cost (Ji et al., 2023). BFPO builds a global ranking that prioritizes safe-and-helpful responses over safe-but-unhelpful, harmful-but-unhelpful, and harmful-but-helpful responses, and evaluates the resulting balance with AlpacaEval helpfulness and a safety benchmark averaging discriminative and generative harmlessness tasks (Zhang et al., 2024).

Other work measures the trade-off directly rather than hiding it. Preference Vector trains separate positive and negative models for helpfulness and harmlessness, extracts preference vectors, and combines them at test time with user-controlled coefficients $H$0 and $H$1. The paper reports smooth, interpretable changes in helpfulness and harmlessness as those coefficients are swept over $H$2, with refusal rate used as an explicit measure of over-conservatism (Liang et al., 27 Apr 2025). The controllable LLM framework likewise avoids collapsing the problem to one scalar and instead asks whether outputs match requested safety-helpfulness coordinates (Tuan et al., 2024).

Agentic post-training provides a particularly sharp view of the frontier. In ToolEmu, safety-only, helpfulness-only, sequential, and combined DPO configurations all end up near a linear Pareto frontier with $H$3, and safety training persists strongly through subsequent helpfulness training, with average safety persistence around 0.90 for $H$4 and 0.94 for $H$5 (Plaut, 13 Feb 2026). This suggests that post-training can preserve earlier safety gains while still moving along an approximately one-dimensional trade-off surface.

Human evaluation can reveal misalignment between relative and absolute metrics. In HRLAIF, basic RLAIF raises preference win ratio against the SFT baseline from 50% to 58.13% but lowers satisfaction rate from 62.92% to 58.33%; HRLAIF raises satisfaction rate to 65.00% while retaining a 56.87% preference win ratio. Satisfaction is defined more strictly: a response counts as satisfied only if it is honest, harmless, and effectively helpful (Li et al., 2024). This makes explicit that pairwise preference and absolute fulfillment are not interchangeable.

Mechanistic work on reward models adds a further layer. Mixed-objective reward models often underperform helpful-only and harmless-only reward models, have weaker and more compressed score distributions, and retain only part of the behavior captured by specialized models. Reported retention averages are around 0.5656 on helpfulness-oriented tasks and 0.6287 on harmlessness-oriented tasks, while targeted neuron ablations show that objective-specific neurons support their own objective and often hurt the opposing one (Tanwar et al., 11 Jun 2026). This suggests that some composite metric failures reflect representational interference rather than merely poor benchmark choice.

5. Reliability, bias, and failure modes

A central controversy is whether the metric itself is safe to use. Concatenation tests show that several harmfulness judges are unreliable under trivial input transformations: harmful prompt-response pairs judged unsafe individually can be judged safe after concatenation, and GPT-based judges can exhibit strong positional bias. In the reported experiments, GPT-3.5 misclassified unsafe concatenated 4-tuples as safe in 203 of 1000 cases, while GPT-4o showed strong order sensitivity in longer concatenations (Fandina et al., 2024). Safe-to-Serve reports an analogous problem for pairwise win-rate judging: an LLM-as-judge setup exhibited significant positional bias and was therefore not used as the final metric (Amballa et al., 2024).

Judge quality can also invert common expectations. HarmMetric Eval finds that reference-based metrics, especially METEOR and ROUGE-1, outperform LLM-based judges on its benchmark of harmfulness as unsafe + relevant + useful content, with METEOR achieving the best overall effectiveness score of 0.634 (Yang et al., 29 Sep 2025). This directly challenges the assumption that LLM judges are automatically superior because they possess broader semantic understanding.

Several papers warn that automated proxies can distort the target construct. The OPT-350M HmR/HpR/CAS evaluation depends on empirical thresholds $H$6 and $H$7, uses only 50 harmful and 50 helpful prompts, and relies on a reward model rather than direct human ratings (Pant, 10 Sep 2025). HILL argues that ASR can be misleading for defense evaluation if a defense changes the prompt’s intent so much that “attack success” is no longer measuring the original harmful goal (Luo et al., 17 Sep 2025). The controllable LLM study similarly warns that optimization metrics on the reward models used for synthetic data generation can overstate controllability relative to held-out reward models (Tuan et al., 2024).

Dataset construction introduces additional bias. The audit of Anthropic’s HH dataset finds that, under a manual taxonomy, only 11.8% of selected “harmless” responses are acceptable, while 44.5% are unhelpful and 43.7% remain harmful in their own right (Chehbouni et al., 2024). The same audit reports demographic skews in identity-term distributions and downstream disparate refusal behavior across groups. PluriHarms generalizes this concern by treating disagreement itself as signal: prompts in the intermediate-to-high harm range produce greater variance and entropy, and annotator traits such as toxicity experience and education help explain systematic disagreement (Li et al., 13 Jan 2026). A plausible implication is that a single binary harmfulness label can erase value-sensitive structure that is present in human judgments.

6. Extensions and open problems

Recent work is pushing helpfulness-harmfulness metrics beyond single-turn refusal benchmarks. Inference-time reviewer systems move evaluation into the execution loop and quantify whether feedback produces net value before tool execution, but extending those metrics to multi-turn settings requires partial credit and explicit handling of error propagation (Ta et al., 29 Apr 2026). Agentic safety benchmarks already score trajectories rather than isolated completions, and their results indicate that chat-style refusal metrics are insufficient for action-taking agents (Plaut, 13 Feb 2026).

Another active direction is controllability and disentanglement. Preference Vector treats helpfulness and harmlessness as modular preference directions in parameter space, while mechanistic reward-model studies argue that mixed-objective interference is driven partly by shared neurons in late layers (Liang et al., 27 Apr 2025, Tanwar et al., 11 Jun 2026). This suggests that better metrics may need to track not only output-level trade-offs, but also calibration, retention, and representational overlap.

A further extension concerns value retention under post-training. The animal-compassion study reports that helpfulness-domain post-training degrades animal compassion on AHB 2.2 more than coding-domain post-training does, across both SFT and GRPO, and recommends explicitly adding benchmarks such as AHB to evaluation suites when a model has value-laden mid-training (Brazilek et al., 30 Apr 2026). That result broadens the meaning of “harmfulness”: harmfulness may include loss of moral sensitivity, not only production of overtly toxic or unsafe content.

The literature also points toward pluralistic and context-sensitive safety metrics. PluriHarms argues for moving beyond one-size-fits-all binary labels toward benchmarks that jointly model harm level and disagreement (Li et al., 13 Jan 2026). The HH audit argues for subgroup-aware and context-sensitive safety measurement rather than overall refusal statistics alone (Chehbouni et al., 2024). Collectively, these works suggest that future helpfulness-harmfulness metrics will likely remain multidimensional: they will need to separate utility from safety, model disagreement rather than discard it, and account for the fact that intervention quality, refusal behavior, semantic relevance, and latent value preservation are not the same phenomenon.