Harmful-Call Rate in Safety Metrics

Updated 12 July 2026

Harmful-call rate is a rate-based safety metric that quantifies the frequency of harmful outputs or actions, with definitions varying across domains like telecom, LLM safety, and medical decision-making.
Methodologies rely on specific decision rules—such as QoS thresholds, moderation classifiers, or causal estimands—to accurately measure harm in context-dependent systems.
Empirical studies demonstrate that balancing harmful-call rate reductions with operational performance is crucial for system design improvements across diverse applications.

Searching arXiv for papers on harmful-call rate and related metrics across telecom, LLM safety, agents, and medical decision-making. Harmful-call rate denotes, in a domain-dependent sense, the frequency with which a system produces calls, responses, actions, or assignments that cross an operational harm criterion. In telephony quality studies, it is tied to unacceptable perceived quality or to dropped calls; in spam and fraud detection, to malicious calls that evade blocking; in LLM safety, to the proportion of harmful answers, jailbreak successes, forbidden tool calls, or policy-violating skills; and in individualized treatment rules, to the proportion of treated individuals who are harmed by the assignment rule (Cipressi et al., 2019, Cartagena et al., 18 Feb 2026, Wu et al., 8 May 2025). This suggests that the concept is best understood as a family of rate-based safety metrics rather than a single standardized quantity.

1. Terminological scope and operationalization

The literature does not use a single canonical definition. Instead, the counted unit and the harm criterion vary with the system under study. In VoLTE quality analysis, harmful calls are inferred from low perceptual quality, typically via an R-factor threshold such as $R < 70$ , with $MOS < 3.5$ often used as a threshold for unacceptable quality (Cipressi et al., 2019). In harmful fine-tuning studies for LLMs, the corresponding quantity is often called the harmful score, defined as the ratio of harmful questions for which the model delivers harmful answers, with moderation used to classify outputs (Huang et al., 29 Jan 2025). In LLM agent studies, the relevant event may be a forbidden tool call rather than unsafe text, and divergence between the two is formalized explicitly (Cartagena et al., 18 Feb 2026). In medical decision-making, the analogous construct is the harm rate of an individualized treatment rule (Wu et al., 8 May 2025).

Domain	Counted unit	Harm criterion
VoLTE quality	Call	R-factor below threshold
LLM response safety	Model answer	Moderation flags harmful output
LLM agent safety	Tool-call trajectory	Forbidden tool call attempted
Skill or code ecosystem	Skill or API sequence	Policy-violating or insecure use
Individualized treatment	Treatment assignment	Worse outcome under treatment

A recurrent methodological consequence is that harmful-call rate is rarely meaningful without an accompanying decision rule. The rule may be a QoS threshold, a moderation classifier, a refusal detector, a deterministic predicate over tool calls, a security specification such as CrySL, or a causal estimand over potential outcomes. This is why several papers pair the rate itself with secondary metrics such as leakage ratio, gradient similarity, BLEU, refusal rate, downstream accuracy, channel utilization, or expected reward (Huang et al., 29 Jan 2025, Tony et al., 2022, Wu et al., 8 May 2025).

2. Telephony quality, dropped calls, and queueing-based harms

In packet voice networks, harmful-call rate arises from quality impairment. A large-scale analysis of over 10 million real-world VoLTE calls used VQmon, an enhanced version of the standardized E-Model, to compute the R-factor, and found that packet loss rate $P_{loss}$ is the dominant factor affecting call quality, more influential than jitter (Cipressi et al., 2019). For AMR, the dependence of the R-factor on packet loss is best described by an exponential decay,

$y(x) = 17.953 + 71.63 e^{-x/0.12},$

whereas for AMR-WB it is nearly linear,

$y(x) = 99.01 - 340.70x.$

With a harmful threshold such as $R = 70$ , the AMR-WB relation yields $x \approx 0.085$ , or an 8.5% loss rate, as the onset at which harmful calls become common; AMR degrades more sharply at lower loss (Cipressi et al., 2019). The practical implication stated in the study is that AMR-WB yields higher R-factors and a lower rate of harmful calls under matching network conditions.

Wireless cellular QoS papers operationalize harmful-call rate more directly as dropped handoffs. In one admission-control formulation, the key QoS parameters are Call Dropping Probability (CDP) and Handoff Dropping Probability (HDP); the study argues that controlling HDP also controls CDP to a minimum extent while maintaining lower blocking rates for new calls (Malathy et al., 2010). A related guard-band policy defines Call Dropping Probability as $P_D = P(C)$ , the probability that all channels are occupied so an incoming handoff is dropped, and explicitly connects CDP to the harmful-call rate of already accepted calls (Rahman et al., 2018). The later QoS provisioning work using Uniform Fractional Band (UFB) likewise treats handover call dropping probability $P_D$ as the harmful-call rate to be minimized while allowing lower blocking of new calls and higher channel utilization (Rahman, 2015).

Call-center studies extend the notion from dropped radio calls to abandonment. A fluid approximation with redials and reconnects models the total effective arrival rate as

$E\Lambda(t) = \lambda(t) + \delta_{RD} E Z_{RD}(t) + \delta_{RC} E Z_{RC}(t),$

and then uses Erlang A to estimate abandonment and service levels (Ding et al., 2013). A two-way functional hazards model represents waiting hazards as a function of both waiting duration and time of day,

$MOS < 3.5$ 0

with penalized likelihood and ADMM estimation (Li et al., 2017). In the bank call-center data analyzed there, abandonment hazards decrease in the first minute and then show periodic sharp peaks; the paper links these peaks to automatic announcements and uses the resulting hazard surfaces as staffing inputs (Li et al., 2017). Across these queueing papers, harmful-call rate is thus tied to abandonment or forced termination rather than content.

3. Malicious telephony and the rate of harmful calls that evade filtering

In spam and scam telephony, harmful-call rate refers to the proportion of malicious calls that reach users or remain unblocked. A machine-learning approach based on 9 billion call-log records and 29 engineered features reported that the best model can reduce up to 90% unblocked malicious calls while maintaining a precision over 99.99% on the benign call traffic (Li et al., 2018). The paper’s central metric is therefore not quality degradation but residual malicious-call exposure after automated filtering.

International robocall analysis broadens the notion from per-system filtering performance to cross-country prevalence. Using 8.7 million call detail records, 839 robocall transcripts from 28 identified robocall campaign clusters, and 677 robocall recordings, the study concludes that robocalls are an international problem but that the severity is significantly higher in the US than in other countries (Altwlkany et al., 30 Jun 2026). Within the honeypot deployment, US numbers received 8,314,813 calls, or 95.1% of all calls in the experiment, while non-US numbers received 432,538 calls, or 4.9%; this corresponded to 692.6 calls per available US number versus 4.25 per available non-US number over nine months (Altwlkany et al., 30 Jun 2026). The same study also reports that 61.2% of 2.6 million Do Not Call complaints in 2025 were robocalls, and that reported US losses due to phone fraud reached USD 1.1 billion in 2025 (Altwlkany et al., 30 Jun 2026).

These telephony papers also highlight an important distinction between incidence and severity. A system may lower the proportion of malicious calls reaching users, as in the 90% reduction result, while still operating in an environment where aggregate exposure is high because call volumes themselves are massive (Li et al., 2018, Altwlkany et al., 30 Jun 2026). Conversely, a cellular admission-control policy may hold call dropping probability nearly constant while substantially reducing call blocking probability, changing inconvenience and harm in different ways (Rahman et al., 2018). The term harmful-call rate therefore often requires explicit statement of the denominator: all calls, unblocked calls, handoff attempts, or answered calls.

4. Harmful-call rate in LLM responses, jailbreaks, and harmful fine-tuning

In LLM safety research, harmful-call rate is most often instantiated as the rate at which a model gives harmful answers to harmful prompts. The Virus attack paper defines Harmful Score as “the ratio of harmful questions that the LLM will deliver harmful answers,” using the BeaverTails moderation model to classify responses (Huang et al., 29 Jan 2025). Its three-stage pipeline comprises safety alignment, guardrail moderation, and user fine-tuning. The study distinguishes leakage ratio, the percentage of harmful samples passed by the guardrail, from harmful score, and shows that guardrail moderation alone is insufficient: with Llama Guard2, harmful data had leakage ratio approximately 38%, while Virus achieved 100% leakage ratio and a harmful score of 30.40, compared with 14.10 for a jailbreak-only baseline and 14.40 for a mixing baseline (Huang et al., 29 Jan 2025). The paper’s methodological point is explicit: leakage alone is not enough, because bypassed samples must also preserve the harmful gradient.

Jailbreak studies at inference time use closely related metrics. Jigsaw Puzzles (JSP) defines Attack Success Rate by Attempt,

$MOS < 3.5$ 1

and Attack Success Rate by Question,

$MOS < 3.5$ 2

where a response is harmful if it directly answers the unsplit harmful question rather than refusing (Yang et al., 2024). On 189 harmful queries across five advanced LLMs, JSP reports an average ASR-q of 93.76%, and on GPT-4 it reports ASR-q of 99.65% and ASR-a of 93.65% (Yang et al., 2024). The paper notes that the term “harmful-call rate” is not used explicitly, but ASR-a and ASR-q directly measure how often a model generates a harmful answer.

Post-attack mitigation studies formalize the same quantity as a model-level harmful-output rate. Antidote defines Harmful Score (HS) as the proportion of model outputs to malicious prompts that the BeaverTails moderation model flags as unsafe, using 1000 unseen harmful instructions for evaluation (Huang et al., 2024). Antidote is a post-fine-tuning pruning method based on Wanda importance scores,

$MOS < 3.5$ 3

followed by top- $MOS < 3.5$ 4 masking and pruning (Huang et al., 2024). Averaged over settings in Table 1, HS decreases from 73.52 for SFT to 60.88 for Antidote, while finetune accuracy changes from 95.00 to 93.05 (Huang et al., 2024).

Reinforcement-learning attacks use yet another but compatible formalization. HarmRLVR introduces a harmfulness evaluator $MOS < 3.5$ 5, average harmfulness score

$MOS < 3.5$ 6

and attack success rate

$MOS < 3.5$ 7

Across five models, harmful RLVR raises the average harmfulness score to 4.94 and the attack success rate to 96.01%, substantially above the harmful fine-tuning baseline (Liu et al., 17 Oct 2025). A common misconception addressed by these papers is that refusal measured at the prompt-text level fully characterizes safety. The measurements instead show that harmful-call rate depends strongly on how harm is operationalized and on whether the evaluation tracks outputs, gradients, or downstream action.

5. Tool-call safety, harmful skills, and action-level divergence

Agentic LLMs introduce a stricter notion of harmful-call rate because the relevant event is an external action rather than an answer. The GAP benchmark defines TC-safe for an interaction $MOS < 3.5$ 8 as

$MOS < 3.5$ 9

where $P_{loss}$ 0 is the set of forbidden tool calls attempted (Cartagena et al., 18 Feb 2026). It defines T-safe as textual refusal without PII leakage,

$P_{loss}$ 1

and formalizes divergence with

$P_{loss}$ 2

The study evaluates six frontier models across six regulated domains, seven jailbreak scenarios per domain, three system prompt conditions, two prompt variants, and three governance modes, producing 17,420 analysis-ready datapoints (Cartagena et al., 18 Feb 2026). Its main empirical result is that text safety does not transfer to tool-call safety: even under safety-reinforced prompts, 219 GAP cases persist across all six models, and prompt wording shifts TC-safe rates by 21 percentage points for the most robust model and 57 for the most prompt-sensitive one (Cartagena et al., 18 Feb 2026). Runtime governance contracts reduce information leakage but show no detectable deterrent effect on forbidden tool-call attempts themselves.

A complementary ecosystem-level measure appears in HarmfulSkillBench, which studies harmful skills in public registries rather than harmful outputs from a single model. The paper analyzes 98,440 skills across ClawHub and Skills.Rest and defines an overall skill risk score as the maximum category-level risk score, with skills at score $P_{loss}$ 3 classified as harmful (Jiang et al., 16 Apr 2026). It finds that 4.93% of all skills, or 4,858 skills, are harmful; ClawHub has an 8.84% harmful rate and Skills.Rest 3.49% (Jiang et al., 16 Apr 2026). The benchmark results show that harmful skills lower refusal rates and raise harm scores: the average harm score rises from 0.27 without the skill to 0.47 with it, and further to 0.76 when the harmful intent is implicit rather than stated as an explicit user request (Jiang et al., 16 Apr 2026).

Taken together, these papers redefine harmful-call rate from “how often the model says something harmful” to “how often the system does something harmful.” That shift is substantive rather than cosmetic. A model may appear safe under refusal-style text evaluation while still attempting forbidden tool calls, and a registry may look benign at the malware or vulnerability level while still hosting a measurable fraction of policy-violating skills (Cartagena et al., 18 Feb 2026, Jiang et al., 16 Apr 2026).

6. Generalizations to code generation, treatment rules, and image generation

The concept also appears in domains where the “call” is neither a phone connection nor an LLM response. In the analysis of cryptographic API sequences from GitHub, Harmful-Call Rate (HCR) is the proportion of extracted call sequences with at least one misuse: $P_{loss}$ 4 Among 206 analyzed sequences, 123 contained at least one misuse, giving $P_{loss}$ 5 (Tony et al., 2022). The paper groups these misuses into six categories, including missing predicate, insecure default implementation, incorrect encoding, incorrect randomization, incorrect method call, and missing method call, and argues that BLEU is not a reliable security metric because high sequence similarity does not guarantee secure API use (Tony et al., 2022). Here harmful-call rate measures insecure operational patterns in software ecosystems.

In causal decision-making, the directly analogous quantity is the harm rate of an individualized treatment rule. The treatment harm rate is defined as

$P_{loss}$ 6

and the optimal constrained rule takes the form

$P_{loss}$ 7

when the harm constraint is binding (Wu et al., 8 May 2025). The paper’s objective is to maximize reward while ensuring that the harm rate remains below a pre-specified threshold, using plug-in estimation when harm rate is identifiable and three strategies under partial identification (Wu et al., 8 May 2025). Although the paper does not foreground the term harmful-call rate, its structure is directly analogous: a rate of harmful decisions constrained by design.

Text-to-image safety extends the same logic to harmful outputs in generative vision. The CASG paper defines harmful rate as

$P_{loss}$ 8

with images flagged if either Q16 or NudeNet marks them harmful (Xiang et al., 24 Feb 2026). On T2I safety benchmarks, CASG reduces harmful rate by up to 15.4% compared to existing methods; for example, on T2VSafetyBench, SLD records 25.2% and CASG+SLD 9.8% (Xiang et al., 24 Feb 2026). The paper explicitly notes that there is no separate harmful-call rate in this setting, only harmful rate. A plausible implication is that the literature increasingly treats harmful-call rate as a transferable measurement pattern: count harmful events, specify the detector or causal criterion, and report the resulting frequency under realistic operating conditions.

Across these non-telephony domains, two methodological lessons recur. First, correctness proxies can be misleading: BLEU does not ensure security, text refusal does not ensure tool-call safety, and category-averaged safety guidance can even amplify overall harmful rate under “harmful conflicts” (Tony et al., 2022, Cartagena et al., 18 Feb 2026, Xiang et al., 24 Feb 2026). Second, controllability matters as much as measurement: treatment rules can be designed to satisfy a harm threshold, pruning can lower harmful score post fine-tuning, and category-aligned safety guidance can reduce harmful image generation without relying on a single averaged safety direction (Wu et al., 8 May 2025, Huang et al., 2024, Xiang et al., 24 Feb 2026).