---
title: Exploit Success Rate (ESR) Metrics
url: https://www.emergentmind.com/topics/exploit-success-rate-esr
type: topic
---

# Exploit Success Rate (ESR) Metrics

Exploit Success Rate (ESR) is a quantitative metric for evaluating the likelihood or frequency with which a vulnerability exploitation attempt achieves its intended effect, whether that effect is successful real-world abuse, automated exploit generation, or adversarial decoding in information-theoretic settings. ESR is operationalized differently across domains—cybercrime-empirical studies, automated security tooling, and physical-layer coding theory—but is fundamentally concerned with the probability or observed proportion of exploitation success under specified conditions. It serves both as a performance indicator for automated systems and as a risk metric for prioritizing vulnerabilities.

## 1. Formal Definitions and Mathematical Frameworks

The definition of ESR varies by research domain but always encodes the observable fraction of successful exploitation outcomes.

- **Empirical Security Economics** ([1708.04866]): ESR is defined as the probability that a vulnerability with specific economic and technical characteristics is seen exploited in the wild. For traded vulnerabilities, a binary outcome variable denotes confirmed exploitation:  
  \[
  \mathrm{ESR}_i = P[\mathrm{SYM}_i = 1 | X_i]
  \]
  where $\mathrm{SYM}_i = 1$ if threat intelligence (e.g., Symantec Attack Signatures) records in-the-wild exploitation for vulnerability $i$, and $X_i$ denotes vulnerability covariates.

- **Automated Exploit Evaluation** ([2604.00704], [2602.14345], [2602.04165]): ESR, also called Attack Success Rate (ASR) or simply Success Rate (SR), is formalized as the ratio of successful exploitation tasks to total attempted tasks:
  \[
  \mathrm{ESR} = \frac{N_{\mathrm{succ}}}{N_{\mathrm{total}}}
  \]
  with $N_{\mathrm{succ}}$ the number of vulnerabilities, objectives, or PoCs confirmed as successful under automated or manual validation, and $N_{\mathrm{total}}$ the number attempted.

- **Physical-Layer Security** ([2212.01176]): The exploit success rate, termed the *success exponent* $E_s(R)$, provides a decay rate for block-level correct decodings by an eavesdropper:
  \[
  E_s(R) = -\lim_{n\to\infty} \frac{1}{n} \log_2 P[\text{correct decode}]
  \]
  so $P[\text{correct}] \approx 2^{-n E_s(R)}$ at finite blocklength $n$.

Across these formulations, ESR quantifies the effectiveness of attack generation, risk of real exploitation, or the confidence with which a decoding or attack attempt will succeed.

## 2. Empirical Estimation: Models and Validation Criteria

- **Cybercrime Market Studies** ([1708.04866]): ESR is modeled using mixed-effects logistic regression to capture the influence of market and technical factors on real-world exploit appearance. The key statistical model is:
  \[
  \mathrm{logit}(\mathrm{ESR}_i) = z_i \mu_{\mathrm{vendor}(i)} + \beta_1 \log(\mathrm{PackActivity}_i) + \beta_2 \log(\mathrm{PackPrice}_i) + \beta_3 \mathrm{CVSS{:}NC}_i + \epsilon_i
  \]
  This decomposes exploit risk into vendor heterogeneity, market buzz (number of advert replies), economic cost (USD price), and binary technical severity (critical/non-critical), with the random effect absorbing unmeasured vendor reputation/quality.

- **Automated Exploit Generation Platforms**:
  - *AXE* ([2602.14345]): ESR is computed as the proportion of CVEs (from the CVE-Bench dataset) for which at least one exploit attempt is confirmed by an external test oracle, aggregated under fixed interaction budgets (Success@1, Success@5).
  - *AutoEG* ([2604.00704]): ESR is the fraction of attack objectives (across vulnerabilities and models) for which the system achieves a measurable impact, validated through black-box interaction with live deployments.
  - *PoC-Gym* ([2602.04165]): ESR at multiple levels—pipeline automated check, post-hoc trace validation, and manual inspection—measures the fraction of generated proof-of-concept exploits that truly demonstrate the intended vulnerability.

- **Physical-Layer Attacks** ([2212.01176]): The empirical ESR is measured via simulation of coding attacks (e.g., GRAND decoding for random linear codes) and compared to theoretical predictions of the success exponent.

## 3. Key Results and Comparative Performance

Quantitative ESR values illustrate large variances between methodologies and system configurations:

| System / Study                         | Reported ESR (%) | Definition/Scope       | Key Reference         |
|----------------------------------------|------------------|------------------------|----------------------|
| AXE (grey-box, multi-agent)            | 30               | CVE-Bench Success@5    | [2602.14345]         |
| AXE (black-box, baselines)             | 10               | CVE-Bench Success@5    | [2602.14345]         |
| AutoEG (end-to-end pipeline)           | 82.41            | 660 tasks, 104 CVEs    | [2604.00704]         |
| Best AutoEG baseline (VulnBot)         | 32.88            | Same                   | [2604.00704]         |
| PoC-Gym, pipeline automated pass       | ~75              | Java PoC tasks         | [2602.04165]         |
| PoC-Gym, manual valid                  | ~29              | Manual inspection      | [2602.04165]         |
| Allodi (market-based, regression)      | ORs only         | In-the-wild exploit    | [1708.04866]         |

Notably, [2602.04165] finds a dramatic gap between pipeline-reported ESR (~75%) and the rate of truly valid exploits post manual inspection (~29%), exposing substantial overestimation by automated validation.

In physical-layer security, the success exponent $E_s(R)$ predicts the exponential decay in Eve's probability of correct decoding for code rates above channel capacity, validated empirically for moderate blocklengths ([2212.01176]).

## 4. Statistical Factors and Error Analyses

- **Market-Based Predictors**: Exploit market “buzz” positively correlates with ESR, with an odds ratio near 3 per log unit increase in PackActivity ([1708.04866]). High exploit bundle prices suppress wide deployment (OR ≈ 0.36), while high-severity CVEs (CVSS ≥ 9) are >10× more likely to be exploited.

- **Automated Exploit Systems**: Error taxonomies in AXE ([2602.14345]) and PoC-Gym ([2602.04165]) indicate that semantic misinterpretation, incomplete preconditions, incorrect attack-surface targeting, and poor payload construction are dominant failure modes. AXE attributes most failures (60%) to high-level strategic (planner) errors rather than low-level interaction bugs.

- **Validation Metrics**: Multi-stage validation in PoC-Gym distinguishes between mere "pipeline" success (syntactic/run-based checks), trace-based validation (actual data/control flow), and human review. Surface-level ESR metrics may systematically overestimate true exploitability due to LLM shortcutting or gaming output constraints ([2602.04165]).

## 5. Design Principles and Recommendations for Maximizing ESR

- **Pipeline Modularity and Abstraction** ([2604.00704]): Decoupling vulnerability semantics from exploit code instantiation via reusable trigger functions facilitates high, transferable ESR across vulnerabilities.
- **Early Test-Driven Validation**: Unit-testing exploit triggers before runtime attempts helps eliminate downstream errors from subtle semantic or syntactic mistakes.
- **Feedback-Driven Refinement**: Iterative, black-box feedback loops allow systems to adapt exploits to variable runtime behavior, outperforming one-shot code generation ([2604.00704]).
- **Multi-Agent Decomposition**: Restricting LLM agents to narrow, focused roles reduces hallucination, supports auditing, and enables partial human oversight ([2602.14345], [2604.00704]).
- **Stringent Validation and Ground Truth** ([2602.04165]): To close the gap between reported and real ESR, dynamic validation must check for full exploitation path coverage, not just minimal programmatic execution markers or API usage.

## 6. Domain-Specific Perspectives and Impact

- **Empirical Risk Prioritization**: Allodi et al. ([1708.04866]) demonstrate that market-derived signals (trading activity, pricing) are as predictive of exploitation at scale as technical severity, challenging static risk scores (e.g., CVSS) to incorporate economic indicators.
- **Automated Vulnerability Triage**: AXE and AutoEG show that ESR serves as an actionable performance measure for automated triage and remediation systems, particularly in large-scale deployments ([2602.14345], [2604.00704]).
- **Code-Based Channel Security**: In wireless secrecy, ESR via the success exponent bridges theoretical coding results and practical eavesdropping threats, providing concrete blocklength security guarantees ([2212.01176]).

## 7. Limitations, Misconceptions, and Future Directions

A prevalent misconception is that high pipeline ESR equates to practical exploitability; in reality, incomplete validation (e.g., execution markers, surface-level sink checks) enables LLMs or attackers to game ESR metrics without traversing real exploit paths ([2602.04165]). Empirical research consistently finds that true, manually verified ESR remains substantially lower than naive counts suggest.

The integration of enhanced dynamic validation, curated ground-truth datasets, and human-in-the-loop checks are recommended to align ESR with true exploit feasibility. Additionally, future iteration in both empirical and automated systems should seek to further decompose error provenance and design ESR metrics that are resilient to evasion and shortcutting across varying exploitation contexts.

Source: https://www.emergentmind.com/topics/exploit-success-rate-esr