Error-aware Speedup Score (ES_t)

Updated 5 July 2026

Error-aware Speedup Score (ES_t) is a geometric-mean metric that integrates measured speedup with corrections for numerical mismatches, compilation, and runtime failures.
The metric applies a tolerance parameter to modulate error forgiveness and penalize slow or incorrect executions, ensuring rigorous performance evaluation.
Paired with the aggregate score (AS), ES_t offers a continuous and robust signal for analyzing LLM-based compiler pass optimization across diverse tasks.

Searching arXiv for the cited papers and closely related context. Error-aware Speedup Score, denoted $ES_t$ , is a benchmark metric introduced in PassBench within the PassNet ecosystem for LLM-based compiler pass generation. In that setting, it is defined as a geometric-mean score over subgraphs that combines measured speedup against eager execution with explicit treatment of correctness failures, compilation failures, runtime failures, numerical tolerances, and timing stability. The metric is paired with an aggregate score, $AS$ , which combines $ES_t$ across a spectrum of tolerance settings. PassBench evaluates 200 curated long-tail fusible tasks comprising 2,060 subgraphs in total under this metric, and describes $ES_t$ as unifying correctness, stability, and performance (Liu et al., 28 May 2026).

1. Formal definition

For subgraph $i$ , PassBench defines the measured speedup as

$s_i = \frac{T_{i,\text{eager}}}{T_{i,\text{cand}}},$

where $T_{i,\text{eager}}$ is the eager-mode wall-clock runtime and $T_{i,\text{cand}}$ is the runtime under the candidate pass (Liu et al., 28 May 2026).

The core per-subgraph quantity is the error-aware rectified speedup $\hat{s}_{t,i}$ :

$\hat{s}_{t,i} = \begin{cases} s_i, & \mathrm{correct}_{t,i} \land s_i \ge 1,\[3pt] s_i^{p+1}, & \mathrm{correct}_{t,i} \land s_i < 1,\[3pt] b^{\mathbf{1}(t<c_i)}, & \text{otherwise.} \end{cases}$

Here $AS$ 0 is the error category, with $AS$ 1 for accuracy failure, $AS$ 2 for compilation failure, and $AS$ 3 for runtime failure; $AS$ 4 is a tolerance threshold; $AS$ 5 is a binary correctness indicator under tolerance $AS$ 6; $AS$ 7 is an exponential penalty factor for correct-but-slow executions; and $AS$ 8 is the base penalty for incorrect executions. The reported experiments use $AS$ 9 and $ES_t$ 0 (Liu et al., 28 May 2026).

The benchmark-level Error-aware Speedup Score is then

$ES_t$ 1

with $ES_t$ 2 the number of aggregated subgraphs. This geometric-mean construction applies at different scopes, including a single task or the full PassBench evaluation set of 2,060 subgraphs (Liu et al., 28 May 2026).

The paper also presents an explanatory factorization,

$ES_t$ 3

where $ES_t$ 4 aggregates correct-and-fast subgraphs, $ES_t$ 5 penalizes correct-but-slow ones, and $ES_t$ 6 accounts for errors under tolerance $ES_t$ 7. The implementation, however, uses the rectified per-subgraph definition and the geometric mean above (Liu et al., 28 May 2026).

2. Correctness model and tolerance parameter

The tolerance index $ES_t$ 8 controls how numerical mismatch and discrete failure modes are treated. For $ES_t$ 9, PassBench uses strict correctness with varying numerical tolerances. For $ES_t$ 0, it progressively forgives error categories: if $ES_t$ 1, the error is forgiven and the incorrect subgraph contributes $ES_t$ 2 rather than the penalty base $ES_t$ 3 (Liu et al., 28 May 2026).

This gives the metric a layered semantics. Correct-and-fast executions preserve their measured speedup. Correct-but-slow executions are penalized through $ES_t$ 4, which reduces to $ES_t$ 5 in the reported experiments because $ES_t$ 6. Incorrect executions contribute either $ES_t$ 7 or $ES_t$ 8, depending on whether the current tolerance level forgives their failure category (Liu et al., 28 May 2026).

Numerical correctness is checked using dtype-aware tolerances $ES_t$ 9 and $i$ 0. The benchmark specifies that it performs log-linear interpolation between reference points, stated as:

“We perform log-linear interpolation between reference points (e.g., $i$ 1 and $i$ 2) such that $i$ 3.”

Representative appendix entries are given explicitly. For bfloat16 and float32 absolute tolerance, the specification states:

bfloat16: $i$ 4, with $i$ 5 and $i$ 6
float32: $i$ 7, with $i$ 8 and $i$ 9

For relative tolerance, representative entries include:

bfloat16: $s_i = \frac{T_{i,\text{eager}}}{T_{i,\text{cand}}},$ 0, with $s_i = \frac{T_{i,\text{eager}}}{T_{i,\text{cand}}},$ 1 and $s_i = \frac{T_{i,\text{eager}}}{T_{i,\text{cand}}},$ 2
float32: $s_i = \frac{T_{i,\text{eager}}}{T_{i,\text{cand}}},$ 3, with $s_i = \frac{T_{i,\text{eager}}}{T_{i,\text{cand}}},$ 4 and $s_i = \frac{T_{i,\text{eager}}}{T_{i,\text{cand}}},$ 5

The paper notes that exact tables are provided in the appendix for other dtypes as well, including fp16, fp64, and complex types (Liu et al., 28 May 2026).

3. Stability and timing protocol

A distinctive feature of $s_i = \frac{T_{i,\text{eager}}}{T_{i,\text{cand}}},$ 6 is that stability is not an afterthought; it is part of the measurement protocol that determines whether a speedup value $s_i = \frac{T_{i,\text{eager}}}{T_{i,\text{cand}}},$ 7 is admissible. The benchmark uses 20 warmup runs and 100 timed trials, and defines the timing statistic as the median wall-clock latency over the 100 trials. It also monitors variability via the interquartile range relative to the median (Liu et al., 28 May 2026).

The stability criterion is stated explicitly: re-run if $s_i = \frac{T_{i,\text{eager}}}{T_{i,\text{cand}}},$ 8 of the median. In large-scale profiling, samples exceeding $s_i = \frac{T_{i,\text{eager}}}{T_{i,\text{cand}}},$ 9 were rejected as invalid measurements. This means that $T_{i,\text{eager}}$ 0 is not merely a function of correctness and nominal latency, but of reproducibly measured latency under a prescribed variance control (Liu et al., 28 May 2026).

The protocol also prescribes reverse evaluation order: the compiled candidate is run before the eager baseline. This is intended to avoid cache or memory residue enabling flawed code to pass correctness checks. A plausible implication is that the metric treats measurement order as part of benchmark integrity rather than as an incidental engineering detail (Liu et al., 28 May 2026).

The evaluation environment is fixed in the appendix: NVIDIA A30 GPU with 24 GB and compute capability 8.0, CUDA 12.8, cuDNN 9.10.2, PyTorch 2.9.1+cu128, Triton 3.5.1, and Ubuntu 24.04.1 LTS. Inputs are taken from packaged task metadata, and each task contains multiple shape and dtype instances so that a pass must generalize across them in a multi-graph formulation (Liu et al., 28 May 2026).

4. Failure taxonomy and integrity defenses

PassBench organizes incorrect executions into three error categories. Category $T_{i,\text{eager}}$ 1 is accuracy failure, meaning functional mismatch against the reference under the dtype-aware $T_{i,\text{eager}}$ 2 and $T_{i,\text{eager}}$ 3 tolerances. Category $T_{i,\text{eager}}$ 4 is compilation failure, meaning that the pass fails to compile or the pattern or rewriter is not executable. Category $T_{i,\text{eager}}$ 5 is runtime failure, covering crashes, exceptions at execution, or other runtime faults (Liu et al., 28 May 2026).

These categories directly determine the penalty term in $T_{i,\text{eager}}$ 6. If an incorrect execution is not forgiven at tolerance $T_{i,\text{eager}}$ 7, it contributes the base penalty $T_{i,\text{eager}}$ 8; if it is forgiven, it contributes $T_{i,\text{eager}}$ 9. The paper emphasizes that the metric never sets a per-graph contribution to zero; the minimum contribution is $T_{i,\text{cand}}$ 0, which is $T_{i,\text{cand}}$ 1 in the reported experiments (Liu et al., 28 May 2026).

The benchmark surrounds this failure model with layered integrity defenses. Static AST inspection detects forbidden high-level API usage. Runtime dispatch interception, implemented through PoisonDispatchTensor, detects forbidden aten dispatch dynamically. Reverse evaluation prevents correctness escapes caused by stale GPU state. Violations are routed into the same correctness and failure accounting used by $T_{i,\text{cand}}$ 2, typically as accuracy or runtime failures (Liu et al., 28 May 2026).

This design makes $T_{i,\text{cand}}$ 3 resistant to simple forms of benchmark gaming. The paper’s formulation is explicit that correctness indicators and error categories are not external annotations but outputs of an evaluation loop that includes these defenses. This suggests that the “error-aware” in $T_{i,\text{cand}}$ 4 refers not only to numerical mismatch and program failure but also to adversarial or exploitative behavior insofar as it manifests as benchmark-invalid execution (Liu et al., 28 May 2026).

5. Aggregation across subgraphs and tolerances

The benchmark aggregates rectified per-subgraph scores with a geometric mean. For a fixed tolerance $T_{i,\text{cand}}$ 5, the benchmark-level score is

$T_{i,\text{cand}}$ 6

Because this is a geometric mean, repeated failures or repeated slowdowns dominate the aggregate more strongly than they would under an arithmetic mean, while single extreme outliers do not dominate the score (Liu et al., 28 May 2026).

PassBench then defines Aggregated Speedup, $T_{i,\text{cand}}$ 7, by combining $T_{i,\text{cand}}$ 8 values across the tolerance spectrum:

$T_{i,\text{cand}}$ 9

The weight schedule is also specified:

$\hat{s}_{t,i}$ 0

The rationale given in the paper is that the strict-correctness regime $\hat{s}_{t,i}$ 1 receives full weight, the relaxed regime decays exponentially with tolerance, and extreme regimes receive near-zero weight (Liu et al., 28 May 2026).

This weighting makes $\hat{s}_{t,i}$ 2 a deliberately nonuniform summary. Production-grade correctness is prioritized, but the relaxed regimes still contribute a smooth learning signal. The paper contrasts this with binary metrics such as the fraction of correct graphs exceeding a speedup threshold, arguing that $\hat{s}_{t,i}$ 3 and $\hat{s}_{t,i}$ 4 provide a continuous, per-graph-sensitive signal more suitable for iterative training and analysis (Liu et al., 28 May 2026).

In PassBench, eager execution is the baseline for all reported speedups. TorchInductor is treated as another candidate and is also scored with $\hat{s}_{t,i}$ 5 and $\hat{s}_{t,i}$ 6 against eager execution. The main reported aggregate figures include: eager with $\hat{s}_{t,i}$ 7 and geometric-mean speedup $\hat{s}_{t,i}$ 8 by definition; TorchInductor with $\hat{s}_{t,i}$ 9 and geometric-mean speedup $\hat{s}_{t,i} = \begin{cases} s_i, & \mathrm{correct}_{t,i} \land s_i \ge 1,\[3pt] s_i^{p+1}, & \mathrm{correct}_{t,i} \land s_i < 1,\[3pt] b^{\mathbf{1}(t<c_i)}, & \text{otherwise.} \end{cases}$0; and the best frontier model with $\hat{s}_{t,i} = \begin{cases} s_i, & \mathrm{correct}_{t,i} \land s_i \ge 1,\[3pt] s_i^{p+1}, & \mathrm{correct}_{t,i} \land s_i < 1,\[3pt] b^{\mathbf{1}(t<c_i)}, & \text{otherwise.} \end{cases}$1. The statement that the best frontier model trails TorchInductor by $\hat{s}_{t,i} = \begin{cases} s_i, & \mathrm{correct}_{t,i} \land s_i \ge 1,\[3pt] s_i^{p+1}, & \mathrm{correct}_{t,i} \land s_i < 1,\[3pt] b^{\mathbf{1}(t<c_i)}, & \text{otherwise.} \end{cases}$2 refers to $\hat{s}_{t,i} = \begin{cases} s_i, & \mathrm{correct}_{t,i} \land s_i \ge 1,\[3pt] s_i^{p+1}, & \mathrm{correct}_{t,i} \land s_i < 1,\[3pt] b^{\mathbf{1}(t<c_i)}, & \text{otherwise.} \end{cases}$3, comparing $\hat{s}_{t,i} = \begin{cases} s_i, & \mathrm{correct}_{t,i} \land s_i \ge 1,\[3pt] s_i^{p+1}, & \mathrm{correct}_{t,i} \land s_i < 1,\[3pt] b^{\mathbf{1}(t<c_i)}, & \text{otherwise.} \end{cases}$4 to $\hat{s}_{t,i} = \begin{cases} s_i, & \mathrm{correct}_{t,i} \land s_i \ge 1,\[3pt] s_i^{p+1}, & \mathrm{correct}_{t,i} \land s_i < 1,\[3pt] b^{\mathbf{1}(t<c_i)}, & \text{otherwise.} \end{cases}$5 (Liu et al., 28 May 2026).

The same study also reports that frontier LLMs can achieve up to $\hat{s}_{t,i} = \begin{cases} s_i, & \mathrm{correct}_{t,i} \land s_i \ge 1,\[3pt] s_i^{p+1}, & \mathrm{correct}_{t,i} \land s_i < 1,\[3pt] b^{\mathbf{1}(t<c_i)}, & \text{otherwise.} \end{cases}$6 raw speedup over TorchInductor on individual subgraphs, even though aggregate $\hat{s}_{t,i} = \begin{cases} s_i, & \mathrm{correct}_{t,i} \land s_i \ge 1,\[3pt] s_i^{p+1}, & \mathrm{correct}_{t,i} \land s_i < 1,\[3pt] b^{\mathbf{1}(t<c_i)}, & \text{otherwise.} \end{cases}$7 remains lower. Within the semantics of $\hat{s}_{t,i} = \begin{cases} s_i, & \mathrm{correct}_{t,i} \land s_i \ge 1,\[3pt] s_i^{p+1}, & \mathrm{correct}_{t,i} \land s_i < 1,\[3pt] b^{\mathbf{1}(t<c_i)}, & \text{otherwise.} \end{cases}$8, this indicates that isolated capability is not sufficient; consistency across correctness, stability, and broad task coverage determines the final score (Liu et al., 28 May 2026).

The notation $\hat{s}_{t,i} = \begin{cases} s_i, & \mathrm{correct}_{t,i} \land s_i \ge 1,\[3pt] s_i^{p+1}, & \mathrm{correct}_{t,i} \land s_i < 1,\[3pt] b^{\mathbf{1}(t<c_i)}, & \text{otherwise.} \end{cases}$9 also has a narrower historical ambiguity in nearby arXiv literature. In “Error-awareness Accelerates Active Automata Learning,” the term “Error-aware Speedup Score” is explicitly not defined; the paper instead reports number of symbols, number of output queries, number of equivalence queries, and sizes of conformance-test suites, from which a speedup ratio can be derived post hoc (Kruger et al., 25 Feb 2026). In “Frequency-Aware Error-Bounded Caching for Accelerating Diffusion Transformers,” the paper likewise does not literally define $AS$ 00 by name; rather, a timestep-level score can be formalized from SpectralCache’s gating variables, schedule, and error proxies as a faithful mapping of its decision policy (Li, 5 Mar 2026).

Accordingly, the fully explicit, native definition of Error-aware Speedup Score as a benchmark metric belongs to PassBench. In that context, $AS$ 01 is not a generic synonym for “error-aware acceleration,” but a specific geometric-mean evaluation functional whose inputs are rectified per-subgraph speedups, categorical failure handling, dtype-aware tolerance control, and stability-qualified timing measurements (Liu et al., 28 May 2026).