LLM Efficiency Benchmark

• LLM Efficiency Benchmark is a comprehensive framework that evaluates LLMs on resource usage, including execution time, memory, and energy consumption.
• It categorizes evaluations by artifact quality, serving stack performance, and agent interactions to capture diverse computational tradeoffs.
• It employs stress testing, reference baselines, and normalization techniques to expose efficiency gaps relative to human or domain-specific standards.

An LLM efficiency benchmark is a benchmarking framework that evaluates resource-sensitive behavior rather than treating correctness or task success as the only endpoint. In the recent literature, efficiency has been operationalized at several distinct levels: execution time and memory of generated or translated programs, energy consumed per served request, inference and hardware costs of compressed models, time- and token-normalized task performance for agents, and hardware utilization of generated GPU kernels (Gong et al., 17 Mar 2026, Pronk et al., 10 Sep 2025, Yang et al., 2024, Ferrag et al., 1 Jan 2026, Wang et al., 6 May 2026). The field has therefore moved from a single question—whether an LLM can produce a valid output—to a broader one: whether it can do so with competitive computational, memory, latency, communication, or deployment characteristics.

1. Scope and taxonomy of efficiency evaluation

Recent benchmarks divide naturally by what is being measured. Some evaluate the efficiency of artifacts produced by LLMs, such as translated source code, generated programs, Verilog RTL, or Triton kernels. Others evaluate the model or serving stack itself, focusing on energy, throughput, memory, and compression. A third group evaluates agentic systems, where efficiency depends on multi-turn interaction, tool use, latency, token budgets, and environmental constraints (Gong et al., 17 Mar 2026, Pronk et al., 10 Sep 2025, Ferrag et al., 1 Jan 2026, Abdelatty et al., 16 Oct 2025, Wang et al., 6 May 2026).

Benchmark family	Primary object	Representative metrics
Code and translation benchmarks	Generated or translated programs	ET, PM, Beyond, eff@k, ET/MP/MI
Deployment and compression benchmarks	Model inference and serving	Energy per request, model size, GPU memory, tokens/s
Agent and systems benchmarks	Multi-turn control loops and tool use	$\alpha^3_{\text{per-sec}}$, $\alpha^3_{\text{per-1k}}$, LCBA-Efficiency, ARB/AMB, IOU/MFU

This diversity matters because “efficiency” is not a single invariant quantity across benchmark families. TRACE measures wall-clock execution time and peak resident set size of translated programs (Gong et al., 17 Mar 2026). The vLLM-based “LLM Efficiency Benchmark” measures energy per request under burst concurrency and production-style serving (Pronk et al., 10 Sep 2025). LLMCBench evaluates compression through training consumption, inference consumption, hardware acceleration, and trustworthiness tracks (Yang et al., 2024). $\alpha^3$-Bench and LoCoBench-Agent treat efficiency as an end-to-end property of an interactive agent, incorporating latency, tokens, tool calls, and communication cost (Ferrag et al., 1 Jan 2026, Qiu et al., 17 Nov 2025).

A plausible implication is that benchmark results are only comparable within a well-specified efficiency regime. A model that is efficient as a code generator need not be efficient as a serving system, and a benchmark that compresses prompt sets efficiently does not necessarily evaluate model deployment efficiency.

2. Core design patterns and metric construction

Efficiency benchmarks in this area typically share four design principles: strong reference baselines, stress-inducing workloads, correctness gating, and per-task normalization. ENAMEL formalizes efficiency with eff@k, extending pass@k from binary correctness to a continuous efficiency score while handling right-censored execution time; it also provides an unbiased, variance-reduced estimator via Rao–Blackwellization (Qiu et al., 2024). TRACE adopts the Beyond score, normalizing each candidate’s execution time or peak memory against the spectrum of correct reference translations for the same task (Gong et al., 17 Mar 2026). EffiBench-X normalizes execution time, peak memory, and memory integral against human-expert implementations in the same language, clipping scores to $[0,1]$ before aggregation (Qing et al., 19 May 2025).

A representative normalization used in TRACE is:

$$B_X(c)=\frac{\max \mathcal{R}_X-\mathrm{clip}(x_c,\min \mathcal{R}_X,\max \mathcal{R}_X)}{\max \mathcal{R}_X-\min \mathcal{R}_X}\times 100\%,$$

where $X \in \{\mathrm{ET}, \mathrm{PM}\}$, $\mathcal{R}_X$ is the task-specific reference spectrum, and incorrect translations receive zero (Gong et al., 17 Mar 2026). ENAMEL’s central move is analogous: efficiency is defined only for correct code, and the best among $k$ samples is summarized by eff@k rather than raw runtime alone (Qiu et al., 2024).

Stress testing is the other recurrent design. TRACE uses Progressive Stress Test Generation, where GPT-4o iteratively synthesizes Python programs that generate large or adversarial test inputs; candidate inputs are retained only if all three ground-truth implementations execute successfully and produce identical outputs (Gong et al., 17 Mar 2026). TRACY uses a closely related two-stage pipeline with stress-test generation followed by efficiency-oriented task pruning, ending with 1,011 translation tasks and an average of 22.1 verified reference translations per task (Gong et al., 15 Aug 2025). The common idea is that small unit tests systematically hide asymptotic and constant-factor inefficiencies.

Controlled execution environments are equally central. TRACE profiles C++, Java, and Python under fixed toolchains, perf stat, and RSS monitoring, with 5 runs per stress test and hard caps of 180 seconds and 4096 MB (Gong et al., 17 Mar 2026). EffiBench-X runs language-specific Docker images with CPU pinning, a 1024 MiB memory limit, and a custom profiler sampling every 0.1 ms (Qing et al., 19 May 2025). The vLLM energy benchmark fixes hardware to a single PC with 2×RTX 3090 and samples power every 15 seconds through CodeCarbon (Pronk et al., 10 Sep 2025). These design choices indicate that efficiency benchmarking is inseparable from execution protocol.

3. Program-generation and code-translation benchmarks

Program-level efficiency benchmarks have become the most mature subfield. ENAMEL builds a high-standard benchmark from 142 non-trivial HumanEval/HumanEval+ problems, expert-designed reference implementations, and strong human-curated test generators; canonical HumanEval and HumanEval+ solutions achieve only $0.455$ and $0.513$ eff@1, while expert references are normalized to $\alpha^3_{\text{per-1k}}$0 (Qiu et al., 2024). EffiBench-X extends this logic across six languages—Python, C++, Java, JavaScript, Ruby, and Go—using 623 competitive-programming tasks and three relative metrics: execution time, memory peak, and memory integral (Qing et al., 19 May 2025). Its headline result is that even the most efficient LLM-generated solutions achieve only around $\alpha^3_{\text{per-1k}}$1 of human efficiency on average, with systematically stronger efficiency in Python, Ruby, and JavaScript than in Java, C++, and Go (Qing et al., 19 May 2025).

For code translation, TRACE makes execution efficiency a first-class target. It derives 1,000 efficiency-critical translation tasks across six directions among C++, Java, and Python, each with 10 default tests, 10 stress tests, and an average of 23 correct translations per task in the reference set (Gong et al., 17 Mar 2026). Its empirical findings are sharply at odds with correctness-only evaluation: Claude-4-think leads in pass rate at 95.5% yet achieves only mid-tier time efficiency, while Qwen2.5-Coder-14B-Instruct attains the best time efficiency among open-source models (Gong et al., 17 Mar 2026). Moreover, 23.5% of functionally correct translations are at least $\alpha^3_{\text{per-1k}}$2 worse in execution time or peak memory than the best translation for the same task, with the dominant failure class being language construct mismatch (Gong et al., 17 Mar 2026). TRACY reaches a closely related conclusion in a similarly structured benchmark, reporting that algorithmic flaws and improper resource handling cause a median time slowdown of $\alpha^3_{\text{per-1k}}$3 and a median memory increase of $\alpha^3_{\text{per-1k}}$4 (Gong et al., 15 Aug 2025).

Hardware-oriented code generation extends the same benchmark logic beyond software runtime. Pluto evaluates LLM-generated Verilog against expert Pareto-optimal RTL in area, delay, and power across 114 problems with self-checking, latency-agnostic testbenches (Abdelatty et al., 16 Oct 2025). It reports 78.3% pass@1 for functional correctness, but only 63.8% area efficiency, 65.9% delay efficiency, and 64.0% power efficiency at eff@1, showing that syntactically and functionally correct RTL still lags expert optimization by roughly one third (Abdelatty et al., 16 Oct 2025). This suggests that efficiency benchmarks consistently expose deficits that correctness benchmarks compress into a single pass/fail label.

4. Serving, compression, and deployment efficiency

At the model-serving layer, the benchmark explicitly titled “LLM Efficiency Benchmark” measures energy per request rather than energy per token, arguing that tokenizer differences and output verbosity complicate cross-model comparison (Pronk et al., 10 Sep 2025). Implemented on vLLM, it varies concurrency from 5 to 5000 requests and finds that energy per request falls sharply with concurrency, plateauing around 100 concurrent requests on a 2×3090 system, with larger models stabilizing even earlier (Pronk et al., 10 Sep 2025). Within the Pythia family, energy per request scales close to linearly with parameter count, except for a 410M vs 1B anomaly attributable to layer-depth differences, and similar-sized architectures around 3B parameters show no significant energy differences under vLLM (Pronk et al., 10 Sep 2025).

Compression benchmarks treat deployment efficiency as a multi-objective problem. LLMCBench defines six tracks—compression performance, generalization ability, training consumption, inference consumption, hardware acceleration, and trustworthiness—each summarized by a quadratic-mean overall metric (Yang et al., 2024). Across those tracks, quantization consistently dominates sparsity in overall performance preservation and inference consumption, while sparsity can remain advantageous in training-time cost and some reasoning-oriented regimes (Yang et al., 2024). Its key contribution is not a single winner but a standardized protocol for asking which compression method is preferable under a specific deployment constraint.

A broader conceptual expansion is provided by “Democratizing LLM Efficiency,” which proposes Overhead-Aware Efficiency (OAE) as a benchmarking agenda rather than a fixed leaderboard (Huang, 3 Nov 2025). OAE argues that efficiency should include adoption cost, robustness under constraint, and talent dependence alongside throughput, memory, and energy. This suggests that many existing serving benchmarks remain hyperscale-centric: they measure tokens/sec or GPU utilization but omit the engineering complexity and maintenance burden that can dominate small-scale deployments.

5. Agentic, interactive, and hardware-aware efficiency benchmarks

Agent benchmarks reframe efficiency as utility per unit time, token, or communication. $\alpha^3_{\text{per-1k}}$5-Bench evaluates UAV agents under dynamic 6G conditions with a six-pillar composite score spanning Task Outcome, Safety Policy, Tool Consistency, Interaction Quality, Network Robustness, and Communication Cost (Ferrag et al., 1 Jan 2026). It then normalizes the composite score by generation time and total tokens to obtain $\alpha^3_{\text{per-1k}}$6 and $\alpha^3_{\text{per-1k}}$7 (Ferrag et al., 1 Jan 2026). The benchmark shows that raw task quality saturates while efficiency varies sharply: ChatGPT-4o-latest has the highest $\alpha^3_{\text{per-1k}}$8, but GPT-5.1-Chat attains the highest token efficiency, and several reasoning-heavy models have extremely low per-second efficiency because of very long generation times (Ferrag et al., 1 Jan 2026).

LoCoBench-Agent transfers the problem to long-context software engineering. It evaluates 8,000 interactive scenarios over repositories ranging from 10K to 1M tokens, with nine metrics split into five comprehension and four efficiency dimensions (Qiu et al., 17 Nov 2025). Its aggregate efficiency score,

$\alpha^3_{\text{per-1k}}$9

combines runtime efficiency, memory efficiency, information coverage, and long-range dependency resolution (Qiu et al., 17 Nov 2025). The benchmark reports a negative correlation $\alpha^3$0 between comprehension and efficiency, and a stronger negative correlation of about $\alpha^3$1 between turns and efficiency beyond roughly 12 turns, showing that exhaustive exploration improves understanding but degrades interaction efficiency (Qiu et al., 17 Nov 2025).

EvoCodeBench introduces a human-performance frame for self-evolving coding systems. Built on 100 LeetCode problems across Python3, C++, Java, Go, and Kotlin, it tracks pass rate, time and memory failures, average runtime and memory on accepted solutions, and human-relative metrics such as Average Runtime Beats and Average Memory Beats (Zhang et al., 10 Feb 2026). Its self-evolving agent, allowed up to three reflection–revision rounds, shows simultaneous gains in pass rate and runtime efficiency; for gemini-3-flash-preview, C++ average runtime improves from 266.04 ms to 142.60 ms while pass rate rises from 84 to 99 (Zhang et al., 10 Feb 2026).

KernelBench-X carries efficiency benchmarking into LLM-generated Triton kernels. Across 176 tasks in 15 categories and six GPUs, it separates compile success, semantic correctness, and hardware efficiency through speedup against PyTorch eager plus memory and compute utilization normalized by device peaks (Wang et al., 6 May 2026). Its most striking result is that 46.6% of semantically correct kernels are slower than the PyTorch eager baseline, while task category explains nearly three times more deviance in semantic correctness than method identity, and quantization remains completely unsolved at 0/30 successes (Wang et al., 6 May 2026). Here, correctness is necessary but plainly insufficient for efficiency.

6. Methodological disputes, benchmark efficiency, and future directions

The strongest recent critique comes from “Rethinking Code Performance Benchmarks for LLMs,” which re-evaluates 1,538 tasks from EffiBench, ENAMEL, EvalPerf, and Mercury under 30-run measurement and Mann–Whitney U testing (Le et al., 8 Jul 2026). With the original benchmark-provided tests, only 6.11% of benchmark “performant” implementations are significantly faster than canonical solutions, and manual analysis of 308 non-significant tasks finds 209 with plausible performance improvements that the original tests fail to expose (Le et al., 8 Jul 2026). A three-agent framework for generating, diagnosing, and repairing deterministic performance-oriented tests then raises the fraction of newly exposed significant improvements to 24.01% with DeepSeek-v3.1 and 25.43% with GPT-4o on tasks that previously showed no difference (Le et al., 8 Jul 2026). The implication is direct: benchmark insufficiency, not only model insufficiency, currently limits what “LLM efficiency” results mean.

A different strand asks whether the benchmarks themselves can be made cheaper without distorting rankings. The MIS-based prompt-selection framework compresses GPQA, IFEval, MMLU-Pro, and Omni-MATH by constructing similarity graphs and selecting maximum independent sets (Kjorvezir et al., 31 May 2026). Across 2,563 configurations, Kendall’s $\alpha^3$2 in 99.2% of stochastic settings, while higher-percentile thresholds reduce prompt count by 25–48% on average and ranking divergence from the full benchmark occurs in only 15.95% of configurations (Kjorvezir et al., 31 May 2026). This is benchmark efficiency in a different sense: reducing evaluation cost while preserving ordering.

Across benchmark families, the stated limitations are recurrent. TRACE measures only execution time and peak memory and only for C++, Java, and Python (Gong et al., 17 Mar 2026). The vLLM energy benchmark uses a single 2×RTX 3090 system and HellaSwag only (Pronk et al., 10 Sep 2025). $\alpha^3$3-Bench uses time, tokens, and tool counts rather than direct FLOP or energy measurement (Ferrag et al., 1 Jan 2026). LoCoBench-Agent uses interaction-level proxies instead of wall-clock latency or API cost (Qiu et al., 17 Nov 2025). KernelBench-X covers Triton kernels rather than full GPU software stacks (Wang et al., 6 May 2026). OAE, in turn, argues that future benchmarks should incorporate adoption cost, robustness under constraint, and talent dependence alongside conventional runtime metrics (Huang, 3 Nov 2025).

Taken together, the literature suggests three durable directions. First, efficiency evaluation is moving from raw runtime toward multi-dimensional measurement that includes memory, energy, hardware utilization, communication, and deployment overhead. Second, correctness-gated, stress-tested, statistically rigorous protocols are becoming necessary, because weak tests systematically understate performance differences. Third, efficiency is increasingly being normalized against a task-specific or human-specific frontier rather than reported as raw latency alone. In that sense, the modern LLM efficiency benchmark is less a single benchmark format than a family of evaluation regimes built around the same principle: a system should be judged not only by what it can do, but by the computational terms on which it does it.