Fair comparison and principled assessment of efficiency–effectiveness trade-offs

Establish fair comparison and principled assessment frameworks for the efficiency–effectiveness trade-off in efficient reasoning methods for large language models, ensuring reliable evaluation across paradigms, backbone scales, and reasoning domains.

Background

The paper critiques existing metrics such as Accuracy per Computation Unit (ACU) and Accuracy–Efficiency Score (AES) for misrepresenting improvements and relying on heuristic thresholds, leading to instability and unfair comparisons.

Motivated by these limitations, the authors call out the lack of a principled, smooth, and stable evaluation framework for the efficiency–effectiveness balance as an open challenge, which they aim to address with the proposed E3-Score.

References

As a result, both fair comparison and principled assessment of efficiency–effectiveness balance for efficient reasoning remain open challenges.

— EffiReason-Bench: A Unified Benchmark for Evaluating and Advancing Efficient Reasoning in Large Language Models  (2511.10201 - Huang et al., 13 Nov 2025) in Related Work, Benchmarks for Reasoning Efficiency

Evaluations per run reflect each agent's own search strategy rather than a budget imposed by the harness, so efficiency differences below are observational; equalizing that budget across conditions is left to future work.

— Do AI Agents Understand Computer Architecture?  (2609.19387 - Sharan et al., 16 Sep 2026) in Section 3, Results, immediately before Table 1