---
title: 'EffiReason-Bench: Efficient Reasoning in LLMs'
url: https://www.emergentmind.com/topics/effireason-bench
type: topic
---

# EffiReason-Bench: Efficient Reasoning in LLMs

EffiReason-Bench is a unified benchmark for evaluating efficient reasoning in large language models, with the explicit goal of comparing methods that reduce the cost of Chain-of-Thought (CoT) reasoning without sacrificing accuracy [2511.10201]. It is motivated by the observation that CoT often yields unnecessarily long and redundant reasoning traces, increasing inference cost and, in some cases, degrading correctness. The benchmark addresses a methodological gap in prior work: efficiency-oriented approaches were typically assessed under fragmented conditions, with different models, tasks, and metrics, making cross-paradigm comparison unreliable. EffiReason-Bench standardizes the comparison space across three method categories—Reasoning Blueprints, Dynamic Execution, and Post-hoc Refinement—while also introducing verified CoT annotations for CommonsenseQA and LogiQA and a trade-off metric, E$^3$-Score, for joint evaluation of effectiveness and efficiency [2511.10201].

## 1. Motivation and conceptual scope

EffiReason-Bench is framed around a specific failure mode of contemporary CoT prompting: strong reasoning performance is often accompanied by excessive token generation rather than calibrated inference. In the benchmark’s formulation, the problem is not simply answer quality, but the efficiency–accuracy trade-off under standardized evaluation. This positions efficient reasoning as a first-class benchmarking target rather than a secondary implementation concern [2511.10201].

The benchmark evaluates methods that intervene at different points in the reasoning pipeline. This organizational choice reflects the paper’s claim that efficient reasoning is not a single technique but a family of approaches distinguished by *when* they compress, constrain, or restructure reasoning. EffiReason-Bench therefore serves not merely as a leaderboard instrument, but as a common experimental substrate for comparing train-free and train-based methods under aligned prompting, formatting, backbones, and datasets [2511.10201].

A central conclusion of the benchmark is that efficient reasoning is not governed by a universal best method. Instead, the best-performing strategy depends on backbone scale, task difficulty, and architecture. This suggests that efficiency interventions interact with model internals and task structure in a highly non-uniform way, rather than expressing a stable Pareto improvement across conditions [2511.10201].

## 2. Method taxonomy and benchmarked paradigms

EffiReason-Bench organizes efficient reasoning methods into three paradigms, corresponding to the stage at which the intervention occurs [2511.10201].

| Paradigm | Intervention point | Methods listed |
|---|---|---|
| Reasoning Blueprints | Before or at the start of reasoning | Chain-of-Draft (CoD), Sketch-of-Thought (SoT), O1-Pruner |
| Dynamic Execution | During inference | Soft Thinking, Skeleton-of-Thought (SloT) |
| Post-hoc Refinement | After rationale generation | TokenSkip |

**Reasoning Blueprints** are methods that shape reasoning from the outset. **O1-Pruner** is described as a train-based approach that fine-tunes with a length-aware reward so the model learns to prune or shorten reasoning while preserving correctness. **Sketch-of-Thought (SoT)** uses a lightweight router to map inputs to high-level reasoning sketches, encouraging compressed templates. **Chain-of-Draft (CoD)** uses prompt instructions that force very short intermediate drafts. The benchmark notes that CoD and SoT are especially aggressive at compression and often trade away accuracy, whereas O1-Pruner is more conservative and tends to preserve performance better, though at the cost of additional training [2511.10201].

**Dynamic Execution** methods alter the reasoning process during decoding. **Soft Thinking** reasons in a continuous concept space through “concept tokens,” described as soft, probability-weighted mixtures of token embeddings. **Skeleton-of-Thought (SloT)** first generates a high-level skeleton and then expands each part, which introduces a more structured, potentially parallelizable inference process. Within the reported results, Soft Thinking is characterized as the most robust dynamic method, while SloT often achieves substantial compression but is more unstable on difficult tasks [2511.10201].

**Post-hoc Refinement** is represented by **TokenSkip**, which removes tokens with low semantic contribution from a completed rationale. Unlike the blueprint and dynamic categories, TokenSkip does not modify reasoning generation upstream. Its reported behavior is notably architecture-sensitive: it is highly effective on commonsense and logic tasks and often strong on LLaMA backbones, yet it can be harmful on some Qwen mathematical settings, particularly on MATH500 [2511.10201].

The reference condition is **Standard CoT**, yielding a seven-method comparison suite in total: CoD, SoT, O1-Pruner, Soft Thinking, SloT, TokenSkip, and Standard CoT [2511.10201].

## 3. Benchmark construction and verified CoT annotations

EffiReason-Bench evaluates four datasets: **GSM8K**, **MATH500**, **CommonsenseQA**, and **LogiQA** [2511.10201]. These cover mathematics, commonsense reasoning, and logical reasoning. A major construction challenge arises from the fact that CommonsenseQA and LogiQA do not natively provide reasoning traces. To support step-by-step evaluation, the benchmark introduces verified CoT annotations for these two datasets via a three-stage pipeline [2511.10201].

The first stage imposes a **standardized reasoning structure** with four components:

1. **Premise identification**  
2. **Relationship formalization**  
3. **Condition evaluation**  
4. **Option mapping**

This standardization is intended to regularize annotations across questions and annotators, preventing free-form explanations from obscuring logical structure [2511.10201].

The second stage performs **option-wise comparative analysis**. For each multiple-choice item, the solution must analyze all answer options, not just justify the selected one. The annotation must explain why the correct option is correct, why distractors are wrong, and how the reasoning aligns with the ground-truth answer. This design explicitly forces comparative reasoning instead of post hoc rationalization of the selected option [2511.10201].

The third stage is **dual human verification**, used to remove logical inconsistencies, unsupported inferences, and mismatches between the reasoning trace and the gold answer. The benchmark emphasizes this because automatically generated CoT traces may contain contradictions or logical gaps even when they sound plausible [2511.10201].

The experiments also standardize output formatting. Models must produce a single-line final answer beginning with `Final Answer:`. For multiple-choice tasks, the answer is a single option letter (`A/B/C/D`), while for open-ended mathematics it is a number or short span. In few-shot settings, exemplars are drafted by GPT-4o and human-audited; a fixed global seed is used; the same exemplar lists are reused across methods, backbones, and regimes; and each exemplar follows the pattern **Question → Reasoning (numbered steps) → Final Answer** [2511.10201].

The appendix further specifies **noise-injection variants** for robustness testing. These include **reasoning-step noise**—deletion, permutation, and repetition of steps—and **semantic noise**. For mathematics, semantic noise consists of small numeric perturbations; for commonsense and logic, it includes logical connective reversals such as “therefore $\rightarrow$ however,” “thus $\rightarrow$ but,” “because $\rightarrow$ despite,” “implies $\rightarrow$ contradicts,” and “leads to $\rightarrow$ prevents” [2511.10201].

## 4. Experimental protocol and comparison design

The benchmark evaluates six open-source instruction-tuned backbones:

- **Qwen2.5-Instruct 7B**
- **Qwen2.5-Instruct 14B**
- **Qwen2.5-Instruct 32B**
- **LLaMA-Instruct 1B**
- **LLaMA-Instruct 8B**
- **LLaMA-Instruct 70B**

This spans a scale range from **1B to 70B**, enabling analysis of how efficient reasoning interacts with model size and architecture [2511.10201].

The evaluation includes both **train-free** and **train-based** regimes. Under this protocol, each efficient reasoning method is compared against Standard CoT across all six backbones and all four datasets, using consistent formatting and prompting rules. This standardized matrix is a core contribution of EffiReason-Bench, since the paper explicitly identifies fragmented prior evaluation as a barrier to fair comparison [2511.10201].

The empirical results support a non-uniform view of efficient reasoning. On **GSM8K**, many methods reduce token counts, but some incur accuracy loss. On **MATH500**, trade-offs become sharper, indicating that more demanding mathematical reasoning is less tolerant of aggressive compression. **Soft Thinking** is reported as the most robust dynamic method and usually matches or exceeds CoT accuracy. **TokenSkip** performs well on **LLaMA** backbones but can catastrophically harm **Qwen**, especially on MATH500. **O1-Pruner** tends to preserve accuracy better than heuristic compression methods, but it is resource-intensive and caused OOM on larger models in the reported setup [2511.10201].

For **CommonsenseQA**, the benchmark reports substantially greater compression tolerance. **CoD** and **SoT** can reduce tokens by over 90%. **TokenSkip** is especially strong in this regime, often preserving or improving accuracy while shortening outputs, and **Soft Thinking** remains stable and often matches Standard CoT. **O1-Pruner** is reported to help weaker backbones, especially **LLaMA 8B** [2511.10201].

For **LogiQA**, compression is again more delicate. **CoD** tends to hurt accuracy despite extreme compression. **Soft Thinking** remains strong on accuracy but yields limited token savings, while **TokenSkip** is reported as the best-balanced method overall, improving or preserving accuracy with moderate token reduction [2511.10201].

The few-shot analysis adds further structure to these findings. **SloT** scales well with more demonstrations in some tasks and transfer settings. By contrast, **CoD** and **SoT** are described as rigid: they often learn only “be short” and do not benefit substantially from additional shots. Under noisy exemplars, **SloT** is more robust than CoD and SoT. In cross-domain transfer, **CoT** is the most stable baseline, while blueprint methods transfer poorly [2511.10201].

## 5. E$^3$-Score and efficiency–effectiveness evaluation

EffiReason-Bench introduces **E$^3$-Score**—the **Efficiency–Effectiveness Equilibrium Score**—to evaluate the trade-off between accuracy and token efficiency [2511.10201]. The metric is inspired by the **Constant Elasticity of Substitution (CES)** formulation from economics and is designed to avoid two problems identified in prior efficiency metrics: over-rewarding token savings and introducing discontinuities or heuristic thresholds.

Let \(A_0, T_0\) denote baseline accuracy and token count, and \(A, T\) denote the method’s accuracy and token count. The paper defines the normalized ratios as

\[
r_{\text{acc}} = \frac{A/(1-A)}{A_0/(1-A_0)}, \qquad
r_{\text{tok}} = \frac{T_0}{T}.
\]

Here, \(r_{\text{acc}}\) measures relative accuracy improvement under odds normalization, and \(r_{\text{tok}}\) measures relative token efficiency improvement [2511.10201].

The E$^3$-Score is then defined as

\[
\mathrm{E^3}(A,T) =
\left(
w \cdot r_{\text{acc}}^{\rho} + (1-w)\cdot r_{\text{tok}}^{\rho}
\right)^{\tfrac{1}{\rho}}.
\]

The parameter \(w\) controls the weight assigned to accuracy versus efficiency, and the paper sets \(w = A_0\), so stronger baselines place greater emphasis on accuracy. The parameter \(\rho\) controls substitutability; with \(\rho < 0\), the two components behave more like complements, meaning token savings cannot fully compensate for accuracy degradation. The benchmark uses **\(\rho = -1\)** by default, which makes the score a weighted harmonic mean and yields a deliberately conservative aggregation [2511.10201].

The paper contrasts E$^3$-Score with **ACU** and **AES**. **ACU (Accuracy per Computation Unit)** is said to undervalue small but meaningful gains when baselines are already strong, while **AES (Accuracy–Efficiency Score)** is characterized as piecewise and dependent on manual thresholds and heuristic hyperparameters. By construction, E$^3$-Score is intended to be smooth, stable, principled, and less heuristic-dependent [2511.10201].

A sensitivity theorem is also provided. With \(a=r_{\text{acc}}>0\), \(b=r_{\text{tok}}>0\), and \(\Delta = |\log(a/b)|\), the paper gives the bound

\[
\left|\log E_3(A,T;\rho_1)-\log E_3(A,T;\rho_2)\right|
\le \frac{\Delta^2}{8}\,|\rho_1-\rho_2|,
\]

equivalently,

\[
\left|\partial_\rho \log E_3(A,T;\rho)\right|\le \frac{\Delta^2}{8},
\]

and

\[
e^{-\frac{\Delta^2}{8}\,|\rho_1-\rho_2|}
\le
\frac{E_3(A,T;\rho_1)}{E_3(A,T;\rho_2)}
\le
e^{\frac{\Delta^2}{8}\,|\rho_1-\rho_2|}.
\]

These expressions are used to justify that E$^3$ is **Lipschitz-stable** in \(\rho\), with a bound independent of \(w\) [2511.10201]. This suggests that the metric is less sensitive to parameter perturbations than piecewise heuristic alternatives.

## 6. Empirical patterns, limitations, and significance

The principal empirical conclusion is that **no single method dominates everywhere** [2511.10201]. The benchmark repeatedly shows that the effect of an efficiency intervention depends on task complexity, model scale, and architecture. **CommonsenseQA** is reported as compression-tolerant; **LogiQA** is more fragile; and **MATH500** is the hardest setting, exposing the weaknesses of aggressive compression most clearly [2511.10201].

Several method-specific patterns recur across the reported experiments. **Soft Thinking** is the most robust dynamic method, especially when preserving accuracy is the primary concern. **TokenSkip** is often the strongest post-hoc method on commonsense and logic tasks, but its performance is architecture-sensitive and can be harmful on some Qwen mathematics settings. **O1-Pruner** is accurate and conservative but training-heavy and resource-intensive. **CoD** and **SoT** achieve extreme compression, yet they are also the methods most likely to sacrifice accuracy on harder reasoning problems [2511.10201].

Model architecture matters independently of scale. A major insight of the benchmark is that the same method can behave very differently on **Qwen** and **LLaMA** backbones. The paper identifies TokenSkip as the clearest example: excellent on LLaMA, but harmful on Qwen in some mathematical settings [2511.10201]. This suggests that efficient reasoning interventions are not plug-compatible abstractions but architecture-contingent procedures.

The benchmark also yields practical guidance for future evaluation. It suggests that efficient reasoning methods should be assessed across multiple backbones rather than a single host model; that both accuracy and token cost should be reported; that architecture-specific behavior should be treated as central rather than incidental; that robustness to noisy and cross-domain exemplars should be included; and that trade-off assessment should use a stable metric such as E$^3$ rather than ad hoc heuristics [2511.10201].

A plausible implication is that EffiReason-Bench shifts efficient reasoning from an informal optimization objective into a reproducible benchmarking problem. Its contribution is not only the evaluation of seven methods across six backbones and four datasets, but the establishment of a standardized cross-paradigm framework in which efficiency is measured jointly with correctness, under aligned reasoning traces and a stable trade-off metric [2511.10201].

Source: https://www.emergentmind.com/topics/effireason-bench