HRBench: Benchmarking and Understanding Thinking-Mode Switch Strategies in Hybrid-Reasoning LLMs

Abstract: Hybrid-reasoning LLMs expose explicit controls over reasoning effort, allowing users or systems to trade off answer quality against inference cost. However, existing methods for adaptive thinking-mode selection are typically evaluated under different models, datasets, and implementation assumptions, making it difficult to compare their practical behavior. We introduce HRBench, a unified evaluation framework for studying thinking-mode switching in hybrid-reasoning LLMs. HRBench organizes the design space along two axes: three switching strategy families, prompt-based selection, external routing, and speculative execution, and four training regimes, training-free, SFT, offline and online RL, yielding 12 controlled evaluation settings. We evaluate these settings across 6 LLMs, from Qwen3.5-2B to Kimi-K2.5-1.1T, and 5 reasoning benchmarks covering mathematics, science, and code, while reimplementing 12+ representative prior methods within the same pipeline. Our analysis characterizes how different switching strategies occupy distinct effectiveness-efficiency trade-off regions: prompt-based methods often provide favorable token-accuracy trade-offs, routing methods offer more stable cost reduction, and speculative methods tend to improve accuracy at higher token cost. We further find that training affects strategies differently, and that the preferred strategy varies with model scale and task domain. HRBench provides reference implementations and a unified evaluation platform to support more controlled research on efficient reasoning in hybrid-reasoning LLMs. Our data, code and repository are available at https://github.com/usail-hkust/HRBench.

• The paper introduces HRBench, a unified framework that systematically benchmarks thinking-mode switch strategies in hybrid-reasoning LLMs.
• It evaluates methods like prompt-tuning, routing, and speculative approaches across diverse tasks in math, science, and code using 12 controlled configurations.
• Empirical findings reveal that strategy performance is scale- and domain-dependent, with prompt-tuning offering optimal trade-offs and speculative methods excelling in code tasks.

HRBench: Systematic Benchmarking of Thinking-Mode Switch Strategies in Hybrid-Reasoning LLMs

Motivation and Problem Statement

Hybrid-reasoning LLMs represent a paradigm wherein explicit controls for reasoning depth are exposed, enabling selective allocation of computational resources during inference. As the complexity and token cost of extended CoT reasoning increases in modern LLMs, practical deployment demands dynamic strategies to switch between shallow (direct answer) and deep reasoning modes. HRBench addresses the substantial gap in the field: prior adaptive thinking-mode switch methods have been tested under disparate models, datasets, and configurations, preventing rigorous comparative analysis. The paper introduces a unified evaluation framework to systematically study three major families of thinking-mode switch strategies across a broad experimental taxonomy.

Figure 1: HRBench overview showing three switch strategies, four training regimes, and wide evaluation coverage across models, datasets, and experiment runs.

Taxonomy of Thinking-Mode Switch Strategies

HRBench categorizes adaptive mode switching along two axes:

• Strategy Family: Prompt-Tuning, Routing, Speculative
• Training Regime: Training-free, SFT, Offline RL (GRPO/DPO), Online RL

Prompt-Tuning (PT) leverages model-internal decisions stimulated by carefully engineered prompts, guiding reasoning depth per input instance. Routing (RT) deploys an explicit classifier to assess problem difficulty and select a reasoning mode for subsequent generation. Speculative (Spec) executes initial fast inference, then triggers escalation to deep reasoning upon detection of uncertainty signals, either via heuristics or learned policies. HRBench orthogonally combines these axes, yielding 12 controlled evaluation configurations.

Experimental Setup

HRBench includes 6 contemporary hybrid-reasoning LLMs spanning 2B–1.1T parameters, and benchmarks spanning mathematics (MATH500, AIME 2025), science (GPQA-Diamond), and code (LiveCodeBench, Codeforces). Each configuration is evaluated on accuracy (Pass@1) and output token cost, providing a direct measure of the effectiveness–efficiency trade-off.

Comparative Analysis of Switch Strategies

The unified empirical study uncovers fundamentally differentiated trade-off profiles.

Figure 2: Efficiency–effectiveness trade-off for Qwen3.5-9B; methods map to distinct regions along the Pareto frontier.

• Prompt-Tuning: Achieves Pareto-optimal trade-off, simultaneously increasing accuracy and reducing token usage versus naive baselines. Models autonomously allocate reasoning effort, eliminating unnecessary CoT for simple instances.
• Routing: Yields moderate token reduction with preserved accuracy, leveraging conservative dispatching based on difficulty estimation.
• Speculative: Improves accuracy, especially in domains such as code, at the cost of additional token overhead due to retriggered reasoning.

The scale effect is pronounced: strategy rankings shift with model size, and token savings by PT and RT are present only in larger models.

Figure 3: Effectiveness and efficiency trends across LLM scales, highlighting the non-universal dominance of any strategy.

Task domain analysis reveals further modulation. PT provides best trade-offs for math and science, while Speculative methods excel in code benchmarks, exploiting the “try-then-verify” dynamic.

Effect of Training Regimes

Three principal training paradigms are studied: SFT, DPO, and GRPO. Training universally improves switching capacity, with substantive gains in efficiency (up to 65% token reduction for RT-GRPO) but marginal improvement in accuracy (typically within 1–2 percentage points).

Figure 4: Training regime effect on accuracy and token reduction for Qwen3.5-9B, averaged across benchmarks.

• GRPO (RL) is most effective for RT, sharply sharpening the router’s decision boundary.
• DPO maximizes accuracy improvements.
• SFT offers balanced performance.

Training effects are strategy-dependent: RT benefits the most in efficiency, while PT and Spec show more modest gains.

Unified Comparison of External Methods

HRBench integrates over 12 representative prior methods, enabling fair comparison through identical pipeline and parameterization. External PT methods collectively span the widest Pareto frontier; RT methods consistently offer moderate improvements; Speculative methods primarily enhance accuracy at the expense of efficiency except ADR, which achieves token savings via learned switching.

Figure 5: Unified benchmark comparison of 12 switch methods by accuracy and token reduction; methods cluster by strategy.

No single external method outperforms all others across every domain. Selection must be matched to task and deployment constraints.

Implications and Future Directions

HRBench’s systematic findings underscore that strategy selection must be context-dependent, jointly considering LLM scale and task domain. The distinct trade-off profiles suggest that hybrid deployment is beneficial: an optimal switch mechanism adapts not only per input but also per model. Training-based improvement is most impactful for routing, and domain-specialized strategies (e.g., speculative for code) warrant further investigation.

The unified platform provides strong baselines and reference implementations, facilitating reproducibility and extendability. Future theoretical developments may explore meta-routing policies, domain-general switch calibration, and compound switch mechanisms for agentic multi-turn scenarios. Practically, the efficiency gains enabled can directly reduce inference cost and carbon footprint, critical for scaling LLM deployment.

Conclusion

HRBench sets a new standard for controlled benchmarking and analysis of thinking-mode switches in hybrid-reasoning LLMs. By thoroughly characterizing the effectiveness–efficiency landscape and identifying key dependencies on strategy, training, model scale, and task domain, the work provides valuable guidance for both researchers and practitioners. The open-source platform anchors future progress in efficient reasoning for LLMs, and motivates exploration in adaptive inference policies, domain transferability, and agentic composition.