GlimpRouter: Efficient Inference Routing

• GlimpRouter is a training-free framework that uses the initial-token entropy as a proxy for reasoning step difficulty, enabling dynamic model selection.
• The system assigns lightweight and heavyweight models based on a calibrated threshold to balance computational efficiency and task accuracy.
• Empirical evaluations show up to 30% latency reduction and improved Pass@1 accuracy across diverse benchmarks including mathematical and code generation tasks.

GlimpRouter is a training-free, step-wise collaborative inference framework designed to efficiently allocate computational resources between small and large reasoning models (SLMs and LRMs) during multi-step chain-of-thought reasoning. Its central innovation lies in using the entropy of the first token generated by the lightweight model as a proxy for reasoning step difficulty, enabling rapid routing decisions that substantially lower inference latency while maintaining or improving task accuracy (Zeng et al., 8 Jan 2026).

1. Motivation and Theoretical Foundations

Large Reasoning Models (LRMs) excel at explicit chain-of-thought generation, but their multi-step reasoning capabilities incur substantial computational cost and inference delay. Collaborative inference—deploying complementary small (SLM) and large (LLM) models—addresses this by dividing labor. A principal challenge is dynamically determining which model to assign at each reasoning step.

Empirically, LRMs exhibit a pronounced spike in uncertainty at the onset of difficult steps, corresponding to what psychologists denote as an "Aha Moment": a discrete cognitive bifurcation at the first token, after which the step completion typically proceeds deterministically. Let $s_k=(t_{k,1},t_{k,2},\dots)$ denote the $k^\text{th}$ reasoning step, conditioned on context $\mathbf c_k$. The initial-token entropy is defined as: $$H_{\rm init}(s_k) = -\sum_{v\in V}P_\theta(t_{k,1}=v\mid \mathbf c_k)\, \log P_\theta(t_{k,1}=v\mid \mathbf c_k)$$ where $V$ is the vocabulary. Low $H_{\rm init}$ indicates routine steps with high model agreement; high $H_{\rm init}$ signifies difficult steps likely requiring LLM intervention. This entropy is calculated using the SLM’s output logits.

2. GlimpRouter Architecture and Step-wise Routing Algorithm

GlimpRouter operates in a probe-then-dispatch paradigm, leveraging two models:

• Small Model ($M_S$): E.g., Qwen3-4B (4B parameters), DeepSeek-1.5B.
• Large Model ($M_L$): E.g., Qwen3-32B, DeepSeek-32B (32B parameters).

At each reasoning step $k$ in a session of $K$ steps:

1. The SLM probes by generating only the first token and outputs its probability distribution $P_{\rm init}$.
2. The initial-token entropy $H_{\rm init}(s_k)$ is computed.
3. The entropy is compared to a threshold $\tau$:
   • If $H_{\rm init}(s_k) \leq \tau$, the full step is delegated to $M_S$.
   • Otherwise, $M_L$ is invoked for the complete step.
4. The generated step is appended to the context and the process repeats.

Pseudocode formalizing this process: $$\begin{aligned} &\texttt{def GlimpRouter}(q,M_S,M_L,\tau):\ &\quad \mathcal T\leftarrow[]\quad//\text{accumulated chain}\ &\quad\text{while not done:}\ &\qquad \mathbf c\leftarrow(q,\mathcal T)\ &\qquad P_{\rm init}\leftarrow M_S(\mathbf c)\ &\qquad H_{\rm init}\leftarrow -\sum_vP_{\rm init}(v)\log P_{\rm init}(v)\ &\qquad\text{if }H_{\rm init}>\tau:\ &\qquad\quad s\leftarrow M_L(\mathbf c)\ &\qquad\text{else:}\ &\qquad\quad s\leftarrow M_S(\mathbf c)\ &\qquad \mathcal T.\text{append}(s)\ &\quad \text{return }\mathcal T \end{aligned}$$

The threshold $\tau$ is calibrated via a sweep over a validation set to target an intervention rate $r \approx 20\textrm{–}30\%$, which determines the trade-off between computational savings and accuracy.

3. Computational Cost and Latency Dynamics

Let $K$ denote the total reasoning steps per session, $r$ the fraction of steps routed to the LLM, and $\alpha$ the speedup factor ($M_S$ is $\alpha$ times faster and requires $\alpha$ times fewer FLOPs than $M_L$). The expected FLOPs per session is: $$\mathbb E[\mathrm{FLOPs}] = (1-r)\,K\cdot C_S \;+\; r\,K\cdot C_L = K\,C_S\,[(1-r)+r\,\alpha]$$ with $C_S$ and $C_L$ as per-step costs for SLM and LLM respectively. End-to-end latency is: $T\approx(1-r)\,T_S\;+\;r\,T_L$ where $T_S$, $T_L$ are SLM and LLM step execution times. The probe (one-token SLM) and KV-cache switching overhead are negligible. Empirical results show that with $\alpha \approx 3$ and $r \approx 25\%$, wall-clock time is reduced by 25–30%.

4. Empirical Evaluation and Benchmarks

GlimpRouter’s performance was validated on several benchmarks:

• Mathematical reasoning: AIME24, AIME25
• General reasoning: GPQA-Diamond
• Code generation: LiveCodeBench v5/v6

Model pairings tested include Qwen3-4B ↔ Qwen3-32B and DeepSeek-1.5B ↔ DeepSeek-32B. Evaluation metrics comprise Pass@1 accuracy and latency per query. Comparative baselines include SLM-only, LLM-only, random routing, RSD (reward-guided), SpecCoT (multi-candidate), and SpecReason (post-hoc verification).

Key results on AIME25 (DeepSeek-32B as LLM, Qwen3-4B as SLM):

Routing Strategy	Accuracy (Pass@1 %)	Latency (s)
LLM-only	46.7	220
GlimpRouter	51.7 (+10.7%)	147 (−25.9%)

Analogous Pareto-optimal improvements (15–30% speedup with equal or greater accuracy) were replicated across all tested tasks and model pairs.

5. Ablation Studies

Several ablation experiments were conducted:

• Threshold sweep ($\tau$): Varying $\tau$ modulates $r$ from $\approx 90\%$ to $\approx 5\%$, tracing a Pareto frontier uniformly superior to SpecReason.
• Metric selection: Substituting $H_{\rm init}$ with mean step-wise entropy $H_{\rm step}=\frac1L\sum_iH(p(t_i))$ or step-wise perplexity results in 8–10% lower accuracy and 10–15% longer latency, implying that entropy dilution over the full step weakens the routing signal.
• Orthogonal optimizations: The incorporation of Speculative Decoding during $M_L$-routed steps further reduces latency by approximately 15% without any accuracy compromise, indicating the possibility for synergistic hierarchical acceleration.

6. Limitations and Prospective Enhancements

A static global threshold $\tau$ may not respond effectively to domain shifts; adaptive or context-dependent thresholds constitute a promising direction for future research. Step boundaries in GlimpRouter presently rely on double-newline delimiters, which are architecture-specific; advancing toward semantic segmentation could broaden applicability.

The probe-first routing framework of GlimpRouter, using initial-token entropy as the decision criterion, yields consistent latency reductions of $\sim$25% and frequently enhances final accuracy by judicious large-model interventions and implicit self-correction. Its simplicity and generality offer a robust foundation for efficient chain-of-thought inference in collaborative model settings.