Select to Think: Unlocking SLM Potential with Local Sufficiency

Abstract: Small LLMs (SLMs) offer computational efficiency for scalable deployment, yet they often fall short of the reasoning power exhibited by their larger counterparts (LLMs). To mitigate this gap, current approaches invoke an LLM to generate tokens at points of reasoning divergence, but these external calls introduce substantial latency and costs. Alternatively, standard distillation is often hindered by the capacity limitation, as SLMs struggle to accurately mimic the LLM's complex generative distribution. We address this dilemma by identifying local sufficiency: at divergence points, the LLM's preferred token consistently resides within the SLM's top-K next-token predictions, even when failing to emerge as the SLM top-1 choice. We therefore propose SELECT TO THINK (S2T), which reframes the LLM's role from open-ended generation to selection among the SLM's proposals, simplifying the supervision signal to discrete candidate rankings. Leveraging this, we introduce S2T-LOCAL, which distills the selection logic into the SLM, empowering it to perform autonomous re-ranking without inference-time LLM dependency. Empirically, we demonstrate that a 1.5B SLM's top-8 candidates capture the 32B LLM's choice with 95% hit rate. Translating this potential into performance, S2T-LOCAL improves greedy decoding by 24.1% on average across benchmarks, effectively matching the efficacy of 8-path self-consistency while operating with single-trajectory efficiency.

• The paper demonstrates that SLM candidate sets inherently include optimal LLM tokens, with top-8 proposals capturing over 95% of targets.
• It introduces the S2T framework that employs local re-ranking, replacing costly external LLM interventions with a distilled local selector.
• Experimental results reveal a 24.1% to 38% accuracy improvement, matching self-consistency performance at significantly lower compute overhead.

Select to Think: Unlocking SLM Potential with Local Sufficiency

Introduction and Motivation

The increasing deployment of Small LLMs (SLMs) is motivated by the need to reduce inference costs while maintaining acceptable performance, particularly for step-intensive reasoning tasks where LLMs excel. However, SLMs suffer significant accuracy degradation in multi-step reasoning due to both capacity limitations and failures in token ranking during decoding. Existing methods for bridging this gap either rely on distribution-level knowledge distillation (which is ineffective for large capacity gaps) or expensive collaborative inference that synchronously queries an external LLM at divergence points, incurring substantial latency and cost.

The "Select to Think: Unlocking SLM Potential with Local Sufficiency" paper (2604.26940) identifies a critical failure mode: SLMs generate viable tokens but frequently mis-rank them, suppressing superior options available in their own distribution. The authors propose a paradigm shift—selective local re-ranking based on LLM guidance—leading to the Select to Think (S2T) framework and its fully local S2T-Local variant.

Figure 1: The S2T paradigm replaces high-latency external calls and brittle distribution distillation with local candidate selection and re-ranking.

Local Sufficiency: Empirical Analysis

The central hypothesis is "local sufficiency": at steps where the SLM's output diverges from the LLM, the LLM's preferred token almost always exists within the SLM's top-$K$ proposals, even if it is not ranked highest by the SLM's greedy decoding. Empirical validation shows that with $K=8$, a 1.5B SLM captures the 32B LLM's choice with a hit rate exceeding 95%. This finding is robust across scales (including 0.5B SLMs), architectures (e.g., Gemma), and domains (math, general reasoning, code synthesis).

Figure 2: At intervention points, the LLM's preferred token is found within the SLM's top-8 candidates in over 95% of cases, vastly exceeding greedy decoding alignment.

Additional analysis on smaller models reinforces this observation.

Figure 3: Even with a 0.5B SLM, over 80% top-8 coverage of LLM targets is retained across diverse tasks, confirming the universality of local sufficiency.

Figure 4: The local sufficiency hypothesis holds across the Gemma model family, confirming model-agnosticity.

Rank-based CDF analysis shows that coverage surges between ranks 2 and 8, with diminishing returns beyond $K=8$, suggesting that the target knowledge is present but suppressed or closely mis-ranked.

Figure 5: Despite poor Hit@1 in complex reasoning, top-8 candidate sets consistently recover LLM-preferred tokens with >83% coverage across scales and domains.

Figure 6: The optimal tokens are rarely absent, but are often just below the greedy top-1—top-3 candidate sets capture over 80% of LLM choices.

The Select to Think (S2T) Framework

The S2T framework reframes LLM assistance: instead of generating open-ended tokens at divergence points, the LLM selects from the SLM's candidate proposals. This transforms the problem from distribution matching to discrete candidate ranking. The candidate set $\mathcal{C}_t$ is constructed by selecting the $K$ highest-probability tokens from the SLM at step $t$. The LLM then assigns preference scores to candidates, and the top-scoring token is selected.

For production scenarios, S2T-Local replaces the full LLM with a distilled local selector. This selector is implemented using reserved tokens whose logits are re-purposed for preference scoring, leveraging the Zero-overhead Inference-time Prediction (ZIP) framework.

Selector Distillation and Training

Training the S2T-Local selector is formulated as a ranking problem: the objective is to maximize the likelihood of selecting the teacher's top candidate from $\mathcal{C}_t$ (cross-entropy over candidate set), regularized by a KL penalty to preserve the SLM's base distribution on standard vocabulary.

Optimization requires a high-quality preference dataset, constructed by rolling out SLM trajectories, identifying high-divergence steps, and annotating candidates with teacher probabilities. The authors demonstrate that discrete Top-1 selection loss outperforms soft distribution-based losses, especially when teacher confidence is high. They also find that regularization and margin filtering further boosts agreement by suppressing noisy gradients when the LLM is itself uncertain.

Experimental Results

S2T-Local achieves substantial gains over all SLM-only baselines on mathematical reasoning, code, and general knowledge:

• 24.1% improvement in greedy decoding accuracy on 1.5B SLMs, with gains amplifying to 38% on 0.5B SLMs.
• Hit@8 covers 95% of teacher targets on 1.5B, and 83% on 0.5B models.
• The accuracy matches 8-path self-consistency (majority voting over repeated sampling) at $1/8$ the compute cost.
• Selector alignment, as measured by Top-1 agreement with the teacher, reaches 67–72% across benchmarks. Rank correlation (Spearman $\rho$) is 0.61–0.65.

Selector confidence and quantitative analysis demonstrate decisive winner-takes-all separation between optimal and sub-optimal candidates.

Figure 7: The predictor scores are sharply peaked on the teacher's preferred candidate, suppressing all others.

Efficiency is maintained: S2T-Local introduces only marginal compute overhead compared to standard SLM inference and is up to 75% faster than collaborative LLM interventions.

Theoretical Implications

The error decomposition provided highlights two sources of failure in sequential decoding: candidate hit failure ($\delta_\text{hit}$) and selection failure ($K=8$0). By expanding the candidate set, selection error becomes the dominant failure mode for SLMs. S2T targets this with explicit selection training, sidestepping the brittle distributional forces of vanilla distillation by focusing on local decision boundaries.

Unlike test-time scaling or reranking by outcome verifiers, S2T operates internally and incrementally, without needing multiple samples or external teacher calls at inference. This shift preserves efficiency and allows the SLM to act as an autonomous agent, not a passive imitator.

Practical and Theoretical Impact

S2T and S2T-Local demonstrate that the gap between SLMs and LLMs in reasoning is not primarily one of knowledge, but of local decision misranking. This recasts the optimal use of SLMs towards discriminative filtering over their own candidate space, and away from capacity-unmatched distribution matching. The findings suggest that future techniques should focus on selector architectures and ranking-specific distillation targets to further shrink the performance gap in resource-constrained environments.

By internalizing LLM selection logic, S2T provides a scalable mechanism for deploying strong, autonomous SLMs—making heavy LLM dependency unnecessary at inference.

Conclusion

"Select to Think" (2604.26940) provides compelling evidence that most SLM reasoning failures are selection rather than generation errors. By reframing the problem around local sufficiency and discriminative selection, the S2T framework converts previously untapped candidate diversity into substantial accuracy and efficiency gains. The work points toward selector-centric architectures for future SLM optimization and establishes selection-based distillation as an essential path for practical, resource-constrained LLM deployment.