---
title: 'VTSR: Adaptive Temperature Sampling for LLMs'
url: https://www.emergentmind.com/topics/variational-temperature-sampling-router-vtsr
type: topic
---

# VTSR: Adaptive Temperature Sampling for LLMs

A Variational Temperature Sampling Router (VTSR) is an adaptive decoding strategy for large language models (LLMs) that dynamically selects the sampling temperature at each generation step to optimize the accuracy–diversity trade-off. By modulating temperature explicitly as a learnable or learned function—either via hierarchical reinforcement learning from verifiable rewards or through tokenwise risk-based routing—VTSRs enable efficient exploration and exploitation during LLM generation, particularly in settings requiring precise reasoning or diverse outputs [2602.13035][2510.01218].

## 1. Foundations and Motivation

Temperature-based sampling is a standard mechanism for controlling the entropy of an autoregressive LLM’s next-token distribution, enabling a continuum from deterministic (greedy, low-temperature) to highly stochastic (high-temperature) generation. Conventional protocols employ a fixed or heuristically annealed temperature, which can be effective for generic text but is suboptimal when generation quality must be maximized at sensitive decision points, as in mathematical or logical reasoning. Empirical evidence demonstrates that uncontrolled high-temperature sampling often degrades output quality by producing erroneous choices at specific “high-risk” positions, motivating approaches where temperature policy itself is a first-class, adaptive component of the decoding process [2510.01218].

VTSR formalizes and generalizes this idea by treating temperature selection as a learnable routing decision at each step, optimizing it in a variational or reinforcement learning framework to maximize downstream, verifiable reward [2602.13035].

## 2. Hierarchical Reinforcement Learning Formulation

The introspective LLM approach models the decoding process as a hierarchical Markov decision process under reinforcement learning from verifiable rewards (RLVR). At each step $t$, two coupled policies operate:

- **High-level temperature policy**: $\pi_\phi(\tau_t \mid h_t, \tau_{t-1})$ chooses the sampling temperature $\tau_t > 0$ based on the decoder hidden state $h_t \in \mathbb{R}^d$ and the previous temperature $\tau_{t-1}$.
- **Low-level token policy**: $\pi_\theta(y_t \mid h_t, \tau_t)$ samples the next token $y_t$ from the softmax distribution with scaling parameter $\tau_t$:
  $$
  p_\theta(y_t \mid h_t, \tau_t) = \mathrm{softmax}\!\left(\ell_t/\tau_t\right)
  $$

The joint trajectory likelihood under this model is
$$
\pi(y_{1:T}, \tau_{1:T} \mid x) = \prod_{t=1}^T \pi_\phi(\tau_t \mid h_t, \tau_{t-1}) \, \pi_\theta(y_t \mid h_t, \tau_t)
$$
where $x$ is the prompt.

The objective is to maximize expected verifiable reward:
$$
J(\theta, \phi) = \mathbb{E}_{x \sim \mathcal{D}}\,\mathbb{E}_{\tau, y \sim \pi(\cdot \mid x)} \left[ R(x, y) \right]
$$
Coordinate ascent optimization (Group Relative Policy Optimization, GRPO) is used, alternating between fixing temperature trajectories and optimizing over token policies, and vice versa, with clipped surrogate gradients [2602.13035].

## 3. Policy Parameterization and Learning

The VTSR mechanism utilizes a mixed discrete–continuous policy for the temperature:

- A two-layer MLP head $f_\phi$ maps $h_t$ to parameters $(u_c, u_\alpha, u_\beta)$.
- A binary variable $c_t \sim \mathrm{Bernoulli}(\sigma(u_c))$ selects whether to update $\tau_t$ or keep it as $\tau_{t-1}$.
- If updated, $z_t \sim \mathrm{Beta}(\alpha=\mathrm{softplus}(u_\alpha),\,\beta=\mathrm{softplus}(u_\beta))$ samples the temperature in a bounded interval $[\tau_{\min}, \tau_{\max}]$.
- The joint log-probability for policy gradient updates is given by
$$
\log\pi_\phi(c_t, z_t \mid h_t) = \log\mathrm{Bern}(c_t; \sigma(u_c)) + c_t \log\mathrm{Beta}(z_t; \alpha, \beta)
$$

Gradient estimation proceeds on-policy via PPO-style clipped surrogates and group-relative advantage computation.

For selective sampling [2510.01218], the router is implemented via a lightweight risk-classifier $s_\theta$ that receives hidden states, outputting a risk score $r_t \in [0,1]$. If $r_t$ exceeds a threshold $\tau$, decoding is greedy; otherwise, high-temperature sampling with min-$p$ truncation is deployed.

## 4. Interpretability and Behavioral Analysis

The learned temperature schedule exhibits interpretable structure:

- **Difficulty-awareness**: On the MATH-500 benchmark, median $\tau_t$ increases monotonically from easier (L1) to harder (L5) problems, indicating more exploration where reasoning is more uncertain.
- **Reasoning rhythm**: Per-token $\tau_t$ traces exhibit peaks at logical pivots (e.g., “assume”, “consider”, “finding”) and dips during arithmetic or factual computation phases, aligning temperature with the model’s internal uncertainty and information requirements.
- **Emergent exploration cycles**: During training, non-monotonic “exploration–exploitation–diversity” cycles in $\mathbb{E}[\tau]$ emerge, in sharp contrast with fixed or annealed schedules [2602.13035].
- *This suggests that VTSR adapts not only to global task difficulty but also to local context shifts within the reasoning process.*

## 5. Empirical Evaluation and Comparisons

Extensive benchmarking demonstrates that VTSR mechanisms confer statistically significant improvements in both reasoning accuracy and sample diversity:

- **Benchmarks**: AIME24, AMC23, MATH-500, Minerva, OlympiadBench, Omni-Math, and out-of-domain (OOD) datasets such as GPQA, MMLU-Pro, and HumanEval.
- **Baselines**: Static-temperature GRPO baselines ($\tau \in \{0.6, 1.0, 1.2\}$), heuristic entropy annealing, sequence-level temperature policies (TAMPO), and risk-based routers (min-$p$, top-$p$, EDT).
- **Metrics**: Avg@8, Pass@8 (multi-sample accuracy), area under the quality–diversity curve, and perplexity.
- **Results**: VTSR (IntroLLM [2602.13035] or selective sampling [2510.01218]) yields Avg@8 and Pass@8 gains of approximately 2–5% over the strongest static or heuristic protocols, with the largest margin on high-difficulty and OOD cases.

Ablation studies show that prompt-level temperature control is too coarse, always-updating at every token introduces high variance, and token-level selective updating yields the best accuracy–diversity trade-off.

Key quality–diversity AUC scores for selective VTSR vs. baselines [2510.01218]:

| Task           | top-p | min-p | EDT  | VTSR  |
|:---------------|:-----:|:-----:|:----:|:-----:|
| GSM8K          | 0.32  | 0.38  | 0.35 | 0.42  |
| Symbolic GSM   | 0.32  | 0.40  | 0.36 | 0.47  |
| Minerva-Alg    | 0.21  | 0.25  | 0.24 | 0.30  |

Greedy routing constitutes a higher fraction of output on harder tasks or at very high $T$ (e.g., 44% for Minerva at $T=3.0$), supporting the adaptive precision/diversity trade-off hypothesis.

## 6. Practical Implementation and Pseudocode

The VTSR architecture is amenable to practical LLM deployment with minimal overhead. In the RLVR-based setting, inference simply samples $\tau_t$ and $y_t$ sequentially from the learned temperature and token policies [2602.13035]. For selective sampling [2510.01218], a single linear classifier runs per token. The decision logic follows:

```python
for t in 1..T:
    logits = LLM.logits(x)
    hidden = LLM.hidden(x)
    risk = classifier(hidden)
    if risk >= threshold:
        y = argmax(logits)
    else:
        y = categorical_sample(temp_scale(logits, T_high), min_p)
    x = x + [y]
    if y == EOS:
        break
```

Such architectures require only hundreds to a few thousand labeled prompts for effective routing and generalize well across tasks—single-head routers can match per-task heads in cross-domain evaluation [2510.01218].

## 7. Theoretical Justification and Generalization

The core theoretical underpinning of VTSR is the minimization of expected task regret at each position, viewed as a local variational selection of the decoding mode. Latent routing variables partition generation trajectories into high-precision and high-diversity regions, picking the mode with maximal expected downstream reward. This formulation prevents catastrophic errors at high-risk positions and allows diversity enhancements at low-risk points.

Empirical results confirm a Pareto-improved frontier in the quality–diversity space: VTSR consistently dominates canonical min-$p$, top-$p$, and static-temperature sampling schemes in both mathematical reasoning and general QA settings [2510.01218][2602.13035].

Source: https://www.emergentmind.com/topics/variational-temperature-sampling-router-vtsr