GIFT: Finite-Temperature Gibbs Initialization

• GIFT is a finite-temperature Gibbs reweighting method that initializes large reasoning models by softly biasing them towards expert demonstrations while retaining base model structure.
• It reformulates standard supervised fine-tuning as a thermodynamic process where the temperature parameter controls the balance between deterministic imitation and exploratory diversity during reinforcement learning.
• By deriving a closed-form token-level approximation of a KL-regularized objective, GIFT bridges the gap between low-entropy imitation and exploration-dependent RL, leading to improved performance on complex benchmarks.

Searching arXiv for the cited GIFT paper and closely related Gibbs-state preparation work. {"query":"(Zhao et al., 14 Jan 2026) GIFT Unlocking Global Optimality in Post-Training via Finite-Temperature Gibbs Initialization"} {"query":"(Warren et al., 2022) Adaptive variational algorithms for quantum Gibbs state preparation"} Gibbs Initialization with Finite Temperature (GIFT) is a post-training method for Large Reasoning Models (LRMs) that reformulates supervised fine-tuning (SFT) as a finite-temperature Gibbs reweighting of a pretrained base policy rather than as rigid one-hot imitation. In this formulation, standard SFT is treated as a degenerate zero-temperature limit that suppresses base priors, whereas GIFT incorporates supervision as a finite-temperature energy potential and uses that softened initialization for subsequent reinforcement learning (RL). The method is designed for the common SFT-then-RL pipeline, with the stated aim of removing the optimization mismatch between low-entropy imitation and exploration-dependent RL (Zhao et al., 14 Jan 2026).

1. Conceptual basis

GIFT is motivated by the claim that the prevailing post-training pipeline for LRMs contains an intrinsic optimization mismatch. In the paper’s formulation, SFT and RL are not merely sequential stages; they induce different distributional geometries. Standard SFT is defined by the token-level negative log-likelihood objective

$\begin{split} \mathcal{L}_{\text{SFT}}(\theta) &= - \mathbb{E}_{(x, y^*) \sim \mathcal{D}} \log \pi_\theta(y^* | x) \ &= - \mathbb{E}_{(x, y^*) \sim \mathcal{D}} \sum_{t=1}^{|y^*|} \left[\log \pi_\theta(y^*_{t} | x,y_{<t}^*) \right]. \end{split}$

The diagnosis is that this objective drives the model toward deterministic imitation of expert trajectories, producing what the paper calls “distributional collapse.” In that view, SFT suppresses non-target mass and thereby exhausts the exploration space needed by RL (Zhao et al., 14 Jan 2026).

The corresponding RL stage is modeled as a KL-regularized reward maximization problem,

$$J_{\text{RL}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta} \left[ R(x, y) - \frac{1}{\eta} D_{\text{KL}}(\pi_\theta \| \pi_{\text{sft}}) \right],$$

where $R(x,y)$ is the reward and $\eta > 0$ controls regularization. The paper’s central claim is that a low-entropy SFT initializer is poorly aligned with this RL objective because RL requires a policy with enough diversity to explore alternative reasoning paths. GIFT is therefore introduced as an initialization rule that preserves the base model’s structural priors while still biasing the policy toward expert demonstrations.

A common misconception is to read GIFT as a generic entropy-injection heuristic. The paper instead frames it as a thermodynamic policy-construction principle: supervision is not added as a penalty on a trained policy, but encoded as a finite-temperature exponential tilt of the base policy. This makes “finite temperature” the central control parameter for how much of the pretrained distribution is retained.

2. Unified post-training objective and Gibbs form

The formal core of GIFT is a unified global post-training objective. The paper defines the globally optimal post-training policy as

$$\begin{aligned} \pi_{\text{global}}^*(\cdot | x)& = \arg\max_{\pi} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi(\cdot | x)} [ \ &R(x, y) - \frac{1}{\eta}D_{\text{KL}}(\pi(\cdot | x) \| \pi_{\text{base}}(\cdot | x))]. \end{aligned}$$

This objective is solved in closed form by

$$\pi_{\text{global}}^*(y|x) = \frac{1}{Z_{\text{base}}(x)} \pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)},$$

with partition function

$$Z_{\text{base}}(x) = \sum_{y}\pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)}.$$

Thus the global optimum is a Gibbs distribution formed by exponentially tilting the base policy by reward (Zhao et al., 14 Jan 2026).

The RL stage, when started from an initialization policy $\pi_{\text{sft}}$, is modeled as converging to

$$\pi_{\text{stage2}}^*(y|x) = \frac{1}{Z_{\text{sft}}(x)} \pi_{\text{sft}}(y|x) \cdot \mathrm{e}^{\lambda R(x, y)},$$

where $\lambda > 0$ is the RL-stage reward/KL tradeoff parameter. Enforcing

$$J_{\text{RL}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta} \left[ R(x, y) - \frac{1}{\eta} D_{\text{KL}}(\pi_\theta \| \pi_{\text{sft}}) \right],$$0

implies the optimal initialization policy

$$J_{\text{RL}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta} \left[ R(x, y) - \frac{1}{\eta} D_{\text{KL}}(\pi_\theta \| \pi_{\text{sft}}) \right],$$1

with

$$J_{\text{RL}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta} \left[ R(x, y) - \frac{1}{\eta} D_{\text{KL}}(\pi_\theta \| \pi_{\text{sft}}) \right],$$2

This is the defining equation of GIFT.

In this construction, $$J_{\text{RL}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta} \left[ R(x, y) - \frac{1}{\eta} D_{\text{KL}}(\pi_\theta \| \pi_{\text{sft}}) \right],$$3 is the inverse temperature of the initialization distribution. Larger $$J_{\text{RL}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta} \left[ R(x, y) - \frac{1}{\eta} D_{\text{KL}}(\pi_\theta \| \pi_{\text{sft}}) \right],$$4 concentrates mass more strongly on high-reward or expert-consistent trajectories; smaller $$J_{\text{RL}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta} \left[ R(x, y) - \frac{1}{\eta} D_{\text{KL}}(\pi_\theta \| \pi_{\text{sft}}) \right],$$5 leaves the policy closer to the base model. The paper interprets standard SFT as the limit $$J_{\text{RL}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta} \left[ R(x, y) - \frac{1}{\eta} D_{\text{KL}}(\pi_\theta \| \pi_{\text{sft}}) \right],$$6, i.e. a zero-temperature degeneracy in which the distribution collapses onto supervised targets. Conversely, $$J_{\text{RL}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta} \left[ R(x, y) - \frac{1}{\eta} D_{\text{KL}}(\pi_\theta \| \pi_{\text{sft}}) \right],$$7 recovers the base policy. GIFT occupies the intermediate regime in which supervision biases the model without erasing its prior support.

The paper’s use of “global optimality” is precise in this section: it denotes the exact optimizer of the KL-regularized objective above. A plausible implication is that the empirical role of GIFT is to make the initialization policy consistent with that closed-form optimum, rather than to prove that nonconvex training attains a global optimum in the optimization-theoretic sense.

3. Token-level approximation and practical training target

Because the sequence-level Gibbs distribution is not directly tractable, the paper derives a token-level form. Starting from

$$J_{\text{RL}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta} \left[ R(x, y) - \frac{1}{\eta} D_{\text{KL}}(\pi_\theta \| \pi_{\text{sft}}) \right],$$8

it introduces the soft $$J_{\text{RL}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta} \left[ R(x, y) - \frac{1}{\eta} D_{\text{KL}}(\pi_\theta \| \pi_{\text{sft}}) \right],$$9-function

$R(x,y)$0

and obtains

$R(x,y)$1

with soft advantage

$R(x,y)$2

The paper then introduces two assumptions, Oracle Path Optimality and Sparse Recovery, under which

$R(x,y)$3

This yields the practical token-level target

$R(x,y)$4

The expert token receives a logit bonus of $R(x,y)$5, while all other tokens retain the relative structure of the base model (Zhao et al., 14 Jan 2026).

The corresponding training objective is soft-target cross-entropy: $R(x,y)$6 Operationally, GIFT therefore differs from standard SFT not in optimizer type or autoregressive factorization, but in the target distribution. Standard SFT uses one-hot supervision; GIFT uses a base-model-reweighted soft target.

This distinction matters for interpretation. GIFT is not merely “SFT with soft labels.” The soft labels are not arbitrary smoothing distributions; they are intended as token-level approximations to a finite-temperature Gibbs tilt of the base policy.

4. Algorithmic instantiation

The paper’s practical procedure starts from base parameters

$R(x,y)$7

samples demonstration pairs $R(x,y)$8, and computes base-model logits $R(x,y)$9 on the demonstration prefixes. These are converted to log-probabilities

$\eta > 0$0

The adjusted target logits are then defined by

$\eta > 0$1

and normalized as

$\eta > 0$2

Training then proceeds with cross-entropy against that target, followed by standard gradient updates (Zhao et al., 14 Jan 2026).

An optional smoothing variant is introduced for numerical stability: $\eta > 0$3 with $\eta > 0$4 in the reported experiments. This is used when the base model assigns near-zero probability to target tokens.

In the reported setup, the SFT-stage optimizer is AdamW with $\eta > 0$5, $\eta > 0$6, weight decay $\eta > 0$7, global batch size $\eta > 0$8, learning rate $\eta > 0$9, and maximum sequence length $$\begin{aligned} \pi_{\text{global}}^*(\cdot | x)& = \arg\max_{\pi} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi(\cdot | x)} [ \ &R(x, y) - \frac{1}{\eta}D_{\text{KL}}(\pi(\cdot | x) \| \pi_{\text{base}}(\cdot | x))]. \end{aligned}$$0. After initialization, RL is run with GRPO in the VeRL framework, using group size $$\begin{aligned} \pi_{\text{global}}^*(\cdot | x)& = \arg\max_{\pi} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi(\cdot | x)} [ \ &R(x, y) - \frac{1}{\eta}D_{\text{KL}}(\pi(\cdot | x) \| \pi_{\text{base}}(\cdot | x))]. \end{aligned}$$1, learning rate $$\begin{aligned} \pi_{\text{global}}^*(\cdot | x)& = \arg\max_{\pi} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi(\cdot | x)} [ \ &R(x, y) - \frac{1}{\eta}D_{\text{KL}}(\pi(\cdot | x) \| \pi_{\text{base}}(\cdot | x))]. \end{aligned}$$2, PPO clip ratio $$\begin{aligned} \pi_{\text{global}}^*(\cdot | x)& = \arg\max_{\pi} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi(\cdot | x)} [ \ &R(x, y) - \frac{1}{\eta}D_{\text{KL}}(\pi(\cdot | x) \| \pi_{\text{base}}(\cdot | x))]. \end{aligned}$$3, exploration temperature $$\begin{aligned} \pi_{\text{global}}^*(\cdot | x)& = \arg\max_{\pi} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi(\cdot | x)} [ \ &R(x, y) - \frac{1}{\eta}D_{\text{KL}}(\pi(\cdot | x) \| \pi_{\text{base}}(\cdot | x))]. \end{aligned}$$4, explicit KL coefficient $$\begin{aligned} \pi_{\text{global}}^*(\cdot | x)& = \arg\max_{\pi} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi(\cdot | x)} [ \ &R(x, y) - \frac{1}{\eta}D_{\text{KL}}(\pi(\cdot | x) \| \pi_{\text{base}}(\cdot | x))]. \end{aligned}$$5, and 1 epoch of RL. The theory is framed with KL-regularized RL, whereas the practical GRPO stage sets the explicit KL coefficient to zero; the paper notes this implementation gap without fully resolving it.

The main hyperparameter is $$\begin{aligned} \pi_{\text{global}}^*(\cdot | x)& = \arg\max_{\pi} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi(\cdot | x)} [ \ &R(x, y) - \frac{1}{\eta}D_{\text{KL}}(\pi(\cdot | x) \| \pi_{\text{base}}(\cdot | x))]. \end{aligned}$$6. It is fixed rather than scheduled, and the paper identifies this as a limitation. Empirically, the best values are reported as approximately $$\begin{aligned} \pi_{\text{global}}^*(\cdot | x)& = \arg\max_{\pi} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi(\cdot | x)} [ \ &R(x, y) - \frac{1}{\eta}D_{\text{KL}}(\pi(\cdot | x) \| \pi_{\text{base}}(\cdot | x))]. \end{aligned}$$7 for Qwen2.5-7B and $$\begin{aligned} \pi_{\text{global}}^*(\cdot | x)& = \arg\max_{\pi} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi(\cdot | x)} [ \ &R(x, y) - \frac{1}{\eta}D_{\text{KL}}(\pi(\cdot | x) \| \pi_{\text{base}}(\cdot | x))]. \end{aligned}$$8 for Llama-3.1-8B.

5. Empirical behavior

The experiments use DeepMath-103k, with 10,000 examples for SFT, 10,000 for RL, and 1,000 for validation. The backbone models are Qwen2.5-7B and Llama-3.1-8B. Evaluation is performed on GSM8K, Math500, OlympiadBench, AIME24, AIME25, GPQA, MMLU-Pro, MMLU-Redux, and ARC-Challenge (Zhao et al., 14 Jan 2026).

For average pass@1 on math benchmarks, GIFT outperforms standard SFT and the other reported baselines.

Method	Qwen2.5-7B	Llama-3.1-8B
Direct SFT	47.09	29.20
Direct RL	45.43	20.68
LUFFY	50.15	—
ReLIFT	49.62	—
SFT	48.58	29.62
SFT + Entropy	49.27	30.97
DFT	40.24	23.74
ASFT	44.62	21.87
PSFT	50.76	30.21
GIFT	52.43	35.60

On Qwen2.5-7B, the paper highlights AIME24, where standard SFT scores $$\begin{aligned} \pi_{\text{global}}^*(\cdot | x)& = \arg\max_{\pi} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi(\cdot | x)} [ \ &R(x, y) - \frac{1}{\eta}D_{\text{KL}}(\pi(\cdot | x) \| \pi_{\text{base}}(\cdot | x))]. \end{aligned}$$9 and GIFT scores $$\pi_{\text{global}}^*(y|x) = \frac{1}{Z_{\text{base}}(x)} \pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)},$$0. On OOD benchmarks, GIFT is also competitive or best among the reported initializers.

Method	Qwen2.5-7B OOD average	Llama-3.1-8B OOD average
Direct SFT	55.41	50.14
Direct RL	57.75	42.53
LUFFY	64.27	—
ReLIFT	64.55	—
SFT	59.78	46.97
SFT + Entropy	62.11	53.32
DFT	49.89	46.39
ASFT	57.60	39.05
PSFT	63.74	33.37
GIFT	64.10	55.24

The paper also supplies direct evidence for the exploration argument. Before RL, on Qwen2.5-7B, pass@1 is $$\pi_{\text{global}}^*(y|x) = \frac{1}{Z_{\text{base}}(x)} \pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)},$$1 for standard SFT and $$\pi_{\text{global}}^*(y|x) = \frac{1}{Z_{\text{base}}(x)} \pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)},$$2 for GIFT, while pass@8 is $$\pi_{\text{global}}^*(y|x) = \frac{1}{Z_{\text{base}}(x)} \pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)},$$3 for standard SFT and $$\pi_{\text{global}}^*(y|x) = \frac{1}{Z_{\text{base}}(x)} \pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)},$$4 for GIFT. This pattern is used to argue that GIFT sacrifices little at greedy decoding while preserving a broader set of successful trajectories.

A temperature ablation shows a non-monotonic relationship between $$\pi_{\text{global}}^*(y|x) = \frac{1}{Z_{\text{base}}(x)} \pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)},$$5 and downstream performance. Low $$\pi_{\text{global}}^*(y|x) = \frac{1}{Z_{\text{base}}(x)} \pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)},$$6 remains too close to the base model; very high $$\pi_{\text{global}}^*(y|x) = \frac{1}{Z_{\text{base}}(x)} \pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)},$$7 approaches the SFT-like limit; intermediate $$\pi_{\text{global}}^*(y|x) = \frac{1}{Z_{\text{base}}(x)} \pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)},$$8 performs best. This is one of the paper’s strongest empirical supports for the finite-temperature interpretation.

The paper also reports policy-consistency diagnostics across stages. For Qwen2.5-7B, in the stage from base to initialized policy, standard SFT gives cosine similarity $$\pi_{\text{global}}^*(y|x) = \frac{1}{Z_{\text{base}}(x)} \pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)},$$9, L2 distance $$Z_{\text{base}}(x) = \sum_{y}\pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)}.$$0, and KL divergence $$Z_{\text{base}}(x) = \sum_{y}\pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)}.$$1, whereas GIFT gives cosine similarity $$Z_{\text{base}}(x) = \sum_{y}\pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)}.$$2, L2 distance $$Z_{\text{base}}(x) = \sum_{y}\pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)}.$$3, and KL divergence $$Z_{\text{base}}(x) = \sum_{y}\pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)}.$$4. In the stage from initialized policy to RL policy, standard SFT gives cosine similarity $$Z_{\text{base}}(x) = \sum_{y}\pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)}.$$5, L2 distance $$Z_{\text{base}}(x) = \sum_{y}\pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)}.$$6, and KL divergence $$Z_{\text{base}}(x) = \sum_{y}\pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)}.$$7, whereas GIFT gives cosine similarity $$Z_{\text{base}}(x) = \sum_{y}\pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)}.$$8, L2 distance $$Z_{\text{base}}(x) = \sum_{y}\pi_{\text{base}}(y|x) \mathrm{e}^{\eta R(x, y)}.$$9, and KL divergence $\pi_{\text{sft}}$0. The corresponding Llama-3.1-8B metrics show the same trend. The intended interpretation is that GIFT induces less destructive policy drift and better continuity across the post-training pipeline.

6. Relation to adjacent methods, terminology, and limitations

GIFT is positioned against several classes of baselines: direct SFT, direct RL, unified paradigms such as LUFFY and ReLIFT, and SFT-then-RL variants including SFT + Entropy, DFT, ASFT, and PSFT. The paper’s claim is not simply that GIFT injects entropy, but that it preserves the base model inside the initialization target itself. In formal terms, it replaces one-hot imitation by

$\pi_{\text{sft}}$1

This is why the method is presented as a distributional bridge rather than a regularization add-on (Zhao et al., 14 Jan 2026).

Several caveats are explicit. First, $\pi_{\text{sft}}$2 is a fixed manually chosen hyperparameter, with no adaptive schedule. Second, the token-level derivation depends on the assumptions Oracle Path Optimality and Sparse Recovery. Third, the practical RL stage uses GRPO with KL coefficient $\pi_{\text{sft}}$3, while the theory is formulated with KL-regularized RL. Fourth, some notation and sign conventions in the paper are inconsistent, although the intended construction remains clear. These limitations constrain how literally the global-optimality argument can be transferred to deployed post-training systems.

The acronym also admits a broader technical context. Outside LRM post-training, finite-temperature Gibbs initialization refers to thermal-state preparation in quantum computation, including adaptive variational Gibbs preparation (Warren et al., 2022), free-energy-based NISQ Gibbs preparation on two registers (Consiglio et al., 2023), continuous-time quantum Gibbs samplers and purified thermal field double preparation (Chen et al., 2023), and low-temperature truncated-spectrum Gibbs initialization with minimal ancilla usage (Sambasivam et al., 18 Mar 2025). This suggests that the phrase “Gibbs initialization with finite temperature” is domain-specific: in the LRM setting it denotes a policy-initialization rule, whereas in quantum information it denotes preparation of $\pi_{\text{sft}}$4.

In its own domain, GIFT is best understood as a thermodynamic reinterpretation of SFT for RL initialization. Its principal claim is that supervision should act as a finite-temperature energy potential over the base policy rather than as a zero-temperature collapse onto demonstrations. The empirical results support that view, but the strongest statement of “global optimality” remains the closed-form optimizer of the KL-regularized objective, not a proof that large-scale post-training reaches a global optimum in practice.