Globally Normalized Distillation Policy Optimization

• GNDPO is defined as an on-policy post-training method that converts raw KL-derived token rewards into globally normalized advantages.
• It stabilizes training by mitigating gradient explosions caused by extreme teacher-student mismatches during token-level distillation.
• Empirical results show that GNDPO improves training robustness and downstream multimodal reasoning performance compared to naive OPD.

Searching arXiv for the specified GNDPO paper and a closely related distillation-policy-optimization paper to ground the article in current literature. arXiv search query: (Hao et al., 8 Jun 2026) "Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization" Globally Normalized Distillation Policy Optimization (GNDPO) is an on-policy post-training method for LLMs and multimodal LLMs (MLLMs) that stabilizes token-level distillation from a stronger teacher by converting raw KL-derived token rewards into batch-level normalized advantages. It is introduced as a practical response to a specific failure mode of naive on-policy distillation (OPD): outlier states in which the teacher assigns near-zero probability to student-sampled tokens can induce disproportionately large gradients, producing volatility, gradient explosions, and divergence. GNDPO preserves the dense token-level supervision characteristic of OPD while importing variance-control machinery from policy-gradient optimization, and is reported to improve both training robustness and downstream performance on multimodal reasoning benchmarks (Hao et al., 8 Jun 2026).

1. Conceptual setting and motivation

The method is formulated in the setting of on-policy distillation. A student policy $\pi_\theta$ is trained using supervision from a larger and stronger teacher model $\pi_{\text{teacher}}$. For each question $q$, trajectories are sampled from the current student policy $\pi_{\theta_{\text{old}}}$ rather than from the teacher or from a static dataset. Every token in those student-generated trajectories is then evaluated by the teacher at the token level, so the student receives dense supervision over the entire reasoning chain rather than only a terminal reward.

This positioning is central to the method’s rationale. The paper places OPD between supervised fine-tuning and on-policy reinforcement learning. Relative to supervised fine-tuning, on-policy sampling mitigates exposure bias and better matches training and inference distributions. Relative to reinforcement learning with verifiable rewards (RLVR) or related RLHF-style pipelines, teacher probabilities provide a per-token target distribution rather than a sparse sequence-level reward. This is especially consequential for long-chain reasoning, where sequence-level credit assignment is indirect and noisy.

The motivation for GNDPO is the observation that naive token-level OPD is unstable. The paper identifies the source as magnitude misalignment in outlier states. If the teacher assigns extremely small probability to a token sampled by the student, the associated log-ratio becomes very large in magnitude. Since the policy-gradient update multiplies $\nabla_\theta \log \pi_\theta$ by this token-wise score, a small number of extreme mismatches can dominate the batch update. The resulting behavior is described as gradient explosions, highly volatile loss curves, and training divergence. In the paper’s terminology, the scale of token-wise KL-derived rewards is heterogeneous across contexts, and rare outliers can rupture the shared representations.

2. Formal objective and normalized advantage construction

The baseline OPD objective is written in RL language. A trajectory $o_i = (o_i^1, o_i^2, \dots, o_i^{|o_i|})$ is sampled on-policy from the student, and each token receives a reward derived from the teacher via the negative reverse KL term

$$\hat{A}_{i,t} = -\left( \log \pi_\theta(o_i^t \mid q, o_i^{<t}) - \log \pi_{\text{teacher}}(o_i^t \mid q, o_i^{<t}) \right).$$

This quantity is maximized when the student matches the teacher. In an RL-style formulation, the corresponding gradient has the form

$$\nabla_\theta \mathcal{L}_{\text{OPD}} \propto \mathbb{E}\left[ \sum_{t=1}^{|o_i|} \hat{A}_{i,t}\; \nabla_\theta \log \pi_\theta(o_i^t \mid q, o_i^{<t}) \right].$$

The key difficulty is that $\hat{A}_{i,t}$ is used at its raw scale. In out-of-distribution contexts, where $\pi_{\text{teacher}}(o_i^t \mid q, o_i^{<t}) \to 0$, the log-ratio can become extremely large in magnitude, and the update becomes dominated by a tiny subset of tokens.

GNDPO retains the same token-level signal but reinterprets it as a relative quantity within the batch. Let $\pi_{\text{teacher}}$0 denote the mini-batch containing all trajectories and all tokens. The method defines a normalized distillation advantage

$\pi_{\text{teacher}}$1

where

$\pi_{\text{teacher}}$2

The update then becomes

$\pi_{\text{teacher}}$3

The paper explicitly aligns this construction with advantage normalization in policy-gradient methods, but the “reward” here is KL-derived token-level supervision rather than environment feedback.

3. Meaning of “global normalization” and its stabilizing role

In GNDPO, “global” refers to normalization across the entire mini-batch of tokens: multiple questions, multiple sampled trajectories per question, and all tokens in those trajectories. This point matters because a common misunderstanding would be to read “global” as sequence-local normalization or position-wise normalization. The method does neither. Mean and standard deviation are computed over all token-level rewards in $\pi_{\text{teacher}}$4, not per trajectory, not per question, and not per time step.

Operationally, a training step consists of sampling questions, generating one or more responses per question on-policy from $\pi_{\text{teacher}}$5, evaluating student and teacher log-probabilities token by token, computing raw rewards $\pi_{\text{teacher}}$6, aggregating these rewards over the full batch, and finally replacing them by z-scored advantages before the policy-gradient update. The batch statistics are

$\pi_{\text{teacher}}$7

The paper attributes two stabilizing effects to this transformation. First, it performs adaptive gradient rescaling. If a batch contains a few catastrophic token-level mismatches, these inflate $\pi_{\text{teacher}}$8, and all normalized advantages are correspondingly scaled down. Extreme outliers therefore lose their ability to dominate the gradient. Second, centering by $\pi_{\text{teacher}}$9 converts absolute KL magnitudes into relative signals. Tokens above the batch average receive positive advantages, and tokens below the batch average receive negative ones. Optimization is therefore directed toward relative improvements rather than raw, context-dependent magnitudes.

This suggests that the method treats token-level distillation as a variance-control problem rather than only a target-matching problem. A plausible implication is that GNDPO is best understood not as altering the teacher signal itself, but as altering the scale on which that signal is permitted to influence policy updates.

4. Training loop, implementation profile, and multimodal scope

The training algorithm is implemented in the VERL/HybridFlow framework and follows an RL-style loop (Hao et al., 8 Jun 2026). For each problem $q$0, the student samples 16 responses with temperature $q$1, up to max response length $q$2 and prompt length $q$3, for a context length of $q$4 during training. The teacher is then run on the same multimodal input and token prefix to produce logits or probabilities for each generated token. Raw token-level rewards are computed from the student and teacher log-probabilities, normalized across the full batch, and used in the policy-gradient objective. Optimization uses Adam with learning rate $q$5, batch size $q$6, PPO mini-batch size $q$7, and $q$8 training steps. The reported KL coefficient $q$9 is none, because KL enters implicitly through the distillation reward rather than as a separate regularizer.

Evaluation uses max new tokens $\pi_{\theta_{\text{old}}}$0, temperature $\pi_{\theta_{\text{old}}}$1, top-$\pi_{\theta_{\text{old}}}$2, and top-$\pi_{\theta_{\text{old}}}$3 with no limit. The GNDPO-specific hyperparameterization is deliberately minimal: beyond the normalization denominator’s $\pi_{\theta_{\text{old}}}$4, the paper does not introduce extra clipping thresholds or additional coefficients.

The experimental models are InternVL3.5 MLLMs. The teacher is InternVL3.5-8B, and the students are InternVL3.5-1B-Instruct, InternVL3.5-2B-Instruct, and InternVL3.5-4B-Instruct. The training dataset is Geometry3K, described as a multimodal mathematical reasoning dataset with geometry problems and diagrams. Inputs combine images and text prompts, and outputs are textual reasoning chains and final answers with a dedicated reasoning prompt using > tags and \boxed{} final answers.

Although the experiments are specific to the InternVL3.5 family, the method is described as operating at the policy level on token outputs and their probabilities. It does not modify the visual encoder architecture directly; instead, gradients from the language head propagate through the network, including visual components. The paper therefore characterizes GNDPO as agnostic to the multimodal architecture so long as the model exposes a token-level policy. Code is released at https://github.com/OPPO-Mente-Lab/GNDPO.

5. Empirical results and observed training dynamics

The evaluation suite consists of seven multimodal reasoning benchmarks: MathVista (MINI), MMMU (VAL), MathVision, MathVerse (vision-only), DynaMath (worst case), WeMath, and LogicVista. The overall score is the average across these seven benchmarks (Hao et al., 8 Jun 2026). Baselines are the original InternVL3.5-X-Instruct model, GSPO with binary outcome rewards, naive OPD with raw KL-based rewards, and GNDPO.

For InternVL3.5-4B, the reported overall scores are 49.4 for Base, 52.5 for GSPO, 53.9 for OPD, and 54.5 for GNDPO. Relative to OPD, GNDPO improves MMMU from 67.2 to 68.3, MathVision from 50.5 to 51.6, and WeMath from 47.2 to 48.8. For InternVL3.5-2B, the overall scores are 38.5 for Base, 40.5 for GSPO, 42.9 for OPD, and 43.7 for GNDPO, with MathVista improving from 64.7 to 66.0 and MathVerse from 42.0 to 43.7. For InternVL3.5-1B, the overall scores are 25.7 for Base, 27.3 for GSPO, 31.6 for OPD, and 32.0 for GNDPO, with MathVista improving from 55.9 to 56.6 and MMMU from 47.1 to 47.6. Across scales, the reported pattern is consistent: OPD outperforms GSPO, and GNDPO outperforms OPD by a small but steady margin.

The training-dynamics evidence is more diagnostic than the final scores. Over 65 steps, naive OPD exhibits severe volatility in training loss, including spikes and oscillations. The $\pi_{\theta_{\text{old}}}$5 gradient norm shows a catastrophic spike above 2.5 around step 15, which the paper interprets as gradient explosion caused by localized logit mismatches. GNDPO suppresses these spikes and produces smoother gradient norms and loss descent. Entropy is also smoother under GNDPO, which the paper takes as evidence that stabilization does not simply suppress exploration. There are no reported ablations over alternative normalization granularities, such as per-sequence or per-question normalization; the principal ablation is OPD without normalization versus GNDPO with batch-wise normalization.

6. Relation to adjacent methods, limitations, and interpretation

GNDPO is mathematically a policy-gradient method, but its advantage signal does not come from environment returns or a learned value function. Instead, the “reward” is the reverse-KL-derived token log-ratio between student and teacher, and the advantage is that reward after global normalization. In this respect, the method is positioned as conceptually related to global advantage normalization, critic-free policy optimization, and group-relative methods such as GSPO and GRPO, while differing from RLVR by using dense KL-based token supervision rather than sparse outcome rewards. It is also placed near direct preference optimization in the broader post-training landscape, although the paper describes it more precisely as a token-level, teacher-driven counterpart rather than as a preference-learning method.

A useful comparison point is "Extreme Region Policy Distillation" (Chen et al., 25 May 2026), which also treats distillation-based policy optimization as a way to reconcile rich token-level supervision with KL-efficient trust-region updates. ERPD explicitly decouples aggressive signal extraction from constrained student optimization in two stages, whereas GNDPO addresses instability within on-policy token-level distillation by normalizing rewards globally inside the training loop. The relationship is conceptual rather than terminological: both methods emphasize dense token-level signals, fixed or reference-relative log-ratio constructions, and the importance of controlling drift or variance so that informative supervision is not overwhelmed by pathological updates.

The reported limitations of GNDPO are concrete. Its normalization relies on empirical batch statistics, so very small batch sizes may make the mean and standard deviation noisy and weaken the stabilization effect. The method is inherently on-policy and therefore carries nontrivial inference cost, because student generation and teacher evaluation are required at every step, especially for long sequences and large models. The experiments are confined to math and multimodal reasoning tasks, so behavior on open-ended creative generation, multi-turn dialogue, or other domains is not yet systematically studied. Teacher quality dependence also remains: GNDPO stabilizes how teacher signals are used, but it does not correct teacher biases or poor teacher supervision.

Taken together, these properties define GNDPO as a hybrid post-training framework that combines on-policy sampling, dense token-level teacher guidance, and batch-wise advantage normalization. Its core claim is not that distillation alone is sufficient, nor that RL-style stabilization alone is sufficient, but that token-level distillation becomes materially more usable when its scale is normalized globally before entering the policy-gradient update.