Adaptive Gradient Veto: Enhancing Stability

• Adaptive Gradient Veto is a family of mechanisms that modulate gradient updates via a logit-space bridge to prevent instabilities.
• It applies veto strategies in tasks like model distillation, feature attribution, and unconstrained optimization to balance convergence speed and output diversity.
• Empirical results demonstrate improved performance in language modeling and attribution by mitigating gradient explosion and ensuring robust training.

Adaptive Gradient Veto (AGV) encompasses a family of mechanisms that suppress, modulate, or steer gradient-based updates during optimization, distillation, or feature attribution, often with the specific goal of stabilizing training, removing noise, or balancing convergence speed versus diversity. Principal instantiations span model distillation (particularly in language modeling), attribution for neural explanations, and unconstrained optimization—each utilizing distinct veto strategies to counteract the detrimental effects of pathological or spurious gradients.

1. Instabilities in Gradient-Based Optimization and Distillation

On-policy knowledge distillation, a procedure used to transfer behaviors from a teacher policy $P_T$ to a student policy $P_S$, is susceptible to instabilities arising from the Kullback-Leibler (KL) objectives. The two canonical losses,

$$\mathcal{L}_{\rm FWD} = \mathbb{E}_{y\sim P_S}\Bigl[\sum_{t=1}^T D_{\rm KL}\bigl(P_T(\cdot\mid x,y_{<t})\;\|\;P_S(\cdot\mid x,y_{<t})\bigr)\Bigr],$$

and

$$\mathcal{L}_{\rm REV} = \mathbb{E}_{y\sim P_S}\Bigl[\sum_{t=1}^T D_{\rm KL}\bigl(P_S(\cdot\mid x,y_{<t})\;\|\;P_T(\cdot\mid x,y_{<t})\bigr)\Bigr],$$

fail due to (i) "zero-avoiding" gradients that explode when $P_S(v) \to 0$ for teacher-preferred tokens, and (ii) diversity collapse, where the student mimics only a single mode favored by the teacher (Jang et al., 12 Jan 2026). These regimes exhibit the following gradient forms:

• Forward-KL: $$-\sum_{v} \frac{P_T(v)}{P_S(v)}\,\nabla_\theta P_S(v)$$ (pathological when $P_S(v)\approx 0$).
• Reverse-KL: $$\mathbb{E}_{v\sim P_S}[\nabla_\theta\log P_S(v)\,(\log P_S(v)-\log P_T(v))]$$, which promotes mode seeking and suppresses diversity.

This instability motivates alternative gradient veto formulations to enable robust student learning without collapse or explosion.

2. Adaptive Gradient Veto via Logit-Space Bridging

The Adaptive Gradient Veto objective constructs an intermediate "bridge" distribution $Q$ by interpolating between teacher and student policies in logit space. Let $z_T(v) = \log P_T(v)$ and $z_S(v) = \log P_S(v)$; then define $Q$ at each token step $t$ as

$$Q(v) \propto \exp(z_T(v) + \beta z_S(v)) = \frac{1}{Z} P_T(v) P_S(v)^\beta$$

where $\beta \geq 0$ is a tunable parameter. This interpolative Product-of-Experts formulation ensures that $Q$ assigns high probability only where teacher and student agree, suppressing low-confidence discrepancies.

Associated KL objectives use this $Q$ in place of $P_T$:

• Veto-Forward: $$\mathcal{L}_t^{\rm VetoF} = D_{\rm KL}(Q \;\|\; P_S)$$
• Veto-Reverse: $$\mathcal{L}_t^{\rm VetoR} = D_{\rm KL}(P_S \;\|\; Q)$$

Summing these losses over time and sample trajectories provides the aggregate training objective (Jang et al., 12 Jan 2026).

3. Suppression of Pathological Gradients and Diversity Modulation

The parameter $\beta$ in the Veto formulation operates as an adaptive knob:

• Gradient Suppression: In forward-KL, as $P_S(v) \to 0$, $Q(v) \propto P_S(v)^\beta$ ensures that the gradient term

$$- Q(v) \log P_S(v) \approx - P_S(v)^\beta \log P_S(v) \to 0$$

as $P_S(v) \to 0$, mitigating gradient explosion and stabilizing early training.

• Decisiveness/Entropy Control: In reverse-KL, the gradient is

$$\mathbb{E}_{v\sim P_S}\left[\nabla_\theta\log P_S(v)\;\left(-\log P_T(v) + (1-\beta)\log P_S(v)\right)\right]$$

By varying $\beta$ within $[0,1]$, the objective interpolates between pure KL ($\beta=0$) and high-entropy policy gradient regimes, thus balancing reward maximization and output diversity.

Theorems formalize these properties:

• Boundedness: For $\beta>0$, Veto-Forward loss per token remains finite as $P_S(v)\to 0$.
• Sharpening Effect: At optimum, $$P_S^*(v|x) \propto \left[P_T(v|x)\right]^{1/(1-\beta)}$$ constitutes a sharpened teacher distribution.
• Policy Gradient Bridge: The Veto-Reverse gradient is directly equivalent to a REINFORCE update with explicit entropy regularization (Jang et al., 12 Jan 2026).

4. Algorithmic Implementations and Pseudocode

The AGV formulation is direct to implement, utilizing geometric interpolation and adaptive $\beta$. Below is high-level pseudocode for Veto as given in (Jang et al., 12 Jan 2026):

Input: student Pθ, teacher PT, data X, initial β₀, total steps N, lr η Initialize θ for i in 1..N: β = β₀ * (1-i/N) x = sample(X) y = sample Pθ(·|x) ℓ = 0 for t in 1..len(y): zT = log PT(·|x,y< t) zS = log Pθ(·|x,y< t) Q ∝ exp(zT + β*zS) if Forward KL: ℓ_t = KL(Q || Pθ(·|x,y< t)) else: ℓ_t = KL(Pθ(·|x,y< t) || Q) ℓ += ℓ_t θ -= η * ∇θ ℓ

In feature attribution, the "Adaptive Gradient Veto" editor’s term is realized in Guided Integrated Gradients (Guided IG) (Kapishnikov et al., 2021), which selects the lowest-gradient coordinates for updates at each step, thus vetoing noisy gradient directions during path integration.

5. Empirical Performance and Application Domains

Empirical results indicate consistent superiority of AGV approaches over conventional methods:

• LLM Distillation: On GSM8K, forward-KL Veto (β from 0.8$\rightarrow$0) yields 39.9% student accuracy vs. 33.4% for supervised KD and 35.1% for reverse-KL on-policy KD. On HumanEval, Veto improves pass@1 from 1.4 to 2.1; pass@10 from 10.6 to 16.4. In DialogSum, win-rate rises to 56.5% from 54.3% (Jang et al., 12 Jan 2026).
• Feature Attribution: In Guided IG, attribution error on closed-path tests reduced 3–4×, localization AUC increased from 0.63–0.67 (IG) to ~0.70–0.71 (Guided IG), and model-aligned saliency improved 5–10% (SIC-AUC); visualizations and human-mask AUCs corroborate robustness to spurious gradients (Kapishnikov et al., 2021).
• Optimization: In unconstrained convex/nonconvex problems, adaptive curvature-veto mechanisms track local geometry for step size selection, improving stability and convergence over standard linesearch or Polyak/Barzilai–Borwein rules, with automatic step-size adaptation based solely on gradients (Malitsky et al., 2019).

These outcomes suggest the veto principle—where high-magnitude or unreliable gradients are systematically suppressed—yields robust performance across domains involving distillation, attribution, and generic optimization.

6. Connections to Adaptive Path Methods and Curvature Vetoes

AGV is a specific instantiation of the broader class of Adaptive Path Methods (APMs) (Kapishnikov et al., 2021). In attribution, APMs generalize path-integral metrics to allow the integration curve to depend on model gradient geometry, rather than input alone. Guided IG, for instance, dynamically constructs paths that avoid noisy gradient regions, anchoring or segmenting as needed to modulate adaptation strength.

In optimization, the "curvature veto" restricts gradient step size using secant estimates of local Hessian smoothness, preventing uncontrolled overshoot even when global smoothness fails (Malitsky et al., 2019). This automated adaptation parallels AGV principles in knowledge distillation and feature attribution.

7. Theoretical Properties, Limitations, and Misconceptions

All major AGV instantiations inherit or extend path-integral axioms:

• Completeness: Attribution sum recovers output difference.
• Implementation Invariance, Sensitivity, Symmetry: AGV preserves these under its adaptive suppression regime (Kapishnikov et al., 2021).
• Stability: For $\beta>0$, forward Veto provably constraints gradient explosion; reverse Veto implements explicit entropy regularization (Jang et al., 12 Jan 2026).
• Convergence: Curvature-veto adaptive methods possess rates $O(1/k)$ (convex), linear contraction (strong-convexity), and handle locally smooth but globally ill-behaved functions (Malitsky et al., 2019).

A common misconception is that veto mechanisms always limit expressivity or convergence; empirical results demonstrate no loss and often notable gains in both stability and model alignment. A plausible implication is that broader adoption of AGV and vetoing approaches may provide uniform improvements in deep model distillation, attribution, and optimization, especially in domains otherwise prone to instability or noise accumulation.