Generalization-Pivot Decoupling (GPD)

• GPD is a training paradigm that decouples and interleaves objectives to optimize natural generalization and adversarial robustness in deep neural networks.
• It employs distinct phases with tailored learning rate regimes and EMA-based mixing to preserve the strengths of clean and robust experts.
• Empirical results demonstrate improved clean and adversarial accuracy with reduced training cost compared to traditional joint-training methods.

Generalization-Pivot Decoupling (GPD) is a training paradigm designed to resolve the canonical trade-off between natural (clean) generalization and adversarial robustness in deep neural networks. GPD decouples the training objectives or stages involved in promoting standard generalization versus adversarial robustness, then strategically interleaves or mixes them to derive a model displaying a favorable accuracy/robustness profile. This methodology has been independently instantiated in adversarially robust fine-tuning for vision-LLMs and in general image classification scenarios, most notably in CLIP-based model distillation and bi-expert learning frameworks (Fu et al., 19 Jan 2026, Wang et al., 2023).

1. Formalization and Motivation

The general adversarial training objective is formulated in terms of natural and robust risk. Let $\mathcal{D}$ be the clean data distribution, $f_{\theta}$ a parameterized model, $\ell_1$ the natural surrogate loss (e.g., cross-entropy), and $\ell_2$ a robust surrogate loss (e.g., cross-entropy on adversarial inputs). The standard practice minimizes a joint risk

$$R_{\mathrm{joint}}(\theta) = R_{\mathrm{nat}}(\theta) + \beta R_{\mathrm{rob}}(\theta)$$

where $R_{\mathrm{nat}}(\theta)$ is natural risk, $R_{\mathrm{rob}}(\theta)$ is adversarial risk, and $\beta$ balances the terms. Empirical evidence reveals inherent trade-offs: reducing $R_{\mathrm{rob}}$ via joint optimization tends to incur increased $R_{\mathrm{nat}}$ (Wang et al., 2023). GPD addresses this by decoupling the competing objectives and incorporating a parameter mixing (pivot) mechanism, either temporally or structurally, to mediate knowledge transfer and preserve their individual strengths.

2. GPD in Vision-LLMs: HPT-GPD Framework

In vision-LLMs, such as CLIP, GPD is integrated within the Heterogeneous Proxy Transfer (HPT) framework to address the overfitting and natural generalization collapse that plague adversarial robustness distillation. The procedure operates on two CLIP models: a fixed proxy $P(\cdot; \theta_P, \phi_P)$ demonstrating strong zero-shot generalization and a target $T(\cdot; \theta, \phi)$ to be robustified.

GPD introduces a two-phase optimization with distinct learning rate regimes (Fu et al., 19 Jan 2026):

• Generalization-Anchored Warm-up (Phase I): Target undergoes fine-tuning with a low learning rate $\eta_1$, minimizing

$$L_{\mathrm{GA}} = \frac{1}{N} \sum_{i=1}^N \mathrm{KL}[P_D(T(x_i^a, t_i)) \, \| \, P_D(P(x_i,t_i))]$$

where adversarial inputs $x_i^a$ are crafted via PGD.

• Generalization-Pulled HPT (Phase II): The process switches to a high learning rate $\eta_2$ optimizing

$$L_{\mathrm{RT}} = \frac{1}{N} \sum_{i=1}^N \mathrm{KL}[P_D(T(x_i^a, t_i)) \, \| \, P_D(P(x_i^a, t_i))]$$

To prevent drift from the natural manifold, target parameters are periodically mixed with an EMA (exponential moving average) of the Phase I state:

$$\{\theta, \phi\}_{\mathrm{hpt}}^e = \beta \{\theta, \phi\}^{e-1} + (1-\beta) \{\theta, \phi\}_{\mathrm{ema}}^{K}$$

3. GPD in General Image Classification: Generalist Bi-Expert Framework

The Generalist framework instantiates GPD by maintaining two separate learners: a clean expert ($\theta_n$) for natural data and a robust expert ($\theta_r$) for adversarially perturbed data, along with a global model ($\theta_g$). Each expert is updated with task-specific losses and optimizers. After each minibatch, the global model is updated via EMA mixing:

$$\theta_g \leftarrow \alpha' \theta_g + (1-\alpha') [\gamma \theta_r + (1-\gamma) \theta_n]$$

with $\alpha' \approx 0.999$ and mixing ratio $\gamma$. Periodically, both base learners are re-initialized ("pivoted") to the global model, ensuring bidirectional transfer of acquired knowledge (Wang et al., 2023). This structural decoupling enables utilization of optimizers, augmentations, and schedules tailored separately to each objective, yielding a single model by fusion.

4. Algorithmic Procedures

Below are process summaries for the primary GPD variants:

Setting	Phase 1: Generalization	Phase 2: Robustness/Transfer	Pivot/Mixing Mechanism
HPT-GPD/CLIP (Fu et al., 19 Jan 2026)	Warm-up with $\eta_1$, $L_{\mathrm{GA}}$	HPT with $\eta_2$, $L_{\mathrm{RT}}$	EMA mixing of parameters ($\gamma$, $\beta$)
Generalist (Wang et al., 2023)	Clean expert trains on natural data	Robust expert on adversarial data	EMA update of global, periodic pivot into experts

The detailed procedural steps for HPT-GPD involve initializing from pretrained CLIP weights, iterated phase-wise adversarial batch generation, parameter updates, and scheduled mixing with EMA anchors. In the bi-expert setting, both clean and robust experts perform their updates, after which their parameters are combined to update the global model with scheduled redistribution.

5. Empirical Findings and Effectiveness

Experimental results confirm that GPD methodologies achieve favorable trade-offs in clean and adversarial accuracy, often outperforming conventional joint-training or distillation baselines.

• In HPT-GPD (ViT-B/32 CLIP on TinyImageNet, evaluated zero-shot on 15 datasets, PGD-10 $\ell_\infty=1/255$):
  • Average adversarial accuracy increases from 29.95% (pre-HPT SOTA) to 35.16%.
  • Clean accuracy is raised from 55.18% to 57.75%.
  • Under AutoAttack, robust accuracy doubles (5.58% $\to$ 11.43%).
  • Training cost is reduced relative to alternatives (Fu et al., 19 Jan 2026).
• Generalist shows on CIFAR-10 (ResNet-18):
  • NAT (clean): 89.09% (vs. 93.04% NT, but higher than TRADES at comparable AA)
  • AA (AutoAttack): 46.07% (vs. 48.2% for TRADES $\lambda=6$; TRADES drops NAT to ~30% at this AA level)
  • Overhead is no greater than baseline adversarial training due to negligible cost for clean-expert steps (Wang et al., 2023).

Ablation studies indicate that loss term selection and learning rate schedules are critical: GPD consistently requires a low learning rate for generalization-anchoring and a high learning rate for robust transfer. Removal of either loss term or improper learning rate causes collapse of desired properties.

6. Theoretical Considerations and Limitations

Theoretical analysis in the Generalist framework establishes that the global model's risk converges to the minimum of the two sub-problems (natural and robust risk) to within averaged regret, given convex and bounded losses. No analogous guarantee has yet been established for the two-phase, two-step min-max problem posed in HPT-GPD; a formal convergence proof is outstanding (Fu et al., 19 Jan 2026, Wang et al., 2023).

Observed phenomena in vision-LLMs—such as the emergence of "proxy adversarial robustness" between vanilla CLIP variants—are not theoretically resolved. The conjecture attributes this to similarities in feature spaces due to multimodal contrastive pretraining, a property absent in standard image classifiers (Fu et al., 19 Jan 2026).

7. Open Problems and Future Directions

Several open research avenues remain. Rigorous theoretical analysis of the min-max behavior in phase-decoupled learning is unresolved. The criteria for proxy selection, particularly outside of CLIP variants or for broader multimodal/scale regimes, are not established. Extensions to more complex forms of robust generalization, such as distributional robustness beyond $\ell_p$ perturbations, are currently unexplored.

The applicability of GPD to additional domains, its integration into various model architectures, and comprehensive understanding of the mechanisms underlying proxy robustness and parameter mixing represent substantive directions for future work (Fu et al., 19 Jan 2026, Wang et al., 2023).