Distillation-guided Gradient Surgery Network (DGS-Net)

• The paper presents DGS-Net—a novel fine-tuning framework that uses gradient decomposition and selective distillation to mitigate catastrophic forgetting in CLIP-based models.
• It leverages a multi-branch configuration and LoRA adapters to control gradient flow, suppress harmful semantic features, and preserve transferable pre-training priors.
• Empirical results demonstrate significant improvements in accuracy and robustness across diverse generative models and image degradation scenarios.

The Distillation-guided Gradient Surgery Network (DGS-Net) is a specialized fine-tuning framework constructed on top of pre-trained CLIP image–text encoders to address the problem of catastrophic forgetting during adaptation for AI-generated image detection. By introducing a novel gradient decomposition and selective distillation strategy, DGS-Net ensures preservation of transferable pre-training priors while suppressing task-irrelevant semantic features, thereby achieving robust cross-domain generalization and improved detection accuracy across a large spectrum of generative models (Yan et al., 17 Nov 2025).

1. Architecture and Training Pipeline

DGS-Net employs the CLIP ViT-L/14 backbone, integrating a unique multi-branch configuration to permit fine-grained control over gradient flow and knowledge retention. The three principal frozen components are: the CLIP text encoder $E_\text{text}(\cdot;\varphi)$, the CLIP image encoder teacher $E_\text{img}^T(\cdot)$ (a static copy of the image encoder), while only the lightweight student image encoder $E_\text{img}(\cdot;\theta)$—augmented with LoRA adapters—and two small linear classification heads are updated.

The training loss is composed as

$$L = L_\text{img} + L_\text{text} + \lambda L_\text{align}$$

where $L_\text{img}$ is the binary cross-entropy (BCE) loss for the student image branch, $L_\text{text}$ is BCE for the text branch, and $L_\text{align}$ is a linear alignment term encoding distillation of “beneficial” gradients from the frozen teacher, balanced by the hyperparameter $\lambda$.

During inference, only the LoRA-adapted image encoder $E_\text{img}(\cdot;\theta)$ and image head $h_\text{img}(\cdot)$ are used.

The training pipeline involves:

• Caption generation for each input image using BLIP, with $t = E_\text{text}(\text{caption})$.
• Extraction of image features: student $f = E_\text{img}(x;\theta)$, teacher $f^T = E_\text{img}^T(x)$.
• Computation of three BCE losses: $L_\text{img}$, $L_\text{text}$, $L_\text{img}^T$.
• Gradients calculated for each branch form the basis of subsequent gradient surgery and distillation steps.

2. Gradient Decomposition and Surgery Strategy

A central element of DGS-Net is explicit gradient-space decomposition, separating update directions into those deemed “harmful” (task-irrelevant, often associated with high-level semantics or dataset shortcuts) and “beneficial” (representing pre-training priors aligned with robust image statistics).

Let

• $$g_\text{task} = \nabla_f L_\text{img} \in \mathbb{R}^d$$
• $$g_\text{text} = \nabla_t L_\text{text} \in \mathbb{R}^d$$
• $$g_\text{img} = \nabla_{f^T} L_\text{img}^T \in \mathbb{R}^d$$

Define, elementwise, positive and negative parts: $[a]_+ = \max(a, 0)$, $[a]_- = \min(a, 0)$. Then set

• $g_\text{harm} = [g_\text{text}]_+$ (harmful directions to suppress)
• $g_\text{help} = [g_\text{img}]_-$ (beneficial directions to preserve)

The gradient update is manipulated as follows:

• Suppress harmful directions by projecting $g_\text{task}$ onto the orthogonal complement of $$\hat{u}_\text{harm} = g_\text{harm} / \| g_\text{harm} \|$$
• Final combined gradient:

$$g_\text{final} = (I - \hat{u}_\text{harm} \hat{u}_\text{harm}^\top) g_\text{task} + \lambda g_\text{help}$$

where the first term enforces orthogonal suppression, and the second injects the distilled, beneficial gradient-based alignment.

3. Selective Distillation Through Negative-Gradient Alignment

Unlike traditional distillation schemes that align entire feature vectors or encourage generic similarity between student and teacher, DGS-Net specifically distills only the negative-part gradient from the frozen image-encoder branch ($g_\text{help}$). This is realized through the alignment loss:

$$L_\text{align}(f) = \langle f, g_\text{help} \rangle$$

with the corresponding backpropagated gradient exactly matching $g_\text{help}$. The scalar alignment weight $\lambda$ modulates the degree of prior enforcement, striking a balance between rigid preservation and adaptability during fine-tuning.

The rationale is that negative component gradients encode priors such as frequency sensitivity and global structure intrinsic to CLIP's self-supervised pre-training, without reinstating overfit or semantically correlational cues.

4. Implementation Details and Hyperparameter Settings

DGS-Net is instantiated with the following configurations:

Component	Setting	Notes
Pre-trained backbone	CLIP ViT-L/14
LoRA adaptation	$r=6$, $\alpha=6$	Dropout $=0.8$
Optimizer	Adam	Learning rate $=1e^{-4}$
Batch size	32
Epochs	1
Alignment weight ($\lambda$)	0.2
Data processing	Patch Selection, resize $224\times224$
Caption generator	BLIP	For text branch

All baselines are retrained under identical settings to ensure comparability. The student branch utilizes LoRA adapters in each transformer block for parameter efficiency, with only these adapters and linear heads subject to gradient updates.

5. Empirical Results Across Multiple Benchmarks

Benchmarks span 50 generative models and three major datasets, demonstrating consistent improvements over prior approaches:

• GenImage (17 generators): Mean accuracy (mAcc) $=97.6\%$, mean AP (mAP) $=99.8\%$, with $+4.4\%$ mAcc and $+0.5\%$ mAP improvement over NS-Net. On DeepFake subsets, accuracy reaches $96.7\%$.
• AIGIBench (34 generators): Accuracy $=81.6\%$, an increase of $+10.1\%$ over UnivFD ($71.5\%$). Notably, accuracy on BlendFace increases by $>50\%$, where many competing methods underperform.
• UniversalFakeDetect (8 diffusion sources): mAcc $=99.0\%$, mAP $=100.0\%$, improving over the best baseline by $+1.5\%$ mAcc and eliminating failure cases on Guided and Glide.

Robustness to common image degradations is also improved. DGS-Net shows lower accuracy drops under JPEG compression (QF=75: $82.4\%$ vs. $79.6\%$ for NS-Net) and Gaussian blur ($\sigma=1.5$: $73.1\%$ vs. $70.2\%$ on AIGIBench).

6. Preservation of Pre-training Priors and Suppression of Shortcut Semantics

DGS-Net’s design enables it to avoid catastrophic forgetting typical of vanilla CLIP fine-tuning. By projecting away gradient directions associated with text-branch “harmful” shortcuts—high-level semantic features that may encode spurious dataset correlations—and selectively distilling negative-part gradients from the teacher, DGS-Net remains close to the original CLIP embedding manifold. This suggests gradual adaptation that preserves universal image priors such as frequency response and geometric coherence, while suppressing those associated with dataset-specific biases.

The net effect is improved cross-model and cross-domain generalization, since the model emphasizes forensically relevant, low-level cues over high-level semantics, crucial for robust AI-generated image detection.

7. Summary and Novel Contributions

DGS-Net introduces a principled, gradient-based approach to fine-tuning large pre-trained models like CLIP for classification of AI-generated content. Its main innovations include:

• Gradient-space decomposition into harmful and beneficial directions, enabling selective suppression and preservation.
• Linear alignment distillation of only the negative-part gradient from a frozen teacher image encoder.
• Integration of LoRA adaptation for parameter efficiency.

The result is a lightweight and effective methodology that advances the state of the art in universal detection of synthetic images, as quantitatively validated on numerous generative models and benchmarks (Yan et al., 17 Nov 2025).