FERD: Fairness-Enhanced Robustness Distillation

• The paper demonstrates that fairness-enhanced distillation effectively transfers adversarial robustness from a teacher to a student without needing the original training data.
• It introduces a robustness-guided class reweighting and Fairness-Aware Example generation to address per-class robustness disparities by adjusting synthetic sample distributions.
• Uniform-target adversarial example construction diversifies attack directions, ensuring balanced defense and improved worst-class robustness across benchmark datasets.

Searching arXiv for the specified paper to ground the article in the current record. Fairness-Enhanced Data-Free Robustness Distillation (FERD) is a framework for Data-Free Robustness Distillation (DFRD) that aims to transfer adversarial robustness from a robust teacher model to a lightweight student without access to the teacher’s original training data, while explicitly addressing robust fairness across categories (Li et al., 25 Sep 2025). In DFRD, a generator synthesizes samples for distillation; FERD modifies this paradigm by adjusting both the proportion and the distribution of adversarial examples. It is presented as the first fairness-enhanced data-free robustness distillation framework, motivated by two observed problems in existing methods: severe disparity of robustness across classes and instability of the student’s robustness across attack targets (Li et al., 25 Sep 2025).

1. Problem formulation and motivating failure modes

In the FERD setting, the goal is to transfer adversarial robustness from a teacher $f^T$ to a student $f^S$ without any access to the training data. The basic DFRD approach is to train a generator $G$ to synthesize “fake” samples $\{(x_i,y_i)\}$, and then use both clean and adversarially perturbed versions of these samples for distillation (Li et al., 25 Sep 2025).

FERD is motivated by two fairness-related deficiencies in existing DFRD. First, even when synthetic samples are drawn with equal class proportions, the student’s per-class robustness under attack often shows large gaps. Some classes rapidly inherit robustness, whereas others remain comparatively vulnerable. Second, standard adversarial-example generation tends to push samples toward the most confusable target class, so the student can become specialized in defending against only a small subset of target directions. The reported consequence is poor defense when the attacker chooses other target classes (Li et al., 25 Sep 2025).

The framework therefore addresses fairness in two distinct senses. One concerns disparity across source categories, where some classes receive weaker robustness transfer than others. The other concerns imbalance across attack targets, where the induced adversarial distribution is concentrated on a narrow set of vulnerable directions. This suggests that FERD treats robust fairness not merely as class-balanced sampling, but as a joint problem of sample allocation and attack-direction diversity.

2. Robustness-guided class reweighting

FERD’s first component is a robustness-guided class reweighting strategy designed to allocate more synthetic samples to weakly robust categories. The procedure begins by generating adversarial examples $x_i^{adv}$ from each synthetic sample $x_i$, for example with PGD-20. For each such example, the teacher’s adversarial margin is computed as

$$m_i \;=\;\bigl(f^T(x_i^{adv})\bigr)_{\,y_i}\;-\;\max_{j\neq y_i}\bigl(f^T(x_i^{adv})\bigr)_j.$$

A smaller, or negative, margin indicates greater vulnerability. Class-level vulnerability is then defined by averaging the negative margins within each class:

$D_c \;=\;\frac{1}{N_c}\sum_{i:\,y_i=c}(-m_i),$

where $N_c$ is the number of synthesized samples with label $c$. These vulnerability scores are transformed into a sampling distribution through a softmax,

$f^S$0

and subsequent generator updates sample labels from $f^S$1 rather than from a uniform distribution (Li et al., 25 Sep 2025).

The operational effect is that $f^S$2 synthesizes more samples for classes with high vulnerability $f^S$3. In the formulation given for FERD, this component directly targets the observed disparity in per-class robustness. The paper further states that reweighting compensates weak classes by supplying more training samples and thereby closes the inter-class robustness gap, empirically reducing the normalized standard deviation of per-class robustness (Li et al., 25 Sep 2025).

3. Fairness-Aware Example generation

FERD’s second component is Fairness-Aware Example (FAE) generation. Its stated goal is to produce benign samples whose non-robust features do not strongly bias toward any single class. The construction begins by identifying non-robust feature channels using an information-bottleneck style scheme at an intermediate layer $f^S$4 of the teacher network (Li et al., 25 Sep 2025).

An intermediate representation is extracted as $f^S$5. Gaussian noise scaled by a learnable vector $f^S$6 is then injected:

$f^S$7

The vector $f^S$8 is optimized to preserve predictiveness of $f^S$9 for $G$0 under the teacher while limiting information flow:

$G$1

where $G$2 is the variance of channel $G$3. A channel $G$4 is marked as non-robust if

$G$5

and the non-robust feature $G$6 is then formed by channel-wise masking (Li et al., 25 Sep 2025).

FERD next imposes a uniformity constraint on the teacher’s output when only the non-robust features are provided:

$G$7

where $G$8 denotes the uniform distribution over the $G$9 classes. The full generator loss for synthesizing FAEs $\{(x_i,y_i)\}$0 is

$\{(x_i,y_i)\}$1

Here, $\{(x_i,y_i)\}$2 matches BatchNorm statistics of teacher and student, while the $\{(x_i,y_i)\}$3 and KL terms enforce diversity and correctness (Li et al., 25 Sep 2025).

The stated interpretation is that FAEs suppress the dominance of class-specific non-robust features and provide a more balanced representation across all categories. A plausible implication is that FERD reorients the generator away from artifacts that are predictive yet unevenly distributed across classes, which in turn is intended to improve worst-class robustness rather than only average robustness.

4. Uniform-Target Adversarial Example construction

FERD’s third component is the construction of Uniform-Target Adversarial Examples (UTAEs). The purpose is to diversify attack targets so that the student learns to defend against perturbations pushing toward every class equally, rather than predominantly toward the most vulnerable or most confusable classes (Li et al., 25 Sep 2025).

Starting from a generated FAE $\{(x_i,y_i)\}$4, FERD applies a modified PGD procedure:

$\{(x_i,y_i)\}$5

where $\{(x_i,y_i)\}$6 projects back into the $\{(x_i,y_i)\}$7 ball $\{(x_i,y_i)\}$8. In the description of the method, the first KL term pulls the adversarial example away from the original FAE label, while the second term, weighted by $\{(x_i,y_i)\}$9, pushes the attack toward a uniform target distribution. Repeating the update for $x_i^{adv}$0 steps yields the UTAE $x_i^{adv}$1 (Li et al., 25 Sep 2025).

The student is then distilled with both FAEs and UTAEs according to

$x_i^{adv}$2

In the paper’s own justification, UTAE diversification ensures that the student is exposed to adversarial shifts toward all possible target classes, thereby reducing vulnerability to any single target. This indicates that FERD treats robustness transfer as sensitive not only to perturbation magnitude and source label, but also to the geometry of target-class allocation in adversarial space (Li et al., 25 Sep 2025).

5. End-to-end algorithmic structure and rationale

The overall FERD algorithm takes as input a teacher $x_i^{adv}$3, a student $x_i^{adv}$4, a generator $x_i^{adv}$5, and a number of epochs $x_i^{adv}$6, together with the hyperparameters $x_i^{adv}$7. For each epoch, the procedure is described as follows: sample a batch of class labels according to the reweighting distribution $x_i^{adv}$8; generate benign FAEs $x_i^{adv}$9; update $x_i$0 and compute $x_i$1; update the generator by minimizing $x_i$2; generate UTAEs from each FAE via the uniform-target PGD; compute $x_i$3 and update the student; and periodically re-estimate margins $x_i$4 and recompute $x_i$5 (Li et al., 25 Sep 2025).

The rationale offered for this design is explicit. Reweighting addresses weak classes by increasing their synthetic-sample allocation. FAEs suppress class-specific non-robust features, preventing the generator from over-emphasizing a few easy classes. UTAE diversification broadens the set of target directions encountered during robustness distillation. The paper additionally reports empirical bounds in Appendix A and B indicating that both uniform class sampling and uniform target distribution tighten the worst-case generalization bound under adversarial shifts (Li et al., 25 Sep 2025).

Taken together, these components define a coupled mechanism in which class-level vulnerability estimates influence sample generation, feature-level uniformity reshapes benign synthetic data, and target-level uniformity reshapes adversarial synthetic data. This suggests that FERD frames fairness-enhanced robustness distillation as a multi-level balancing problem spanning class frequency, feature salience, and attack-direction coverage.

6. Experimental setting, reported results, and implementation details

FERD is evaluated on three public datasets: CIFAR-10, CIFAR-100, and Tiny-ImageNet. The teacher architectures are WideResNet-34-10 for CIFAR-10 and CIFAR-100, and PreActResNet-34 for Tiny-ImageNet. The student architectures are ResNet-18 and MobileNet-V2 for CIFAR, and PreActResNet-18 for Tiny-ImageNet. The attacks used for evaluation are FGSM, PGD-20, $x_i$6, and AutoAttack (AA) (Li et al., 25 Sep 2025).

The paper reports that FERD achieves state-of-the-art worst-class robustness under all adversarial attack settings. For MobileNet-V2 on CIFAR-10, the key quantitative gains in worst-class robustness are reported as $x_i$7 under FGSM, $x_i$8 under PGD-20, $x_i$9 under $$m_i \;=\;\bigl(f^T(x_i^{adv})\bigr)_{\,y_i}\;-\;\max_{j\neq y_i}\bigl(f^T(x_i^{adv})\bigr)_j.$$0, and $$m_i \;=\;\bigl(f^T(x_i^{adv})\bigr)_{\,y_i}\;-\;\max_{j\neq y_i}\bigl(f^T(x_i^{adv})\bigr)_j.$$1 under AutoAttack. It is also stated that FERD yields the lowest Normalized Standard Deviation (NSD) in most cases, which is presented as signifying higher robust fairness (Li et al., 25 Sep 2025).

The implementation details provided are specific. The generator is optimized with Adam using learning rate $$m_i \;=\;\bigl(f^T(x_i^{adv})\bigr)_{\,y_i}\;-\;\max_{j\neq y_i}\bigl(f^T(x_i^{adv})\bigr)_j.$$2, $$m_i \;=\;\bigl(f^T(x_i^{adv})\bigr)_{\,y_i}\;-\;\max_{j\neq y_i}\bigl(f^T(x_i^{adv})\bigr)_j.$$3, and $$m_i \;=\;\bigl(f^T(x_i^{adv})\bigr)_{\,y_i}\;-\;\max_{j\neq y_i}\bigl(f^T(x_i^{adv})\bigr)_j.$$4. The student is optimized with SGD using initial learning rate $$m_i \;=\;\bigl(f^T(x_i^{adv})\bigr)_{\,y_i}\;-\;\max_{j\neq y_i}\bigl(f^T(x_i^{adv})\bigr)_j.$$5, momentum $$m_i \;=\;\bigl(f^T(x_i^{adv})\bigr)_{\,y_i}\;-\;\max_{j\neq y_i}\bigl(f^T(x_i^{adv})\bigr)_j.$$6, and weight decay $$m_i \;=\;\bigl(f^T(x_i^{adv})\bigr)_{\,y_i}\;-\;\max_{j\neq y_i}\bigl(f^T(x_i^{adv})\bigr)_j.$$7. Distillation runs for 220 epochs, with 400 iterations per epoch for both $$m_i \;=\;\bigl(f^T(x_i^{adv})\bigr)_{\,y_i}\;-\;\max_{j\neq y_i}\bigl(f^T(x_i^{adv})\bigr)_j.$$8 and $$m_i \;=\;\bigl(f^T(x_i^{adv})\bigr)_{\,y_i}\;-\;\max_{j\neq y_i}\bigl(f^T(x_i^{adv})\bigr)_j.$$9. The batch size is 256 on CIFAR-10 and 512 on CIFAR-100 and Tiny-ImageNet. The feature extraction layer $D_c \;=\;\frac{1}{N_c}\sum_{i:\,y_i=c}(-m_i),$0 is the last convolutional block of $D_c \;=\;\frac{1}{N_c}\sum_{i:\,y_i=c}(-m_i),$1. The reported hyperparameters are $D_c \;=\;\frac{1}{N_c}\sum_{i:\,y_i=c}(-m_i),$2, $D_c \;=\;\frac{1}{N_c}\sum_{i:\,y_i=c}(-m_i),$3, $D_c \;=\;\frac{1}{N_c}\sum_{i:\,y_i=c}(-m_i),$4, $D_c \;=\;\frac{1}{N_c}\sum_{i:\,y_i=c}(-m_i),$5, $D_c \;=\;\frac{1}{N_c}\sum_{i:\,y_i=c}(-m_i),$6, $D_c \;=\;\frac{1}{N_c}\sum_{i:\,y_i=c}(-m_i),$7, and $D_c \;=\;\frac{1}{N_c}\sum_{i:\,y_i=c}(-m_i),$8 (Li et al., 25 Sep 2025).

A common misconception in robustness distillation is that balanced class counts alone suffice to ensure balanced robustness outcomes. FERD is explicitly motivated by the observation that equal class proportion data can still produce substantial differences across categories, and that target imbalance during attack generation constitutes a separate failure mode. Within the scope of the reported experiments, the framework is therefore positioned not merely as a stronger DFRD variant, but as a method that reformulates robustness transfer in terms of worst-class robustness and robust fairness rather than overall robustness alone (Li et al., 25 Sep 2025).