Fairness-Aware Examples in Robust Distillation

• FAEs are synthetic samples defined to enforce a uniform predictive distribution on non-robust features, thereby mitigating class-specific feature dominance.
• Within the FERD framework, FAEs help distribute adversarial pressure evenly across classes, addressing variability in worst-class robustness.
• Empirical results show that incorporating FAEs enhances worst-class robustness by up to 1.1% and boosts overall fairness in robustness distillation.

Fairness-Aware Examples (FAEs) are synthetic samples introduced within the FERD framework for data-free robustness distillation. In this setting, a generator produces pseudo-data for transferring robustness from a pre-trained teacher $f^T$ to a student $f^S$ without access to the original training set. FAEs are defined by a specific regularization criterion: when the teacher classifies them using only the identified non-robust features, the resulting predictive distribution is driven toward the uniform distribution over the $C$ classes. The intended effect is to suppress the dominance of class-specific non-robust features, distribute adversarial pressure more evenly across categories, and improve worst-class robustness under attacks such as FGSM, PGD, and AutoAttack (Li et al., 25 Sep 2025).

1. Problem setting and motivating fairness criterion

FERD studies fairness in a robustness-distillation regime rather than in the protected-attribute sense used in much of the fairness literature. The paper considers data-free robustness distillation (DFRD), where pseudo-data are synthesized by a generator and used to transfer both clean and adversarial knowledge from a teacher to a student. Existing DFRD methods typically sample each class equally, yet the distilled student still exhibits wide inter-class gaps in adversarial robustness. Some categories inherit much of the teacher’s robustness, while others remain highly vulnerable. The paper identifies two key problems: the student distilled with equal class proportion data behaves significantly different across distinct categories, and the robustness of the student model is not stable across different attack targets (Li et al., 25 Sep 2025).

The central observation behind FAEs is that the non-robust subspace of features tends to concentrate on a few classes unless explicitly regularized. In the paper’s formulation, this concentration causes unfairness across categories because attackable directions are not distributed evenly. FAEs address this by forcing the non-robust feature predictions to be as uniform as possible across classes. This implies that no single class’s non-robust features should dominate the synthesized sample distribution. A plausible implication is that fairness here is operationalized as reducing disparity in robustness across categories rather than enforcing a parity criterion over demographic groups.

2. Formal definition of FAEs

Let $x \in \mathbb{R}^d$ be a synthetic sample produced by the generator $G$, and let

$Z = f^T_\ell(x)$

denote the teacher’s pre-activations at layer $\ell$. FAEs are defined through an intermediate step that identifies non-robust feature channels. The method injects isotropic Gaussian noise into $Z$: $$Z_I = f^T_\ell(x) + \text{softplus}(\lambda_r)\,\epsilon,\quad \epsilon \sim \mathcal N(0,I).$$

The noise scale $\lambda_r \in \mathbb{R}^{C_\ell}$ is fit by minimizing a variational Information-Bottleneck–style objective. After optimization, channel $f^S$0 is marked as non-robust if

$f^S$1

and the non-robust feature subset is defined by channel-wise masking: $f^S$2

The defining condition for FAEs is then imposed on the teacher’s prediction using only these non-robust features. If $f^S$3 denotes the post-non-robust-feature logits after softmax, the uniformity loss is

$f^S$4

where $f^S$5. FAEs are therefore those generator outputs $f^S$6 for which the teacher’s non-robust-feature classifier is maximally uncertain (Li et al., 25 Sep 2025).

3. Synthesis procedure within FERD

FAEs are produced as part of the generator optimization loop in FERD. A batch of noise vectors is sampled, class labels are drawn from a categorical distribution with class-sampling weights $f^S$7, and synthetic inputs $f^S$8 are generated. The teacher’s intermediate features $f^S$9 are computed, noise is injected through $C$0, and the variational Information-Bottleneck objective is used to update $C$1. The resulting non-robust mask $C$2 defines $C$3, from which the uniformity loss $C$4 is evaluated. This loss is combined with standard data-free synthesis terms and used to update the generator (Li et al., 25 Sep 2025).

The generator-side losses appearing in FERD are summarized below.

Component	Expression	Role
$C$5	$C$6	Teacher–student alignment
$C$7	$C$8	Batch-norm statistics matching
$C$9	$x \in \mathbb{R}^d$0	Label consistency
$x \in \mathbb{R}^d$1	$x \in \mathbb{R}^d$2	FAE uniformity constraint

These terms are combined as

$x \in \mathbb{R}^d$3

For CIFAR-10 experiments, the paper reports typical settings of generator Adam with $x \in \mathbb{R}^d$4, $x \in \mathbb{R}^d$5, $x \in \mathbb{R}^d$6, together with $x \in \mathbb{R}^d$7, $x \in \mathbb{R}^d$8, $x \in \mathbb{R}^d$9, $G$0, $G$1, and $G$2 iterations. After these generator updates, the output batch $G$3 is treated as a set of FAEs.

4. Mechanism and interaction with UTAEs

By construction, FAEs satisfy two properties. First, their non-robust features $G$4 produce a nearly uniform teacher prediction under the loss $G$5. The paper interprets this as suppressing class-specific non-robust shortcuts, so that no single class’s non-robust subspace can confidently predict the true label. Second, FAEs are used as the starting point for Uniform-Target Adversarial Examples (UTAEs), which apply a uniform target class constraint to avoid biased attack directions and distribute attack targets across all categories (Li et al., 25 Sep 2025).

The relationship between FAEs and UTAEs is central to FERD’s mechanism of action. FAEs regularize the benign synthetic samples so that their non-robust content is semantically balanced across classes. UTAEs then transform these balanced starting points into adversarial examples whose attack directions are not concentrated on a few vulnerable categories. When these diverse adversarial directions are used in the student’s robust distillation, the student is exposed to a broader spectrum of weak directions. This suggests that FAEs alone are not the entire fairness mechanism in FERD; rather, they are the generator-side component that prepares a more equitable substrate for subsequent adversarial synthesis.

5. Theoretical status and empirical evidence

The paper does not state a formal theorem solely about FAEs, but it places them inside two proved conjectures in the appendices. Appendix A shows that re-weighting sample proportions based on empirical adversarial margins provably reduces the variance of class-wise robustness under mild smoothness assumptions on the teacher. Appendix B shows that distributing adversarial targets uniformly maximizes a lower bound on the student’s worst-case robust risk. In this structure, FAEs support the second component by making uniform attack targeting more effective (Li et al., 25 Sep 2025).

The empirical effect of FAEs is isolated by ablations on CIFAR-10 with WideResNet-34-10 $G$6 ResNet-18.

Setting	PGD-worst
Full FERD	38.60%
w/o FAEs (but with reweight + UTAEs)	37.50%
w/o FAEs & UTAEs	36.20%

In this ablation, FAEs alone improve worst-class PGD robustness by roughly $G$7 percentage points, and the combination of FAEs with UTAEs yields the full $G$8 percentage-point gain. In the main CIFAR-10 results, FERD reports worst-class robustness of $G$9 under FGSM versus $Z = f^T_\ell(x)$0 for DFHL, $Z = f^T_\ell(x)$1 under PGD versus $Z = f^T_\ell(x)$2, and $Z = f^T_\ell(x)$3 under AutoAttack versus $Z = f^T_\ell(x)$4. The normalized standard deviation (NSD) of robustness across classes is reported as $Z = f^T_\ell(x)$5 for FGSM and $Z = f^T_\ell(x)$6 for PGD, the lowest among the benchmarks listed in the summary. The paper also states that visualizations in Figure 1 show FAEs with more diverse, semantically balanced patterns than competing data-free generators.

6. Conceptual scope, interpretation, and common confusions

A common source of confusion is the meaning of fairness in FAEs. In FERD, the relevant disparity is across categories or classes, measured through robustness gaps under adversarial attacks. This differs from fairness formulations that are defined over protected variables such as race, gender, or age. For example, FairOD formalizes fairness-aware outlier detection through desiderata including treatment parity, statistical parity, and within-group utility, with protected labels $Z = f^T_\ell(x)$7 entering the fairness constraints during training but not at test time (Shekhar et al., 2020).

This comparison clarifies what FAEs are and are not. FAEs are not examples designed to equalize protected-group flag rates, nor are they framed as a demographic fairness intervention. They are synthetic examples designed to enforce a uniform predictive distribution on non-robust feature subsets, so that robustness distillation does not overfit to a few categories. The shared phrase “fairness-aware” therefore spans distinct technical agendas in the literature: FERD addresses class-wise robustness disparity in data-free distillation, whereas FairOD addresses unjust outcomes and disparate impact in outlier detection over protected groups. A plausible implication is that the term “fairness-aware” has become methodology-dependent, and careful identification of the protected object—categories, groups, or both—is necessary when interpreting claims.

7. Significance and limitations

Within FERD, FAEs are characterized as a lightweight, generator-side regularization. Their role is to suppress class-specific non-robust shortcuts by uniformizing feature-space predictions and to seed more equitable adversarial directions for downstream robustness distillation. The reported experimental results indicate that this intervention materially improves worst-class robustness while reducing variability across categories (Li et al., 25 Sep 2025).

The available evidence also bounds the scope of the concept. The strongest claims in the paper concern worst-class robustness, class-wise fairness, and compatibility with UTAEs and robustness-guided class reweighting. The paper does not present a theorem solely about FAEs, and its formulation is embedded in the larger FERD framework rather than as a standalone method. This suggests that FAEs are best understood as one component in a composite data-free distillation pipeline: they regularize the generator, shape the distribution of non-robust features, and enable more balanced adversarial targeting, but their full effect is realized in conjunction with the other elements of FERD.