Dice Adversarial Robustness Distillation (DARD)

• Dice Adversarial Robustness Distillation (DARD) is a method that transfers robustness from a large adversarially trained teacher to a compact student via a Dice-mixed target combining clean and adversarial predictions.
• It fuses supervised adversarial training with knowledge distillation by using a balanced soft target, enhancing both natural and robust feature learning in image classification tasks.
• Experimental results on CIFAR-10 and CIFAR-100 show that DARD boosts robust accuracy and improves the clean–robust trade-off compared to standard adversarial defenses.

Dice Adversarial Robustness Distillation (DARD) is a teacher–student adversarial robustness method for supervised classification that distills robustness from a larger, adversarially trained teacher into a compact student by combining adversarial training with knowledge distillation on clean and adversarial teacher outputs (Zou et al., 15 Sep 2025). In the formulation reported for CIFAR-10 and CIFAR-100, DARD trains the student on adversarial examples while matching a Dice-mixed soft target formed by averaging the teacher’s softened predictions on clean and adversarial inputs. The same work also introduces Dice Projected Gradient Descent (DPGD), an iterative attack that uses dynamic Dice-style weighting during adversarial example generation. A separate 2024 paper uses the acronym DARD for “Demonstration Augmentation Retrieval Defences” in retrieval-based in-context learning; that method is unrelated to distillation and addresses a different threat model (Yu et al., 2024).

1. Nomenclature and scope

In the 2025 usage, DARD denotes “Dice Adversarial Robustness Distillation against Adversarial Attacks,” a method situated in adversarially robust knowledge distillation for image classification (Zou et al., 15 Sep 2025). The paper considers a $C$-class supervised classification problem over inputs $x \in [0,1]^d$ and labels $y \in \{1,\dots,C\}$, with a teacher model $f_T$ and a student model $f_S$. The teacher maps inputs to logits $z_T(x) \in \mathbb{R}^C$ and softened probabilities

$p_T^{(\tau)}(x)=\mathrm{softmax}(z_T(x)/\tau),$

while the student analogously produces

$p_S^{(\tau)}(x)=\mathrm{softmax}(z_S(x)/\tau).$

The threat model is norm-bounded adversarial perturbation, with emphasis on the $\ell_\infty$ setting $\|\delta\|_\infty \le \epsilon$, using typical $x \in [0,1]^d$0, iterative attack length $x \in [0,1]^d$1, and step size $x \in [0,1]^d$2 (Zou et al., 15 Sep 2025). The adversarial example is denoted $x \in [0,1]^d$3, and perturbations are constrained by projection $x \in [0,1]^d$4 and clipping to $x \in [0,1]^d$5.

A common misconception arises from acronym reuse. In the 2024 literature on retrieval-based in-context learning, DARD explicitly means “Demonstration Augmentation Retrieval Defences,” a training-free defence based on enriching a retrieval pool with attacked samples rather than distilling robustness between neural classifiers (Yu et al., 2024). The two methods share only the acronym.

2. Core objective and Dice-mixed distillation

The central claim of DARD is that robustness can be distilled from a larger, adversarially trained teacher into a smaller student by combining supervised adversarial training with distillation to teacher soft labels (Zou et al., 15 Sep 2025). The paper states that larger models exhibit enhanced robustness over smaller models and that this robustness can be systematically transferred.

The distinctive component is the Dice-based mixture of teacher distributions. Let $x \in [0,1]^d$6 be the teacher’s softened output on clean input $x \in [0,1]^d$7, and $x \in [0,1]^d$8 its softened output on the corresponding adversarial example. DARD forms the mixed target

$x \in [0,1]^d$9

with $y \in \{1,\dots,C\}$0 in the reported experiments (Zou et al., 15 Sep 2025). The paper characterizes this as the “Dice” component: a balanced soft target that averages clean and adversarial teacher signals.

The supervised term is cross-entropy on adversarial inputs,

$y \in \{1,\dots,C\}$1

The distillation term is a KL divergence from the mixed teacher target to the student on the same adversarial input,

$y \in \{1,\dots,C\}$2

The full DARD objective is

$y \in \{1,\dots,C\}$3

where $y \in \{1,\dots,C\}$4 balances adversarial cross-entropy and distillation, and the $y \in \{1,\dots,C\}$5 factor follows standard knowledge-distillation practice (Zou et al., 15 Sep 2025).

This construction suggests that DARD is not merely adversarial training with a teacher regularizer. The method specifically uses a mixed target intended to preserve information from both natural and adversarial regimes, rather than relying exclusively on one or the other.

3. Variants and relation to prior robust training

The paper’s main DARD variant uses adversarial examples exclusively during student training (Zou et al., 15 Sep 2025). A related variant, TDARD, averages clean and adversarial objectives:

$y \in \{1,\dots,C\}$6

Within the experimental comparison, DARD is evaluated against Natural training, SAT, ARD, and TDARD (Zou et al., 15 Sep 2025). The paper also reports an ablation in which DPGD is replaced by standard PGD; this variant is called “PGDARD,” and it reduces robustness relative to DARD. Another ablation uses only teacher adversarial soft labels; the paper refers to it as “OnlyadvlaARD,” and reports that it underperforms DARD. According to the reported interpretation, mixing clean and adversarial soft labels with $y \in \{1,\dots,C\}$7 yields more balanced and higher robustness and convergence.

The paper further argues that pure adversarial training often sacrifices clean accuracy, whereas DARD addresses this trade-off by distilling from a robust, well-calibrated teacher, combining clean and adversarial soft labels, and employing DPGD to generate stronger training attacks (Zou et al., 15 Sep 2025). Formal guarantees are not provided; the justification is empirical and framed in terms of margin transfer, representation alignment via knowledge distillation, and curriculum-like attack generation.

4. Dice Projected Gradient Descent (DPGD)

DPGD generalizes standard PGD by introducing a Dice-style dynamic weighting in the attack loss used to craft adversarial examples (Zou et al., 15 Sep 2025). At attack step $y \in \{1,\dots,C\}$8, the batch is partitioned into correctly classified and misclassified subsets:

• $y \in \{1,\dots,C\}$9,
• $f_T$0.

The dynamic Dice weight is

$f_T$1

Using this weight, the DPGD mini-batch loss is

$f_T$2

For the $f_T$3 setting, the update is

$f_T$4

followed by element-wise clipping to $f_T$5 (Zou et al., 15 Sep 2025).

The reported motivation is that early iterations emphasize correctly classified samples, while later iterations shift weight toward misclassified ones, thereby avoiding saturation and strengthening attack efficacy. The paper also mentions dynamic step-size tuning and channel-wise gradient masking as heuristics inherited from segmentation contexts, but states that the core improvements in the provided experiments come from the dynamic Dice weighting together with standard PGD hyperparameters.

The training loop is correspondingly structured. For each mini-batch, adversarial examples are generated for the student via DPGD; the teacher is then evaluated on both clean and adversarial inputs; the Dice-mixed teacher target is formed with $f_T$6; and the student is updated by SGD using the DARD loss (Zou et al., 15 Sep 2025). No $f_T$7 ramp-up is used in the reported experiments.

5. Experimental configuration and empirical results

The reported experiments use CIFAR-10 and CIFAR-100, with a ResNet-56 teacher and a ResNet-18 student (Zou et al., 15 Sep 2025). The student is optimized with SGD, learning rate $f_T$8, momentum $f_T$9, and weight decay $f_S$0. Adversarial example generation for both training and evaluation uses the $f_S$1 setting with $f_S$2, $f_S$3, and $f_S$4. The evaluated attacks are FGSM, PGD-20, T-PGD, BIM, and AutoAttack.

On CIFAR-10, the student ResNet-18 under DARD attains Clean $f_S$5, FGSM $f_S$6, PGD20 $f_S$7, T-PGD $f_S$8, BIM $f_S$9, and AutoAttack $z_T(x) \in \mathbb{R}^C$0 (Zou et al., 15 Sep 2025). The corresponding values for SAT are Clean $z_T(x) \in \mathbb{R}^C$1, FGSM $z_T(x) \in \mathbb{R}^C$2, PGD20 $z_T(x) \in \mathbb{R}^C$3, T-PGD $z_T(x) \in \mathbb{R}^C$4, BIM $z_T(x) \in \mathbb{R}^C$5, and AutoAttack $z_T(x) \in \mathbb{R}^C$6; for ARD, Clean $z_T(x) \in \mathbb{R}^C$7, FGSM $z_T(x) \in \mathbb{R}^C$8, PGD20 $z_T(x) \in \mathbb{R}^C$9, T-PGD $p_T^{(\tau)}(x)=\mathrm{softmax}(z_T(x)/\tau),$0, BIM $p_T^{(\tau)}(x)=\mathrm{softmax}(z_T(x)/\tau),$1, and AutoAttack $p_T^{(\tau)}(x)=\mathrm{softmax}(z_T(x)/\tau),$2; for TDARD, Clean $p_T^{(\tau)}(x)=\mathrm{softmax}(z_T(x)/\tau),$3, FGSM $p_T^{(\tau)}(x)=\mathrm{softmax}(z_T(x)/\tau),$4, PGD20 $p_T^{(\tau)}(x)=\mathrm{softmax}(z_T(x)/\tau),$5, T-PGD $p_T^{(\tau)}(x)=\mathrm{softmax}(z_T(x)/\tau),$6, BIM $p_T^{(\tau)}(x)=\mathrm{softmax}(z_T(x)/\tau),$7, and AutoAttack $p_T^{(\tau)}(x)=\mathrm{softmax}(z_T(x)/\tau),$8. Natural training collapses under iterative attacks, with PGD20, BIM, and AutoAttack all reported as $p_T^{(\tau)}(x)=\mathrm{softmax}(z_T(x)/\tau),$9.

On CIFAR-100, DARD attains Clean $p_S^{(\tau)}(x)=\mathrm{softmax}(z_S(x)/\tau).$0, FGSM $p_S^{(\tau)}(x)=\mathrm{softmax}(z_S(x)/\tau).$1, PGD20 $p_S^{(\tau)}(x)=\mathrm{softmax}(z_S(x)/\tau).$2, T-PGD $p_S^{(\tau)}(x)=\mathrm{softmax}(z_S(x)/\tau).$3, BIM $p_S^{(\tau)}(x)=\mathrm{softmax}(z_S(x)/\tau).$4, and AutoAttack $p_S^{(\tau)}(x)=\mathrm{softmax}(z_S(x)/\tau).$5 (Zou et al., 15 Sep 2025). SAT reports Clean $p_S^{(\tau)}(x)=\mathrm{softmax}(z_S(x)/\tau).$6, FGSM $p_S^{(\tau)}(x)=\mathrm{softmax}(z_S(x)/\tau).$7, PGD20 $p_S^{(\tau)}(x)=\mathrm{softmax}(z_S(x)/\tau).$8, T-PGD $p_S^{(\tau)}(x)=\mathrm{softmax}(z_S(x)/\tau).$9, BIM $\ell_\infty$0, and AutoAttack $\ell_\infty$1; ARD reports Clean $\ell_\infty$2, FGSM $\ell_\infty$3, PGD20 $\ell_\infty$4, T-PGD $\ell_\infty$5, BIM $\ell_\infty$6, and AutoAttack $\ell_\infty$7; TDARD reports Clean $\ell_\infty$8, FGSM $\ell_\infty$9, PGD20 $\|\delta\|_\infty \le \epsilon$0, T-PGD $\|\delta\|_\infty \le \epsilon$1, BIM $\|\delta\|_\infty \le \epsilon$2, and AutoAttack $\|\delta\|_\infty \le \epsilon$3. Natural training again shows near-zero robustness under stronger attacks.

The paper’s stated observations are specific. On CIFAR-10, DARD achieves the highest robust accuracy under PGD-20 ($\|\delta\|_\infty \le \epsilon$4) and AutoAttack ($\|\delta\|_\infty \le \epsilon$5) among the compared defences, with competitive clean accuracy (Zou et al., 15 Sep 2025). On CIFAR-100, DARD improves robust accuracy across all attacks relative to SAT and ARD, with clean accuracy above SAT and comparable to ARD.

The trade-off analysis on CIFAR-10 uses Weighted Robust Accuracy (W-Robust). Under AutoAttack, DARD reports $\|\delta\|_\infty \le \epsilon$6 versus SAT $\|\delta\|_\infty \le \epsilon$7 and ARD $\|\delta\|_\infty \le \epsilon$8; under PGD20, DARD reports $\|\delta\|_\infty \le \epsilon$9 versus SAT $x \in [0,1]^d$00 and ARD $x \in [0,1]^d$01; under FGSM, DARD reports $x \in [0,1]^d$02 versus SAT $x \in [0,1]^d$03 and ARD $x \in [0,1]^d$04 (Zou et al., 15 Sep 2025). This suggests that the method is optimized not only for worst-case robustness but also for the clean–robust trade-off emphasized by the paper.

6. Interpretation, limitations, and future directions

The paper interprets DARD as a mechanism for transferring smoother decision boundaries and larger logit margins from large, adversarially trained teachers to smaller students (Zou et al., 15 Sep 2025). Distilling a balanced mixture of clean and adversarial teacher outputs is argued to help the student preserve natural-data structure while learning robust features. A plausible implication is that the method’s benefit depends strongly on teacher quality: the reported guidance explicitly states that the better and more robust the teacher, the more effective the distillation.

Several limitations are explicit. The reported experiments use the $x \in [0,1]^d$05 threat model with fixed $x \in [0,1]^d$06, so extension to $x \in [0,1]^d$07, $x \in [0,1]^d$08, or physical-world constraints requires tuning (Zou et al., 15 Sep 2025). Performance depends on the trade-off coefficient $x \in [0,1]^d$09, temperature $x \in [0,1]^d$10, the Dice mixture weight $x \in [0,1]^d$11, and the DPGD schedule $x \in [0,1]^d$12; mis-tuning may degrade either clean or robust accuracy. DARD also incurs non-trivial compute overhead, because it requires iterative adversarial example generation per mini-batch and teacher forward passes on both clean and adversarial inputs. Results are reported on CIFAR-10 and CIFAR-100 only, and the paper identifies scaling to ImageNet and other modalities as a natural next step.

The paper also identifies several extensions: integration with certified robustness such as randomized smoothing, alternative Dice formulations including class-wise or margin-aware mixtures, teacher ensembles, and adaptive schedules for $x \in [0,1]^d$13 across epochs (Zou et al., 15 Sep 2025). These directions remain prospective rather than established. More generally, the absence of formal guarantees means that DARD presently stands as an empirically motivated robust distillation framework rather than a certified defence.

In the current literature, DARD therefore denotes a specific robust knowledge-distillation paradigm: adversarial-only student training combined with a Dice-mixed teacher target and strengthened by DPGD attack generation (Zou et al., 15 Sep 2025). Its reported contribution is a consistent empirical improvement in robustness over adversarially trained networks with the same student architecture, while maintaining competitive standard accuracy.