---
title: Dice Adversarial Robustness Distillation (DARD)
url: https://www.emergentmind.com/topics/dice-adversarial-robustness-distillation-dard
type: topic
---

# Dice Adversarial Robustness Distillation (DARD)

Dice Adversarial Robustness Distillation (DARD) is a teacher–student adversarial robustness method for supervised classification that distills robustness from a larger, adversarially trained teacher into a compact student by combining adversarial training with knowledge distillation on clean and adversarial teacher outputs [2509.11525]. In the formulation reported for CIFAR-10 and CIFAR-100, DARD trains the student on adversarial examples while matching a Dice-mixed soft target formed by averaging the teacher’s softened predictions on clean and adversarial inputs. The same work also introduces Dice Projected Gradient Descent (DPGD), an iterative attack that uses dynamic Dice-style weighting during adversarial example generation. A separate 2024 paper uses the acronym DARD for “Demonstration Augmentation Retrieval Defences” in retrieval-based in-context learning; that method is unrelated to distillation and addresses a different threat model [2405.15984].

## 1. Nomenclature and scope

In the 2025 usage, DARD denotes “Dice Adversarial Robustness Distillation against Adversarial Attacks,” a method situated in adversarially robust knowledge distillation for image classification [2509.11525]. The paper considers a $C$-class supervised classification problem over inputs $x \in [0,1]^d$ and labels $y \in \{1,\dots,C\}$, with a teacher model $f_T$ and a student model $f_S$. The teacher maps inputs to logits $z_T(x) \in \mathbb{R}^C$ and softened probabilities
$$
p_T^{(\tau)}(x)=\mathrm{softmax}(z_T(x)/\tau),
$$
while the student analogously produces
$$
p_S^{(\tau)}(x)=\mathrm{softmax}(z_S(x)/\tau).
$$

The threat model is norm-bounded adversarial perturbation, with emphasis on the $\ell_\infty$ setting $\|\delta\|_\infty \le \epsilon$, using typical $\epsilon=8/255$, iterative attack length $K=20$, and step size $\alpha=2/255$ [2509.11525]. The adversarial example is denoted $x_{\mathrm{adv}}$, and perturbations are constrained by projection $\Pi_{B_\epsilon(x)}$ and clipping to $[0,1]^d$.

A common misconception arises from acronym reuse. In the 2024 literature on retrieval-based in-context learning, DARD explicitly means “Demonstration Augmentation Retrieval Defences,” a training-free defence based on enriching a retrieval pool with attacked samples rather than distilling robustness between neural classifiers [2405.15984]. The two methods share only the acronym.

## 2. Core objective and Dice-mixed distillation

The central claim of DARD is that robustness can be distilled from a larger, adversarially trained teacher into a smaller student by combining supervised adversarial training with distillation to teacher soft labels [2509.11525]. The paper states that larger models exhibit enhanced robustness over smaller models and that this robustness can be systematically transferred.

The distinctive component is the Dice-based mixture of teacher distributions. Let $p_T^{(\tau)}(x)$ be the teacher’s softened output on clean input $x$, and $p_T^{(\tau)}(x_{\mathrm{adv}})$ its softened output on the corresponding adversarial example. DARD forms the mixed target
$$
\tilde p_T^{(\tau)}(x)=(1-\lambda_{KD})\,p_T^{(\tau)}(x)+\lambda_{KD}\,p_T^{(\tau)}(x_{\mathrm{adv}}),
$$
with $\lambda_{KD}=0.5$ in the reported experiments [2509.11525]. The paper characterizes this as the “Dice” component: a balanced soft target that averages clean and adversarial teacher signals.

The supervised term is cross-entropy on adversarial inputs,
$$
L_{CE}^{adv}(x,y;f_S)=CE(p_S(x_{\mathrm{adv}}),y)=-\log p_S(x_{\mathrm{adv}})_y.
$$
The distillation term is a KL divergence from the mixed teacher target to the student on the same adversarial input,
$$
L_{KD}^{adv}(x;f_S,f_T,\tau)=KL(\tilde p_T^{(\tau)}(x)\,\|\,p_S^{(\tau)}(x_{\mathrm{adv}})).
$$
The full DARD objective is
$$
L_{DARD}(x,y;f_S,f_T)=(1-\alpha)\cdot CE(p_S(x_{\mathrm{adv}}),y)+\alpha\cdot\tau^2\cdot KL(\tilde p_T^{(\tau)}(x)\,\|\,p_S^{(\tau)}(x_{\mathrm{adv}})),
$$
where $\alpha \in [0,1]$ balances adversarial cross-entropy and distillation, and the $\tau^2$ factor follows standard knowledge-distillation practice [2509.11525].

This construction suggests that DARD is not merely adversarial training with a teacher regularizer. The method specifically uses a mixed target intended to preserve information from both natural and adversarial regimes, rather than relying exclusively on one or the other.

## 3. Variants and relation to prior robust training

The paper’s main DARD variant uses adversarial examples exclusively during student training [2509.11525]. A related variant, TDARD, averages clean and adversarial objectives:
$$
L_{TDARD}(x,y;f_S,f_T)= (1-\alpha)\cdot\frac{CE(p_S(x),y)+CE(p_S(x_{\mathrm{adv}}),y)}{2}
+\alpha\cdot\tau^2\cdot\frac{KL(p_T^{(\tau)}(x)\,\|\,p_S^{(\tau)}(x))+KL(p_T^{(\tau)}(x_{\mathrm{adv}})\,\|\,p_S^{(\tau)}(x_{\mathrm{adv}}))}{2}.
$$

Within the experimental comparison, DARD is evaluated against Natural training, SAT, ARD, and TDARD [2509.11525]. The paper also reports an ablation in which DPGD is replaced by standard PGD; this variant is called “PGDARD,” and it reduces robustness relative to DARD. Another ablation uses only teacher adversarial soft labels; the paper refers to it as “OnlyadvlaARD,” and reports that it underperforms DARD. According to the reported interpretation, mixing clean and adversarial soft labels with $\lambda_{KD}=0.5$ yields more balanced and higher robustness and convergence.

The paper further argues that pure adversarial training often sacrifices clean accuracy, whereas DARD addresses this trade-off by distilling from a robust, well-calibrated teacher, combining clean and adversarial soft labels, and employing DPGD to generate stronger training attacks [2509.11525]. Formal guarantees are not provided; the justification is empirical and framed in terms of margin transfer, representation alignment via knowledge distillation, and curriculum-like attack generation.

## 4. Dice Projected Gradient Descent (DPGD)

DPGD generalizes standard PGD by introducing a Dice-style dynamic weighting in the attack loss used to craft adversarial examples [2509.11525]. At attack step $t$, the batch is partitioned into correctly classified and misclassified subsets:
- $S_{\mathrm{corr}}^{(t)}=\{i:\arg\max_c p_S(x_i^{(t)})_c=y_i\}$,
- $S_{\mathrm{wrong}}^{(t)}=\{i:\arg\max_c p_S(x_i^{(t)})_c\neq y_i\}$.

The dynamic Dice weight is
$$
\lambda(t)=\frac{t-1}{2K}.
$$
Using this weight, the DPGD mini-batch loss is
$$
L_{DPGD}^{(t)}=
(1-\lambda(t))\cdot \frac{1}{|S_{\mathrm{corr}}^{(t)}|}\sum_{i\in S_{\mathrm{corr}}^{(t)}} CE(p_S(x_i^{(t)}),y_i)
+\lambda(t)\cdot \frac{1}{|S_{\mathrm{wrong}}^{(t)}|}\sum_{i\in S_{\mathrm{wrong}}^{(t)}} CE(p_S(x_i^{(t)}),y_i).
$$
For the $\ell_\infty$ setting, the update is
$$
x^{(t+1)}=\Pi_{B_\epsilon(x)}\Big(x^{(t)}+\alpha\cdot \mathrm{sign}(\nabla_{x^{(t)}}L_{DPGD}^{(t)})\Big),
$$
followed by element-wise clipping to $[0,1]$ [2509.11525].

The reported motivation is that early iterations emphasize correctly classified samples, while later iterations shift weight toward misclassified ones, thereby avoiding saturation and strengthening attack efficacy. The paper also mentions dynamic step-size tuning and channel-wise gradient masking as heuristics inherited from segmentation contexts, but states that the core improvements in the provided experiments come from the dynamic Dice weighting together with standard PGD hyperparameters.

The training loop is correspondingly structured. For each mini-batch, adversarial examples are generated for the student via DPGD; the teacher is then evaluated on both clean and adversarial inputs; the Dice-mixed teacher target is formed with $\lambda_{KD}=0.5$; and the student is updated by SGD using the DARD loss [2509.11525]. No $\epsilon$ ramp-up is used in the reported experiments.

## 5. Experimental configuration and empirical results

The reported experiments use CIFAR-10 and CIFAR-100, with a ResNet-56 teacher and a ResNet-18 student [2509.11525]. The student is optimized with SGD, learning rate $0.1$, momentum $0.9$, and weight decay $5\times 10^{-4}$. Adversarial example generation for both training and evaluation uses the $\ell_\infty$ setting with $\epsilon=8/255$, $\alpha=2/255$, and $K=20$. The evaluated attacks are FGSM, PGD-20, T-PGD, BIM, and AutoAttack.

On CIFAR-10, the student ResNet-18 under DARD attains Clean $83.56\%$, FGSM $57.53\%$, PGD20 $52.63\%$, T-PGD $81.11\%$, BIM $53.07\%$, and AutoAttack $47.75\%$ [2509.11525]. The corresponding values for SAT are Clean $81.33\%$, FGSM $55.62\%$, PGD20 $49.68\%$, T-PGD $78.42\%$, BIM $50.27\%$, and AutoAttack $46.63\%$; for ARD, Clean $84.01\%$, FGSM $57.40\%$, PGD20 $50.71\%$, T-PGD $76.42\%$, BIM $50.12\%$, and AutoAttack $45.78\%$; for TDARD, Clean $85.88\%$, FGSM $54.78\%$, PGD20 $44.35\%$, T-PGD $82.55\%$, BIM $44.12\%$, and AutoAttack $42.58\%$. Natural training collapses under iterative attacks, with PGD20, BIM, and AutoAttack all reported as $0.00\%$.

On CIFAR-100, DARD attains Clean $61.13\%$, FGSM $32.42\%$, PGD20 $28.80\%$, T-PGD $57.40\%$, BIM $28.74\%$, and AutoAttack $23.44\%$ [2509.11525]. SAT reports Clean $58.59\%$, FGSM $28.18\%$, PGD20 $23.79\%$, T-PGD $55.64\%$, BIM $23.80\%$, and AutoAttack $20.85\%$; ARD reports Clean $59.50\%$, FGSM $29.30\%$, PGD20 $25.61\%$, T-PGD $53.64\%$, BIM $21.60\%$, and AutoAttack $21.54\%$; TDARD reports Clean $65.06\%$, FGSM $29.90\%$, PGD20 $24.33\%$, T-PGD $57.96\%$, BIM $23.80\%$, and AutoAttack $19.60\%$. Natural training again shows near-zero robustness under stronger attacks.

The paper’s stated observations are specific. On CIFAR-10, DARD achieves the highest robust accuracy under PGD-20 ($52.63\%$) and AutoAttack ($47.75\%$) among the compared defences, with competitive clean accuracy [2509.11525]. On CIFAR-100, DARD improves robust accuracy across all attacks relative to SAT and ARD, with clean accuracy above SAT and comparable to ARD.

The trade-off analysis on CIFAR-10 uses Weighted Robust Accuracy (W-Robust). Under AutoAttack, DARD reports $65.66\%$ versus SAT $63.98\%$ and ARD $64.90\%$; under PGD20, DARD reports $68.10\%$ versus SAT $65.51\%$ and ARD $67.36\%$; under FGSM, DARD reports $70.55\%$ versus SAT $68.48\%$ and ARD $70.71\%$ [2509.11525]. This suggests that the method is optimized not only for worst-case robustness but also for the clean–robust trade-off emphasized by the paper.

## 6. Interpretation, limitations, and future directions

The paper interprets DARD as a mechanism for transferring smoother decision boundaries and larger logit margins from large, adversarially trained teachers to smaller students [2509.11525]. Distilling a balanced mixture of clean and adversarial teacher outputs is argued to help the student preserve natural-data structure while learning robust features. A plausible implication is that the method’s benefit depends strongly on teacher quality: the reported guidance explicitly states that the better and more robust the teacher, the more effective the distillation.

Several limitations are explicit. The reported experiments use the $\ell_\infty$ threat model with fixed $\epsilon$, so extension to $\ell_2$, $\ell_1$, or physical-world constraints requires tuning [2509.11525]. Performance depends on the trade-off coefficient $\alpha$, temperature $\tau$, the Dice mixture weight $\lambda_{KD}$, and the DPGD schedule $\lambda(t)$; mis-tuning may degrade either clean or robust accuracy. DARD also incurs non-trivial compute overhead, because it requires iterative adversarial example generation per mini-batch and teacher forward passes on both clean and adversarial inputs. Results are reported on CIFAR-10 and CIFAR-100 only, and the paper identifies scaling to ImageNet and other modalities as a natural next step.

The paper also identifies several extensions: integration with certified robustness such as randomized smoothing, alternative Dice formulations including class-wise or margin-aware mixtures, teacher ensembles, and adaptive schedules for $\lambda_{KD}$ across epochs [2509.11525]. These directions remain prospective rather than established. More generally, the absence of formal guarantees means that DARD presently stands as an empirically motivated robust distillation framework rather than a certified defence.

In the current literature, DARD therefore denotes a specific robust knowledge-distillation paradigm: adversarial-only student training combined with a Dice-mixed teacher target and strengthened by DPGD attack generation [2509.11525]. Its reported contribution is a consistent empirical improvement in robustness over adversarially trained networks with the same student architecture, while maintaining competitive standard accuracy.

Source: https://www.emergentmind.com/topics/dice-adversarial-robustness-distillation-dard