Multi-Teacher Adversarial Robustness Distillation

• The paper demonstrates that multi-teacher setups combine distinct clean and robust guidance, addressing the inherent accuracy–robustness trade-off.
• It employs various configurations—dual-teacher, ensemble, peer-to-peer, and multimodal—to effectively decouple and blend multiple sources of supervision.
• Balancing mechanisms and adaptive weighting in loss design are crucial to mitigate conflicting signals and achieve superior empirical performance.

Multi-Teacher Adversarial Robustness Distillation denotes a family of training schemes in which a student model, or a set of jointly trained peers, receives supervision from more than one source in order to improve adversarial robustness without collapsing clean performance. Across the literature, the central rationale is that a single teacher typically encodes only one point on the accuracy–robustness spectrum, whereas multiple teachers can separate clean-discriminative guidance from robust guidance, or aggregate robustness induced by different attacks, architectures, and modalities. Representative formulations include balanced dual-teacher distillation for image classifiers, ensemble-based adversarial knowledge distillation, reliability-aware or entropy-aware teacher aggregation, peer-to-peer mutual adversarial training, and multimodal variants for vision–LLMs (Zhao et al., 2023, Maroto et al., 2022, Li et al., 21 Nov 2025).

1. Problem setting and formal objective families

The underlying optimization problem inherits the min–max structure of adversarial training. A canonical formulation minimizes

$$\min_\theta \mathbb{E}_{(x,y)\sim D}\left[\max_{\|\delta\|\le \epsilon}\mathcal{L}(f_\theta(x+\delta), y)\right],$$

which improves robustness but typically reduces standard accuracy, producing the accuracy–robustness trade-off explicitly highlighted in balanced multi-teacher distillation work (Zhao et al., 2023).

Multi-teacher distillation augments that base objective by replacing a single supervisory source with multiple soft targets. In one representative ensemble formulation, adversarially trained student updates use a mixed target composed of teacher predictions and hard labels:

$$AKD\text{-}ens(x,y)=CE\!\left(f_S(x'), \alpha \sum_{i=1}^M \beta_i f_{T_i}(x') + (1-\alpha)y\right),$$

with $\beta_i\ge 0$ and $\sum_i \beta_i=1$ (Maroto et al., 2022). In balanced dual-teacher adversarial robustness distillation, the final objective is instead written as

$$\mathcal{L}_{total}=w_{nat}\,KL\!\left(S(x_{nat};\tau_s)\,\|\,T_{nat}(x_{nat};\tau_{nat})\right)+w_{adv}\,KL\!\left(S(x_{adv};\tau_s)\,\|\,T_{adv}(x_{adv};\tau_{adv})\right),$$

so that clean and adversarial supervision remain explicitly separated and dynamically reweighted (Zhao et al., 2023).

The literature uses at least three distinct supervision patterns. One pattern fixes multiple pretrained teachers and distills them into one student. A second trains multiple peer models simultaneously so that each model is both teacher and student; the abstract of “Mutual Adversarial Training: Learning together is better than going alone” describes this as models that “learn together” and “teach each other” by sharing knowledge of adversarial examples (Liu et al., 2021). A third pattern keeps two semantically different teachers, typically a clean teacher and a robust teacher, and couples them with balancing mechanisms, as in B-MTARD, CIARD, and MMT-ARD (Lu et al., 16 Sep 2025).

2. Teacher configurations and architectural patterns

The most common architecture in vision classification is a dual-teacher configuration with a clean teacher and a robust teacher. In B-MTARD, a “strong clean teacher and a strong robust teacher” guide clean examples and adversarial examples respectively, with both teachers frozen during student training (Zhao et al., 2023). CIARD keeps the same high-level division, naming the two sources $T_{nat}$ and $T_{adv}$, but additionally refreshes the robust teacher during training to counter degradation under evolving adversarial examples (Lu et al., 16 Sep 2025).

A second pattern uses multiple robust teachers rather than one clean and one robust source. “On the benefits of knowledge distillation for adversarial robustness” studies ensembles of $4$ teachers, with diversity coming from different random initializations, heterogeneous robustness levels, and different early-stopping epochs; it also reports mixtures of standard and robust teachers to obtain smooth control of the clean–robust trade-off (Maroto et al., 2022). “Improving Adversarial Robustness Through Adaptive Learning-Driven Multi-Teacher Knowledge Distillation” uses $M=4$ lightweight CNN teachers trained under FGSM, FFGSM, RFGSM, and PGD, then distills them into a clean-data-only student (Ullah et al., 28 Jul 2025).

A third pattern is multimodal. MMT-ARD introduces a dual-teacher knowledge fusion architecture for vision–LLMs consisting of a clean-accuracy oriented teacher and an adversarially trained robust teacher, both CLIP-like, supervising a lighter student through image and text encoders, projection heads, feature losses, and CLIP-style contrastive alignment (Li et al., 21 Nov 2025).

A fourth pattern is reliability-aware aggregation. Reliable Introspective Adversarial Distillation distinguishes between teachers that are good on natural data and adversarial data, good on natural data but not adversarial data, or unreliable on both, and the multi-teacher extension in the detailed description aggregates teachers through reliability-weighted mixtures on adversarial inputs (Zhu et al., 2021). The theoretical study “Toward Understanding Adversarial Distillation: Why Robust Teachers Fail” derives a multi-teacher extension in which teacher weights depend on whether a sample is likely to belong to the Robustly Unlearnable Set and on each teacher’s predictive entropy (Lee et al., 21 May 2026).

Framework	Teacher configuration	Distinctive mechanism
B-MTARD (Zhao et al., 2023)	Strong clean teacher + strong robust teacher	Entropy-Based Balance and Normalization Loss Balance
AKD ensembles (Maroto et al., 2022)	Multiple standard, weakly robust, or strongly robust teachers	Mixed soft targets on adversarial examples
MTKD-AR (Ullah et al., 28 Jul 2025)	Four adversarially trained CNN clones	Cosine-similarity adaptive weighting
MMT-ARD (Li et al., 21 Nov 2025)	Clean CLIP-like teacher + robust CLIP-like teacher	Multimodal feature and contrastive alignment
CIARD (Lu et al., 16 Sep 2025)	Clean teacher + cyclically refreshed robust teacher	Push-loss alignment and iterative retraining

This diversity of teacher topologies suggests that “multi-teacher” is not a single algorithmic template but a design principle: complementary supervision sources are arranged so that clean discrimination, adversarial stability, and, in multimodal settings, cross-modal consistency can be transferred under separate controls.

3. Loss design, balancing mechanisms, and adaptive weighting

The central technical difficulty is that multiple teachers usually provide signals on different scales and with conflicting preferences. B-MTARD addresses this with two explicit balancing algorithms. Entropy-Based Balance adjusts teacher temperatures so that the teachers’ “knowledge scales” become comparable; the update uses the monotonic increase of softmax entropy with temperature and applies gradient-sign steps to $\tau_{nat}$ and $$AKD\text{-}ens(x,y)=CE\!\left(f_S(x'), \alpha \sum_{i=1}^M \beta_i f_{T_i}(x') + (1-\alpha)y\right),$$0 while clamping them to $$AKD\text{-}ens(x,y)=CE\!\left(f_S(x'), \alpha \sum_{i=1}^M \beta_i f_{T_i}(x') + (1-\alpha)y\right),$$1. Normalization Loss Balance synchronizes the student’s learning speeds by computing relative losses $$AKD\text{-}ens(x,y)=CE\!\left(f_S(x'), \alpha \sum_{i=1}^M \beta_i f_{T_i}(x') + (1-\alpha)y\right),$$2 and $$AKD\text{-}ens(x,y)=CE\!\left(f_S(x'), \alpha \sum_{i=1}^M \beta_i f_{T_i}(x') + (1-\alpha)y\right),$$3, converting them into target ratios, and updating $$AKD\text{-}ens(x,y)=CE\!\left(f_S(x'), \alpha \sum_{i=1}^M \beta_i f_{T_i}(x') + (1-\alpha)y\right),$$4 and $$AKD\text{-}ens(x,y)=CE\!\left(f_S(x'), \alpha \sum_{i=1}^M \beta_i f_{T_i}(x') + (1-\alpha)y\right),$$5 with an EMA rule (Zhao et al., 2023).

MMT-ARD generalizes balancing into a multimodal objective. Its per-sample loss contains clean KD, robust KD, feature-level alignment on normalized embeddings, CLIP-style contrastive alignment for clean and adversarial image–text pairs, and a cross-modal consistency constraint:

$$AKD\text{-}ens(x,y)=CE\!\left(f_S(x'), \alpha \sum_{i=1}^M \beta_i f_{T_i}(x') + (1-\alpha)y\right),$$6

The full loss weights the clean and robust branches by teacher-confidence-derived $$AKD\text{-}ens(x,y)=CE\!\left(f_S(x'), \alpha \sum_{i=1}^M \beta_i f_{T_i}(x') + (1-\alpha)y\right),$$7 and $$AKD\text{-}ens(x,y)=CE\!\left(f_S(x'), \alpha \sum_{i=1}^M \beta_i f_{T_i}(x') + (1-\alpha)y\right),$$8, while feature losses are further decomposed by modality weights $$AKD\text{-}ens(x,y)=CE\!\left(f_S(x'), \alpha \sum_{i=1}^M \beta_i f_{T_i}(x') + (1-\alpha)y\right),$$9 and $\beta_i\ge 0$0 computed from a sigmoid transform of the teacher-driven clean-to-adversarial alignment gap (Li et al., 21 Nov 2025).

CIARD treats teacher disagreement as an optimization conflict rather than merely a weighting problem. It defines

$\beta_i\ge 0$1

and a push term

$\beta_i\ge 0$2

The total student loss is

$\beta_i\ge 0$3

with $\beta_i\ge 0$4 updated toward a normalized difficulty share $\beta_i\ge 0$5 (Lu et al., 16 Sep 2025). The negative coefficient on $\beta_i\ge 0$6 operationalizes a contrastive effect: the student is attracted to the robust teacher on adversarial views while being repelled from the clean teacher’s vulnerable adversarial behavior.

Reliable Introspective Adversarial Distillation uses a different remedy: partial trust. The teacher trust coefficient is defined as $\beta_i\ge 0$7, and the student also aligns its adversarial prediction with its own natural prediction through an introspective KL term,

$\beta_i\ge 0$8

so that unreliable teacher supervision can be discounted rather than averaged indiscriminately (Zhu et al., 2021).

The 2026 theoretical analysis reframes balancing as a sample-dependent entropy selection problem. It identifies the Robustly Unlearnable Set and proposes an entropy-aware multi-teacher weighting rule

$\beta_i\ge 0$9

where $\sum_i \beta_i=1$0 indicates whether the sample is likely to belong to the Robustly Unlearnable Set and $\sum_i \beta_i=1$1 is normalized teacher entropy (Lee et al., 21 May 2026). On this view, high-entropy teachers should dominate on robustly unlearnable samples, whereas low-entropy confident teachers should dominate on learnable samples.

4. Training procedures, adversarial generation, and threat models

Most methods generate adversarial examples against the current student rather than against the teachers. B-MTARD uses standard PGD-AT inner maximization with $\sum_i \beta_i=1$2 perturbations, employing PGD-10 steps, step size $\sum_i \beta_i=1$3, random start magnitude $\sum_i \beta_i=1$4, and $\sum_i \beta_i=1$5 during training on CIFAR-10, CIFAR-100, and Tiny-ImageNet (Zhao et al., 2023). IAD uses essentially the same student-driven PGD-10 training setup and evaluates with FGSM, PGD-20, $\sum_i \beta_i=1$6, and AutoAttack (Zhu et al., 2021). AKD adversarially trains the student with PGD-7 under $\sum_i \beta_i=1$7, $\sum_i \beta_i=1$8, and evaluates with AutoAttack and PGD (Maroto et al., 2022).

The specific relation between adversarial generation and distillation varies sharply across methods. In MTKD-AR, each teacher is adversarially trained with one attack from the set $\sum_i \beta_i=1$9FGSM, FFGSM, RFGSM, PGD$$\mathcal{L}_{total}=w_{nat}\,KL\!\left(S(x_{nat};\tau_s)\,\|\,T_{nat}(x_{nat};\tau_{nat})\right)+w_{adv}\,KL\!\left(S(x_{adv};\tau_s)\,\|\,T_{adv}(x_{adv};\tau_{adv})\right),$$0 using a 50–50 mixture of clean and adversarial examples per batch, but the student is trained exclusively on clean data and only receives weighted soft targets from the teachers (Ullah et al., 28 Jul 2025). This contrasts with CIARD, B-MTARD, AKD, and IAD, where student training remains embedded in an adversarial inner loop (Lu et al., 16 Sep 2025).

MMT-ARD adapts this template to vision–LLMs. Adversarial images are generated by untargeted PGD under $$\mathcal{L}_{total}=w_{nat}\,KL\!\left(S(x_{nat};\tau_s)\,\|\,T_{nat}(x_{nat};\tau_{nat})\right)+w_{adv}\,KL\!\left(S(x_{adv};\tau_s)\,\|\,T_{adv}(x_{adv};\tau_{adv})\right),$$1 against the student’s classification loss, with initialization $$\mathcal{L}_{total}=w_{nat}\,KL\!\left(S(x_{nat};\tau_s)\,\|\,T_{nat}(x_{nat};\tau_{nat})\right)+w_{adv}\,KL\!\left(S(x_{adv};\tau_s)\,\|\,T_{adv}(x_{adv};\tau_{adv})\right),$$2 and evaluation across $$\mathcal{L}_{total}=w_{nat}\,KL\!\left(S(x_{nat};\tau_s)\,\|\,T_{nat}(x_{nat};\tau_{nat})\right)+w_{adv}\,KL\!\left(S(x_{adv};\tau_s)\,\|\,T_{adv}(x_{adv};\tau_{adv})\right),$$3. Text inputs are kept clean, and the robust teacher is fed adversarial images while the clean teacher is fed clean images (Li et al., 21 Nov 2025). This path separation is a defining training protocol rather than merely an architectural choice.

Peer-based mutual training is more expansive. The detailed guide associated with the MAT abstract describes per-model adversarial crafting, shared or pooled adversarial batches, and knowledge distillation on clean inputs, on self-generated adversarial inputs, or on peer-generated adversarial inputs. The same guide also describes a union-of-norms variant in which the worst perturbation over $$\mathcal{L}_{total}=w_{nat}\,KL\!\left(S(x_{nat};\tau_s)\,\|\,T_{nat}(x_{nat};\tau_{nat})\right)+w_{adv}\,KL\!\left(S(x_{adv};\tau_s)\,\|\,T_{adv}(x_{adv};\tau_{adv})\right),$$4, $$\mathcal{L}_{total}=w_{nat}\,KL\!\left(S(x_{nat};\tau_s)\,\|\,T_{nat}(x_{nat};\tau_{nat})\right)+w_{adv}\,KL\!\left(S(x_{adv};\tau_s)\,\|\,T_{adv}(x_{adv};\tau_{adv})\right),$$5, and $$\mathcal{L}_{total}=w_{nat}\,KL\!\left(S(x_{nat};\tau_s)\,\|\,T_{nat}(x_{nat};\tau_{nat})\right)+w_{adv}\,KL\!\left(S(x_{adv};\tau_s)\,\|\,T_{adv}(x_{adv};\tau_{adv})\right),$$6 is selected per sample for training (Liu et al., 2021).

Evaluation protocols are correspondingly heterogeneous. B-MTARD and CIARD emphasize white-box FGSM, PGD variants, $$\mathcal{L}_{total}=w_{nat}\,KL\!\left(S(x_{nat};\tau_s)\,\|\,T_{nat}(x_{nat};\tau_{nat})\right)+w_{adv}\,KL\!\left(S(x_{adv};\tau_s)\,\|\,T_{adv}(x_{adv};\tau_{adv})\right),$$7, AutoAttack, and black-box Square Attack, with reporting based on clean accuracy, robust accuracy, and Weighted Robust Accuracy (Zhao et al., 2023, Lu et al., 16 Sep 2025). MMT-ARD reports clean accuracy, robust accuracy under PGD at multiple $$\mathcal{L}_{total}=w_{nat}\,KL\!\left(S(x_{nat};\tau_s)\,\|\,T_{nat}(x_{nat};\tau_{nat})\right)+w_{adv}\,KL\!\left(S(x_{adv};\tau_s)\,\|\,T_{adv}(x_{adv};\tau_{adv})\right),$$8, Sum-ACC, and zero-shot accuracy on ImageNet and standard zero-shot benchmarks (Li et al., 21 Nov 2025).

5. Empirical behavior and reported results

The empirical record consistently shows that multi-teacher methods are used to improve the clean–robust trade-off rather than robustness alone. B-MTARD reports, for a ResNet-18 student on CIFAR-10 under AutoAttack, 67.82% W-Robust (Clean 88.20%, Robust 47.44%), exceeding RSLAD (67.49%) and SCORE (66.84%); on CIFAR-100 the same student reaches 44.53% (Clean 65.08%, Robust 23.98%) versus ARD 42.91% (+1.62%) (Zhao et al., 2023). Its black-box gains are larger: under Square Attack with 100 queries, the ResNet-18 student reaches 84.01% on CIFAR-10 versus 79.56% (+4.45%) for the next best method, and the MobileNet-v2 student reaches 84.00% versus 78.44% (+5.56%) (Zhao et al., 2023).

AKD shows a different but related pattern. On CIFAR-10 with a ResNet-18 student under AutoAttack, a single PGD-7 teacher yields 82.70% clean, 47.66% AA, while a 4-teacher PGD-7 ensemble yields 82.96% clean, 47.96% AA. On CIFAR-100, the corresponding improvement is 56.87%/24.61% to 58.34%/24.93% (Maroto et al., 2022). The same study also reports that increasing the mixture weight assigned to robust teachers smoothly increases robust accuracy while decreasing clean accuracy, making the trade-off tunable rather than fixed (Maroto et al., 2022).

MMT-ARD extends these gains to vision–LLMs. The abstract reports that on ViT-B-32 it improves robust accuracy by +4.32%, zero-shot accuracy by +3.5%, and training efficiency by 2.3x relative to traditional single-teacher methods (Li et al., 21 Nov 2025). The detailed results report, for ViT-B/32, acc: baseline 61.84 → MMT-ARD 63.48 (+1.64), racc ($$\mathcal{L}_{total}=w_{nat}\,KL\!\left(S(x_{nat};\tau_s)\,\|\,T_{nat}(x_{nat};\tau_{nat})\right)+w_{adv}\,KL\!\left(S(x_{adv};\tau_s)\,\|\,T_{adv}(x_{adv};\tau_{adv})\right),$$9): 49.00 → 49.34 (+0.34), racc ($T_{nat}$0): 34.56 → 34.78 (+0.22), and Zero-shot: 26.40 → 27.10 (+0.70) (Li et al., 21 Nov 2025).

MTKD-AR demonstrates that robustness can be transferred even when the student never sees adversarial examples. On MNIST at $T_{nat}$1, the reported clean-only student achieves Clean ≈ 97.66%, with FGSM ≈ 95.55%, FFGSM ≈ 93.10%, RFGSM ≈ 93.76%, and PGD ≈ 92.98%, while single-teacher KD remains strongly attack-specific (Ullah et al., 28 Jul 2025). On Fashion-MNIST at $T_{nat}$2, the same method reports Clean ≈ 91.2% and robustness values around 90% across FGSM, FFGSM, RFGSM, and PGD (Ullah et al., 28 Jul 2025).

CIARD focuses on the dual-teacher conflict itself. Its abstract reports an average 3.53 improvement in adversarial defense rates and a 5.87 increase in clean sample accuracy across CIFAR-10, CIFAR-100, and Tiny-ImageNet (Lu et al., 16 Sep 2025). Under AutoAttack on CIFAR-10, the reported ResNet-18 student obtains Clean 88.87%, Robust 48.88%, W-R 68.88%, described as +1.06% W-R over B-MTARD, while MobileNet-V2 achieves Robust 46.31%, W-R 67.61%, +0.77% W-R over B-MTARD (Lu et al., 16 Sep 2025).

These results do not imply a universal dominance of multi-teacher methods. Rather, they indicate that carefully balanced multi-source supervision can outperform both single-teacher adversarial distillation and naive dual-teacher averaging on standard white-box and black-box evaluations.

6. Limitations, misconceptions, and current research directions

A recurrent misconception is that a more robust teacher necessarily yields a more robust student. The theoretical analysis in “Toward Understanding Adversarial Distillation: Why Robust Teachers Fail” rejects that view: teacher standalone robustness is reported not to correlate with student robustness and can even correlate negatively, whereas teacher predictive entropy on the Robustly Unlearnable Set is a strong predictor of student robust accuracy after distillation (Lee et al., 21 May 2026). The same study argues that confident supervision on unlearnable samples forces the student to memorize spurious noise patterns, while high uncertainty suppresses that memorization (Lee et al., 21 May 2026).

Another misconception is that simply increasing teacher scale guarantees better distillation. B-MTARD explicitly reports that larger or stronger teachers do not always yield better students; on CIFAR-10, a clean ResNet-56 plus a robust WRN-34-10 produced higher W-Robust Acc than pairings with a stronger clean WRN-34-10-C and/or stronger robust WRN-70-16 (Zhao et al., 2023). This suggests that complementarity and compatibility matter as much as raw teacher strength.

The literature also distinguishes genuine robustness from gradient masking. Work on defensive distillation with teacher assistants states that later attacks such as Carlini–Wagner exposed the weakness of defensive distillation because it often induces gradient masking rather than true robustness, and strong optimization-based attacks can still reliably craft adversarial examples (Mandal et al., 2023). B-MTARD separately reports that stronger attacks have higher success than weaker ones, white-box attacks are stronger than black-box attacks, and gradient estimation attacks yield higher robust accuracies than PGD-based attacks, which it interprets as evidence against obfuscated gradients (Zhao et al., 2023).

Multi-teacher methods introduce their own costs and failure modes. B-MTARD notes the overhead of two teacher forward passes plus the student forward/backward and PGD inner loop, although EBB and NLB add negligible compute and inference time remains unchanged because only the student is used (Zhao et al., 2023). MMT-ARD attributes its reported 2.3× efficiency gain over single-teacher ARD to faster convergence, dynamic weighting, and parallelization/caching, but also notes that extreme distribution shifts or very strong attacks can reduce both teachers’ confidence, making dynamic weights less informative (Li et al., 21 Nov 2025). CIARD reports per-epoch time increasing from 111.06 s/epoch without iterative teacher training to 140.56 s/epoch with iterative teacher training for WRN-34-10 on a single RTX 4090 (Lu et al., 16 Sep 2025).

Current directions expand the idea rather than replacing it. MMT-ARD states that extending beyond two teachers, including multiple robust teachers trained with different adversarial strategies, naturally fits its dynamic weighting mechanism and can be combined with curriculum learning and LoRA-compatible backbones (Li et al., 21 Nov 2025). The entropy-based theory paper similarly proposes ranking candidate teachers by average entropy on adversarial examples of a proxy Robustly Unlearnable Set and choosing the top-$T_{nat}$3 teachers (Lee et al., 21 May 2026). Peer-based mutual training, dual-teacher balancing, reliability-aware introspection, and multimodal confidence fusion therefore represent different answers to the same central question: how to exploit diverse supervisory signals without letting any one teacher’s failure mode dominate adversarial generalization.