Consensus-Oriented Masked Distillation (CoMAD)

• The paper demonstrates that CoMAD distills knowledge from three strong ViT-Base teachers (MAE, MoCo v3, iBOT) into a lightweight ViT-Tiny student using dual-level KL-based objectives.
• It introduces asymmetric masking for student and teachers, enabling complementary representations to be selectively fused via a non-parametric token-wise consensus gating mechanism.
• Empirical results on ImageNet-1K, ADE20K, and MS-COCO show enhanced performance over single-teacher methods by effectively integrating heterogeneous SSL priors.

Searching arXiv for the CoMAD paper and key related works to ground citations. arxiv.search ^^^^^^^^3query3 "^^^^^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^^^^^" arxiv.search({"^^^^^^^^3query3 A Multiple-Teacher Self-Supervised Distillation Framework\"","max_results":5}) Consensus-oriented Masked Distillation (CoMAD) is a multiple-teacher self-supervised distillation framework for compressing several large self-supervised Vision Transformers into a compact student while preserving complementary representational priors. Introduced in "CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework" (&&&^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^&&&), the method distills from three pretrained ViT-Base teachers—MAE, MoCo v3, and iBOT—into a small student such as ViT-Tiny by combining asymmetric masking, lightweight feature alignment, token-wise consensus gating, and dual-level KL-based distillation. The framework is positioned at the intersection of masked image modeling, multi-teacher knowledge distillation, and compact SSL transfer, and is evaluated primarily on ImageNet-^^^^3query3^^^^K, ADE^^^^3ti:\3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^K, and MS-COCO; the teacher families themselves derive from masked image modeling, contrastive or EMA-based SSL, and hybrid masked-token plus clustering paradigms (&&&^^^^3query3^^^^&&&, &&&^^^^3ti:\3^^^^&&&, Zhou et al., 2021).

^^^^3query3^^^^. Motivation and problem formulation

The framework addresses two properties of contemporary visual self-supervised learning. First, major SSL paradigms such as contrastive learning, masked image modeling, and clustering-based methods are typically pretrained in isolation, so several strong but different ViT-Base models emerge, each capturing complementary structure. Second, ViT-Base and ViT-Large encoders remain expensive at inference, which is problematic for deployment in edge or mobile scenarios (&&&^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^&&&).

Within this setting, CoMAD is defined by three design goals. It seeks to unify the knowledge of multiple heterogeneous SSL teachers—specifically MAE, MoCo v3, and iBOT—into one compact student; it aims to exploit complementary priors from different SSL paradigms rather than relying on a single teacher or static fusion; and it aims to retain a lightweight distillation mechanism, with a token-wise gating module that has no learnable parameters. The compact target model emphasized in the experiments is a ViT-Tiny with approximately 5.4M parameters.

A central premise of the method is that naive multi-teacher fusion is suboptimal. The paper explicitly contrasts CoMAD with single-teacher distillation, uniform averaging, static weights, and symmetric masking. This suggests that the method treats disagreement among heterogeneous SSL teachers as a primary optimization problem rather than as noise to be averaged away. In the paper’s formulation, CoMAD is conceptually “masked reconstruction + multi-teacher KD” with a non-parametric, token-wise consensus mechanism (&&&^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^&&&).

^^^^3ti:\3^^^^. Architectural configuration and teacher–student roles

The default student is a ViT-Tiny with patch size PRESERVED_PLACEHOLDER_^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^, depth PRESERVED_PLACEHOLDER_^^^^3query3^^^^, embedding dimension PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^, and approximately 5.4M parameters. The teachers are three frozen ViT-Base encoders with patch size $16$, depth $L_T=12$, and embedding dimension $D_T=768$. Each teacher is pretrained on ImageNet-^^^^3query3^^^^K for 3^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^ epochs with a distinct SSL objective: MAE emphasizes masked-image reconstruction and contextual modeling from partial views; MoCo v3 emphasizes contrastive or EMA-style discriminative semantics and invariances; iBOT emphasizes hybrid masked-token learning with clustering or online tokenization and is described as strong in patch-level semantics and token grouping (&&&^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^&&&).

The pipeline begins from an input image $x\in\mathbb{R}^{H\times W\times 3}$, for example $224\times224$, which is split into $N=\frac{HW}{P^2}$ patches. For $P=16$, this yields a PRESERVED_PLACEHOLDER_^^^^3query3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^ grid of tokens. Student and teacher token sequences are then masked independently, forwarded through their respective encoders, projected into a common student-dimensional space, fused token-wise, and matched through two KL objectives. Only the student and small teacher-side adapters are trained; all teacher encoders remain frozen.

Feature alignment is handled by a simple linear adapter followed by LayerNorm, shared across token positions for each teacher branch:

PRESERVED_PLACEHOLDER_^^^^3query3query3^^^^

with PRESERVED_PLACEHOLDER_^^^^3query3ti:\3^^^^ and PRESERVED_PLACEHOLDER_^^^^3query33^^^^. This alignment stage is the only trainable component on the teacher side. The description of CoMAD as “lightweight” and “essentially parameter-free” therefore refers specifically to the fusion or gating mechanism, not to the complete absence of auxiliary trainable components.

3. Asymmetric masking and consensus-oriented fusion

CoMAD’s masking policy is asymmetric. The student uses mask ratio PRESERVED_PLACEHOLDER_^^^^3query3^^^^4, so it keeps only PRESERVED_PLACEHOLDER_^^^^3query3^^^^5 of image patches. The three teachers use lighter masks with ratios

PRESERVED_PLACEHOLDER_^^^^3query3^^^^6

so they keep PRESERVED_PLACEHOLDER_^^^^3query3^^^^7, PRESERVED_PLACEHOLDER_^^^^3query3^^^^8, and PRESERVED_PLACEHOLDER_^^^^3query3^^^^9 of patches respectively (&&&^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^&&&). Masking is applied after patch embedding and is independent across teachers; the class token is always retained:

PRESERVED_PLACEHOLDER_^^^^3ti:\3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^

The stated rationale is that the teachers operate with richer context than the student and that different teachers observe different subsets of patches. Because the student sees only one quarter of the image tokens, it must interpolate missing features from the fused teacher signals. The paper reports that this asymmetry is important empirically: a student mask ratio of PRESERVED_PLACEHOLDER_^^^^3ti:\3query3^^^^ degrades performance because the student is starved of input, whereas PRESERVED_PLACEHOLDER_^^^^3ti:\3ti:\3^^^^ reduces the challenge of distillation and also lowers gains.

Fusion is performed per sample and per token, including both patch and class tokens. For each token, CoMAD computes a student–teacher cosine affinity

PRESERVED_PLACEHOLDER_^^^^3ti:\33^^^^

and an inter-teacher consensus score

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^4

These are added,

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^5

and normalized over teachers by a softmax with temperature PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^6:

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^7

The fused target token is then

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^8

This mechanism is adaptive, token-wise, and non-parametric. In the paper’s interpretation, a teacher is up-weighted when it aligns both with the student’s current token representation and with the other teachers; a teacher that disagrees with both receives a smaller weight. Ablation results reported in the paper show that uniform averaging is weaker, affinity-only or consensus-only gating is stronger than uniform weighting, and combining both signals gives the best performance (&&&^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^&&&).

4. Distillation objective and optimization protocol

The student is trained against the fused teacher representation at two levels. The first is token-level KL divergence over visible student tokens. The second is spatial feature-map KL divergence over reshaped patch-token grids. The total objective is

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^9

For token-level matching, the method applies a lightweight projection $16$^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^ that converts each token embedding into a probability distribution. If $16$^^^^3query3^^^^ indicates that a token is visible to the student, then the loss is

$16$^^^^3ti:\3^^^^

The second term reshapes patch tokens into spatial maps $16$3 and applies another projection $16$4 to obtain channel-wise distributions at each spatial location, which are then matched by KL divergence over the $16$5 grid.

The training configuration reported for distillation uses unlabeled ImageNet-^^^^3query3^^^^K for 3^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^ epochs, AdamW, an initial learning rate of $16$6, weight decay $16$7, batch size $16$8, ^^^^3query3^^^^5 epochs of linear warmup, cosine decay, and mixed-precision training on $16$9 V^^^^3query3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^ GPUs. Inputs use resolution $L_T=12$^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^ and patch size $L_T=12$^^^^3query3^^^^. The data augmentation listed for student and teachers is random resized crop and horizontal flip, with additional color jitter for the student. After distillation, the student is fine-tuned on labeled ImageNet-^^^^3query3^^^^K for ^^^^3query3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^ epochs with learning rate $L_T=12$^^^^3ti:\3^^^^ and weight decay $L_T=12$3 (&&&^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^&&&).

Ablation results reported in the paper state that dual-level KL is superior to using only token-level fusion distillation or only spatial fusion distillation, and that replacing KL with MSE weakens performance. This suggests that the method depends not just on multi-scale alignment but specifically on distributional matching.

5. Empirical results and ablation findings

The reported quantitative results cover ImageNet-^^^^3query3^^^^K classification and dense prediction transfers to ADE^^^^3ti:\3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^K and MS-COCO. On ImageNet-^^^^3query3^^^^K, CoMAD with a ViT-Tiny student achieves $L_T=12$4 Top-^^^^3query3^^^^ accuracy, compared with $L_T=12$5 for DMT and $L_T=12$6, $L_T=12$7, and $L_T=12$8 for TinyMIM distilled respectively from MAE, iBOT, and MoCo v3 teachers. For larger students, the paper reports $L_T=12$9 for ViT-Small and $D_T=768$^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^ for ViT-Base under CoMAD, exceeding DMT by $D_T=768$^^^^3query3^^^^ and $D_T=768$^^^^3ti:\3^^^^ percentage points respectively (&&&^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^&&&).

On dense prediction, distilled backbones are evaluated with UPerNet on ADE^^^^3ti:\3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^K and with Cascade R-CNN plus FPN on MS-COCO. The reported best compact-distillation results are $D_T=768$3 mIoU and $D_T=768$4 aAcc on ADE^^^^3ti:\3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^K, and $D_T=768$5 and $D_T=768$6 on COCO. These values are reported as improvements over DMT, which reaches $D_T=768$7 mIoU and $D_T=768$8 aAcc on ADE^^^^3ti:\3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^K and $D_T=768$9 box AP and $x\in\mathbb{R}^{H\times W\times 3}$^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^ mask AP on COCO.

Evaluation	Baseline reported in the paper	CoMAD
ImageNet-^^^^3query3^^^^K, ViT-Tiny Top-^^^^3query3^^^^	DMT: 75.^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^%	75.4%
ADE^^^^3ti:\3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^K mIoU	DMT: 46.9%	47.3%
COCO box AP	DMT: 44.3%	44.5%
COCO mask AP	DMT: 4^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^.3%	4^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^.5%

The ablations isolate three key mechanisms. First, teacher composition matters: single teachers yield approximately $x\in\mathbb{R}^{H\times W\times 3}$^^^^3query3^^^^ mIoU for MAE only, $x\in\mathbb{R}^{H\times W\times 3}$^^^^3ti:\3^^^^ for MoCo v3 only, and $x\in\mathbb{R}^{H\times W\times 3}$3 for iBOT only, while combining all three teachers yields $x\in\mathbb{R}^{H\times W\times 3}$4. Second, mask ratios matter: the configuration $x\in\mathbb{R}^{H\times W\times 3}$5 and $x\in\mathbb{R}^{H\times W\times 3}$6 gives the best reported combination at Top-^^^^3query3^^^^ $x\in\mathbb{R}^{H\times W\times 3}$7 and mIoU $x\in\mathbb{R}^{H\times W\times 3}$8, outperforming both heavier and lighter student masking. Third, the gating mechanism matters: uniform weighting gives Top-^^^^3query3^^^^ $x\in\mathbb{R}^{H\times W\times 3}$9 and mIoU $224\times224$^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^, affinity-only gives $224\times224$^^^^3query3^^^^, consensus-only gives $224\times224$^^^^3ti:\3^^^^, and combined affinity plus consensus gives $224\times224$3. The paper’s conclusion from these experiments is that the gains arise from the interaction of heterogeneous teachers, asymmetric masking, and joint consensus gating rather than from multi-teacher aggregation alone.

6. Relation to prior SSL distillation, limitations, and extensions

CoMAD is explicitly positioned relative to three lines of work. The first is SSL pretraining, represented in the paper by MAE, MoCo v3, and iBOT. The second is single-teacher SSL distillation, represented by TinyMIM. The third is multi-teacher distillation, represented in the comparisons by DMT and by prior approaches that use fixed or parametric teacher weighting. In this landscape, CoMAD differs by simultaneously using multiple SSL teachers, applying asymmetric masking so that teachers have richer context than the student, and performing token-wise parameter-free gating based on both student–teacher affinity and inter-teacher consensus (&&&^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^&&&).

The paper also distinguishes CoMAD from masking-based distillation methods that use the same masking pattern for teacher and student or that operate in supervised rather than self-supervised settings. A plausible implication is that CoMAD treats masking not merely as a regularizer or FLOP-reduction device but as a mechanism for creating an informational asymmetry that teachers can exploit during distillation.

Several limitations and assumptions are stated directly. All teachers and students are ViTs with the same patch size $224\times224$4; heterogeneous architectures such as CNNs or Swin are not explored. The method relies on strong off-the-shelf SSL ViT-Base teachers, and the benefit may diminish if the teachers are weak or not complementary. Distillation is computationally expensive because each iteration runs three ViT-Base teachers plus the student, even though the teachers are frozen. The mask ratios are fixed rather than adaptive, and experiments are limited to standard vision benchmarks rather than video, multimodal data, or explicit domain-shift settings.

The paper identifies several extensions. These include adaptive masking schedules, more or different teachers such as CLIP-like multimodal teachers or larger ViTs, heterogeneous teacher ensembles such as CNN-plus-ViT mixtures, and applications beyond standard image benchmarks to video SSL, multimodal learning, medical imaging, robotics, few-shot transfer, long-tailed settings, and domain adaptation. These are presented as future directions rather than validated properties of the reported system.

In summary, CoMAD defines a compact SSL distillation regime in which a heavily masked student learns from multiple lightly and differently masked teachers through non-parametric consensus fusion and dual-level KL matching. Within the experimental scope reported in the paper, its principal contribution is to show that complementary SSL priors from heterogeneous teachers can be integrated at token level into a small ViT while improving both classification and dense-prediction transfer performance (&&&^^^^3ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework3^^^^&&&).