---
title: Consensus-Oriented Masked Distillation (CoMAD)
url: https://www.emergentmind.com/topics/consensus-oriented-masked-distillation-comad
type: topic
---

# Consensus-Oriented Masked Distillation (CoMAD)

Searching arXiv for the CoMAD paper and key related works to ground citations.
arxiv.search query: "ti:CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework"
arxiv.search({"query":"ti:\"CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework\"","max_results":5})
Consensus-oriented Masked Distillation (CoMAD) is a multiple-teacher self-supervised distillation framework for compressing several large self-supervised Vision Transformers into a compact student while preserving complementary representational priors. Introduced in "CoMAD: A Multiple-Teacher Self-Supervised Distillation Framework" [2508.04816], the method distills from three pretrained ViT-Base teachers—MAE, MoCo v3, and iBOT—into a small student such as ViT-Tiny by combining asymmetric masking, lightweight feature alignment, token-wise consensus gating, and dual-level KL-based distillation. The framework is positioned at the intersection of masked image modeling, multi-teacher knowledge distillation, and compact SSL transfer, and is evaluated primarily on ImageNet-1K, ADE20K, and MS-COCO; the teacher families themselves derive from masked image modeling, contrastive or EMA-based SSL, and hybrid masked-token plus clustering paradigms [2111.06377, 2104.02057, 2111.07832].

## 1. Motivation and problem formulation

The framework addresses two properties of contemporary visual self-supervised learning. First, major SSL paradigms such as contrastive learning, masked image modeling, and clustering-based methods are typically pretrained in isolation, so several strong but different ViT-Base models emerge, each capturing complementary structure. Second, ViT-Base and ViT-Large encoders remain expensive at inference, which is problematic for deployment in edge or mobile scenarios [2508.04816].

Within this setting, CoMAD is defined by three design goals. It seeks to unify the knowledge of multiple heterogeneous SSL teachers—specifically MAE, MoCo v3, and iBOT—into one compact student; it aims to exploit complementary priors from different SSL paradigms rather than relying on a single teacher or static fusion; and it aims to retain a lightweight distillation mechanism, with a token-wise gating module that has no learnable parameters. The compact target model emphasized in the experiments is a ViT-Tiny with approximately 5.4M parameters.

A central premise of the method is that naive multi-teacher fusion is suboptimal. The paper explicitly contrasts CoMAD with single-teacher distillation, uniform averaging, static weights, and symmetric masking. This suggests that the method treats disagreement among heterogeneous SSL teachers as a primary optimization problem rather than as noise to be averaged away. In the paper’s formulation, CoMAD is conceptually “masked reconstruction + multi-teacher KD” with a non-parametric, token-wise consensus mechanism [2508.04816].

## 2. Architectural configuration and teacher–student roles

The default student is a ViT-Tiny with patch size $P=16$, depth $L_S=12$, embedding dimension $D_S=192$, and approximately 5.4M parameters. The teachers are three frozen ViT-Base encoders with patch size $16$, depth $L_T=12$, and embedding dimension $D_T=768$. Each teacher is pretrained on ImageNet-1K for 300 epochs with a distinct SSL objective: MAE emphasizes masked-image reconstruction and contextual modeling from partial views; MoCo v3 emphasizes contrastive or EMA-style discriminative semantics and invariances; iBOT emphasizes hybrid masked-token learning with clustering or online tokenization and is described as strong in patch-level semantics and token grouping [2508.04816].

The pipeline begins from an input image $x\in\mathbb{R}^{H\times W\times 3}$, for example $224\times224$, which is split into $N=\frac{HW}{P^2}$ patches. For $P=16$, this yields a $14\times14$ grid of tokens. Student and teacher token sequences are then masked independently, forwarded through their respective encoders, projected into a common student-dimensional space, fused token-wise, and matched through two KL objectives. Only the student and small teacher-side adapters are trained; all teacher encoders remain frozen.

Feature alignment is handled by a simple linear adapter followed by LayerNorm, shared across token positions for each teacher branch:
$$
\hat{z}^{T(m)}_{b,n}=\mathrm{LayerNorm}\!\left(W_{\mathrm{adp}}\,z^{T(m)}_{b,n}+b_{\mathrm{adp}}\right),
$$
with $W_{\mathrm{adp}}\in\mathbb{R}^{D_S\times D_T}$ and $b_{\mathrm{adp}}\in\mathbb{R}^{D_S}$. This alignment stage is the only trainable component on the teacher side. The description of CoMAD as “lightweight” and “essentially parameter-free” therefore refers specifically to the fusion or gating mechanism, not to the complete absence of auxiliary trainable components.

## 3. Asymmetric masking and consensus-oriented fusion

CoMAD’s masking policy is asymmetric. The student uses mask ratio $r_S=0.75$, so it keeps only $25\%$ of image patches. The three teachers use lighter masks with ratios
$$
r_{T(1)}=0.50,\qquad r_{T(2)}=0.40,\qquad r_{T(3)}=0.30,
$$
so they keep $50\%$, $60\%$, and $70\%$ of patches respectively [2508.04816]. Masking is applied after patch embedding and is independent across teachers; the class token is always retained:
$$
\tilde Z^S=M^S\odot Z^S,\qquad \tilde Z^{T(m)}=M^{T(m)}\odot Z^{T(m)}.
$$

The stated rationale is that the teachers operate with richer context than the student and that different teachers observe different subsets of patches. Because the student sees only one quarter of the image tokens, it must interpolate missing features from the fused teacher signals. The paper reports that this asymmetry is important empirically: a student mask ratio of $0.80$ degrades performance because the student is starved of input, whereas $0.70$ reduces the challenge of distillation and also lowers gains.

Fusion is performed per sample and per token, including both patch and class tokens. For each token, CoMAD computes a student–teacher cosine affinity
$$
s_{b,n,m}=\frac{\langle z^S_{b,n},\hat{z}^{T(m)}_{b,n}\rangle}
{\|z^S_{b,n}\|\,\|\hat{z}^{T(m)}_{b,n}\|},
$$
and an inter-teacher consensus score
$$
c_{b,n,m}=\frac{1}{M-1}\sum_{k\neq m}
\frac{\langle \hat{z}^{T(m)}_{b,n},\hat{z}^{T(k)}_{b,n}\rangle}
{\|\hat{z}^{T(m)}_{b,n}\|\,\|\hat{z}^{T(k)}_{b,n}\|}.
$$
These are added,
$$
e_{b,n,m}=s_{b,n,m}+c_{b,n,m},
$$
and normalized over teachers by a softmax with temperature $\tau=0.1$:
$$
\alpha_{b,n,m}=
\frac{\exp(e_{b,n,m}/\tau)}
{\sum_{k=1}^{M}\exp(e_{b,n,k}/\tau)}.
$$
The fused target token is then
$$
z^T_{b,n}=\sum_{m=1}^{M}\alpha_{b,n,m}\hat{z}^{T(m)}_{b,n}.
$$

This mechanism is adaptive, token-wise, and non-parametric. In the paper’s interpretation, a teacher is up-weighted when it aligns both with the student’s current token representation and with the other teachers; a teacher that disagrees with both receives a smaller weight. Ablation results reported in the paper show that uniform averaging is weaker, affinity-only or consensus-only gating is stronger than uniform weighting, and combining both signals gives the best performance [2508.04816].

## 4. Distillation objective and optimization protocol

The student is trained against the fused teacher representation at two levels. The first is token-level KL divergence over visible student tokens. The second is spatial feature-map KL divergence over reshaped patch-token grids. The total objective is
$$
\mathcal{L}=\mathcal{L}_{\mathrm{token}}+\mathcal{L}_{\mathrm{spatial}}.
$$

For token-level matching, the method applies a lightweight projection $\phi:\mathbb{R}^{D_S}\rightarrow\mathbb{R}^{C}$ that converts each token embedding into a probability distribution. If $M^S_{b,n}=1$ indicates that a token is visible to the student, then the loss is
$$
\mathcal{L}_{\mathrm{token}}=
\frac{1}{\sum_{b,n}M^S_{b,n}}
\sum_{b=1}^{B}\sum_{n=0}^{N}
M^S_{b,n}\,
\mathrm{KL}\!\bigl(\phi(z^S_{b,n})\;\|\;\phi(z^T_{b,n})\bigr).
$$
The second term reshapes patch tokens into spatial maps $F^S,F^T\in\mathbb{R}^{B\times D_S\times H'\times W'}$ and applies another projection $\psi:\mathbb{R}^{D_S}\rightarrow\mathbb{R}^{C'}$ to obtain channel-wise distributions at each spatial location, which are then matched by KL divergence over the $H'\times W'$ grid.

The training configuration reported for distillation uses unlabeled ImageNet-1K for 300 epochs, AdamW, an initial learning rate of $1.5\times10^{-4}$, weight decay $0.05$, batch size $4096$, 15 epochs of linear warmup, cosine decay, and mixed-precision training on $8\times$ V100 GPUs. Inputs use resolution $224\times224$ and patch size $16$. The data augmentation listed for student and teachers is random resized crop and horizontal flip, with additional color jitter for the student. After distillation, the student is fine-tuned on labeled ImageNet-1K for 100 epochs with learning rate $1\times10^{-3}$ and weight decay $0.05$ [2508.04816].

Ablation results reported in the paper state that dual-level KL is superior to using only token-level fusion distillation or only spatial fusion distillation, and that replacing KL with MSE weakens performance. This suggests that the method depends not just on multi-scale alignment but specifically on distributional matching.

## 5. Empirical results and ablation findings

The reported quantitative results cover ImageNet-1K classification and dense prediction transfers to ADE20K and MS-COCO. On ImageNet-1K, CoMAD with a ViT-Tiny student achieves $75.4\%$ Top-1 accuracy, compared with $75.0\%$ for DMT and $74.6\%$, $74.6\%$, and $74.5\%$ for TinyMIM distilled respectively from MAE, iBOT, and MoCo v3 teachers. For larger students, the paper reports $82.9\%$ for ViT-Small and $84.7\%$ for ViT-Base under CoMAD, exceeding DMT by $0.7$ and $1.0$ percentage points respectively [2508.04816].

On dense prediction, distilled backbones are evaluated with UPerNet on ADE20K and with Cascade R-CNN plus FPN on MS-COCO. The reported best compact-distillation results are $47.3\%$ mIoU and $83.1\%$ aAcc on ADE20K, and $\mathrm{AP}^{bb}=44.5\%$ and $\mathrm{AP}^{mk}=40.5\%$ on COCO. These values are reported as improvements over DMT, which reaches $46.9\%$ mIoU and $82.9\%$ aAcc on ADE20K and $44.3\%$ box AP and $40.3\%$ mask AP on COCO.

| Evaluation | Baseline reported in the paper | CoMAD |
|---|---:|---:|
| ImageNet-1K, ViT-Tiny Top-1 | DMT: 75.0% | 75.4% |
| ADE20K mIoU | DMT: 46.9% | 47.3% |
| COCO box AP | DMT: 44.3% | 44.5% |
| COCO mask AP | DMT: 40.3% | 40.5% |

The ablations isolate three key mechanisms. First, teacher composition matters: single teachers yield approximately $39.2\%$ mIoU for MAE only, $40.3\%$ for MoCo v3 only, and $40.0\%$ for iBOT only, while combining all three teachers yields $42.0\%$. Second, mask ratios matter: the configuration $r_S=0.75$ and $r_T=\{0.50,0.40,0.30\}$ gives the best reported combination at Top-1 $75.3$ and mIoU $42.0$, outperforming both heavier and lighter student masking. Third, the gating mechanism matters: uniform weighting gives Top-1 $74.5$ and mIoU $41.3$, affinity-only gives $74.8/41.7$, consensus-only gives $74.6/41.5$, and combined affinity plus consensus gives $75.4/42.0$. The paper’s conclusion from these experiments is that the gains arise from the interaction of heterogeneous teachers, asymmetric masking, and joint consensus gating rather than from multi-teacher aggregation alone.

## 6. Relation to prior SSL distillation, limitations, and extensions

CoMAD is explicitly positioned relative to three lines of work. The first is SSL pretraining, represented in the paper by MAE, MoCo v3, and iBOT. The second is single-teacher SSL distillation, represented by TinyMIM. The third is multi-teacher distillation, represented in the comparisons by DMT and by prior approaches that use fixed or parametric teacher weighting. In this landscape, CoMAD differs by simultaneously using multiple SSL teachers, applying asymmetric masking so that teachers have richer context than the student, and performing token-wise parameter-free gating based on both student–teacher affinity and inter-teacher consensus [2508.04816].

The paper also distinguishes CoMAD from masking-based distillation methods that use the same masking pattern for teacher and student or that operate in supervised rather than self-supervised settings. A plausible implication is that CoMAD treats masking not merely as a regularizer or FLOP-reduction device but as a mechanism for creating an informational asymmetry that teachers can exploit during distillation.

Several limitations and assumptions are stated directly. All teachers and students are ViTs with the same patch size $P=16$; heterogeneous architectures such as CNNs or Swin are not explored. The method relies on strong off-the-shelf SSL ViT-Base teachers, and the benefit may diminish if the teachers are weak or not complementary. Distillation is computationally expensive because each iteration runs three ViT-Base teachers plus the student, even though the teachers are frozen. The mask ratios are fixed rather than adaptive, and experiments are limited to standard vision benchmarks rather than video, multimodal data, or explicit domain-shift settings.

The paper identifies several extensions. These include adaptive masking schedules, more or different teachers such as CLIP-like multimodal teachers or larger ViTs, heterogeneous teacher ensembles such as CNN-plus-ViT mixtures, and applications beyond standard image benchmarks to video SSL, multimodal learning, medical imaging, robotics, few-shot transfer, long-tailed settings, and domain adaptation. These are presented as future directions rather than validated properties of the reported system.

In summary, CoMAD defines a compact SSL distillation regime in which a heavily masked student learns from multiple lightly and differently masked teachers through non-parametric consensus fusion and dual-level KL matching. Within the experimental scope reported in the paper, its principal contribution is to show that complementary SSL priors from heterogeneous teachers can be integrated at token level into a small ViT while improving both classification and dense-prediction transfer performance [2508.04816].

Source: https://www.emergentmind.com/topics/consensus-oriented-masked-distillation-comad