Implicit Clustering Distillation (iCD)

• The paper shows that iCD transfers latent structural relationships from local logits, enhancing the student model’s performance over standard KD methods.
• iCD uses Gram matrices of normalized local logits to capture implicit clustering without relying on explicit clustering assignments.
• Empirical evaluations reveal that iCD consistently provides significant gains in fine-grained tasks across various heterogeneous teacher–student architectures.

Implicit Clustering Distillation (iCD) is a logit-based knowledge distillation method that transfers not only a teacher’s class predictions but also the latent structural relationships embedded in local logits. It was introduced as a simple and effective mechanism for mining “interpretable structural knowledge from logits” by operating on decoupled local logit representations and their Gram matrices, rather than relying solely on global softened predictions or on explicit intermediate feature alignment (Xue et al., 16 Sep 2025).

1. Definition and conceptual scope

iCD belongs to the logit-distillation family, but it departs from standard logit KD in two ways. First, it treats global logits as an insufficient summary of the teacher’s decision process, because global prediction is formed by averaging local logit evidence across spatial locations. Second, it interprets the relationships among local logits as a source of latent semantic structure that can be distilled into a student model (Xue et al., 16 Sep 2025).

The method is built on the observation that conventional logit KD is largely a “learn-to-predict” paradigm: it transfers softened class probabilities, but it does not explicitly encode how semantic evidence is organized across local regions. iCD therefore starts from spatially decoupled local logits and then distills structural relations among them. In the paper’s interpretation, this constitutes an “implicit clustering” process: no explicit clustering algorithm is introduced, yet local teacher–student structures are treated as feature clusters and transferred through similarity statistics rather than through hard assignments (Xue et al., 16 Sep 2025).

A common misconception is to read iCD as an explicit clustering method. It is not. The method does not run $k$-means, does not learn prototype assignment distributions, and does not introduce a cluster-label supervision stage. The clustering is implicit in the Gram-based structural matching of normalized local logits. Another common misconception is to read it as a feature-alignment method. The paper explicitly positions iCD as retaining the practical flexibility of logit KD, especially in heterogeneous teacher–student settings where explicit intermediate feature alignment is awkward (Xue et al., 16 Sep 2025).

2. Formal construction

The point of departure is standard logit distillation. Let $P_T$ and $P_S$ denote global teacher and student logits. The paper writes the conventional logit KD loss as

$$L_{\mathcal{LD}} = \mathcal{KL}(\sigma(P_T)\,\|\,\sigma(P_S)).$$

The teacher’s global logits are expressed as an average over local logits,

$$P_T = \sum_{j=0}^{h_T-1}\sum_{k=0}^{w_T-1}\frac{1}{h_Tw_T}L_T(j,k),$$

and similarly for the student,

$$P_S = \sum_{j=0}^{h_S-1}\sum_{k=0}^{w_S-1}\frac{1}{h_Sw_S}L_S(j,k).$$

This decomposition motivates the central critique: global logits collapse diverse local semantics into a single vector, which can obscure the teacher’s structural rationale (Xue et al., 16 Sep 2025).

iCD extends Scale-Decoupled Distillation (SDD). For a scale set $M$, the feature or logit map is partitioned into non-overlapping cells. At scale $m$, the $n$-th cell is denoted $\mathcal{C}(m,n)$, and the corresponding decoupled local logits are

$P_T$0

These are the local logit representations on which iCD operates (Xue et al., 16 Sep 2025).

The structural stage begins by normalizing the local logits: $P_T$1 The method then constructs Gram matrices,

$P_T$2

The stated role of these Gram matrices is to encode the similarity or correlation structure in the local logit representation. Distillation is then applied not to raw local logits alone, but to the structural distributions induced by the Gram matrices: $P_T$3 Here $P_T$4 is a generic logit-based divergence, allowing iCD to be instantiated on top of KD, DKD, or NKD (Xue et al., 16 Sep 2025).

The full iCD loss sums these structural divergences across scales and cells: $P_T$5 with scale weights

$P_T$6

where $P_T$7 is the ordinal position of scale $P_T$8 in the scale set. This weighting gives larger emphasis to finer scales, on the claim that they contain richer fine-grained information (Xue et al., 16 Sep 2025).

3. Objective, variants, and implementation

The training objective combines supervised classification, local logit distillation, and structural distillation: $P_T$9 Here $P_S$0 is the classification loss, $P_S$1 is the scale-decoupled logit distillation term, and $P_S$2 is the implicit clustering distillation term. The paper reports $P_S$3, $P_S$4, and an $P_S$5 that follows the original SDD setting (Xue et al., 16 Sep 2025).

The method is modular. Because $P_S$6 is generic, the paper defines three named variants: iCD-KD, iCD-DKD, and iCD-NKD. This is important for the taxonomy of the method: iCD is not presented as a replacement for logit KD, but as a structural add-on that can strengthen several logit-distillation baselines (Xue et al., 16 Sep 2025).

Training follows a standard image-classification regime. On CIFAR-100 and CUB-200, the reported settings are SGD, 240 epochs, batch size $P_S$7, learning rate $P_S$8 for ShuffleNet and MobileNetV2, learning rate $P_S$9 for VGG, ResNet, and WRN, momentum $$L_{\mathcal{LD}} = \mathcal{KL}(\sigma(P_T)\,\|\,\sigma(P_S)).$$0, and weight decay $$L_{\mathcal{LD}} = \mathcal{KL}(\sigma(P_T)\,\|\,\sigma(P_S)).$$1. The paper also uses a 30-epoch linear warm-up, explicitly because accumulated multi-scale distillation can cause large early gradients (Xue et al., 16 Sep 2025).

The implementation claim of modest overhead is supported by per-batch timings on CUB-200 for a ResNet32$$L_{\mathcal{LD}} = \mathcal{KL}(\sigma(P_T)\,\|\,\sigma(P_S)).$$24 teacher and MobileNetV2 student. For KD, SD-KD, and iCD-KD, the paper reports $$L_{\mathcal{LD}} = \mathcal{KL}(\sigma(P_T)\,\|\,\sigma(P_S)).$$3 ms, $$L_{\mathcal{LD}} = \mathcal{KL}(\sigma(P_T)\,\|\,\sigma(P_S)).$$4 ms, and $$L_{\mathcal{LD}} = \mathcal{KL}(\sigma(P_T)\,\|\,\sigma(P_S)).$$5 ms, respectively. For DKD, SD-DKD, and iCD-DKD, it reports $$L_{\mathcal{LD}} = \mathcal{KL}(\sigma(P_T)\,\|\,\sigma(P_S)).$$6 ms, $$L_{\mathcal{LD}} = \mathcal{KL}(\sigma(P_T)\,\|\,\sigma(P_S)).$$7 ms, and $$L_{\mathcal{LD}} = \mathcal{KL}(\sigma(P_T)\,\|\,\sigma(P_S)).$$8 ms. This places the additional cost of iCD at only about $$L_{\mathcal{LD}} = \mathcal{KL}(\sigma(P_T)\,\|\,\sigma(P_S)).$$9–$$P_T = \sum_{j=0}^{h_T-1}\sum_{k=0}^{w_T-1}\frac{1}{h_Tw_T}L_T(j,k),$$0 ms over SDD in that setting (Xue et al., 16 Sep 2025).

A useful clarification concerns labels. The paper states that iCD can work “without requiring ground-truth labels or feature-space alignment,” but that statement applies to the structural transfer signal itself. The reported end-to-end training objective still includes $$P_T = \sum_{j=0}^{h_T-1}\sum_{k=0}^{w_T-1}\frac{1}{h_Tw_T}L_T(j,k),$$1, so the full training recipe is not label-free in the strict sense (Xue et al., 16 Sep 2025).

4. Empirical behavior

The reported evaluation covers CIFAR-100 and CUB-200-2011, with multiple heterogeneous teacher–student pairs, including ResNet32$$P_T = \sum_{j=0}^{h_T-1}\sum_{k=0}^{w_T-1}\frac{1}{h_Tw_T}L_T(j,k),$$24 $$P_T = \sum_{j=0}^{h_T-1}\sum_{k=0}^{w_T-1}\frac{1}{h_Tw_T}L_T(j,k),$$3 MobileNetV2, WRN40-2 $$P_T = \sum_{j=0}^{h_T-1}\sum_{k=0}^{w_T-1}\frac{1}{h_Tw_T}L_T(j,k),$$4 VGG8, ResNet50 $$P_T = \sum_{j=0}^{h_T-1}\sum_{k=0}^{w_T-1}\frac{1}{h_Tw_T}L_T(j,k),$$5 MobileNetV2, ResNet50 $$P_T = \sum_{j=0}^{h_T-1}\sum_{k=0}^{w_T-1}\frac{1}{h_Tw_T}L_T(j,k),$$6 ShuffleNetV1, VGG13 $$P_T = \sum_{j=0}^{h_T-1}\sum_{k=0}^{w_T-1}\frac{1}{h_Tw_T}L_T(j,k),$$7 MobileNetV2, and others. Across these settings, iCD consistently improves the corresponding KD baselines and usually also improves over SDD (Xue et al., 16 Sep 2025).

Teacher $$P_T = \sum_{j=0}^{h_T-1}\sum_{k=0}^{w_T-1}\frac{1}{h_Tw_T}L_T(j,k),$$8 Student / dataset	Baseline	iCD result
ResNet32$$P_T = \sum_{j=0}^{h_T-1}\sum_{k=0}^{w_T-1}\frac{1}{h_Tw_T}L_T(j,k),$$94 $$P_S = \sum_{j=0}^{h_S-1}\sum_{k=0}^{w_S-1}\frac{1}{h_Sw_S}L_S(j,k).$$0 MobileNetV2 / CIFAR-100	KD 67.72	iCD-KD 69.46
ResNet50 $$P_S = \sum_{j=0}^{h_S-1}\sum_{k=0}^{w_S-1}\frac{1}{h_Sw_S}L_S(j,k).$$1 MobileNetV2 / CIFAR-100	NKD 68.85	iCD-NKD 70.67
ResNet32$$P_S = \sum_{j=0}^{h_S-1}\sum_{k=0}^{w_S-1}\frac{1}{h_Sw_S}L_S(j,k).$$24 $$P_S = \sum_{j=0}^{h_S-1}\sum_{k=0}^{w_S-1}\frac{1}{h_Sw_S}L_S(j,k).$$3 MobileNetV2 / CUB-200	KD 56.09	iCD-KD 61.17
VGG13 $$P_S = \sum_{j=0}^{h_S-1}\sum_{k=0}^{w_S-1}\frac{1}{h_Sw_S}L_S(j,k).$$4 MobileNetV2 / CUB-200	DKD 58.45	iCD-DKD 65.29

The strongest evidence comes from fine-grained classification. The abstract highlights a peak improvement of $$P_S = \sum_{j=0}^{h_S-1}\sum_{k=0}^{w_S-1}\frac{1}{h_Sw_S}L_S(j,k).$$5 over the baseline, specifically for ResNet32$$P_S = \sum_{j=0}^{h_S-1}\sum_{k=0}^{w_S-1}\frac{1}{h_Sw_S}L_S(j,k).$$64 $$P_S = \sum_{j=0}^{h_S-1}\sum_{k=0}^{w_S-1}\frac{1}{h_Sw_S}L_S(j,k).$$7 MobileNetV2 on CUB-200 under KD. The result table also contains larger gains under other comparisons, such as VGG13 $$P_S = \sum_{j=0}^{h_S-1}\sum_{k=0}^{w_S-1}\frac{1}{h_Sw_S}L_S(j,k).$$8 MobileNetV2 under DKD. The paper’s interpretation is that local structural information matters more in fine-grained settings, where subtle semantic differences are often spatially localized (Xue et al., 16 Sep 2025).

Ablation studies reinforce the multi-scale claim. On CUB-200 for ResNet32$$P_S = \sum_{j=0}^{h_S-1}\sum_{k=0}^{w_S-1}\frac{1}{h_Sw_S}L_S(j,k).$$94 $M$0 ShuffleNetV1, KD yields $M$1, while scale choices $M$2, $M$3, $M$4, $M$5, $M$6, $M$7, and $M$8 produce $M$9, $m$0, $m$1, $m$2, $m$3, $m$4, and $m$5, respectively. On VGG13 $m$6 VGG8, the full multi-scale setting $m$7 is again best at $m$8. This suggests that the structural information distilled by iCD is not confined to a single spatial scale (Xue et al., 16 Sep 2025).

The coefficient $m$9 is also tuned empirically. For ResNet32$n$04 $n$1 ShuffleNetV1, the paper reports $n$2, $n$3, $n$4, $n$5, $n$6, $n$7, $n$8, and $n$9 for $\mathcal{C}(m,n)$0, with $\mathcal{C}(m,n)$1 performing best. The authors interpret this as evidence that moderate structural supervision is preferable to either weak or overly dominant structural matching (Xue et al., 16 Sep 2025).

Visualization results are consistent with the numerical findings. The paper notes that iCD may exhibit slightly larger discrepancy than KD in global logit correlation matrices while still achieving better accuracy, and its t-SNE plots show clearer class separation than KD. This suggests that the gain is not simply better global logit imitation, but better transfer of local structural knowledge (Xue et al., 16 Sep 2025).

5. Relation to adjacent research

iCD sits within a broader shift from pointwise imitation toward structural or relational distillation, but its specific mechanism is unusual. Earlier work such as ISD replaced hard instance discrimination with teacher-guided soft neighborhood distributions over sampled anchors, thereby transferring local similarity structure without explicit clustering (Tejankar et al., 2020). A plausible implication is that iCD occupies a different point in the same design space: instead of distilling soft neighborhoods over instances, it distills structural correlations over decoupled local logits.

Another nearby line is the “implicit structured relation distillation” component of EEIKD, which matches batch-level structured differentiation relationships among teacher and student outputs. That method also avoids explicit clustering assignments, but it operates on average inter-sample distances in a mini-batch rather than on multi-scale local logit structure (Wang et al., 2023). Relative to that formulation, iCD is more tightly tied to dense spatial logits and to fine-grained regional semantics.

By contrast, several clustering-distillation methods use explicit cluster machinery. DistilMVC constructs K-means pseudo-labels on momentum-teacher features, aligns them by Hungarian matching, and distills softened cluster-label distributions as “dark knowledge” (Wang et al., 2023). DisClusterDA performs target-domain clustering with centroids, Fisher-like criteria, and centroid ordering in unsupervised domain adaptation (Tang et al., 2023). iCD differs from both because it introduces no explicit cluster assignments, no centroid bank, and no pseudo-label alignment stage.

Theoretical work on relational knowledge distillation is also directly relevant. “Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clustering” interprets RKD as spectral clustering on a teacher-induced graph and formalizes the idea that relation matching can implicitly recover clustering structure (Dong et al., 2023). This gives an external theoretical lens through which iCD can be read: although iCD is formulated operationally through Gram matrices on local logits, it belongs to a broader family in which structural relations, rather than direct labels, are the distilled object.

6. Interpretation, limitations, and common misunderstandings

The most useful way to interpret iCD is as a structure-aware extension of logit KD. It keeps the architecture-agnostic appeal of output-space distillation, but it injects relational guidance by treating multi-scale local logits as the carriers of latent semantic structure. This suggests that iCD is best understood not as a general clustering algorithm, but as an implicit clustering perspective on logit distillation (Xue et al., 16 Sep 2025).

Several limitations follow directly from the formulation. The method depends on spatially decoupled logits and therefore inherits SDD’s multi-scale partitioning machinery. It is consequently more complex than vanilla KD. It also assumes that local logits are semantically meaningful enough for Gram-based structure to be informative. The paper’s strongest gains appear in fine-grained and heterogeneous teacher–student settings, which suggests that its benefits may be especially pronounced when local evidence is semantically rich and explicit feature alignment is difficult (Xue et al., 16 Sep 2025).

There is also a notational subtlety in the Gram construction. The paper writes

$\mathcal{C}(m,n)$2

while the local logit tensors are described tersely. The intended role is nonetheless clear in the paper: these quantities are treated as similarity or correlation statistics that expose structural knowledge embedded in the logits. This suggests that the conceptual contribution of iCD is clearer than any single tensor-shape interpretation in the presentation (Xue et al., 16 Sep 2025).

A final clarification concerns interpretability. The paper’s interpretability claim is not that iCD yields human-readable explanations. Rather, it argues that standard logit KD transfers only final decision tendencies, whereas iCD also transfers how local semantic evidence is structurally organized. In that narrower but technically meaningful sense, iCD is an attempt to make logit distillation more structurally informative and more semantically grounded (Xue et al., 16 Sep 2025).