Confusion Distillation Overview

• Confusion Distillation is a family of techniques that uses class-confusion structures as supervisory signals instead of traditional teacher posteriors.
• It includes teacher-free self-distillation with dataset-level confusion matrices, class-incremental methods with expert models, and triage KD for speaker verification.
• The approach improves model regularization and discrimination, yielding measurable gains in accuracy and reduced error rates across various applications.

Confusion Distillation (CD) denotes a set of distillation strategies that treat class-confusion structure as the informative supervisory signal, rather than relying exclusively on a conventional teacher posterior. In the 2026 teacher-free formulation, CD replaces the usual teacher soft targets with the student’s own dataset-level confusion statistics, using the evolving confusion matrix as dynamic “dark knowledge” (Kang et al., 2 Jun 2026). In class-incremental continual learning, Confusion/Discriminative Distillation identifies confusing old–new class pairs, trains an expert model on those pairs, and transfers a discriminative margin-based signal to the student (Zhong et al., 2021). In speaker verification, Triage KD (TRKD) operationalizes an assess–prioritize–focus confusion-distillation strategy by separating the teacher posterior into target, confusion-set, and background components, then distilling only the informative non-target structure (Kim et al., 21 Jan 2026). Taken together, these works suggest that CD is best understood as a family of methods organized around the selective transfer of inter-class confusion information.

1. Conceptual basis

The central premise of CD is that misclassification structure is not merely an error pattern but a representation of inter-class relationships. In the teacher-free self-distillation formulation, the dataset-level confusion matrix is reported to contain structural information analogous to the teacher’s “Dark Knowledge,” and this observation motivates replacing the external teacher with the student’s own evolving confusion statistics (Kang et al., 2 Jun 2026). In the continual-learning formulation, the motivating claim is different but related: downgraded performance is not attributed solely to catastrophic forgetting, because class confusion among semantically or visually similar classes can also degrade recognition even when old-class knowledge is not entirely lost (Zhong et al., 2021).

This distinction is important. Standard KD transfers a softened class distribution, but the continual-learning work argues that such a signal is global and does not explicitly target the high-similarity class pairs that cause confusion (Zhong et al., 2021). The speaker-verification work sharpens the same criticism in a different setting: classical KD entangles target confidence and non-target structure in one Kullback-Leibler term, and Decoupled KD, although separating target and non-target terms, still treats all non-targets uniformly and remains vulnerable to the long tail of low-probability classes in large-class settings (Kim et al., 21 Jan 2026).

A plausible implication is that CD is not a single loss function but a selection principle: identify which confusion information is structurally meaningful, suppress low-value non-target structure, and distill only the portion that improves discrimination or regularization.

2. Teacher-free CD from dataset-level confusion matrices

In the self-distillation formulation, let the dataset be $\mathcal{D}=\{(x_n,y_n)\}$ with $K$ classes, and for each class $j$ define the index set $\mathcal{D}_j=\{n \mid y_n=j\}$. At epoch $t$, the student’s hard predictions are collected as $\hat y_n^{(t)}=\arg\max_i p_i(x_n)$, and the raw confusion ratio matrix $C^{(t)}\in\mathbb{R}^{K\times K}$ is defined by

$$C^{(t)}_{j,i} = \frac{1}{|\mathcal{D}_j|} \sum_{n\in\mathcal{D}_j} \mathbf{1}\bigl(\hat y_n^{(t)}=i\bigr),$$

so that each row $j$ sums to one and records how often true-class $j$ samples are classified as $K$0 (Kang et al., 2 Jun 2026).

Because $K$1 is noisy immediately after switching to CD, the method initializes an isotropic smoothing prior $K$2 with

$K$3

and maintains an exponential moving average

$K$4

with $K$5 at the epoch $K$6 when CD begins, and $K$7 (Kang et al., 2 Jun 2026).

For a sample $K$8 at epoch $K$9, the $j$0-th row of $j$1 is used as the soft target. If $j$2 denotes the student’s logits and

$j$3

then the soft-label loss is

$j$4

The total loss combines this term with the standard hard cross-entropy:

$j$5

In practice, the reported settings are $j$6, $j$7, and $j$8 (Kang et al., 2 Jun 2026).

The training procedure has two phases. First, the network is trained with standard cross-entropy alone for $j$9. Second, after initializing $\mathcal{D}_j=\{n \mid y_n=j\}$0, each epoch computes raw confusion statistics, updates the EMA, and uses the class-conditional row $\mathcal{D}_j=\{n \mid y_n=j\}$1 as the soft target for each sample of true class $\mathcal{D}_j=\{n \mid y_n=j\}$2 (Kang et al., 2 Jun 2026). Reported implementation details include SGD with momentum $\mathcal{D}_j=\{n \mid y_n=j\}$3, weight decay $\mathcal{D}_j=\{n \mid y_n=j\}$4, batch size $\mathcal{D}_j=\{n \mid y_n=j\}$5, random crop and horizontal flip, a multi-step learning-rate schedule with linear warm-up, transition to CD at epoch $\mathcal{D}_j=\{n \mid y_n=j\}$6 of a $\mathcal{D}_j=\{n \mid y_n=j\}$7-epoch run, smoothing $\mathcal{D}_j=\{n \mid y_n=j\}$8, and a phase alternation schedule $\mathcal{D}_j=\{n \mid y_n=j\}$9 of $t$0 scaled to total epochs (Kang et al., 2 Jun 2026).

3. Regularization view and empirical behavior in self-distillation

The self-distillation paper interprets CD through the Interaction Tensor framework. Its main claim is that effective KD acts as a regularizer that prunes low-frequency, sample-specific features and encourages the student to rely on a compact set of highly reusable features; CD is reported to reproduce this principal effect without an external teacher (Kang et al., 2 Jun 2026). More specifically, feature-level analysis indicates that CD sharply reduces the number of low-frequency features and increases the activation frequency of a smaller core feature set.

This interpretation provides the rationale for calling the confusion matrix a form of dark knowledge. The method does not simply recycle hard predictions; instead, it uses the row-wise confusion ratio as a dataset-level summary of how a class is distributed over alternatives. The reported result is that these confusion-based soft targets act as a feature-level regularizer that suppresses idiosyncratic patterns and promotes highly reusable discriminative features (Kang et al., 2 Jun 2026).

On CIFAR-100, the reported Top-1 / Top-5 accuracies are as follows (Kang et al., 2 Jun 2026):

Method	ResNet-34 (200ep)	ResNet-50 (200ep)
Baseline	77.61 / 93.76	78.48 / 94.53
CS-KD	76.73 / 93.32	76.31 / 92.64
PS-KD (300)	77.33 / 94.34	78.41 / 94.93
CD (200)	77.87 / 94.21	78.63 / 94.78
CD (300)	78.53 / 94.44	79.38 / 94.93

The accompanying interpretation is precise. On ResNet-34, CD outperforms CS-KD by $t$1 in Top-1 and $t$2 in Top-5, and matches or exceeds PS-KD with fewer epochs. On ResNet-50, CD yields a $t$3 Top-1 gain over baseline and $t$4 over CS-KD in $t$5 epochs (Kang et al., 2 Jun 2026).

The ablations also delimit the method’s operating regime. EMA smoothing with $t$6 is described as crucial, because without it, per-epoch fluctuations propagate and impair stability. Initializing $t$7 with the smoothing prior $t$8 prevents early noise from biasing the soft targets. Lengthening the confusion phase beyond the optimal schedule degrades performance, indicating that overly reinforcing rare confusion signals can hurt (Kang et al., 2 Jun 2026).

4. Confusion/Discriminative Distillation in continual learning

In class-incremental continual learning, CD is formulated around the distinction between catastrophic forgetting and class confusion. Catastrophic forgetting is quantified as

$t$9

whereas class confusion is defined as confusion among semantically or visually similar classes, old vs. new or old vs. old, even when the old-class knowledge is not entirely lost (Zhong et al., 2021). The motivating example uses CIFAR-100 human-face categories such as “baby,” “girl,” “boy,” “man,” and “woman,” where a model may remember “baby” yet fail to discriminate it from “girl” when the latter appears later.

The algorithm at each incremental round consists of three modules. First, confusing class pairs are identified by extracting old-model features, computing class centroids

$\hat y_n^{(t)}=\arg\max_i p_i(x_n)$0

and, for each new class, finding the top-$\hat y_n^{(t)}=\arg\max_i p_i(x_n)$1 nearest old centroids by Euclidean distance. Second, an expert teacher $\hat y_n^{(t)}=\arg\max_i p_i(x_n)$2 is trained on the new classes together with the selected confusing old classes. Third, the student is trained on all seen classes using a combination of classification loss, standard KD from the old model, and a discriminative loss on the confusing pairs (Zhong et al., 2021).

The three loss terms are

$\hat y_n^{(t)}=\arg\max_i p_i(x_n)$3

$\hat y_n^{(t)}=\arg\max_i p_i(x_n)$4

$\hat y_n^{(t)}=\arg\max_i p_i(x_n)$5

and the hinge-margin discriminative term

$\hat y_n^{(t)}=\arg\max_i p_i(x_n)$6

The total minibatch objective is

$\hat y_n^{(t)}=\arg\max_i p_i(x_n)$7

where $\hat y_n^{(t)}=\arg\max_i p_i(x_n)$8, $\hat y_n^{(t)}=\arg\max_i p_i(x_n)$9, and $C^{(t)}\in\mathbb{R}^{K\times K}$0 (Zhong et al., 2021).

This module is explicitly designed to be added to existing class-incremental learners. The description states that it can be plugged into iCaRL, LwF, and EWC by preserving the original backbone loss and adding $C^{(t)}\in\mathbb{R}^{K\times K}$1; for EWC, the usual regularizer $C^{(t)}\in\mathbb{R}^{K\times K}$2 is retained in addition to $C^{(t)}\in\mathbb{R}^{K\times K}$3 (Zhong et al., 2021).

The reported protocol covers CIFAR-100, mini-ImageNet, and ImageNet-1K, using ResNet32 for CIFAR and ResNet18 for ImageNet. On CIFAR-100 with $C^{(t)}\in\mathbb{R}^{K\times K}$4 classes per round and memory $C^{(t)}\in\mathbb{R}^{K\times K}$5, iCaRL baseline final accuracy is reported as approximately $C^{(t)}\in\mathbb{R}^{K\times K}$6, while iCaRL$C^{(t)}\in\mathbb{R}^{K\times K}$7CD reaches approximately $C^{(t)}\in\mathbb{R}^{K\times K}$8; UCIR baseline is approximately $C^{(t)}\in\mathbb{R}^{K\times K}$9, while UCIR$$C^{(t)}_{j,i} = \frac{1}{|\mathcal{D}_j|} \sum_{n\in\mathcal{D}_j} \mathbf{1}\bigl(\hat y_n^{(t)}=i\bigr),$$0CD reaches approximately $$C^{(t)}_{j,i} = \frac{1}{|\mathcal{D}_j|} \sum_{n\in\mathcal{D}_j} \mathbf{1}\bigl(\hat y_n^{(t)}=i\bigr),$$1 (Zhong et al., 2021). On mini-ImageNet, the average gain is approximately $$C^{(t)}_{j,i} = \frac{1}{|\mathcal{D}_j|} \sum_{n\in\mathcal{D}_j} \mathbf{1}\bigl(\hat y_n^{(t)}=i\bigr),$$2, and on ImageNet-1K approximately $$C^{(t)}_{j,i} = \frac{1}{|\mathcal{D}_j|} \sum_{n\in\mathcal{D}_j} \mathbf{1}\bigl(\hat y_n^{(t)}=i\bigr),$$3.

The work also attempts to isolate confusion reduction. On CIFAR-100, UCIR at round $$C^{(t)}_{j,i} = \frac{1}{|\mathcal{D}_j|} \sum_{n\in\mathcal{D}_j} \mathbf{1}\bigl(\hat y_n^{(t)}=i\bigr),$$4 shows confusion errors decreasing from $$C^{(t)}_{j,i} = \frac{1}{|\mathcal{D}_j|} \sum_{n\in\mathcal{D}_j} \mathbf{1}\bigl(\hat y_n^{(t)}=i\bigr),$$5 to $$C^{(t)}_{j,i} = \frac{1}{|\mathcal{D}_j|} \sum_{n\in\mathcal{D}_j} \mathbf{1}\bigl(\hat y_n^{(t)}=i\bigr),$$6 and forgetting errors from $$C^{(t)}_{j,i} = \frac{1}{|\mathcal{D}_j|} \sum_{n\in\mathcal{D}_j} \mathbf{1}\bigl(\hat y_n^{(t)}=i\bigr),$$7 to $$C^{(t)}_{j,i} = \frac{1}{|\mathcal{D}_j|} \sum_{n\in\mathcal{D}_j} \mathbf{1}\bigl(\hat y_n^{(t)}=i\bigr),$$8 (Zhong et al., 2021). An ablation over the number of similar old classes per new class reports that $$C^{(t)}_{j,i} = \frac{1}{|\mathcal{D}_j|} \sum_{n\in\mathcal{D}_j} \mathbf{1}\bigl(\hat y_n^{(t)}=i\bigr),$$9 gives the best average improvement, whereas larger $j$0 can dilute discriminative focus. Reported hyperparameter studies place the best values around $j$1, margin $j$2, and KD temperature $j$3 (Zhong et al., 2021).

The limitations are explicit. Training an extra expert model at each round roughly doubles training time per round, and extending from $j$4 to large $j$5 or all pairs leads to $j$6 loss terms (Zhong et al., 2021).

5. Triage KD as confusion distillation in speaker verification

The speaker-verification work does not use the bare name “Confusion Distillation” as the method title, but it gives an explicit account of how Triage Knowledge Distillation implements a confusion-distillation strategy (Kim et al., 21 Jan 2026). The problem setting is speaker verification on resource-constrained devices, where high-capacity teachers are effective but costly. The method starts from the claim that classical KD entangles target confidence with non-target structure in one KL term, and that even Decoupled KD still transfers long-tail noise because it treats all non-targets uniformly (Kim et al., 21 Jan 2026).

At each training step $j$7, the teacher’s posterior over $j$8 classes,

$j$9

is partitioned into three groups: the target class $j$0, a confusion-set $j$1 consisting of the smallest subset of non-targets whose cumulative teacher probability is at least $j$2, and the remaining non-targets, the background $j$3 (Kim et al., 21 Jan 2026). Formally,

$j$4

$j$5

The aggregated masses are

$j$6

for $j$7, and the within-set normalized distributions are

$j$8

Once $j$9 is selected, TRKD aligns the student’s conditional distribution over $K$00 to the teacher’s via

$K$01

At the coarse level, it transfers the top-level KL over the three aggregated masses:

$K$02

The background-set conditional term $K$03 is defined analogously but discarded in the final loss. The final distillation objective is

$K$04

typically with $K$05 and $K$06, and the complete training objective is

$K$07

All of these components are stated explicitly in the method description (Kim et al., 21 Jan 2026).

The “focus” stage is implemented by a curriculum on $K$08. A clipped progress variable is defined over $K$09:

$K$10

and $K$11 follows an exponential decay:

$K$12

The reported settings are $K$13, $K$14, $K$15, $K$16, and $K$17 (Kim et al., 21 Jan 2026). The stated rationale is that early training with large $K$18 includes broad non-target context and behaves like DKD, whereas later training with small $K$19 shrinks $K$20 to the hardest impostors.

Empirically, the method is evaluated on VoxCeleb1 original, extended, and hard trials using both homogeneous and heterogeneous teacher–student pairs. Across $K$21 evaluations, TRKD attains the lowest EER in every case; examples include ECAPA1024$K$22ECAPA400, where the student without KD has EER $K$23 and TRKD reaches $K$24; ReDim-B5$K$25ReDim-B2, where $K$26 becomes $K$27; and RN152$K$28MNV2, where $K$29 becomes $K$30 (Kim et al., 21 Jan 2026). The reported average relative EER reduction over the no-KD baseline is approximately $K$31, and ablations show that the three-mass term and confusion-set conditional term are complementary and that the $K$32-curriculum is essential for training stability and peak performance.

6. Comparative interpretation, misconceptions, and directions

The three formulations can be compared by the origin of their confusion signal and the way that signal is used:

Formulation	Source of confusion information	Main distilled signal
Teacher-free CD	Student’s dataset-level confusion matrix $K$33	KL to class-conditional confusion row
Continual-learning CD	Nearest confusing old classes plus expert teacher	Hinge-margin discriminative loss with standard KD
TRKD	Teacher posterior partitioned by cumulative cutoff $K$34	Three-mass KL plus confusion-set conditional KL

A common misconception would be to treat CD as synonymous with teacher-free self-distillation. The literature here does not support that simplification. One formulation is explicitly teacher-free and replaces the teacher by dataset-level confusion statistics (Kang et al., 2 Jun 2026); another depends on an extra expert teacher at each incremental round (Zhong et al., 2021); and TRKD uses a conventional teacher but distills only the target, confusion-set, and background structure judged to be informative (Kim et al., 21 Jan 2026).

A second misconception would be to assume that all non-target information is equally useful. The speaker-verification formulation explicitly discards the background-set conditional term, the teacher-free formulation reports that overly reinforcing rare confusion signals can hurt, and the continual-learning formulation selects only the most similar old classes per new class, with $K$35 reported as the best average setting (Kim et al., 21 Jan 2026). This convergence suggests that the operative principle in CD is selective transfer rather than exhaustive imitation.

The current literature also indicates distinct practical trade-offs. Teacher-free CD avoids the need for a separate pretrained teacher and is presented as a computationally efficient alternative to standard KD (Kang et al., 2 Jun 2026). Continual-learning CD improves discrimination but incurs the overhead of expert-model training and faces scalability concerns when the confusion set grows (Zhong et al., 2021). TRKD is designed for large-class speaker verification and couples confusion selection to a curriculum that changes over training, which is presented as essential for stability and peak performance (Kim et al., 21 Jan 2026).

The explicit future directions appear mainly in the continual-learning line: embedding a metric-learning head in the expert model, using a contrastive-learning loss instead of the hinge-margin loss, automatically adapting $K$36 and the margin based on observed confusion rates, and extending Confusion Distillation to multi-modal or few-shot continual learning (Zhong et al., 2021). More broadly, the collected evidence suggests that CD is emerging as a general strategy for converting confusion structure into supervisory signal, with implementations ranging from dataset-level self-distillation to pairwise discriminative training and posterior triage.