RSD: Redundancy Suppression Distillation

• RSD is a cross-architecture distillation method that extracts shared, input-dependent semantic information while suppressing redundant, model-specific noise.
• It employs a lightweight Architecture-Agnostic Knowledge Decoupler (AAD) to align heterogeneous teacher and student representations without overriding unique student features.
• Empirical results show that RSD improves accuracy and efficiency across diverse architecture pairs, using fewer parameters and lower GPU memory than prior methods.

Searching arXiv for the primary paper and closely related works mentioned in the provided data. Searching arXiv for the primary paper. Searching arXiv for the primary paper and related works named in the source material. Searching arXiv for the cited paper. Querying arXiv metadata for the primary paper and related named works. Redundancy Suppression Distillation (RSD) is a method for cross-architecture knowledge distillation in which knowledge transfer is cast into a redundant information suppression formulation. It is proposed for heterogeneous teacher–student settings, where a frozen teacher and a randomly initialized student may differ substantially in architectural bias, including CNN, ViT, and MLP pairs. The central premise is that heterogeneous representations contain both shared, input-dependent, architecture-agnostic semantic information and model-exclusive redundant patterns; RSD attempts to maximize the former while suppressing the latter through a simple correlation-based objective and a lightweight auxiliary module (Zhang et al., 29 Jul 2025).

1. Problem setting and conceptual basis

RSD is introduced against the background of cross-architecture distillation methods that introduce sophisticated modules, architecture-tailored designs, and excessive parameters, which impair their efficiency and applicability. In contrast, RSD is presented as a simple method that extracts architecture-agnostic knowledge in heterogeneous representations by reducing redundant architecture-exclusive information, and it is explicitly described as devoid of the architecture-specific designs and complex operations in the pioneering method of OFA (Zhang et al., 29 Jul 2025).

The method models the penultimate representations of teacher and student as mixtures of two components: the same, input-dependent, architecture-agnostic “semantic” information, and each model’s own architecture-specific “redundant” patterns. On this view, a CNN may encode local textures, a ViT may encode global patch interactions, and an MLP may emphasize channel-wise mixing, yet all may still encode the same semantic content for the same input. RSD therefore aims to strip away what exists only in the teacher or only in the student and to retain what both representations agree upon.

The paper frames this as a suppression problem rather than as a direct feature imitation problem. Maximizing the shared component is treated as an invariance objective, while discouraging cross-talk among heterogeneous dimensions is treated as a redundancy-reduction objective. This suggests a shift in emphasis from architecture matching to representation purification: instead of forcing a student to reproduce all aspects of a teacher representation, RSD tries to isolate the part of the signal that survives architectural mismatch.

2. Formal loss and optimization target

Let $(x,y)$ be an input–label pair, $T$ a frozen teacher network, and $S$ a randomly initialized student. The teacher penultimate embedding is written as $z^t\in\mathbb{R}^{B\times D_t}$, the student penultimate embedding as $z^s\in\mathbb{R}^{B\times D_s}$, and a small MLP decoupler as $$h(\cdot):\mathbb{R}^{D_s}\rightarrow\mathbb{R}^{D_t}$$. Each feature dimension is center-normalized over the batch of size $B$:

$$\hat z^t_{k,i} = \frac{z^t_{k,i}-\mu^t_i}{\sigma^t_i}, \qquad \hat z^s_{k,j} = \frac{h(z^s)_{k,j}-\mu^s_j}{\sigma^s_j}.$$

The cross-correlation matrix $P\in\mathbb{R}^{D_t\times D_t}$ is then defined by

$$P_{ij} = \frac{1}{B}\sum_{k=1}^B \hat z^t_{k,i}\cdot \hat z^s_{k,j}.$$

RSD consists of two terms. The invariance maximization term operates on the diagonal,

$T$0

and is described as driving each teacher dimension $T$1 to correlate perfectly with student dimension $T$2. The redundancy suppression term operates on the off-diagonals,

$T$3

and pushes different dimensions to be decorrelated. The resulting batch-level RSD loss is

$T$4

or, in matrix form, $T$5 with off-diagonals multiplied by $T$6 before squaring. Student training uses the usual cross-entropy plus RSD:

$T$7

Typical choices in the reported experiments are $T$8 and $T$9, selected by a short grid search. The paper characterizes the diagonal term as an information-maximization step and the off-diagonal term as a classic redundancy-reduction objective. It further states that, taken together, RSD can be viewed as an approximation to extracting the shared subspace of two random variables under an information-bottleneck perspective: maximize $S$0 on the shared dimensions while minimizing $S$1 on all spurious pairings (Zhang et al., 29 Jul 2025).

3. Architecture-Agnostic Knowledge Decoupler

A central auxiliary component is the Architecture-Agnostic Knowledge Decoupler (AAD), a lightweight module designed to prevent the student from entirely losing its architecture-specific capabilities. The paper argues that if $S$2 were attached directly to $S$3, the method might override beneficial student-exclusive traits. To avoid this, the student representation is routed through a tiny MLP $S$4 whose only job is to produce an RSD-compatible representation; this module is discarded at inference (Zhang et al., 29 Jul 2025).

Concretely, AAD contains an “expander” layer $S$5, followed by BatchNorm and GeLU, and then an “adaptor” layer $S$6 with no bias. The paper states that $S$7 or $S$8 is enough. At training time, the transformed student representation is

$S$9

and RSD is applied on $z^t\in\mathbb{R}^{B\times D_t}$0.

This decoupling is methodologically important because it separates the optimization target used for cross-architecture matching from the student’s native internal representation. A recurring misconception in heterogeneous distillation is that stronger alignment should always be imposed directly on the student features; the ablations reported for RSD contradict that simplification. When RSD is applied directly to $z^t\in\mathbb{R}^{B\times D_t}$1 without AAD, the student loses its exclusive inductive biases, with a reported drop of approximately $z^t\in\mathbb{R}^{B\times D_t}$2 on CIFAR and approximately $z^t\in\mathbb{R}^{B\times D_t}$3 on ImageNet.

4. Training protocol and implementation profile

The reported experiments follow the same training schedules as OFA for a fair comparison. On CIFAR-100, the schedule is 200 epochs with SGD and momentum $z^t\in\mathbb{R}^{B\times D_t}$4, initial learning rate $z^t\in\mathbb{R}^{B\times D_t}$5, step decay at epochs $z^t\in\mathbb{R}^{B\times D_t}$6 by $z^t\in\mathbb{R}^{B\times D_t}$7, weight decay $z^t\in\mathbb{R}^{B\times D_t}$8, batch size $z^t\in\mathbb{R}^{B\times D_t}$9, and default $z^s\in\mathbb{R}^{B\times D_s}$0, $z^s\in\mathbb{R}^{B\times D_s}$1. Teacher–student pairs include examples such as Swin-T$z^s\in\mathbb{R}^{B\times D_s}$2ResNet-18, ViT-S$z^s\in\mathbb{R}^{B\times D_s}$3MobileNetV2, Mixer-B/16$z^s\in\mathbb{R}^{B\times D_s}$4ResNet-18, and ConvNeXt-T$z^s\in\mathbb{R}^{B\times D_s}$5DeiT-T (Zhang et al., 29 Jul 2025).

On ImageNet-1k, the schedule is 100 epochs with AdamW for Transformers or SGD, initial learning rate $z^s\in\mathbb{R}^{B\times D_s}$6 with a cosine schedule and 10-epoch warmup, weight decay $z^s\in\mathbb{R}^{B\times D_s}$7, batch size $z^s\in\mathbb{R}^{B\times D_s}$8 per GPU, and $z^s\in\mathbb{R}^{B\times D_s}$9, $$h(\cdot):\mathbb{R}^{D_s}\rightarrow\mathbb{R}^{D_t}$$0. Teacher–student pairs include ResNet50$$h(\cdot):\mathbb{R}^{D_s}\rightarrow\mathbb{R}^{D_t}$$1DeiT-T, ConvNeXt-T$$h(\cdot):\mathbb{R}^{D_s}\rightarrow\mathbb{R}^{D_t}$$2Swin-N, Swin-T$$h(\cdot):\mathbb{R}^{D_s}\rightarrow\mathbb{R}^{D_t}$$3ResMLP-S12, and 12 others.

All experiments are implemented in PyTorch. The paper reports that RSD adds only approximately $$h(\cdot):\mathbb{R}^{D_s}\rightarrow\mathbb{R}^{D_t}$$4–$$h(\cdot):\mathbb{R}^{D_s}\rightarrow\mathbb{R}^{D_t}$$5 parameters for AAD, compared to OFA’s $$h(\cdot):\mathbb{R}^{D_s}\rightarrow\mathbb{R}^{D_t}$$6–$$h(\cdot):\mathbb{R}^{D_s}\rightarrow\mathbb{R}^{D_t}$$7. In the specific ConvNeXt-T$$h(\cdot):\mathbb{R}^{D_s}\rightarrow\mathbb{R}^{D_t}$$8Swin-N example, OFA requires $$h(\cdot):\mathbb{R}^{D_s}\rightarrow\mathbb{R}^{D_t}$$9 extra parameters, whereas RSD requires $B$0, described as $B$1 smaller, and RSD uses approximately $B$2 lower peak GPU memory than OFA. Within the paper’s framing, these numbers support the claim that the method’s simplicity is not merely architectural but also computational.

5. Reported empirical behavior

On CIFAR-100, using Top-1 accuracy with single-crop evaluation, the average un-distilled student is reported at approximately $B$3, OFA at approximately $B$4, and RSD at $B$5, corresponding to $B$6 relative to OFA. The summary further reports, over 12 heterogeneous pairs, a $B$7 margin versus scratch and $B$8 versus OFA (Zhang et al., 29 Jul 2025).

On ImageNet-1k, the average un-distilled student is reported at approximately $B$9, OFA at approximately $$\hat z^t_{k,i} = \frac{z^t_{k,i}-\mu^t_i}{\sigma^t_i}, \qquad \hat z^s_{k,j} = \frac{h(z^s)_{k,j}-\mu^s_j}{\sigma^s_j}.$$0, and RSD at $$\hat z^t_{k,i} = \frac{z^t_{k,i}-\mu^t_i}{\sigma^t_i}, \qquad \hat z^s_{k,j} = \frac{h(z^s)_{k,j}-\mu^s_j}{\sigma^s_j}.$$1, corresponding to $$\hat z^t_{k,i} = \frac{z^t_{k,i}-\mu^t_i}{\sigma^t_i}, \qquad \hat z^s_{k,j} = \frac{h(z^s)_{k,j}-\mu^s_j}{\sigma^s_j}.$$2 versus OFA. The paper presents these benchmark results as evidence that the method remains effective when moving from CIFAR-100 to a larger-scale setting.

The ablation results isolate the contribution of each component. Using only $$\hat z^t_{k,i} = \frac{z^t_{k,i}-\mu^t_i}{\sigma^t_i}, \qquad \hat z^s_{k,j} = \frac{h(z^s)_{k,j}-\mu^s_j}{\sigma^s_j}.$$3 by setting $$\hat z^t_{k,i} = \frac{z^t_{k,i}-\mu^t_i}{\sigma^t_i}, \qquad \hat z^s_{k,j} = \frac{h(z^s)_{k,j}-\mu^s_j}{\sigma^s_j}.$$4 yields typically $$\hat z^t_{k,i} = \frac{z^t_{k,i}-\mu^t_i}{\sigma^t_i}, \qquad \hat z^s_{k,j} = \frac{h(z^s)_{k,j}-\mu^s_j}{\sigma^s_j}.$$5–$$\hat z^t_{k,i} = \frac{z^t_{k,i}-\mu^t_i}{\sigma^t_i}, \qquad \hat z^s_{k,j} = \frac{h(z^s)_{k,j}-\mu^s_j}{\sigma^s_j}.$$6 over scratch. Adding $$\hat z^t_{k,i} = \frac{z^t_{k,i}-\mu^t_i}{\sigma^t_i}, \qquad \hat z^s_{k,j} = \frac{h(z^s)_{k,j}-\mu^s_j}{\sigma^s_j}.$$7 with $$\hat z^t_{k,i} = \frac{z^t_{k,i}-\mu^t_i}{\sigma^t_i}, \qquad \hat z^s_{k,j} = \frac{h(z^s)_{k,j}-\mu^s_j}{\sigma^s_j}.$$8 provides a further $$\hat z^t_{k,i} = \frac{z^t_{k,i}-\mu^t_i}{\sigma^t_i}, \qquad \hat z^s_{k,j} = \frac{h(z^s)_{k,j}-\mu^s_j}{\sigma^s_j}.$$9–$P\in\mathbb{R}^{D_t\times D_t}$0. Removing AAD and applying RSD directly to $P\in\mathbb{R}^{D_t\times D_t}$1 produces the previously noted degradation. The paper also reports that applying RSD as a pure logit distiller, using $P\in\mathbb{R}^{D_t\times D_t}$2, still outperforms KD, DKD, and OFA-logit by $P\in\mathbb{R}^{D_t\times D_t}$3–$P\in\mathbb{R}^{D_t\times D_t}$4. A plausible implication is that the redundancy-suppression principle is not confined to penultimate embeddings, even though the paper recommends those embeddings for the simplest plug-and-play use.

6. Practical usage, interpretation, and scope

The paper gives several practical guidelines for new architecture pairs. It recommends operating on penultimate embeddings when the goal is the simplest plug-and-play deployment, because those embeddings are 1-D, universal, and carry rich semantic information. When $P\in\mathbb{R}^{D_t\times D_t}$5, it recommends using AAD with two small fully connected layers to align dimensions and to avoid overwriting student-only features. It further recommends normalizing each embedding dimension over the batch to zero mean and unit variance before computing $P\in\mathbb{R}^{D_t\times D_t}$6 (Zhang et al., 29 Jul 2025).

For hyperparameter selection, the paper advises keeping the off-diagonal weight $P\in\mathbb{R}^{D_t\times D_t}$7 low, in the range $P\in\mathbb{R}^{D_t\times D_t}$8–$P\in\mathbb{R}^{D_t\times D_t}$9, in order to softly suppress redundancy; making $$P_{ij} = \frac{1}{B}\sum_{k=1}^B \hat z^t_{k,i}\cdot \hat z^s_{k,j}.$$0 too large can force student dimensions to be fully independent and hurt performance. It advises choosing $$P_{ij} = \frac{1}{B}\sum_{k=1}^B \hat z^t_{k,i}\cdot \hat z^s_{k,j}.$$1 by starting at $$P_{ij} = \frac{1}{B}\sum_{k=1}^B \hat z^t_{k,i}\cdot \hat z^s_{k,j}.$$2 or $$P_{ij} = \frac{1}{B}\sum_{k=1}^B \hat z^t_{k,i}\cdot \hat z^s_{k,j}.$$3, and lowering $$P_{ij} = \frac{1}{B}\sum_{k=1}^B \hat z^t_{k,i}\cdot \hat z^s_{k,j}.$$4 if the student’s training loss becomes unstable. It also states that a single RSD loss on the final embedding is often enough and that multi-stage features are rarely needed; if necessary, additional RSD heads can be attached to intermediate embedding outputs, each with its own small adaptor MLP.

The paper finally recommends keeping batch size large enough that the sample correlation estimates in $$P_{ij} = \frac{1}{B}\sum_{k=1}^B \hat z^t_{k,i}\cdot \hat z^s_{k,j}.$$5 are stable, phrased as $$P_{ij} = \frac{1}{B}\sum_{k=1}^B \hat z^t_{k,i}\cdot \hat z^s_{k,j}.$$6, with the example $$P_{ij} = \frac{1}{B}\sum_{k=1}^B \hat z^t_{k,i}\cdot \hat z^s_{k,j}.$$7 for $$P_{ij} = \frac{1}{B}\sum_{k=1}^B \hat z^t_{k,i}\cdot \hat z^s_{k,j}.$$8. This operational detail is important because the entire method depends on batch-estimated cross-correlation. More broadly, RSD is positioned as replacing the heavy, tailored projectors of OFA with one tiny MLP and a two-term correlation loss. Within the paper’s own interpretation, its significance lies in treating cross-architecture distillation as suppression of cross-architecture clutter combined with amplification of shared signal, rather than as explicit architectural reconciliation.