Subspace Knowledge Distillation Strategy

• Subspace knowledge distillation strategies are methods that transfer the teacher’s structure by aligning low-dimensional subspaces instead of mimicking full output distributions.
• They employ techniques like SVD, PCA, and manifold approximations to capture the geometric organization of features for more efficient student learning.
• These methods show practical benefits across domains such as continual learning, structured prediction, and parameter-efficient fine-tuning while prompting choices on optimal subspace design.

Searching arXiv for relevant papers on subspace knowledge distillation and closely related KD strategies. Subspace knowledge distillation strategy denotes a family of knowledge distillation methods in which the transferred object is not restricted to final softened predictions, but is instead formulated as a subspace, coordinate system, low-rank manifold approximation, projected intermediate representation, or student-compatible region of the teacher’s solution space. Across this literature, the common shift is from pointwise imitation toward geometry-aware or structure-aware transfer: features may be aligned in principal subspaces computed by SVD or PCA, teacher layers may be combined into a projected intermediate representation, structured predictors may be matched on local sub-structures rather than the full output space, and low-rank adapters may be regularized by principal angles on the Grassmannian (Roy et al., 2023, Zhou et al., 2024, Roy et al., 9 Mar 2026, Tariq et al., 5 Jul 2026). This suggests that “subspace” in KD is best understood as an umbrella notion for transferring the teacher’s organization of information rather than only its terminal outputs.

1. Conceptual scope and defining forms

A central distinction within knowledge distillation is whether the student is asked to imitate teacher outputs directly or to inherit a representation geometry. Several papers make this distinction explicit. “Subspace Distillation for Continual Learning” states that standard KD usually matches output probabilities, logits, or raw intermediate features, whereas its method preserves the geometry of the feature space through low-dimensional linear subspaces (Roy et al., 2023). “All You Need in Knowledge Distillation Is a Tailored Coordinate System” argues that dark knowledge can be captured by the coordinate system or linear subspace where the features lie in, rather than by raw logits or raw features (Zhou et al., 2024). “SPREAD: Subspace Representation Distillation for Lifelong Imitation Learning” similarly replaces raw-space $L_2$ feature matching with low-rank subspace alignment intended to preserve the underlying geometry of multimodal features (Roy et al., 9 Mar 2026).

The term also extends beyond linear feature geometry. “Triplet Knowledge Distillation” introduces an anchor model that delimits a subspace within the full solution space of the target problem; the teacher is then constrained to express solutions within that student-friendly region (Wang et al., 2023). “Efficient Sub-structured Knowledge Distillation” transfers knowledge by locally matching predictions on all sub-structures rather than the whole output space, so its “subspace” is combinatorial and structural rather than Euclidean (Lin et al., 2022). “SAD-LoRA: Spectral Alignment for Low-Rank Knowledge Distillation” moves the idea into parameter-efficient fine-tuning, where the distilled object is the low-rank adapter subspace itself (Tariq et al., 5 Jul 2026).

A useful synthesis is that subspace KD strategies differ primarily in the object being restricted or aligned. In one group, the relevant object is a linear feature subspace derived from SVD or PCA. In a second group, it is a projected layer mixture or classifier-compatible feature space. In a third group, it is a function-space or parameter-space region made explicitly student-compatible. The family resemblance lies in replacing unrestricted imitation with a structured target that is smaller, more stable, or more relevant.

2. Linear subspaces, coordinate systems, and manifold approximations

The most direct formulations treat teacher and student representations as elements of high-dimensional spaces whose informative content is concentrated in a low-dimensional subspace. In SPREAD, for a feature matrix $f\in\mathbb{R}^{m\times n}$, a reduced SVD

$f=U\Sigma V^\top$

is computed, and the projection onto the dominant subspace is $UU^\top f$. The subspace distillation loss is

$$\mathcal{L}_{\text{SPREAD}} = \|U_tU_t^\top f_t-U_sU_s^\top f_s\|_F^2 + \|U_tU_t^\top f_s-U_sU_s^\top f_t\|_F^2,$$

with $U_t$ and $U_s$ the top-$r$ left singular vectors of teacher and student features (Roy et al., 9 Mar 2026). The method applies this construction separately to HandEye and AgentView image features, text embeddings, and joint and gripper features, thereby distilling the geometry of multimodal latent representations across consecutive policies.

In continual learning, Subspace Distillation uses thin SVD to approximate a local manifold by a linear subspace. For feature matrices $F_i^{t-1}$ and $F_i^t$, the top $f\in\mathbb{R}^{m\times n}$0 left singular vectors form orthonormal bases $f\in\mathbb{R}^{m\times n}$1 and $f\in\mathbb{R}^{m\times n}$2, and the distance between old and new representations is measured by the projection metric on the Grassmannian: $f\in\mathbb{R}^{m\times n}$3 This basis-invariant comparison is explicitly motivated as a first-order manifold approximation, intended to preserve the local tangent structure of old knowledge while learning new tasks (Roy et al., 2023).

TCS replaces direct feature imitation with transfer of a PCA basis. Teacher features $f\in\mathbb{R}^{m\times n}$4 are centered, PCA is computed through

$f\in\mathbb{R}^{m\times n}$5

and the student feature is projected into the teacher coordinate system by

$f\in\mathbb{R}^{m\times n}$6

A trainable mask $f\in\mathbb{R}^{m\times n}$7 then tailors the coordinate system by sparse coordinate selection, with the reported final target ratio $f\in\mathbb{R}^{m\times n}$8 in the experiments (Zhou et al., 2024). The paper’s central claim is that the transferable knowledge is the representation basis itself.

MoKD adopts a related but distinct construction. Rather than a single shared projection, it learns separate teacher and student subspaces $f\in\mathbb{R}^{m\times n}$9 and $f=U\Sigma V^\top$0 with an indefinite metric $f=U\Sigma V^\top$1, and compares features through

$f=U\Sigma V^\top$2

This is presented as a shared high-dimensional latent space that better accommodates teacher-student mismatch (Hayder et al., 13 May 2025).

3. Intermediate representations, structured outputs, and student-compatible regions

A second line of work uses “subspace” more flexibly, as a projected or restricted region of internal computation. ALP-KD addresses intermediate-layer distillation when teacher and student depths differ. For each student layer $f=U\Sigma V^\top$3, it forms a projected teacher representation

$f=U\Sigma V^\top$4

with attention weights

$f=U\Sigma V^\top$5

In its fully global variant, each student layer attends over all teacher layers, avoiding both the skip problem and the search problem of hard layer selection (Passban et al., 2020). The projected teacher-side representation is effectively a learned intermediate subspace assembled from multiple teacher layers.

SimKD adopts a classifier-anchored representation view. It directly reuses the pretrained teacher classifier for student inference and trains the student encoder only through feature alignment,

$f=U\Sigma V^\top$6

The projector $f=U\Sigma V^\top$7 maps student features into the teacher feature dimension, and the student is optimized to inhabit the teacher classifier’s discriminative geometry rather than to learn a fresh classifier (Chen et al., 2022). This suggests a subspace interpretation in which the teacher classifier defines the relevant coordinate system.

In structured prediction, Efficient Sub-structured Knowledge Distillation avoids distilling over an exponentially large structured output space. For CRF sequence labeling, a sub-structure is a neighboring tag pair $f=U\Sigma V^\top$8, and the student matches the teacher locally through

$f=U\Sigma V^\top$9

The method does not require dynamic programming, K-best search, or global structured inference during distillation (Lin et al., 2022).

Triplet Knowledge Distillation defines a still more abstract subspace. An anchor model of the same capacity as the student constrains both teacher and student through KL terms, shrinking their effective function classes to

$UU^\top f$0

The teacher is thus pushed toward a student-friendly region of the larger hypothesis space (Wang et al., 2023).

4. Optimization mechanisms and training schedules

Subspace KD strategies are often distinguished as much by their optimization mechanisms as by their geometric targets. Several rely on explicit spectral or orthogonality constraints. SubDistill matches only the task-relevant projected teacher component at each layer: $UU^\top f$1 where $UU^\top f$2, $UU^\top f$3 has orthogonal columns, centering is built into the objective, and the paper states that optimization is performed on the Stiefel manifold (Chormai et al., 9 Jan 2026). The relevant teacher subspace $UU^\top f$4 is identified with principal relevant component analysis (PRCA), using the teacher’s margin between the top class and the runner-up.

SAD-LoRA makes the adapter column space itself a distillation target. For each adapted layer, a data-weighted student-space reference update $UU^\top f$5 is formed, its leading left singular vectors $UU^\top f$6 define the target subspace, and alignment is enforced through the principal-angle loss

$UU^\top f$7

The method uses spectral initialization $UU^\top f$8, $UU^\top f$9, so the student reproduces the pretrained backbone at step $$\mathcal{L}_{\text{SPREAD}} = \|U_tU_t^\top f_t-U_sU_s^\top f_s\|_F^2 + \|U_tU_t^\top f_s-U_sU_s^\top f_t\|_F^2,$$0 while starting exactly in the target subspace (Tariq et al., 5 Jul 2026).

MoKD addresses a different bottleneck: balancing the task objective and the distillation objective. It reformulates KD as multi-objective optimization over $$\mathcal{L}_{\text{SPREAD}} = \|U_tU_t^\top f_t-U_sU_s^\top f_s\|_F^2 + \|U_tU_t^\top f_s-U_sU_s^\top f_t\|_F^2,$$1 and $$\mathcal{L}_{\text{SPREAD}} = \|U_tU_t^\top f_t-U_sU_s^\top f_s\|_F^2 + \|U_tU_t^\top f_s-U_sU_s^\top f_t\|_F^2,$$2, defines a convex combination $$\mathcal{L}_{\text{SPREAD}} = \|U_tU_t^\top f_t-U_sU_s^\top f_s\|_F^2 + \|U_tU_t^\top f_s-U_sU_s^\top f_t\|_F^2,$$3, and chooses $$\mathcal{L}_{\text{SPREAD}} = \|U_tU_t^\top f_t-U_sU_s^\top f_s\|_F^2 + \|U_tU_t^\top f_s-U_sU_s^\top f_t\|_F^2,$$4 by minimizing the norm of the combined gradient on the simplex. This is explicitly motivated by gradient conflict and gradient dominance (Hayder et al., 13 May 2025).

Curriculum Extraction constructs a subspace-based curriculum from a fully trained teacher by random projection of hidden representations. For each intermediate layer $$\mathcal{L}_{\text{SPREAD}} = \|U_tU_t^\top f_t-U_sU_s^\top f_s\|_F^2 + \|U_tU_t^\top f_s-U_sU_s^\top f_t\|_F^2,$$5, a projection $$\mathcal{L}_{\text{SPREAD}} = \|U_tU_t^\top f_t-U_sU_s^\top f_s\|_F^2 + \|U_tU_t^\top f_s-U_sU_s^\top f_t\|_F^2,$$6 is chosen, and the student prefix is trained with

$$\mathcal{L}_{\text{SPREAD}} = \|U_tU_t^\top f_t-U_sU_s^\top f_s\|_F^2 + \|U_tU_t^\top f_s-U_sU_s^\top f_t\|_F^2,$$7

Only after this layerwise projected curriculum does the student match the teacher output distribution with KL divergence (Gupta et al., 21 Mar 2025). The paper’s terminology is explicit that the random projection is the mechanism that extracts a useful curriculum from a single final teacher.

PWKD offers a different curriculum construction. The teacher is reconstructed into weight-sharing sub-networks with the same depth but increasing channel width, such as $$\mathcal{L}_{\text{SPREAD}} = \|U_tU_t^\top f_t-U_sU_s^\top f_s\|_F^2 + \|U_tU_t^\top f_s-U_sU_s^\top f_t\|_F^2,$$8, $$\mathcal{L}_{\text{SPREAD}} = \|U_tU_t^\top f_t-U_sU_s^\top f_s\|_F^2 + \|U_tU_t^\top f_s-U_sU_s^\top f_t\|_F^2,$$9, $U_t$0, and $U_t$1, and the student distills from these knowledge fragments progressively, aided by cyclical learning rate (Zhang et al., 2021). A plausible interpretation is that width-ordered sub-networks function as a capacity-ordered decomposition of the teacher’s knowledge space.

5. Empirical behavior across application domains

The empirical literature shows that subspace KD is not confined to a single task family. In structured prediction, Efficient Sub-structured Knowledge Distillation was evaluated on CoNLL 2003 named entity recognition and CoNLL 2000 chunking. On CoNLL 2003, the BERT$U_t$2-CRF student baseline scored $U_t$3, K-best KD $U_t$4, Structural KD $U_t$5, and Efficient KD $U_t$6. On CoNLL 2000, the corresponding numbers were $U_t$7, $U_t$8, $U_t$9, and $U_s$0. The reported per-epoch training times on CoNLL 2003 were $U_s$1s for vanilla training, $U_s$2s for K-best KD, $U_s$3s for Structural KD, and $U_s$4s for Efficient KD (Lin et al., 2022).

In continual learning, Subspace Distillation reports gains on both classification and segmentation. On Pascal VOC $U_s$5-$U_s$6, MiB improves from $U_s$7 to $U_s$8 with SD, ILT from $U_s$9 to $r$0, and PLOP from $r$1 to $r$2 (Roy et al., 2023). The same paper also reports that, on Split CIFAR-10 class-incremental learning, SD achieves $r$3 versus DER $r$4 with tiny memory, and DER+SD rises to $r$5 in the small-memory setting.

In lifelong imitation learning, SPREAD is evaluated on LIBERO-OBJECT, LIBERO-GOAL, and LIBERO-SPATIAL. The reported metrics are FWT $r$6, NBT $r$7, AUC $r$8 on LIBERO-OBJECT; FWT $r$9, NBT $F_i^{t-1}$0, AUC $F_i^{t-1}$1 on LIBERO-GOAL; and FWT $F_i^{t-1}$2, NBT $F_i^{t-1}$3, AUC $F_i^{t-1}$4 on LIBERO-SPATIAL. The paper further reports over $F_i^{t-1}$5 reduction in language-embedding drift, HandEye drift kept below $F_i^{t-1}$6 versus baseline peaks above $F_i^{t-1}$7, AgentView mean drift $F_i^{t-1}$8 versus $F_i^{t-1}$9 for M2Distill, and the best subspace rank at $F_i^t$0 (Roy et al., 9 Mar 2026).

TCS reports practical efficiency in image classification. On CIFAR-100 with a ResNet18 student and a ViT-S teacher, TCS reaches $F_i^t$1 top-1, with $F_i^t$2 GB GPU memory and about $F_i^t$3 s per epoch. On ImageNet-1K with a Swin-T teacher, it reaches $F_i^t$4 top-1 with the same $F_i^t$5 GB memory and about $F_i^t$6 s/epoch (Zhou et al., 2024). SubDistill reports that on WResNet101 $F_i^t$7 MbNetv4 on ImageNet subtasks, gains are about $F_i^t$8 to $F_i^t$9 percentage points over baselines, and on the CIFAR-100 People subtask it yields about $f\in\mathbb{R}^{m\times n}$00 points improvement over baselines (Chormai et al., 9 Jan 2026).

For parameter-efficient transfer, SAD-LoRA reports a synthetic flat-spectrum case in which KD-LoRA’s error budget is dominated by subspace misalignment at $f\in\mathbb{R}^{m\times n}$01, while SAD-LoRA reduces the subspace-misalignment term to nearly zero and raises final alignment from $f\in\mathbb{R}^{m\times n}$02 to $f\in\mathbb{R}^{m\times n}$03. On RoBERTa-large to RoBERTa-base GLUE distillation, at $f\in\mathbb{R}^{m\times n}$04 the reported gains over KD-LoRA include STS-B $f\in\mathbb{R}^{m\times n}$05, CoLA $f\in\mathbb{R}^{m\times n}$06, and RTE $f\in\mathbb{R}^{m\times n}$07 (Tariq et al., 5 Jul 2026). These results are consistent with the paper’s claim that better subspace choice can substitute for extra rank.

6. Interpretive issues, misconceptions, and research directions

A recurring misconception is that subspace KD is a single technique. The literature does not support such a narrow view. In different papers, “subspace” refers to principal directions of activation variance, a tailored coordinate system, a projected layer mixture, a local structured factorization, a low-rank adapter column space, or a student-compatible region of function space (Passban et al., 2020, Wang et al., 2023, Tariq et al., 5 Jul 2026). The shared principle is selective transfer of structure, not a single canonical loss.

Another misconception is that subspace KD simply replaces logits with features. Some methods do exactly that, but others combine geometric and behavioral constraints. SPREAD augments subspace representation distillation with a confidence-guided policy loss restricted to the top-$f\in\mathbb{R}^{m\times n}$08 most confident action samples (Roy et al., 9 Mar 2026). Subspace Distillation for Continual Learning is explicitly described as complementary and is combined with ILT, MiB, PLOP, and DER (Roy et al., 2023). SAD-LoRA combines task loss, KD loss, alignment, and an optional coefficient term (Tariq et al., 5 Jul 2026). This suggests that subspace KD is often an additional structural regularizer rather than a wholesale replacement for classical KD objectives.

The literature also identifies concrete limitations. TCS notes that very low-shot settings such as $f\in\mathbb{R}^{m\times n}$09-shot or $f\in\mathbb{R}^{m\times n}$10-shot can be less stable because PCA is estimated from too few samples (Zhou et al., 2024). Subspace Distillation adds SVD overhead with complexity roughly $f\in\mathbb{R}^{m\times n}$11 (Roy et al., 2023). SimKD’s projector adds parameters and reduces compression ratio (Chen et al., 2022). PWKD requires reconstructing the teacher into a slimmable supernet and can suffer from conflicting optimization under weight sharing (Zhang et al., 2021). SubDistill reports experiments limited to image classification and depends on PRCA as the mechanism for defining relevance (Chormai et al., 9 Jan 2026). Curriculum Extraction provides theoretical guarantees in a stylized sparse parity setting rather than general deep LLMs (Gupta et al., 21 Mar 2025).

A plausible implication is that the main open problem is not whether subspace KD works, but which subspace is appropriate for a given deployment constraint. The papers surveyed propose at least five distinct answers: principal low-rank geometry, task-tailored coordinates, teacher-layer compositions, student-friendly hypothesis regions, and low-rank parameter subspaces. The field therefore points less toward a universal recipe than toward a design language for specifying what part of teacher knowledge should be preserved, how that part should be parameterized, and how aggressively the student should be constrained while retaining plasticity.