Subspace-Native Distillation Overview

• Subspace-Native Distillation is a paradigm that represents model features and parameters in a low-dimensional subspace to optimize memory, information sharing, and geometry fidelity.
• It leverages methodologies such as Neural Spectrum Decomposition and anchor-induced projections to enforce low-rank structures and streamline knowledge transfer.
• Empirical results demonstrate significant improvements in dataset synthesis, continual learning, and cross-task performance compared to conventional pixel-native methods.

Subspace-Native Distillation is a paradigm in knowledge distillation in which the transfer, compression, or synthesis of neural representations is performed within an explicit low-dimensional subspace, rather than the native high-dimensional ambient space. This approach formalizes, leverages, and enforces low-rank structure in feature, dataset, or model parameter spaces to optimize memory efficiency, information sharing, transfer fidelity, and generalization. Core instances include Neural Spectrum Decomposition for dataset distillation (Yang et al., 2024), anchor-induced KL-ball projection in triadic knowledge distillation (Wang et al., 2023), explicit construction of solution subspaces for model head transfer (Kalyoncuoglu, 29 Dec 2025), manifold-aligned continual learning (Roy et al., 2023), subnetwork activation-based transfer in sparse modular systems (Xue et al., 17 Dec 2025), and multi-task Pareto-optimized subspace distillation for high-dimensional feature alignment (Hayder et al., 13 May 2025).

1. Formal Definition and Motivation

Subspace-Native Distillation denotes any distillation protocol in which the student, teacher, or synthetic data is parameterized via shared subspace or low-dimensional factorization. Rather than optimizing elements (images, features, neurons) independently or in pixel-native, patch-native, or monolithic dense representations, the entire set of objects to be distilled is embedded or represented in a common subspace—be it linear, manifold, statistical, or network-topological.

• In Neural Spectrum Decomposition (NSD), the synthetic dataset is parameterized by a collection of spectrum tensors $\{\mathcal{T}_i\}$ and separable transformation ("kernel") matrices $\{\mathcal{K}_j\}$, where each image $$x_{j+(i-1)N_\mathcal{K}} = \mathcal{T}_i\mathcal{K}_j$$ lies in the joint low-rank subspace enforced by $t_d \ll u_d$ across all dimensions (Yang et al., 2024).
• For solution subspace compression, subspace $S = \text{span}(U)$ of intrinsic dimension $k \ll d$ is constructed such that classification head weights $w^* \approx Uv^* + w_0$ for basis $U$, providing geometric stability and allowing students to operate entirely in $S$ (Kalyoncuoglu, 29 Dec 2025).

Fundamental motivations include: (i) Storage efficiency: drastic reduction in free parameters, avoiding redundant optimization in high-dimensional ambient spaces. (ii) Gradient and information sharing: combinatorial pairing in factorizations enables cross-sample propagation of optimization signals. (iii) Fidelity to natural data geometry: neural and image data distributions are inherently low-rank; explicit adherence minimizes modelling and transfer error.

2. Mathematical Frameworks of Subspace-Native Distillation

Subspace-native protocols are characterized by one or more of the following mathematical constructs:

Setting	Subspace Representation	Optimization Domain
Dataset Distillation	Spectral tensor $\mathcal{T}_i$ + kernel $\mathcal{K}_j$	Joint low-rank tensor subspace (Yang et al., 2024)
Feature-Level Distill.	Student/teacher projectors $U_s$, $U_t$	Stiefel manifolds, aligned via $D$ (Hayder et al., 13 May 2025)
Final Head/Classifier	Orthonormal basis $U\in\mathbb{R}^{d \times k}$	Solution subspace $S \subset \mathbb{R}^d$ (Kalyoncuoglu, 29 Dec 2025)
Sparse Modular	Top-K indices, activation entropy $H_i$	Task-specific neuron subspace $S_t$ (Xue et al., 17 Dec 2025)
Manifold Alignment	SVD basis $P_k$, Grassmann metric $\delta_p(P,Q)$	Task- or class-local tangent subspaces (Roy et al., 2023)
Anchor-Induced	KL-ball $F_A(\delta)$ around anchor $f_A$	Restricted hypothesis subspaces $F_T', F_S'$ (Wang et al., 2023)

For NSD specifically, each synthetic image is constructed as $$x_{j+(i-1)N_\mathcal{K}} = \mathcal{T}_i\;\mathcal{K}_j$$, with $\mathcal{K}_j$ factorized as $\bigotimes_{d=1}^n \widehat{\mathcal{K}_j^{(d)}}$, yielding overall parameter cost $$\sum_{d=1}^n t_d u_d + \prod_{d=1}^n t_d \ll N_{\mathcal{T}} N_{\mathcal{K}} (\prod_{d=1}^n u_d)$$ for conventional pixel-wise methods (Yang et al., 2024).

In model compression/distillation, the projection operator is $P_S = UU^T$, and student regression is achieved via $$\mathcal{L}_{\rm sub}(\phi) = \mathbb{E}_x \|h_S(x;\phi) - U^T f_\theta(x)\|_2^2$$ (Kalyoncuoglu, 29 Dec 2025).

3. Optimization Objectives and Algorithms

Subspace-native protocols adopt objectives and training schemes dependent on the geometry and structure of the induced subspace.

• NSD minimizes combined trajectory-matching loss and real-guided cross-entropy:

$$\mathcal{L}(T,K) = \sum_{i=0}^{I-1} \|\theta_{i+N}^s(T,K)-\theta_{i+M}^t\|_2^2 + \gamma \sum_{i=0}^{I-1}\Big[-\frac{1}{B}\sum_{b \in B} y_b \log f_b(\theta_{i+N}^s)\Big]$$

subject to parameter budget constraints (Yang et al., 2024).

• In TriKD, anchor-induced KL penalties enforce the teacher and student to remain close to the anchor solution $f_A$:

$$L_S = \alpha L_{CE}(f_S) + \beta L_{T \rightarrow S} + \gamma L_{A \rightarrow S},\quad L_T = \alpha L_{CE}(f_T) + \beta L_{S \rightarrow T} + \gamma L_{A \rightarrow T}$$

for hyperparameters $\alpha, \beta, \gamma > 0$ (Wang et al., 2023).

• Subspace projectors in MoKD are trained end-to-end with a multi-objective, Pareto-optimal aggregation of distillation and task loss gradients, guaranteeing minimization of both objectives and balanced alignment (Hayder et al., 13 May 2025).
• SSD matches activation statistics only within selected neuron sets for each task, reducing interference and improving cross-task transfer while strictly adhering to subnet isolation (Xue et al., 17 Dec 2025).

Algorithmic processes involve construction of subspace elements (e.g., SVD, random projection, Top-K selection, anchor restriction), synthesis of new samples or student features as subspace functionals, and iterative joint backpropagation.

4. Empirical Results and Performance Analysis

Subspace-native protocols yield state-of-the-art results across benchmarks in dataset distillation, knowledge transfer, model compression, and continual learning. Key highlights include:

Method	Benchmark / Setting	Main Result (Metric)
NSD	CIFAR-10/100, TinyImageNet	+22.2pts over MTT (IPC=1), SOTA (Yang et al., 2024)
Solution Subspace	ResNet-50 (CIFAR-100, $16\times$)	Only −1.21pts: 82.40 $\rightarrow$ 81.19% (Kalyoncuoglu, 29 Dec 2025)
SSD (Sparse)	Split CIFAR-10/100, MNIST	+10pts, +11pts accel. + reduced BWT (Xue et al., 17 Dec 2025)
SDCL	Split-CIFAR10/TinyImageNet, VOC segmentation	~+4 points mIoU; robust to buffer size (Roy et al., 2023)
MoKD	ImageNet-1K, COCO	+1.3 AP (subspace), +0.3 multi-task (Hayder et al., 13 May 2025)
TriKD	Face Recog., Classification	Monotonic gains, further KL reduction (Wang et al., 2023)

Ablations in NSD demonstrate that learnable kernels outperform fixed bases (DCT/SVD), and that adding real-guided terms increases accuracy by $\sim$1pt. In Subspace Classification, linear separability and accuracy survived aggressive random projection contractions (up to $16\times$), confirming robustness and suggesting negligible expressivity loss (Kalyoncuoglu, 29 Dec 2025). Continual learning and SSD frameworks reduced forgetting, enhanced alignment, and maintained modular coverage even without replay (Xue et al., 17 Dec 2025, Roy et al., 2023).

5. Comparison to Previous, Non-Subspace Methods

Prior dense or pixel-native methods (Dataset Condensation, DM, MTT for distillation; vanilla KD protocols) optimize synthetic data or feature alignment in full ambient spaces, ignoring the latent low-rank geometry and missing opportunities for information sharing and compression. Parametric approaches introduce sharing via latent codes or auxiliary networks but do not enforce dataset-wise or cross-model subspace structure (Yang et al., 2024). By contrast, subspace-native protocols amalgamate samples, features, network heads, and learning signals—across time and tasks—into explicit shared low-dimensional geometries.

In continual learning, SDCL and SSD outperform classical regularization and replay by aligning first-order tangent planes and activation subspaces, leading to significant reductions in catastrophic forgetting (Roy et al., 2023, Xue et al., 17 Dec 2025). In triadic protocols, anchor-induced subspace constraint provably shrinks risk and transfer-mismatch bounds (Wang et al., 2023). Multi-task optimization in MoKD resolves gradient conflict and dominance endemic to conventional KD (Hayder et al., 13 May 2025).

6. Extensions, Limitations, and Future Directions

Current variants of subspace-native distillation differ in their implementation—scalar versus tensor factorizations, discrete versus continuous subspace selection, random versus learned projections, explicit versus implicit manifold matching. Open directions and limitations include:

• Stability of random versus learned basis constructions under domain shift or heterogeneous data (Kalyoncuoglu, 29 Dec 2025).
• Extension of subspace-native protocols beyond final layers to intermediate representations or multi-modal data (Hayder et al., 13 May 2025).
• Adaptive subspace selection for domain-incremental and task-adaptive continual learning (Roy et al., 2023, Xue et al., 17 Dec 2025).
• Integration with quantization, sparsity, or latent code-based transfer for joint resource efficiency and transfer fidelity (Yang et al., 2024, Kalyoncuoglu, 29 Dec 2025).

A plausible implication is that subspace-native distillation may unify the goals of compact model deployment, transfer learning, and dataset synthesis by explicitly decoupling solution geometry from optimization complexity—realizing "Train Big, Deploy Small" at scale (Kalyoncuoglu, 29 Dec 2025). The approach remains robust to memory constraints, cross-architecture transfer, and catastrophic interference across a variety of application domains.