Directly Constructing Low-Dimensional Solution Subspaces in Deep Neural Networks (2512.23410v1)

Abstract: While it is well-established that the weight matrices and feature manifolds of deep neural networks exhibit a low Intrinsic Dimension (ID), current state-of-the-art models still rely on massive high-dimensional widths. This redundancy is not required for representation, but is strictly necessary to solve the non-convex optimization search problem-finding a global minimum, which remains intractable for compact networks. In this work, we propose a constructive approach to bypass this optimization bottleneck. By decoupling the solution geometry from the ambient search space, we empirically demonstrate across ResNet-50, ViT, and BERT that the classification head can be compressed by even huge factors of 16 with negligible performance degradation. This motivates Subspace-Native Distillation as a novel paradigm: by defining the target directly in this constructed subspace, we provide a stable geometric coordinate system for student models, potentially allowing them to circumvent the high-dimensional search problem entirely and realize the vision of Train Big, Deploy Small.

• The paper demonstrates that high-dimensional networks can be effectively compressed to low-dimensional subspaces with less than 1.2% performance loss.
• It employs fixed Johnson-Lindenstrauss projections to extract intrinsic solution manifolds across vision and NLP models.
• The study introduces Subspace-Native Distillation, enabling efficient model deployment by decoupling optimization from representation.

Direct Construction of Low-Dimensional Solution Subspaces in Deep Neural Networks

Background and Motivation

This paper presents a constructive framework for compressing the high-dimensional solution space of deep neural networks into provably low-dimensional subspaces, with negligible degradation in classification performance. The motivation arises from the empirical observation that, although modern architectures utilize very high embedding dimensions for optimization—embedding sizes ranging from hundreds to thousands—successful weight matrices and feature manifolds tend to be intrinsically low rank, with most useful signal concentrated in a small number of singular directions. Previous works have established the prevalence of heavy-tailed spectral densities [martin2021implicit], Neural Collapse phenomena [papyan2020prevalence], and the general efficacy of low-rank model modifications such as SVD and LoRA [denton2014exploiting, hu2021lora]. However, these approaches operate post hoc and do not fully decouple optimization from representation.

The key paradox addressed is that, despite the existence of sparse "winning tickets" [frankle2018lottery] and the theoretical representational power of compact backbones, training in restricted low-dimensional parameter spaces remains intractable due to the non-convexity and combinatorial complexity of loss landscapes. High-dimensional ambient spaces are only required to assist gradient descent in solving the optimization problem, and not for ultimate inference or representation. This work proposes to directly construct robust, low-dimensional solution subspaces and to leverage these for a new paradigm of "Subspace-Native Distillation."

Methodology

The central experimental approach is to isolate the intrinsic solution subspace of a deep neural network using fixed, data-independent Johnson-Lindenstrauss (JL) projections [johnson1984extensions]. For a trained backbone $f_\theta: \mathcal{X} \to \mathbb{R}^d$ (e.g., ResNet-50, ViT-B/16, or BERT-base), the final latent representation $h \in \mathbb{R}^d$ is projected into a smaller space $\tilde{h} \in \mathbb{R}^k$ via a randomly generated matrix $R \in \mathbb{R}^{k \times d}$, whose entries are sampled i.i.d. from a standard normal distribution and frozen at initialization. The scaling factor $1/\sqrt{k}$ ensures preservation of the norm. The projected solution vector then serves as input to a lightweight linear classifier, trained on the frozen features.

Dimensionality is systematically varied: for ResNet-50 ($d=2048$), subspaces are constructed down to $k=128$ (16× compression); for BERT and ViT ($d=768$), down to $k=64$ (12× compression). The experiments are conducted across three modalities—vision (CIFAR-100, ImageNet-100) and NLP (MNLI)—with fully reproducible and deterministic setups.

Empirical Results

Classification performance in the subspace is consistently robust. Across all three architectures and tasks, the accuracy reduction remains below 1.2% for compression rates up to 16×. Notably, projecting BERT-base and ViT-B/16 features from $d=768$ to $k=64$ leads to accuracy loss of less than 0.2%. In certain cases, such as ViT-B/16, the subspace classifier even marginally outperforms the full-dimensional frozen baseline. This empirical stability, under aggressive random projection, serves as direct evidence that discriminative information is concentrated along a handful of geometric directions, and that the backbone is not reliant on the full ambient dimension for separability.

Theoretical Implications

These findings strongly validate the "flat solution manifold" hypothesis. Prior work measured local intrinsic dimensionality via manifold estimators [ansuini2019intrinsic], but this approach provides a global validation—if the solution geometry were highly nonlinear or locally entangled, random projections would induce frequent class collisions and degrade accuracy. The observed performance preservation signifies that the solution manifolds are globally flat and linearly separable, permitting effective contraction to a random low-dimensional basis. It generalizes Neural Collapse theory, showing that the flattening extends not only to class means but to the space spanned by all features relevant for classification.

Subspace-Native Distillation: Architectural Implications

The paper proposes Subspace-Native Distillation as a future paradigm, wherein the student model is trained not to reconstruct the full high-dimensional output of the teacher, but rather to directly regress the projected signal in the low-dimensional subspace. The distillation loss is expressed as $$\mathcal{L}_{subspace} = \| h_{student} - R h_{teacher}\|^2$$, allowing the student to avoid modeling redundant, noisy, or bulk components of the teacher representation. This decoupling of optimization from representation opens avenues for "Train Big, Deploy Small" workflows: models can be scaled during training for optimization tractability, and deployed in compact form by targeting the explicit intrinsic solution manifold.

Practically, this may significantly decrease student model parameters without sacrificing accuracy, especially for edge or resource-constrained deployment. Theoretically, it further refines our understanding of deep learning generalization and optimization: inference complexity is not dictated by training complexity.

Conclusion

This work establishes that high-dimensional width in deep neural networks serves fundamentally as an optimization scaffold, not as a representational requirement at inference. The final solution can be robustly isolated in a low-dimensional subspace independent of the ambient geometry, and standard classification performance is preserved even with extreme compression via random projection. This empirical and theoretical result motivates future research into subspace-native architectures and distillation, recasting both the theory and practice of efficient deep learning.

By providing an explicit, verifiable coordinate system for solution manifolds, this framework lays the foundation for a new generation of models and knowledge transfer techniques which leverage the distinction between optimization and representation, translating to improved efficiency and broader applicability for deep learning systems.