Dynamic Subspace Composition: Efficient Adaptation via Contractive Basis Expansion (2512.23448v1)

Abstract: Mixture of Experts (MoE) models scale capacity but often suffer from representation collapse and gradient instability. We propose Dynamic Subspace Composition (DSC), a framework that approximates context-dependent weights via a state-dependent, sparse expansion of a shared basis bank. Formally, DSC models the weight update as a residual trajectory within a Star- Shaped Domain, employing a Magnitude-Gated Simplex Interpolation to ensure continuity at the identity. Unlike standard Mixture-of-LoRAs, which incurs O(M rd) parameter complexity by retrieving independent rank-r matrices, DSC constructs a compositional rank-K approximation from decoupled unit-norm basis vectors. This reduces parameter complexity to O(M d) and memory traffic to O(Kd), while Frame-Theoretic regularization and spectral constraints provide rigorous worst-case bounds on the dynamic update. The code is available at https://github. com/VladimerKhasia/DSC

• The paper introduces Dynamic Subspace Composition (DSC), which composes high-rank updates from unit-norm basis atoms, decoupling adaptation expressivity from storage costs.
• The methodology leverages magnitude-gated simplex interpolation and spectral norm bounds to ensure stability and reduce inference latency by 15% compared to traditional MoE architectures.
• Experimental results on WikiText-103 demonstrate DSC’s competitive validation performance and improved deployment efficiency in bandwidth-constrained environments.

Dynamic Subspace Composition: Efficient Adaptation via Contractive Basis Expansion

Problem Context and Motivations

Efficient scaling of neural LLMs has been dominated by Mixture-of-Experts (MoE) architectures, which decouple model capacity from active per-token parameter computation by routing inputs to sparse expert subsets. While MoE enables models with trillions of parameters, it introduces two principal limitations: (1) high inference latency due to memory bandwidth bottlenecks when loading full-rank expert weights, and (2) optimization instability, notably representation collapse, when sparse routers converge to trivial expert utilization. Parameter-efficient fine-tuning methods (e.g., Mixture-of-LoRAs) further mitigate storage burdens but fundamentally couple adaptation and storage ranks, restricting expressive adaptation without proportional memory cost.

Dynamic Subspace Composition: Methodology

Dynamic Subspace Composition (DSC) reformulates the expert selection problem as dynamic, sparse dictionary learning in weight space, leveraging compositionality in a shared basis bank of unit-norm rank-1 atoms. Key departures from standard MoE or MoLoRA include:

• Decoupled Basis Expansion: Instead of fetching $K$ independent rank-$r$ matrices (incurring an $\mathcal{O}(Mrd)$ complexity), DSC assembles high-rank updates from $K$ unit-norm vectors chosen from a shared basis, reducing parameter complexity to $\mathcal{O}(Md)$ and active memory traffic to $\mathcal{O}(Kd)$. Adaptation rank can hence be scaled independently of storage rank.
• Magnitude-Gated Simplex Interpolation: DSC models dynamic updates as residual trajectories within a star-shaped domain, separating the directional and radial components of routing coefficients. The simplex separation ensures continuity at the identity, with strict contraction such that updates vanish for low-confidence routing, unlike Softmax-based routing which is susceptible to noisy activations.
• Spectral Stability and Frame-Theoretic Regularization: DSC imposes spectral norm bounds on the residual update via $\ell_2$-projected normalization of basis atoms, guaranteeing worst-case local Lipschitz constants. Frame potential minimization promotes basis diversity, ensuring the shared atoms span a maximal hypothesis subspace and mitigating collapse.

Theoretical Properties and Algorithmic Details

DSC’s update construction is formally:

$$\Delta \mathbf{W}(\mathbf{z}) = \sum_{j \in \mathcal{I}(\mathbf{x})} \hat{z}_j (\mathbf{u}_j^\top \mathbf{v}_j)$$

with basis indices $\mathcal{I}(\mathbf{x})$ routed per input token, normalized vectors $\mathbf{u}_j, \mathbf{v}_j$, and contractive coefficients $\hat{z}_j$ derived from a magnitude-gated mechanism.

Spectral bounds are proven for both global and channel-wise scaling paradigms, strictly containing $\Delta \mathbf{W}$ within a spectral ball for gradient-stable optimization. Regularization comprises (1) auxiliary load balancing to prevent representation collapse, (2) signal preservation to enforce activations away from trivial zero mappings, (3) frame potential minimization for basis orthogonality even in overcomplete settings, and (4) logit range constraints for robust non-saturating routing.

Experimental Results

DSC is evaluated on WikiText-103’s next-token prediction with controlled iso-active parameter protocols (fixed $\approx 28$M active parameters across models). Baselines are a wide dense Transformer and a standard MoE layer.

DSC matches the generalization performance of MoE (validation loss $5.126$ vs. $5.125$), both outperforming the dense model ($5.171$). Crucially, DSC achieves a $15\%$ reduction in inference latency relative to MoE by replacing matrix retrievals with lightweight vector fetches while maintaining high-rank compositional updates. Parameter efficiency is explicitly realized: DSC requires only $\mathcal{O}(Md)$ static storage rather than $\mathcal{O}(Mrd)$.

Figure 1: DSC (red) closely tracks the loss improvement trajectory of Standard MoE (blue) and diverges from the Dense baseline (green) in validation loss convergence over training.

DSC retains the predictive power of full-rank, conditional computation while substantially improving deployment efficiency in bandwidth-constrained environments.

Practical and Theoretical Implications

DSC represents a structurally efficient alternative to MoE, relevant for LLMs and conditional computation regimes constrained by hardware bandwidth or requiring rapid inference. The decoupling of compositional expressivity from storage complexity enables wider, higher-rank adaptation with negligible parameter inflation.

On the theoretical side, DSC’s architecture introduces robust guarantees for stability (via spectral norm bounding and frame regularization) and continuity (via contractive simplex routing). These design elements curtail common failure modes associated with sparse activation, such as expert and signal collapse, and suggest new directions in adaptive modular architectures where basis adaptation can be tailored to task, data, or deployment constraints.

Future Directions

Open research questions for DSC and related methods include scaling to even larger basis banks with distributed synchronization, generalization to other model families (e.g., vision transformers), and integration with autoregressive sampling frameworks for ultra-low latency applications. Further analysis of basis diversity, signal collapse mitigation, and nuanced regularization regimes will clarify the ultimate limits of conditional adaptation using sparsified dynamic subspace composition.

Conclusion

Dynamic Subspace Composition generalizes mixture-of-experts conditional computation by sparse, contractive composition of shared basis atoms, decoupling adaptation expressivity from storage costs. DSC provides practical latency advantages, spectral stability, and competitive generalization performance, making it a promising approach for efficient parameter adaptation and sparse modular architectures in contemporary and future neural network deployments (2512.23448).