MIC: Maximizing Informational Capacity in Adaptive Representations via Isotropic Subspace Alignment

Abstract: Although multi-scales representation learning enables elastic-dimension embeddings, nested subspaces often suffer from dimensional redundancy and spectral collapse. To address this, we introduce MIC, a framework that optimizes the geometric landscape of multi-granular embeddings through isotropic subspace alignment. MIC employs Soft Collapse Regularization (SCR) to mitigate redundancy between prefix and residual subspaces via cross-correlation penalties, alongside Spectral Isotropy Regularization (SIR) to ensure hyper-spherical uniformity in low-dimensional prefixes. By unifying these strategies through a self-distillation objective, MIC generates semantically dense representations that maintain high discriminative power. Our experiments demonstrate that MIC significantly outperforms standard baselines, particularly in high-compression scenarios where maintaining informational capacity is most critical.

• The paper introduces MIC, a framework that addresses subspace redundancy and spectral collapse in elastic-dimensional representations.
• It employs Soft Collapse Regularization (SCR) to decorrelate prefix-residual subspaces while preserving shared signal.
• Experimental results demonstrate that MIC significantly boosts performance under aggressive truncation across various NLP tasks and backbones.

Maximizing Informational Capacity in Adaptive Representations via Isotropic Subspace Alignment

Introduction and Motivation

The paper "MIC: Maximizing Informational Capacity in Adaptive Representations via Isotropic Subspace Alignment" (2605.29987) focuses on geometric conditioning for Matryoshka Representation Learning (MRL). MRL provides elastic-dimensional representations through nested embeddings, supporting adaptive truncation of representational dimensionality during inference. However, vanilla MRL and its multi-objective derivatives frequently suffer from two detrimental geometric failures: subspace redundancy and spectral collapse. These limitations manifest as highly correlated prefix/residual subspaces and non-uniform variance allocation, ultimately degrading semantic compressibility and feature utility under aggressive dimension truncation.

Contemporary strategies (e.g., SimCSE, ESE) enforce decorrelation globally, failing to preserve structural independence between embedded prefixes and their complementary residuals. Attempts at hard orthogonalization are overly restrictive, suppressing meaningful shared signal. The MIC framework directly targets these bottlenecks by employing a combination of Soft Collapse Regularization (SCR) to decorrelate prefix-residual subspaces and Spectral Isotropy Regularization (SIR) to maintain variance isotropy within every prefix.

Methodology

Soft Collapse Regularization (SCR)

SCR introduces a thresholded cross-correlation penalty over the prefix-residual cross-covariance matrix, attenuating redundancy while avoiding the impractical zero-correlation constraint. For each nested dimension $d$, the SCR loss penalizes off-diagonal correlation entries above margin $T_{corr}$ and applies an explicit variance floor to prevent trivial solutions via variance shrinkage. The regularizer is sensitive to the hierarchical structure of MRL, addressing subspace-specific redundancies through layer- and dimension-level supervision. This soft conditioning promotes non-redundancy without the destructive effects of strict orthogonality, which would starve the representational space of shared pattern utility.

Spectral Isotropy Regularization (SIR)

SIR regularizes the variance distribution across prefix dimensions via two complementary losses. The first loss minimizes the coefficient of variation of the per-dimension variance, ensuring that information is not concentrated within a few dominant axes and reducing the risk of dimensional collapse. The second loss enforces hyperspherical uniformity with a pairwise RBF potential on the $L_2$-normalized prefix embeddings. This dual action yields subspaces that are both variance-balanced and evenly populated across the unit hypersphere, addressing the anisotropy and cone collapse issues observed in deep contextualized encoders.

Overall Objective and Implementation

MIC integrates these geometric regularizers with the canonical MRL contrastive loss, yielding a training objective that aligns multiple intermediate transformer layers and nested dimensions. SCR and SIR are applied selectively to layers and dimensions empirically identified as most semantically informative, trading off maximal alignment with computational efficiency. Regularization loss balancing is governed by searched hyperparameters, with optimal settings provided for diverse backbones (TinyBERT, BERT, BGE-M3).

Experimental Validation

Datasets and Settings

The framework is validated on both in-domain (TweetEval, Banking77, MRPC, STS-B, SICK) and out-of-domain (Emotion, SciTail, STS12-16, SickR) tasks, covering text classification, natural language inference, and semantic textual similarity. All models are trained in a unified setup using unsupervised SimCSE-style contrastive learning. Evaluation spans a broad spectrum of truncation dimensions (16 to 1024) and model variants.

Results

MIC consistently surpasses MRL and ESE, especially at low-dimensional prefixes where informational collapse is most acute. For example, on Banking77 at $d=16$, MIC achieves a substantial improvement (BERT: 59.45 vs. MRL: 46.39; TinyBERT: 44.1 vs. MRL: 40.64). This superiority is observed across classification, NLI, and STS, and on both compact (TinyBERT-6L) and larger (BERT, BGE-M3) backbones. The ablation study indicates that each regularizer independently improves over the baseline, but their combination is strongly synergistic—maximizing preservation of semantic signal at each nested tier.

Performance gains persist under out-of-domain evaluation, confirming the robust generalization of the learned geometry. Results scale to larger encoders (BGE-M3), with MIC maintaining or expanding its advantage as compression increases and the importance of regularized geometry becomes more pronounced.

Training Cost

MIC incurs moderate additional training-time cost due to layer- and dimension-wise covariance/correlation computation (BERT throughput: MIC 105.6 samples/sec vs. MRL 214.4 samples/sec). However, there is no inference-time penalty: inference architecture and latency are identical to vanilla MRL.

Discussion and Implications

The MIC framework establishes that direct geometric conditioning of adaptive representations can substantially improve semantic density and discriminative power, even under aggressive truncation. By explicitly minimizing subspace redundancy and enforcing isotropy, it addresses key bottlenecks that multi-objective contrastive training leaves unresolved.

This result has two principal implications:

• Practical: Embedding-based retrieval, search, and semantic similarity systems can achieve higher memory and compute efficiency without degrading accuracy, supporting deployment within strict latency or resource budgets typical of production systems.
• Theoretical: The optimal informational capacity of adaptive representation frameworks lies not merely in multitask or multigranularity supervision, but in control of internal covariance and variance geometry.

The requirement of layer-specific alignment currently constrains architectural flexibility—layer selection must be adapted for different backbones, and the regularization does not yet dynamically scale to reflect the varying semantic utility of nested prefixes.

Future Directions

Potential extensions include generalization to autoregressive/generative architectures and application to multimodal or cross-modal embedding frameworks. Automating the selection and weighting of regularization across layers and dimensions is a salient direction, aiming for a more architecture-agnostic, adaptive solution.

Conclusion

This work systematically demonstrates that addressing subspace redundancy and variance anisotropy yields significant practical and robustness benefits in elastic-dimensional representation learning. MIC represents a step towards information-theoretically and geometrically optimal embedding frameworks, providing a viable deployment path for highly compressed yet semantically faithful representations in modern NLP and beyond.