DIVE: Embedding Compression via Self-Limiting Gradient Updates

Abstract: High-dimensional embeddings from LLMs impose significant storage and computational costs on vector search systems. Recent embedding compression methods, including Matryoshka-Adaptor (EMNLP 2024), Search-Adaptor (ACL 2024), and SMEC (EMNLP 2025), enable dimensionality reduction through lightweight residual adapters, but their training objectives cause severe overfitting when labeled data is scarce, degrading retrieval performance below the frozen baseline. We propose \textsc{DIVE} (\textbf{D}imensionality reduction with \textbf{I}mplicit \textbf{V}iew \textbf{E}nsembles), a compression adapter that addresses this failure through two mechanisms. First, a self-limiting hinge-based triplet loss produces zero gradient once a triplet satisfies the margin constraint, bounding the total perturbation applied to the pretrained embedding space. Second, a head-wise NT-Xent contrastive loss treats multiple learned projections of each embedding as implicit views, providing dense self-supervised gradients that compensate for the sparsity of the triplet signal on small datasets. Across six BEIR datasets, \textsc{DIVE} outperforms all three baseline adapters on every dataset and at every evaluated compression ratio, with a 14M-parameter open-source implementation.

• The paper introduces a self-limiting hinge triplet loss to safely compress high-dimensional embeddings without semantic degradation.
• It employs a multi-head adapter architecture with contrastive loss to maintain dense gradient signals and robust learning across datasets.
• Empirical results demonstrate that DIVE outperforms existing methods, achieving high retrieval performance even at extreme compression ratios.

DIVE: Embedding Compression via Self-Limiting Gradient Updates

Overview

DIVE (Dimensionality reduction with Implicit View Ensembles) addresses the challenge of compressing high-dimensional text embeddings produced by LLMs for dense retrieval, with specific attention to the risks of overfitting and semantic degradation in low-resource supervision regimes. The method fundamentally departs from previous supervised adapter-based compression techniques by introducing a self-limiting hinge triplet loss and a multi-head contrastive learning paradigm that together provide safe yet effective dimensionality reduction. DIVE achieves superior retrieval performance compared to prior adapters across all tested BEIR datasets and compression ratios, even outperforming the original, uncompressed, high-dimensional embeddings.

Figure 1: Architecture of DIVE. During training, the adapter maps each frozen embedding to $H$ projection heads; the self-limiting triplet loss supervises head~1 only, while the NT-Xent contrastive loss applies to all $H$ heads. At inference, heads~2 through~$H$ are discarded and only head~1 is used for retrieval.

Technical Contributions

Self-Limiting Triplet Loss

DIVE's core innovation is the replacement of continuous ranking or pairwise losses, used by methods such as Matryoshka-Adaptor and Search-Adaptor, with a margin-based hinge triplet loss applied only to a designated head of the adapter. This loss yields exactly zero gradient for triplets already satisfying the margin, thereby bounding the perturbation applied to the pretrained semantic space. This is critical for preventing the overfitting and performance collapse observed in competitive methods when labeled data is scarce; e.g., Matryoshka-Adaptor degrades to nDCG@10 = 0.098 (vs. frozen baseline 0.292) on nfcorpus.

Theoretical analysis establishes a cumulative perturbation bound, indicating that DIVE sharply reduces the effective signal received by the adapter after the initial learning phase — by a factor of 10–15× relative to the typical, ungated objectives of prior work.

Multi-Head Implicit View Ensembles and Dense Contrastive Regularization

To mitigate the sparse gradient signal resulting from the gating effect of the hinge loss, DIVE employs a multi-head adapter architecture. Multiple projection heads are trained per embedding, with the first head supervised via the self-limiting triplet loss and all heads regularized with a head-wise NT-Xent contrastive loss. This loss treats multiple heads as "implicit views" of a single sample, enforcing consistency while encouraging representational diversity.

The contrastive regularization functions as a self-supervised learning signal, providing $O(BH^2)$ positive gradient contributions per batch—critical for preventing mode collapse in data-sparse settings. At inference, only the first head is used, ensuring no increase in storage or computational burden.

Figure 2: Training dynamics of DIVE on three representative datasets, illustrating that the active triplet ratio $\rho(t)$ falls below 1% (left) and that the contrastive loss remains substantial after triplet loss sparsifies (right).

Adapter Architecture

The adapter comprises a three-layer MLP with decreasing hidden dimensions and batch normalization on the initial layer for stability across varying sequence lengths. The output is partitioned into $H$ normalized projection heads. Only one head is used at inference for each document/query, equivalently efficient to a standard, non-ensemble adapter. Model size is approximately 14M parameters.

Empirical Results

DIVE is evaluated on six BEIR tasks: scifact, arguana, nfcorpus, scidocs, fiqa, and quora, using two dissimilar LLM2Vec backbones (Mistral-7B and Sheared-LLaMA-1.3B), and at three compression ratios (128, 256, 512). Across all configurations, DIVE:

• Outperforms Matryoshka-Adaptor, Search-Adaptor, and SMEC on every dataset at every compression level.
• Never falls below the performance of the frozen baseline, while all baselines collapse on at least one dataset.
• Surpasses original high-dimensional embeddings on all datasets, achieving up to 32× compression without sacrificing accuracy.
• Exhibits largest gains on data-sparse datasets: e.g., on nfcorpus (323 queries), DIVE achieves nDCG@10 = 0.7584 (vs. 0.098/0.432/0.303 for baselines at 128d).
• Retains these gains across backbone architectures, demonstrating robustness to encoder choice and scale.

Ablation studies confirm that both the multi-head contrastive loss and the self-limiting triplet loss are essential. Removing either reduces nDCG@10 by ~0.17, with catastrophic degradation on small datasets.

Training Dynamics and Analysis

The training dynamics monitored across datasets reveal that the fraction of active triplets decays rapidly: typically within 5–15 epochs, fewer than 10% of triplets remain active, concentrating the update signal on genuinely misranked examples and avoiding over-perturbation. The contrastive loss ensures a persistent, dense gradient supply, stabilizing learning even as the triplet signal sparsifies.

Implications and Future Perspectives

DIVE's approach to gradient self-limitation fundamentally alters the risks associated with adapter-based embedding compression in low-supervision regimes, enabling aggressive dimensionality reduction without semantic collapse. The explicit use of architecture-induced multi-view contrastive objectives sidesteps the difficulty of data augmentation in text and offers a pathway for self-supervised signal machinery in other representation learning tasks.

From a theoretical standpoint, DIVE makes the case for gradient gating mechanisms as central to robust PEFT for retrieval, challenging continuous-rank-loss conventions. The method's simplicity, parameter efficiency, and superior empirical performance open up future exploration of self-limiting objectives for broader transfer and compression tasks, potentially extending beyond retrieval (e.g., classification, generation) and into other modalities.

Practical deployment of DIVE will benefit large-scale retrieval systems constrained by memory or latency. Downstream extensions could integrate DIVE's self-limiting backbone with quantization or binary hashing frameworks, or adapt it to cross-lingual, multimodal, or low-resource settings.

Conclusion

DIVE delivers a technically rigorous, empirically validated solution for safe and effective embedding compression under supervision constraints. Its combination of a sparse, bounded supervised loss and dense, architecture-driven contrastive regularization constitutes a new design pattern for adapter-based model compression, with strong implications for the broader field of efficient and robust representation learning.