Is Dimensionality a Barrier for Retrieval Models?

Abstract: Why does the low dimensionality of representations, typically $d\approx 1000$, not prevent modern embedding-based retrieval models from scaling to billions, or even trillions, of data points? To answer this question, we study maximal-margin embeddings in the following retrieval model, classically studied in communication complexity [PS86] and more recently in embedding-based retrieval [WBNL26]. Let $A\in {0,1}^{N\times n}$ be a matrix indicating whether each of $N$ queries is relevant to each of $n$ documents. We are interested in the largest margin $m>0,$ denoted by $\mathsf{m}^{\mathsf{rd}}(d, A),$ for which there exist unit norm embeddings of the queries and documents ${U_j}{j = 1}^N, {V_i}{i = 1}^n$ with the following property. $\langle U_j, V_i\rangle \ge m$ whenever $A_{ji} = 1$ and $\langle U_j, V_i\rangle \le -m$ otherwise. A large margin is a key proxy for representation quality: it controls both robustness to perturbations and compositional generalization across queries. Our main theorem establishes that the best possible margin without a restriction on the dimension, $\mathsf{m}^{\mathsf{rd}}(+\infty, A),$ can be nearly achieved in dimension $d = O(\mathsf{m}^{\mathsf{rd}}(+\infty, A)^{-2}\log n)$ which improves a theorem of [BDES02]. Together with a matching lower bound in Theorem 1.5, we conclude that when $A\in {0,1}^{\binom{n}{k}\times n}$ is the matrix containing all possible $k$-sparse rows once, dimension $d = O(k\log (n/k))$ is necessary and sufficient for the maximal possible margin $\mathsf{m}^{\mathsf{rd}}(+\infty, A) = Θ(k^{-1/2})$ in this setting. This fully resolves the setup of [WBNL26]. We also give several constructions for large margins when $d = o(k\log (n/k)).$ Finally, we empirically test the InfoNCE and sigmoid losses for producing large margin embeddings and demonstrate a clear advantage of the sigmoid loss.

• The paper proves that low-dimensional embeddings, scaling logarithmically with data size, can achieve nearly optimal retrieval margins.
• It presents novel algebraic constructions and sphere packing arguments to establish necessary and sufficient conditions based on k-sparse rows.
• Empirically, the sigmoid loss outperforms InfoNCE by achieving positive margins in low dimensions, highlighting practical benefits for scalable retrieval.

Dimensionality Constraints in Embedding-Based Retrieval Models

Problem Formulation and Margin Complexity

The paper systematically analyzes the relationship between the embedding dimensionality and the retrieval capability of models based on unit-norm embeddings, focusing on maximal-margin embeddings. Given a binary relevance matrix $A \in \{0,1\}^{N \times n}$ describing the association between queries and documents, the core metric is the largest achievable margin $m > 0$ such that there exist unit-norm query $\{U_j\}$ and document $\{V_i\}$ embeddings in dimension $d$ with

$$\langle U_j, V_i\rangle \ge m \text{ if } A_{ji} = 1,\quad \langle U_j, V_i\rangle \le -m \text{ otherwise}.$$

Large margin embeddings confer robustness to perturbations, enable compositional generalization, and guarantee accurate quantization. The paper provides a formal equivalence between dimensionality and achievable margin, situating the analysis in the context of communication complexity, compressed sensing, and learning theory.

Main Theoretical Results: Tight Dimension-Margin Trade-Off

A central claim is that high-quality retrieval embeddings, as measured by maximal margin $m(+\infty, A)$, can be constructed in dimension scaling logarithmically with the data size: $d = O(m(+\infty, A)^{-2} \log n)$ achieves margin arbitrarily close to $m(+\infty, A)$. For $A$ with $m > 0$0-sparse rows, it is shown that $m > 0$1, so that dimension $m > 0$2 is both necessary and sufficient for optimal margin, closing the gap with existing results. The proof improves on prior JL-based dimensionality reduction arguments by leveraging convex geometry and Maurey's Approximation Lemma to obtain sharper logarithmic bounds, rather than quadratic ones.

Matching lower bounds are established using sphere packing volume arguments, demonstrating that achieving margin $m > 0$3 for the exhaustive $m > 0$4-sparse retrieval setting requires dimension $m > 0$5.

Constructions Below the Maximal Margin and Beyond Standard Settings

The paper develops explicit constructions for non-maximal margins in sub-logarithmic dimensions, e.g., margin $m > 0$6 in $m > 0$7 via algebraic embeddings, and applies self-Khatri-Rao lifts for improved restricted isometry properties. These constructions provide smooth trade-offs between margin and dimension, extending applicability to more realistic retrieval settings.

Empirical Evaluation: Sigmoid vs InfoNCE Losses

Experimental results reveal that the sigmoid loss is significantly more efficient in producing high-margin embeddings in low dimensions, compared to InfoNCE, both quantitatively and across a range of $m > 0$8, $m > 0$9, and $\{U_j\}$0. The sigmoid achieves positive margins in dimensions near theoretical minimum, while InfoNCE requires considerably higher dimension for nonzero margin.

Figure 1: Minimal dimension needed to achieve a non-zero margin after 100000 training steps for the InfoNCE and sigmoid losses; sigmoid loss succeeds in much smaller dimensions.

This empirical finding is rationalized theoretically: global minimizers of the sigmoid loss coincide with maximal-margin embeddings, while InfoNCE lacks this property outside the one-hot ($\{U_j\}$1) case. Further, the margins achieved by the sigmoid loss are consistently higher across dimensions.

Figure 2: Comparison of achieved maximal margin for various $\{U_j\}$2 and $\{U_j\}$3, demonstrating clear advantage of the sigmoid loss over InfoNCE.

Practical and Theoretical Implications

The results demonstrate that dimensionality is not a practical barrier for retrieval models: optimal retrieval performance can typically be attained using low-dimensional embeddings, provided they are well-constructed. This has significant implications for the scalability of nearest neighbor search, memory-efficient deployment, and system design for billion-scale datasets. Moreover, the strong connection to restricted isometry and sphere packing theory establishes an explicit link between information-theoretic limits, generalization, and representation geometry.

Theoretical implications extend to robust retrieval under adversarial perturbations, compositional generalization with margin-based embeddings, and quantization-resistant representation learning. The tight characterization of the margin-dimension trade-off enables principled encoder design. The work also clarifies inductive bias effects of contrastive and sigmoid losses, and provides a foundation for analyzing limitations of retrieval systems.

Speculation on Future Developments

Future directions include extending these constructions to practical architectures, real-world datasets, and generalized top-$\{U_j\}$4 retrieval scenarios. Analysis of representation quality for downstream tasks beyond retrieval, as well as deeper connections between margin-based learning, adaptation, and multimodal representation compatibility, are promising avenues. Advances in parameter-efficient transfer learning, robust geometric representation, and further exploration of statistical properties of contrastive losses may leverage these theoretical insights.

Conclusion

The paper provides a comprehensive and tight characterization of how embedding dimensionality governs retrieval margin in dense retrieval models, establishing that logarithmic dimension scaling suffices for nearly optimal margin and performance. It refutes the assumption that low dimensionality is a fundamental barrier for large-scale retrieval, and offers explicit constructions, matching lower bounds, and empirical evidence supporting the theoretical results. The implications for scalable information retrieval, representation learning, and architectural design are substantial, and the framework is extensible to broader settings in AI research.