Provable Accuracy Collapse in Embedding-Based Representations under Dimensionality Mismatch

Abstract: Embedding-based representations in Euclidean space $\mathbb{R}^d$ are a cornerstone of modern machine learning, where a major goal is to use the \emph{smallest dimension} that faithfully captures data relations. In this work, we prove sharp dimension--accuracy tradeoffs and identify a fundamental information-theoretic limitation: unless the embedding dimension $d$ is chosen close to the ground-truth dimension $D$, accuracy undergoes a sudden collapse. Our main result shows that this phenomenon arises even in standard contrastive learning settings, where supervision is limited to a set of $m$ anchor--positive--negative triplets $(i,j,k)$ encoding distance comparisons $\mathrm{dist}(i,j) < \mathrm{dist}(i,k)$. Specifically, given triplets realizable by an unknown ground-truth embedding in $D$ dimensions, we prove that there exists constant $c < 1$, such that \emph{every embedding of dimension at most $cD$ violates half of the triplets}, yielding accuracy as low as a trivial one-dimensional solution that ignores the input. We complement our information-theoretic bounds with strong computational hardness results: under the Unique Games Conjecture, even if the given triplets are nearly realizable in $D=1$ dimension, no polynomial-time algorithm -- \textit{regardless of its dimension} -- can achieve accuracy above the trivial $50\%$ baseline.

• The paper establishes sharp lower bounds showing that low embedding dimensions yield trivial triplet accuracy of 50%.
• It uses probabilistic methods to rigorously link representational accuracy with the intrinsic dimensionality of data.
• Empirical results validate a phase transition in triplet accuracy, underscoring a fundamental barrier in low-dimensional embeddings.

Provable Accuracy Collapse in Embedding-Based Representations under Dimensionality Mismatch

Introduction and Motivation

The analysis of embedding-based representations in $\mathbb{R}^d$ critically depends on the choice of embedding dimension $d$, which directly determines the ability of these representations to capture the relational structure of the input data. This work rigorously characterizes the information-theoretic and computational phenomena governing the tight coupling between representational accuracy and dimensionality, particularly within the context of contrastive (triplet-based) ordinal embedding. The paper establishes foundational lower bounds showing that, unless $d$ is a constant-factor approximation to the ground-truth dimension $D$ required to realize the triplet constraints, no embedding method can surpass a baseline accuracy of $50\%$. This collapse of accuracy is shown to be robust both in the information-theoretic and the algorithmic regime.

Information-Theoretic Limits: Dimension-Accuracy Tradeoff

The main theorem asserts that for any set of triplets exactly realizable in $\mathbb{R}^D$, if the embedding dimension $d$ is reduced to $o(D)$, then the maximal achievable accuracy for triplet satisfaction is bounded above by $1/2 + o(1)$. Notably, even if $d$ is only a constant factor smaller than $d$0, accuracy deteriorates to the trivial level obtained by random projections. This result provides the first sharp theoretical explanation for the empirically observed sudden accuracy drops in embedding truncation experiments for both synthetic and state-of-the-art models.

Figure 1: Ground-truth Euclidean embeddings: triplet accuracy as a function of both ground-truth $d$1 and embedding $d$2, demonstrating catastrophic collapse once $d$3 falls below a threshold—top: unconstrained, bottom: spherical embeddings.

The probabilistic method is leveraged to construct random collections of triplet constraints, which are simultaneously:

• perfectly realizable in $d$4 dimensions, but
• unsatisfiable above a $d$5 fraction by any embedding of dimension $d$6.

The collapse threshold is shown to be independent of model architecture, loss function, or optimization strategy, thus isolating this as a pure representational barrier. This behavior persists for ground-truth dimensions $d$7 scaling up to $d$8 for $d$9 objects, and also extends to analogous quartet-based constraints.

Computational Hardness and Approximation Resistance

Beyond the intrinsic representational limitations, the paper delivers a strong computational inapproximability result. Building on the Unique Games Conjecture, it is shown that even if all but an arbitrarily small $d$0-fraction of triplets are realizable in some (potentially large) dimension, it is NP-hard to efficiently approximate the optimal accuracy better than $d$1. Thus, no polynomial-time algorithm can out-perform the random embedding baseline—even allowing unbounded embedding dimension.

These results formally establish approximation resistance for the triplet embedding problem. The gap reduction to Maximum Acyclic Subgraph (MAS) transfers known approximation lower bounds from the ranking CSP literature to the embedding setting, further solidifying the tightness of the demonstrated limitations.

Empirical Validation

Experimental results on synthetic datasets substantiate the theoretical predictions. As $d$2 decreases below a critical threshold relative to $d$3, triplet accuracy for both unconstrained and spherical embedding models exhibits a pronounced phase transition toward the $d$4 baseline.

Figure 2: Uniformly random triplet instances: as embedding dimension $d$5 varies, triplet accuracy stays near 1 until a sharp fall to the baseline value ($d$6, $d$7).

The methodology involves (i) embedding $d$8 points in ground-truth dimensions $d$9, (ii) generating large sets of triplet constraints, and (iii) optimizing representations using triplet loss (AdamW) with varied target dimension $D$0. The accuracy curve's abrupt collapse for $D$1 is consistent with the theoretical bounding argument.

Extensions and Broader Implications

The analysis extends to quadruplet (ordinal) constraints with minimal technical modification. The conclusions immediately transfer to a broad class of representation learning paradigms and contrastive objectives, such as those employed in modern large-scale retrieval and multimodal alignment. The result circumscribes the performance achievable by models seeking aggressive dimension reduction, irrespective of advances in optimization or architecture, unless domain-specific structure can be leveraged to circumvent the lower bounds.

The theoretical framework exposes a tight inherent link between the intrinsic dimensionality of data and the maximal attainable relational accuracy using low-dimensional embeddings, raising critical considerations for the deployment and further compression of embedding-based systems. In particular, attempts to tune $D$2 below the problem’s ground-truth requirement (as determined by data intrinsic structure) are effectively guaranteed to fail catastrophically.

Conclusion

Embedding-based representations encounter a provable, sudden accuracy collapse whenever the embedding dimension $D$3 undershoots the critical threshold dictated by the ground-truth dimension $D$4 of the input instance. This phenomenon persists even under the mildest supervision—triplet comparisons—and is immune to algorithmic advances under standard complexity assumptions. The results resolve foundational open questions on the limits of representation compression, establishing both an explicit, unavoidable information-theoretic bottleneck and computational hardness. These findings direct future work toward probing how structured constraints or large-margin properties might be harnessed to circumvent accuracy collapse, or inform the optimal design of scalable embedding architectures for high-dimensional retrieval and reasoning tasks.

Reference: "Provable Accuracy Collapse in Embedding-Based Representations under Dimensionality Mismatch" (2605.03346)