Retrieval Needs Multivectors: An Exponential Separation

Abstract: Recent works have highlighted the expressive limitations of embedding based retrieval models through both theoretical analyses and challenging benchmarks such as LIMIT. While multi-vector embeddings consistently outperform single-vector embeddings, the precise representational gap between them remains poorly understood. In this work, following Jayaram's work, we provide the first explicit family of query and document sets, together with their relevance matrices, for which single-vector embeddings that rank all relevant documents above irrelevant ones require exponential size, whereas polynomial-size multi-vector embeddings suffice. Our result establishes an exponential separation between the expressive power of single-vector and multi-vector embeddings for the task of ranking of documents as opposed to approximating numerical scores as in the work of Jayaram. Motivated by our theoretical construction, we introduce ANDOR, a new retrieval benchmark that naturally instantiates these hard examples. We show that state-of-the-art single-vector embedding models perform poorly on ANDOR in the zero-shot setting and exhibit only marginal improvements after fine-tuning, highlighting the inherent difficulty of the benchmark compared to prior work. In contrast, multi-vector models consistently outperform their single-vector counterparts and improve substantially with fine-tuning, closely aligning with our theoretical predictions.

• The paper establishes that multi-vector retrieval systems have a significant advantage over single-vector systems in terms of exact relevance ordering, with single-vector models requiring exponentially larger dimensions.
• It demonstrates that relevance matrices with high sign-rank necessitate single-vector embeddings of dimension $2^{\Omega(m)}$, while multi-vector embeddings achieve the same ordering with $O(m^6)$ representation size.
• The ANDOR benchmark, derived from the theoretical construction, highlights the empirical gap between single and multi-vector models, showing late interaction consistently outperforms single-vector systems, even under fine-tuning.

The paper establishes a formal expressive-power separation between single-vector and multi-vector retrieval for the retrieval-ordering problem. Its central claim is that there are explicit relevance matrices for which any single-vector inner-product representation requires exponentially large dimension, while a multi-vector representation scored by late interaction realizes the same relevance relation with polynomial representation size. The distinction is important because prior theoretical work primarily studied approximation of numerical Chamfer or MaxSim scores, whereas retrieval systems are ordinarily evaluated by whether relevant documents are ranked above irrelevant ones. The paper therefore addresses a weaker and more operational criterion than pointwise score preservation, while still obtaining an exponential separation (2608.21494).

Problem formulation and conceptual contribution

A single-vector retriever assigns a vector $q_i \in \mathbb{R}^d$ to each query and $p_j \in \mathbb{R}^d$ to each document, ranking documents by $\langle q_i,p_j\rangle$. A multi-vector retriever instead assigns sets of vectors to queries and documents. The scoring function is the Chamfer or late-interaction score

$$S(Q_i,P_j)=\frac{1}{|Q_i|}\sum_{q\in Q_i}\max_{p\in P_j}\langle q,p\rangle.$$

The target is not to reproduce a prescribed score matrix. It is to realize a Boolean relevance matrix $R$ such that every relevant document receives a higher score than every irrelevant document for each query. In other words, the representation must satisfy a per-query separating threshold between positive and negative documents.

This formulation separates three notions that are often conflated: numerical score approximation, binary classification of query-document pairs, and complete retrieval ordering. The paper argues that hardness for score approximation does not automatically imply hardness for retrieval ordering. This point is demonstrated constructively through a result concerning the NAND pattern matrix used in prior work: although that matrix is hard for approximating multi-vector Chamfer scores with single vectors, it admits a linear-dimensional single-vector representation that preserves its relevance ordering (Jayaram, 22 Jun 2026). The result is consequently not a restatement of the earlier score-approximation separation, but a distinct lower-bound construction targeted at ranking.

The paper’s primary lower bound uses sign-rank. If a relevance matrix can be represented by single-vector inner products with dimension $d$, then after introducing a query-dependent threshold, the resulting sign matrix has rank at most $d+1$. Therefore, a sign-rank lower bound transfers directly to a lower bound on the embedding dimension. The construction uses the pattern-matrix method and the Minsky–Papert function, whose high sign-rank behavior was established in communication-complexity theory (Sherstov et al., 2019).

The exponential separation

The relevance matrix is a pattern matrix

$R=\operatorname{PM}(N,n,\operatorname{MP}_{m,L}),$

with $L=4m^2$, $n=mL=4m^3$, and $p_j \in \mathbb{R}^d$0. A query is indexed by a selector $p_j \in \mathbb{R}^d$1 and a mask $p_j \in \mathbb{R}^d$2, while a document is indexed by a Boolean vector $p_j \in \mathbb{R}^d$3. The selector extracts one bit from each of $p_j \in \mathbb{R}^d$4 blocks of the document, and the mask applies a bitwise XOR. The resulting $p_j \in \mathbb{R}^d$5-bit string is arranged as an $p_j \in \mathbb{R}^d$6 Boolean array.

The Minsky–Papert predicate computes an AND over $p_j \in \mathbb{R}^d$7 clauses, each of which is an OR over $p_j \in \mathbb{R}^d$8 literals. Thus, a query-document pair is relevant exactly when every row of the array contains at least one satisfied literal. This is precisely an AND-of-OR predicate: all mandatory clauses must be satisfied, but each clause may be satisfied in any one of several ways.

For the selected parameterization, the underlying pattern matrix has sign-rank $p_j \in \mathbb{R}^d$9 (2608.21494). The paper then shows that any single-vector representation preserving the retrieval ordering must have dimension

$\langle q_i,p_j\rangle$0

The threshold argument is technically nontrivial because sign-rank witnesses cannot contain zero entries. Given an inner-product matrix $\langle q_i,p_j\rangle$1 and query-specific thresholds $\langle q_i,p_j\rangle$2, the authors perturb each threshold by a sufficiently small positive amount and construct

$\langle q_i,p_j\rangle$3

The matrix $\langle q_i,p_j\rangle$4 has positive entries on relevant pairs and negative entries on irrelevant pairs, so it is a valid sign-rank witness. Since $\langle q_i,p_j\rangle$5, the exponential sign-rank bound implies the exponential lower bound on $\langle q_i,p_j\rangle$6.

The multi-vector construction directly encodes the clause structure. Each query has $\langle q_i,p_j\rangle$7 vectors, one for each conjunction clause. Each document has $\langle q_i,p_j\rangle$8 one-hot signed vectors, one for each document coordinate. For clause $\langle q_i,p_j\rangle$9, the associated query vector places signed mass on the coordinates selected by $$S(Q_i,P_j)=\frac{1}{|Q_i|}\sum_{q\in Q_i}\max_{p\in P_j}\langle q,p\rangle.$$0 that belong to that clause. The document vectors encode the bits of $$S(Q_i,P_j)=\frac{1}{|Q_i|}\sum_{q\in Q_i}\max_{p\in P_j}\langle q,p\rangle.$$1 through signed standard basis vectors.

For a clause whose extracted row contains at least one satisfying literal, the maximum inner product with the document vectors is $$S(Q_i,P_j)=\frac{1}{|Q_i|}\sum_{q\in Q_i}\max_{p\in P_j}\langle q,p\rangle.$$2. If the clause is unsatisfied, the maximum is zero. Consequently, the Chamfer score is exactly $$S(Q_i,P_j)=\frac{1}{|Q_i|}\sum_{q\in Q_i}\max_{p\in P_j}\langle q,p\rangle.$$3 when all $$S(Q_i,P_j)=\frac{1}{|Q_i|}\sum_{q\in Q_i}\max_{p\in P_j}\langle q,p\rangle.$$4 clauses are satisfied, while it is at most

$$S(Q_i,P_j)=\frac{1}{|Q_i|}\sum_{q\in Q_i}\max_{p\in P_j}\langle q,p\rangle.$$5

when at least one clause fails. A threshold between these values realizes the Boolean relevance matrix exactly. Since $$S(Q_i,P_j)=\frac{1}{|Q_i|}\sum_{q\in Q_i}\max_{p\in P_j}\langle q,p\rangle.$$6, the separation margin is

$$S(Q_i,P_j)=\frac{1}{|Q_i|}\sum_{q\in Q_i}\max_{p\in P_j}\langle q,p\rangle.$$7

The query representation contains $$S(Q_i,P_j)=\frac{1}{|Q_i|}\sum_{q\in Q_i}\max_{p\in P_j}\langle q,p\rangle.$$8 vectors and the document representation $$S(Q_i,P_j)=\frac{1}{|Q_i|}\sum_{q\in Q_i}\max_{p\in P_j}\langle q,p\rangle.$$9 vectors, each in an $R$0-dimensional space. Under the paper’s size convention—number of vectors multiplied by vector dimension—the representation size is $R$1 per query and document. The resulting comparison is therefore exponential versus polynomial:

Representation	Retrieval criterion	Required size
Single-vector	Exact relevance ordering	$R$2 dimensions
Multi-vector	Exact relevance ordering with Chamfer score	$R$3 size
Multi-vector separation	Relevant versus irrelevant scores	$R$4

The result is an explicit exponential separation for ranking, not merely an empirical advantage of late interaction. It also shows why a fixed-dimensional reduction of multi-vector representations cannot automatically preserve retrieval order at low dimension. MUVERA-type results provide single-vector approximations to Chamfer scores with dimension exponential in the inverse squared error (2608.21494). For the present construction, preserving the ordering requires approximation error on the scale of the margin, $R$5, which yields a dimension exponential in a polynomial of $R$6. This does not contradict the paper’s theorem, but it underscores that score approximation and direct ranking preservation involve different parameter regimes.

Separation from prior score-approximation hardness

The paper’s second theoretical result clarifies the boundary of the preceding literature. Jayaram’s construction based on a NAND pattern matrix establishes that approximating multi-vector Chamfer scores may require very high-dimensional single-vector embeddings (Jayaram, 22 Jun 2026). The present paper shows that the same relevance matrix nevertheless admits a unit-norm single-vector representation in dimension $R$7, with a relevance margin of

$R$8

The construction aggregates the selected document bits into a single signed vector and compares it against a similarly signed query vector. If $R$9 of the $d$0 selected bits agree with the query mask, the inner product is

$d$1

For NAND, irrelevance occurs only when $d$2, while every relevant pair has $d$3. A single threshold therefore separates the two classes. The implication is direct: the prior exponential lower bound cannot be interpreted as an exponential lower bound for the weaker retrieval-ordering objective. A matrix can be difficult to approximate numerically while remaining easy to threshold correctly.

This distinction strengthens the paper’s main claim. The exponential separation is not obtained by importing a stronger approximation lower bound and treating it as a ranking lower bound. It requires a relevance matrix whose Boolean decision structure itself has high sign-rank, while its clause structure remains naturally compatible with multi-vector MaxSim computation.

ANDOR as a semantic realization

The theoretical construction motivates ANDOR, a controlled e-commerce retrieval benchmark. The benchmark translates Boolean variables into product attributes, clauses into product categories, and the Minsky–Papert predicate into a faceted-search relevance rule. A query specifies several mandatory categories, with multiple acceptable attribute values per category. A document is relevant if and only if it overlaps with the query in at least one attribute for every queried category:

$d$4

The benchmark therefore implements AND across categories and OR within each category. Its design avoids several shortcuts that could make the task solvable through superficial document statistics. Documents generally contain 11–15 categories and 2–4 attributes per present category, while the corpus contains 50,000 products. Hard negatives retain the relevant category structure but violate exactly one, two, or three queried categories. In particular, level-1 negatives fail only one constraint, requiring models to inspect the disjunctive content rather than count category overlap.

The test set contains 1,000 query skeletons across nine widths, with exactly two relevant documents per query at every width. Query width controls the mean number of acceptable attributes per category, ranging from 3.5 to 11.5. The query suites are nested: wider suites add acceptable values rather than replacing existing ones. This preserves the identity of the two positive documents across widths and isolates the effect of increasing disjunctive complexity. The training sets use 800 queries at widths from 5.5 to 9.5, with substantially more positives per query than the sparse test setting.

This construction gives ANDOR useful experimental control, but it also creates an important interpretive distinction. The benchmark is a semantic instantiation inspired by the pattern-matrix construction, not the formal hard matrix used in the proof. Consequently, the theoretical theorem does not imply that the finite ANDOR relevance matrix has exponential sign-rank. Its empirical role is instead to test whether the same compositional structure produces a measurable gap in learned semantic retrieval.

Empirical evaluation

The evaluation compares seven models: two multi-vector systems, GTE ModernColBERT v1 and Jina Embeddings v4 in late-interaction mode, and five single-vector systems, including Jina Embeddings v4, Qwen3 Embedding 0.6B, Snowflake Arctic Embed L v2, Cohere Embed v4, and OpenAI text-embedding-3-large. Jina v4 is especially informative because its single-vector and multi-vector heads share the same backbone and can be trained jointly. This comparison reduces confounding from parameter count, pretraining data, and architecture.

In the zero-shot setting, late interaction consistently outperforms all single-vector models over the full width sweep. Relative to Cohere Embed v4, the strongest single-vector baseline, GTE ModernColBERT achieves gains of approximately 81% at Recall@2, 87% at Recall@10, and 87% at Recall@100 according to the reported aggregate comparison. The other single-vector models exhibit substantially larger gaps; the reported relative gains over the non-Jina single-vector baselines reach 631.8% at Recall@2, 510.5% at Recall@10, and 414.4% at Recall@100 when averaged over the listed zero-shot baselines.

Fine-tuning improves all evaluated open-source models, but it does not eliminate the gap. Averaged across the train–test grid and the Qwen3 and Snowflake baselines, GTE ModernColBERT retains approximately a 99% relative advantage at Recall@2 and 58% at Recall@100. Its fine-tuned Recall@100 declines from 91.4% at test width 3.5 to 25.4% at width 11.5, a 66.0 percentage-point drop and a factor of 3.6. Qwen3 declines from 67.2% to 17.8%, while Snowflake declines from 68.0% to 14.4%. Thus, increasing the number of acceptable alternatives per mandatory category makes the task harder for every model, but the single-vector models degrade more rapidly over much of the range.

The controlled Jina comparison provides the strongest empirical evidence for a representational effect. After joint fine-tuning from the same checkpoint, the multi-vector head leads the single-vector head at every test width and cutoff. The reported relative gains of multi-vector over single-vector Jina are approximately 105% at Recall@2, 84% at Recall@10, and 62% at Recall@100. The paper further reports that fine-tuning the Jina single-vector head brings it close to the zero-shot multi-vector head at shallow cutoffs, but does not materially surpass it. At Recall@2, the two are within roughly one percentage point across the width sweep, whereas the fine-tuned multi-vector head remains clearly superior.

Independent training produces similar conclusions. The multi-vector-to-single-vector ratios are 2.07, 1.89, and 1.61 at Recall@2, Recall@10, and Recall@100, respectively. Under joint training, the corresponding ratios are 2.05, 1.84, and 1.61. The near invariance of these ratios indicates that the gap is not plausibly attributable only to the shared optimization regime or to the particular joint objective. At the same time, the paper correctly qualifies this comparison: independent and joint runs differ in learning rates, temperatures, and other recipe details, so they should not be treated as a perfectly isolated ablation.

The width sweep also reveals a nontrivial generalization effect. For GTE ModernColBERT, increasing the training width from 5.5 to 9.5 changes Recall@100 at test width 3.5 by only 3.45 percentage points, but improves Recall@100 at test width 11.5 from 17.10% to 32.50%, a 15.40 percentage-point or 90% relative increase. Wider training queries therefore improve robustness to wider test queries, although the improvement does not close the representation gap. The optimal training width is model-dependent: GTE and Snowflake improve monotonically through width 9.5, whereas Qwen3 peaks at 7.5.

The nDCG and MRR results reinforce the recall findings. Because each test query has exactly two positives, nDCG@2 is close to Recall@2. At deeper cutoffs, the decreasing nDCG indicates that retrieved positives increasingly occur at poor ranks as the Boolean structure becomes more complex. MRR yields a particularly clear family-level separation, with late interaction leading single-vector systems by roughly 80% in the pooled comparison. These ranking-sensitive results are consistent with the paper’s theoretical objective rather than merely reflecting the ability to retrieve a positive somewhere in the corpus.

Limitations and open questions

The formal theorem is worst-case and asymptotic. It establishes the existence of a family of relevance matrices parameterized by $d$5, not that arbitrary practical retrieval datasets require exponential single-vector dimension. The multi-vector upper bound also uses a representation in $d$6-dimensional space with $d$7 document vectors, yielding an $d$8 total size under the paper’s accounting. This is polynomial but not necessarily computationally efficient for deployment, and the construction does not address indexing, compression, or candidate-generation costs.

The ANDOR benchmark is controlled and synthetic in its semantic content, even though it is rendered as natural-language shopping data. Its Boolean labels are exact and its corpus is deliberately engineered around hard negatives. This is appropriate for testing compositional retrieval, but it limits direct claims about unconstrained web or production e-commerce distributions. Moreover, every test query has exactly two relevant documents. The paper notes that an existential result from prior work then guarantees a five-dimensional single-vector embedding for the finite relevance matrix, assuming complete access to all relevance labels (S et al., 31 Mar 2026). The experiments do not contradict that result: they evaluate models that must infer relevance from semantic text, rather than an embedding constructed with access to the full matrix. The distinction between representational existence and learnability from semantic content is therefore central.

The experiments also do not establish that multi-vector representations retain an exponential advantage under approximate ordering, where only a $d$9 fraction of relevant and irrelevant comparisons must be correct. Nor do they analyze the stronger objective of preserving a specified total ranking among documents within each query. These are explicit unresolved questions in the paper. A further open issue is whether the width-dependent degradation is caused primarily by the need to maintain multiple alternatives within each category, by the increasing conjunction depth, or by interactions between semantic encoding and hard-negative density.

Conclusion

The paper proves an exponential separation between single-vector and multi-vector retrieval for exact relevance ordering. Its construction combines high-sign-rank pattern matrices with a direct clause-wise Chamfer realization: single-vector embeddings require dimension $d+1$0, whereas multi-vector embeddings achieve exact separation with $d+1$1 representation size and margin $d+1$2. The accompanying ANDOR benchmark translates the same AND-of-OR structure into faceted shopping retrieval and shows persistent multi-vector advantages, including under controlled same-backbone fine-tuning. The results establish that late interaction can provide a genuine representational advantage for ranking, while also delineating the difference between score-approximation hardness, finite-matrix realizability, and learnability from semantic data.