What Limits Does Quantization Place on Dense Top-$k$ Retrieval? A Theoretical Study

Abstract: We establish conditions for embedding a corpus of $N$ documents as $d$-dimensional vectors such that every $k$-subset $S \subseteq [N]$ is realizable as a result of top-$k$ retrieval by some query vector. Recent work shows that $d = O(k)$ suffices for such embeddings to exist in $\mathbb{R}^d$, independently of $N$. We theoretically prove that this corpus-independent bound is specific to infinite precision. With $B$ bits per coordinate, perfect top-$k$ retrieval requires $Bd = Ω(k \ln N)$; thus, at any fixed precision, the dimension must grow at least logarithmically with $N$. Specializing to a $\ell_2$-normalized $B$-bit uniform scalar quantization model, we also identify a threshold on the precision $B^{*} = O(\ln \ln N)$ below which no dimension suffices, together with two further regimes that bound the feasible $(B, d)$ pairs. Our result implies that in practical vector databases and dense retrieval systems where quantization is standard, the embedding dimension and possibly the precision must grow with the corpus size.

• The paper presents an impossibility result, showing that finite-precision quantization requires precision and/or embedding dimension to scale with corpus size to perfectly shatter all top-k subsets.
• It uses information-theoretic and combinatorial counting methods to derive lower bounds, establishing that the bit budget must grow as Ω(k ln N) for full retrieval.
• The study highlights a practical trade-off, warning that standard quantization (e.g. int4) and typical embedding dimensions may be insufficient for large corpora in dense retrieval systems.

Quantization Constraints on Dense Top-$k$ Retrieval: Theoretical Limits

Introduction

This work offers a rigorous theoretical analysis of how quantization restricts the expressive capacity of dense vector embeddings for top-$k$ retrieval from a corpus of $N$ documents. Specifically, it poses the question: for every $k$-subset $S \subseteq [N]$, is there always an embedding and a query vector, within a finite-precision discrete set, such that $S$ is the exact top-$k$ result of the inner product scores? While prior literature established that $d = O(k)$ embedding dimensions suffice in the infinite-precision ($\mathbb{R}^d$) regime [(Wang et al., 28 Jan 2026); weller2026limits], this paper presents the first explicit impossibility results in the quantized setting, making the pivotal claim that precision and/or embedding dimension must scale with the corpus size for full top-$k$ shattering in any practical bit-limited system.

Problem Framing and Existing Results

Let $k$0 corpus elements and queries be embedded as vectors in a space $k$1, with each coordinate quantized using $k$2 bits. The goal is to realize every possible $k$3-subset as a top-$k$4 result—equivalently, to shatter all ground-truth top-$k$5 sets. Previous positive results [(Wang et al., 28 Jan 2026), weller2026limits] guarantee this in $k$6 with infinite precision. However, these constructions fundamentally rely on the continuum structure of $k$7, which breaks down under quantization.

In real IR systems, float and int quantization (int4, int8, fp4, fp8) is the norm due to scalability constraints; therefore, understanding the effect of quantization is crucial.

Main Theoretical Results

The key insight is that the finite alphabet constraint alters the existential question, turning it into a high-dimensional counting problem: can the finite pool of bit-limited codewords realize the combinatorially vast number of subset separation constraints?

An upper bound is established using the first moment method, setting an information-theoretic lower bound on the memory budget ($k$8, bits per vector) required to shatter all top-$k$9 sets. The core results are:

1. General Finite-Alphabet Bound: The bit budget must grow with $N$0.

   $N$1

   At fixed $N$2 (precision), the embedding dimension $N$3 must grow logarithmically with $N$4. Conversely, at fixed $N$5, the minimum quantization precision $N$6 increases with the corpus size.

   Figure 1: $N$7 needed for perfect top-$N$8 shattering as a function of $N$9, for $k$0, under Theorem 1's counting bound.

2. $k$1-Normalized Uniform Quantization: In practical quantized systems with unit-norm embeddings and uniform scalar quantization, the constraints are even tighter. There exists a precision threshold:

- $k$2 is a hard floor: for $k$3, no embedding dimension suffices to shatter all top-$k$4 subsets. - For $k$5, both lower (from subset counting) and upper (from norm-ball grid density) bounds on $k$6 are derived. - For fixed $k$7 and large $k$8, the norm restriction causes the number of available embeddings to plummet, further restricting shatterability above a dimension ceiling.

Figure 2: (Left) Lower and upper bounds for $k$9 vs. $S \subseteq [N]$0 given $S \subseteq [N]$1 (critical dimensions from Theorem 2). (Right) Growth of $S \subseteq [N]$2 with $S \subseteq [N]$3.

Implications for Vector Search and Embedding Design

These results have quantifiable consequences for the architecture and scalability of dense retrieval and vector search systems:

• Practicality Constraint: The $S \subseteq [N]$4-invariant dimension guarantee of the infinite-precision setting does not survive practical quantization. For int4 quantization (a widespread default), beyond a certain corpus size ($S \subseteq [N]$5), increasing the embedding dimension alone cannot compensate for insufficient precision.
• Trade-Off: The bound $S \subseteq [N]$6 is symmetric; decreasing precision must be offset by a proportional increase in dimension, and vice versa, to maintain retrieval completeness.
• Precision Floor: The existence of a threshold $S \subseteq [N]$7 imposes a lower limit on quantization; subcritical precision leads to intrinsic impossibility, irrespective of learned embedding geometry.
• System Sizing: For typical parameter values ($S \subseteq [N]$8, $S \subseteq [N]$9), standard dense retriever dimensions ($S$0) become insufficient for $S$1—matching real-scale applications. Higher values of $S$2 can mitigate this, but storage and compute costs rise.

Connections to Prior Work and Open Problems

• These negative results formally separate the regimes addressed by prior works on low-dimensional top-$S$3 realizability for real-valued embeddings [(Wang et al., 28 Jan 2026), weller2026limits] from the discrete setting analyzed here. The new constraints apply in any finite-alphabet setting, including scalar and product quantization.
• The impossibility results do not depend on the specifics of the learnability or optimization protocol, but purely on information-theoretic covering arguments.
• The possibility of approximate shattering (allowing a controlled error rate) and the extension to learned/product quantization codebooks remain open directions.

Future Developments

• The present analysis is for perfect retrieval; future research could relax this requirement, offering trade-offs between error rates and memory/dimensionality budgets.
• Further exploration of quantization-aware embedding architectures and retrieval algorithms that operate near the derived lower bounds is warranted.
• Extending counting techniques (possibly using the second moment method or alternative combinatorial tools) may tighten the bounds or yield explicit constructions.

Conclusion

This work establishes that finite-precision quantization in vector databases places fundamental lower bounds on the minimal required product of bit precision and embedding dimension for universal support of top-$S$4 retrieval tasks over large corpora. Such constraints are not detectable in the infinite-precision regime, underscoring the need to account for quantization's combinatorial effect in dense retrieval system design and theoretical framework. The precise asymptotic thresholds inform both hardware quantization choices and embedding dimensionality decisions for scalability.