MaxSim: Late-Interaction Similarity Operator

• MaxSim is a late-interaction similarity operator that retains multiple contextualized embeddings and scores queries by summing the maximum dot products over query tokens.
• It enables fine-grained retrieval by independently encoding query and document representations, preserving token-level details and interpretability.
• Despite its computational challenges from full similarity matrix evaluations, indexing and ANN strategies help maximize its effectiveness across diverse modalities.

MaxSim is a late-interaction similarity operator for multi-vector retrieval. Rather than compressing a query and a document into single embeddings, it retains multiple contextualized token, patch, or segment vectors and scores a candidate by summing, over query-side units, the maximum similarity to any document-side unit. In the notation used across late-interaction systems,

$$S(Q,D)=\sum_{i=1}^{|Q|}\max_{j=1}^{|D|}(q_i\cdot d_j),$$

equivalently

$$S_{q,c}=\sum_{i\in[|E_q|]}\max_{j\in[|E_c|]}E_{q_i}\cdot E_{c_j}^{T}.$$

This operator is the defining mechanism of ColBERT-style retrieval and of a wider family of text, multimodal, visual, and biological retrieval systems because it preserves fine-grained evidence while still allowing independent encoding of queries and candidates (Edy et al., 27 Mar 2026, Chaffin et al., 5 Aug 2025).

1. Formal definition and scoring semantics

MaxSim is a two-stage pooling rule. First, for each query-side embedding, it scans all candidate-side embeddings and keeps only the single best match. Second, it aggregates those per-query maxima, typically by summation. In the standard late-interaction formulation, the compared units may be text tokens, visual patches, multimodal fused tokens, residues, or other localized embeddings, but the operator itself remains the same (Edy et al., 27 Mar 2026, Chaffin et al., 5 Aug 2025).

The similarity inside the maximization is usually a dot product. In several systems, embeddings are explicitly $\ell_2$-normalized, so the dot product becomes cosine-like or cosine exactly. AMES, for example, defines

$$\mathrm{MaxSim}(\mathcal{Q},d)=\sum_{i=1}^{|\mathcal{Q}|}\max_{\mathbf{d}_j\in\mathcal{D}_d}\langle \mathbf{q}_i,\mathbf{d}_j\rangle$$

with normalized query and document child embeddings, while OFAR applies the same structural rule to unified multimodal token sequences produced by OFA encoders (Joseph et al., 13 Mar 2026, Dengtian et al., 2023).

The operator is generally asymmetric. PROTOCOL makes this explicit for protein retrieval: with

$$\mathrm{MaxSim}(E^q,E^d)=\sum_{i=1}^{T_q}\max_{j\in[T_d]}\langle e_i^q,e_j^d\rangle,$$

summing over query residues means that $$\mathrm{MaxSim}(E^q,E^d)\neq \mathrm{MaxSim}(E^d,E^q)$$ in general. The score therefore answers a query-conditioned coverage question rather than a symmetric set-similarity question (Cohn et al., 27 May 2026).

A useful consequence of this semantics is token-level or unit-level interpretability. Because each query-side unit routes to exactly one best candidate-side unit under the hard max, the final score can be decomposed into localized match contributions. This localized routing is central to both the precision and the pathologies of late interaction (Edy et al., 27 Mar 2026, Suresh et al., 6 Apr 2026).

2. Role in late interaction retrieval

MaxSim is the mechanism that distinguishes late interaction from single-vector dense retrieval. In the single-vector paradigm, a transformer produces contextualized token embeddings and then a pooling step such as mean, max, or CLS compresses the whole sequence into one fixed-dimensional vector. Late interaction removes that compression bottleneck by preserving individual contextualized embeddings and performing relevance aggregation only after independent encoding. PyLate describes this as the defining difference between dense retrieval and ColBERT-style multi-vector retrieval, and attributes late interaction’s empirical advantages to that preservation of token-level information, particularly for out-of-domain retrieval, long-context retrieval, and reasoning-intensive retrieval (Chaffin et al., 5 Aug 2025).

Because encoding is independent on the two sides, document representations can be precomputed. OFAR uses separate query and document encoders, then applies “MaxSim-based Contrastive Late Intersection” over multimodal token sequences; AMES does the same in a production-style multimodal enterprise setting, where parent documents are reconstructed from child embeddings and exact MaxSim is applied only on shortlisted candidates (Dengtian et al., 2023, Joseph et al., 13 Mar 2026).

This design is not restricted to fixed document representations. Argus retains the standard late-interaction scoring rule,

$$\mathrm{MaxSim}(q,d)=\sum_{i=1}^{m}\max_{j=1,\dots,n} q_i^\top d_j,$$

but makes the document-side multi-vector representation query-conditioned, $\mathbf{D}(q)$, through region-aware routing and mixture-of-experts fusion before MaxSim is applied. The scoring operator is unchanged; the document vectors entering it are rewritten per query (Abdallah et al., 3 Jun 2026).

Across these systems, MaxSim functions as a generic late-interaction primitive rather than a text-only heuristic. This suggests that its importance lies less in any particular modality and more in the architectural decision to postpone aggregation until after local matching has occurred.

3. Computational structure and systems implications

The expressive benefit of MaxSim comes with substantial systems cost. Exact scoring for one query-document pair requires all pairwise unit interactions between the query-side set and the document-side set. In visual late interaction, the cost is written explicitly as

$$\mathrm{Cost}_{\mathrm{search}}=Q\times D\times N\times d$$

multiply-adds over a corpus of $N$ pages, where $$S_{q,c}=\sum_{i\in[|E_q|]}\max_{j\in[|E_c|]}E_{q_i}\cdot E_{c_j}^{T}.$$0 is query-token count, $$S_{q,c}=\sum_{i\in[|E_q|]}\max_{j\in[|E_c|]}E_{q_i}\cdot E_{c_j}^{T}.$$1 is document-vector count, and $$S_{q,c}=\sum_{i\in[|E_q|]}\max_{j\in[|E_c|]}E_{q_i}\cdot E_{c_j}^{T}.$$2 is embedding dimension (Yeroyan, 13 Feb 2026).

The main bottleneck on modern GPUs is not the arithmetic alone but the materialization of the full similarity matrix. TileMaxSim analyzes standard MaxSim as a memory-bound operation and reports that naive GPU implementations reach only $$S_{q,c}=\sum_{i\in[|E_q|]}\max_{j\in[|E_c|]}E_{q_i}\cdot E_{c_j}^{T}.$$3–$$S_{q,c}=\sum_{i\in[|E_q|]}\max_{j\in[|E_c|]}E_{q_i}\cdot E_{c_j}^{T}.$$4 of peak HBM bandwidth because they write and reread the full $$S_{q,c}=\sum_{i\in[|E_q|]}\max_{j\in[|E_c|]}E_{q_i}\cdot E_{c_j}^{T}.$$5 similarity matrix even though only rowwise maxima and a final sum are needed. Its fused Triton kernels avoid this materialization, reach $$S_{q,c}=\sum_{i\in[|E_q|]}\max_{j\in[|E_c|]}E_{q_i}\cdot E_{c_j}^{T}.$$6 of peak HBM bandwidth on H100, score $$S_{q,c}=\sum_{i\in[|E_q|]}\max_{j\in[|E_c|]}E_{q_i}\cdot E_{c_j}^{T}.$$7M documents/second, and preserve exact rankings on MS MARCO and three BEIR benchmarks (Sharma, 24 Jun 2026). Flash-MaxSim applies the same IO-aware principle, computing the exact score without ever materializing the tensor, and reports up to $$S_{q,c}=\sum_{i\in[|E_q|]}\max_{j\in[|E_c|]}E_{q_i}\cdot E_{c_j}^{T}.$$8 speedup on A100, $$S_{q,c}=\sum_{i\in[|E_q|]}\max_{j\in[|E_c|]}E_{q_i}\cdot E_{c_j}^{T}.$$9 on H100, up to $\ell_2$0 less inference memory, and about $\ell_2$1 less training memory while keeping $\ell_2$2 top-20 agreement with an FP32 reference (Pony et al., 28 May 2026).

Practical retrieval systems therefore rely on indexing and approximation. PyLate exposes MaxSim as a first-class scoring module and couples it with PLAID and HNSW/Voyager-based indexes for large-scale search, while emphasizing storage-quality tradeoffs such as post-hoc pooling via pool_factor (Chaffin et al., 5 Aug 2025). Visual RAG Toolkit reduces document-side vectors from hundreds or thousands to dozens using static spatial pooling, then performs exact MaxSim reranking on the full embeddings; for strong 3B visual retrievers this preserves NDCG@5, NDCG@10, Recall@5, and Recall@10 with approximately $\ell_2$3 QPS at common cutoffs (Yeroyan, 13 Feb 2026). AMES similarly uses token-level ANN retrieval and a Top-$\ell_2$4 MaxSim approximation for candidate generation, followed by exact MaxSim reranking on shortlisted parents (Joseph et al., 13 Mar 2026).

Other work rewrites or sparsifies the operator more aggressively. ColBERTSaR replaces token embeddings with anchor identifiers and scores with a residual-free approximation

$\ell_2$5

yielding an index that is $\ell_2$6–$\ell_2$7 smaller than a one-bit PLAID index (Yang et al., 4 Jun 2026). LEMUR learns a latent single-vector MIPS surrogate for corpus-wide MaxSim and reports order-of-magnitude speedups over earlier multi-vector search methods (Jääsaari et al., 29 Jan 2026). Col-Bandit leaves the model unchanged but adaptively reveals only selected document–query-token MaxSim entries at reranking time, reducing MaxSim FLOPs by up to $\ell_2$8 while preserving ranking fidelity (Pony et al., 2 Feb 2026).

4. Empirical behavior: effectiveness, sufficiency, and pathologies

A major line of recent work has examined what MaxSim is actually doing beyond its headline retrieval quality. One result is that the operator can induce length bias. In the NanoBEIR analysis of late interaction dynamics, if a causal multi-vector encoder appends tokens to a document chunk, the set of candidate document embeddings strictly expands while the previous embeddings remain available; therefore each term in

$\ell_2$9

is monotone nondecreasing, creating a theoretical monotonic length bias. Empirically, the causal multi-vector model jina-embeddings-v4 showed the clearest bias, while the causal single-vector baseline did not; bi-directional multi-vector models suppressed strict monotonicity but still showed statistically significant vulnerability at extreme lengths (Edy et al., 27 Mar 2026).

The same study also tested a common objection to hard max pooling: that informative evidence may exist below the top-1 matched document token and be discarded by MaxSim. On NanoBEIR, including analyses of failed and successful retrievals, the reported result was that “no significant similarity trend lies beyond the top-1 document token,” with only a local caveat on NanoArguAna that did not generalize across the benchmark. The paper therefore concluded that “the MaxSim operator efficiently exploits the token-level similarity scores” for the tested models and tasks (Edy et al., 27 Mar 2026).

A more mechanistic critique concerns gradient routing. “Spike Hijacking in Late-Interaction Retrieval” treats MaxSim as a winner-take-all pooling rule and shows that it produces significantly higher patch-level gradient concentration than Top-$$\mathrm{MaxSim}(\mathcal{Q},d)=\sum_{i=1}^{|\mathcal{Q}|}\max_{\mathbf{d}_j\in\mathcal{D}_d}\langle \mathbf{q}_i,\mathbf{d}_j\rangle$$0 pooling or softmax aggregation. In the synthetic environment, the patch-gradient Gini coefficient stayed around $$\mathrm{MaxSim}(\mathcal{Q},d)=\sum_{i=1}^{|\mathcal{Q}|}\max_{\mathbf{d}_j\in\mathcal{D}_d}\langle \mathbf{q}_i,\mathbf{d}_j\rangle$$1 for MaxSim, versus $$\mathrm{MaxSim}(\mathcal{Q},d)=\sum_{i=1}^{|\mathcal{Q}|}\max_{\mathbf{d}_j\in\mathcal{D}_d}\langle \mathbf{q}_i,\mathbf{d}_j\rangle$$2 for Top-$$\mathrm{MaxSim}(\mathcal{Q},d)=\sum_{i=1}^{|\mathcal{Q}|}\max_{\mathbf{d}_j\in\mathcal{D}_d}\langle \mathbf{q}_i,\mathbf{d}_j\rangle$$3 and $$\mathrm{MaxSim}(\mathcal{Q},d)=\sum_{i=1}^{|\mathcal{Q}|}\max_{\mathbf{d}_j\in\mathcal{D}_d}\langle \mathbf{q}_i,\mathbf{d}_j\rangle$$4 for softmax; on a real ColQwen2.5 benchmark the ordering persisted with $$\mathrm{MaxSim}(\mathcal{Q},d)=\sum_{i=1}^{|\mathcal{Q}|}\max_{\mathbf{d}_j\in\mathcal{D}_d}\langle \mathbf{q}_i,\mathbf{d}_j\rangle$$5, $$\mathrm{MaxSim}(\mathcal{Q},d)=\sum_{i=1}^{|\mathcal{Q}|}\max_{\mathbf{d}_j\in\mathcal{D}_d}\langle \mathbf{q}_i,\mathbf{d}_j\rangle$$6, and $$\mathrm{MaxSim}(\mathcal{Q},d)=\sum_{i=1}^{|\mathcal{Q}|}\max_{\mathbf{d}_j\in\mathcal{D}_d}\langle \mathbf{q}_i,\mathbf{d}_j\rangle$$7, respectively. The paper links this sparse routing to increased sensitivity to document length and to “spike hijacking,” where semantically competitive distractor patches steal many token-wise maxima (Suresh et al., 6 Apr 2026).

Taken together, these results give a mixed picture. MaxSim’s hard top-1 pooling often appears sufficient for current late-interaction retrievers, but the same hard routing can create structural brittleness under document-length growth, targeted distractors, or causal contextualization.

5. Modalities, domains, and application-specific formulations

MaxSim has been adapted well beyond text retrieval. The compared units change, but the fundamental rule—best local match per query-side unit, then aggregation—remains stable.

Domain	Query-side units	Candidate-side units
Illegal live-streaming evidence retrieval	multimodal query tokens	multimodal document tokens
Enterprise multimodal search	text query tokens	text tokens, image patches, or video frames
Protein homolog search	residue embeddings	residue embeddings
Text-motion retrieval	text tokens	motion-image patch embeddings
Semantic caching	prompt-segment embeddings	cached-prompt segment embeddings
Agent discovery	query embedding	usage-example embeddings

In OFAR, MaxSim is applied over unified multimodal token sequences generated by OFA encoders, not over separate text-only and image-only channels, and the resulting “MaxSim-based Contrastive Late Intersection” yields large gains over a $$\mathrm{MaxSim}(\mathcal{Q},d)=\sum_{i=1}^{|\mathcal{Q}|}\max_{\mathbf{d}_j\in\mathcal{D}_d}\langle \mathbf{q}_i,\mathbf{d}_j\rangle$$8-similarity ablation on TaoLive (Dengtian et al., 2023). AMES generalizes the same late-interaction principle to enterprise search over text, image, and video child embeddings, using ANN-restricted approximate MaxSim for candidate generation and exact MaxSim reranking for final ranking (Joseph et al., 13 Mar 2026).

PROTOCOL transfers ColBERT-style late interaction to protein homolog retrieval by representing proteins as sets of $$\mathrm{MaxSim}(\mathcal{Q},d)=\sum_{i=1}^{|\mathcal{Q}|}\max_{\mathbf{d}_j\in\mathcal{D}_d}\langle \mathbf{q}_i,\mathbf{d}_j\rangle$$9-dimensional normalized residue embeddings and scoring candidates with asymmetric residue-level MaxSim. On SCOPe and Pfam, this outperformed pooled protein baselines and several alignment-oriented alternatives, supporting the claim that local motif or domain evidence benefits from query-conditioned maximum matching (Cohn et al., 27 May 2026).

Fine-grained motion retrieval uses text-token to motion-patch MaxSim over ViT patch embeddings extracted from a structured joint-angle “motion image,” with the score

$$\mathrm{MaxSim}(E^q,E^d)=\sum_{i=1}^{T_q}\max_{j\in[T_d]}\langle e_i^q,e_j^d\rangle,$$0

The paper attributes its gains to replacing global embedding cosine with localized token-patch late interaction while also enabling interpretable joint/time correspondences (Zhang et al., 10 Mar 2026).

In semantic caching, MVR-cache adapts MaxSim to prompt segments rather than tokens and replaces directional MaxSim with a symmetric normalized form,

$$\mathrm{MaxSim}(E^q,E^d)=\sum_{i=1}^{T_q}\max_{j\in[T_d]}\langle e_i^q,e_j^d\rangle,$$1

because cache reuse requires mutual semantic compatibility rather than one-way coverage. The segmentation model is trained with a theoretically motivated REINFORCE objective, and the resulting system improves cache hit rates while maintaining the same correctness guarantees (Noshad et al., 24 May 2026).

GRAIL moves the abstraction one level higher, replacing token-level units with agent usage examples. Its “MaxSim Resonance” computes the maximum similarity between a query embedding and an agent’s example embeddings, then mixes that score with a coarse context score for final ranking. The paper’s theoretical argument against mean pooling is explicit: if an agent has many approximately orthogonal capabilities, the mean-pooled score decays roughly like $$\mathrm{MaxSim}(E^q,E^d)=\sum_{i=1}^{T_q}\max_{j\in[T_d]}\langle e_i^q,e_j^d\rangle,$$2, whereas MaxSim preserves a score of $$\mathrm{MaxSim}(E^q,E^d)=\sum_{i=1}^{T_q}\max_{j\in[T_d]}\langle e_i^q,e_j^d\rangle,$$3 for a perfectly aligned capability example (Xu, 4 May 2026).

6. Variants, generalizations, and surrogates

Several works retain the MaxSim intuition but alter the aggregation rule or replace the operator with more deployable surrogates. BiMax, for cross-lingual document alignment, defines

$$\mathrm{MaxSim}(E^q,E^d)=\sum_{i=1}^{T_q}\max_{j\in[T_d]}\langle e_i^q,e_j^d\rangle,$$4

then symmetrizes it as

$$\mathrm{MaxSim}(E^q,E^d)=\sum_{i=1}^{T_q}\max_{j\in[T_d]}\langle e_i^q,e_j^d\rangle,$$5

This bidirectional averaging is meant to reward mutual coverage rather than one-sided containment, and on WMT16 it achieved accuracy comparable to Optimal Transport with an approximate $$\mathrm{MaxSim}(E^q,E^d)=\sum_{i=1}^{T_q}\max_{j\in[T_d]}\langle e_i^q,e_j^d\rangle,$$6-fold speed increase (Wang et al., 17 Oct 2025).

Spectral Retrieval treats MaxSim as one endpoint of a broader family. With a pooled single query vector and per-token document embeddings, it defines

$$\mathrm{MaxSim}(E^q,E^d)=\sum_{i=1}^{T_q}\max_{j\in[T_d]}\langle e_i^q,e_j^d\rangle,$$7

as the $$\mathrm{MaxSim}(E^q,E^d)=\sum_{i=1}^{T_q}\max_{j\in[T_d]}\langle e_i^q,e_j^d\rangle,$$8 endpoint of a multi-scale sinc-convolution framework; as $$\mathrm{MaxSim}(E^q,E^d)=\sum_{i=1}^{T_q}\max_{j\in[T_d]}\langle e_i^q,e_j^d\rangle,$$9, the method approaches mean pooling. The paper proves that when both endpoints are present, the final spectral score is pointwise at least as large as either MaxSim or mean-pooled cosine, and argues that intermediate scales can better handle relevance localized to short spans rather than single tokens or entire documents (Morandi, 23 May 2026).

Other papers preserve MaxSim exactly while changing upstream representations. NumColBERT keeps the standard ColBERT score

$$\mathrm{MaxSim}(E^q,E^d)\neq \mathrm{MaxSim}(E^d,E^q)$$0

but gates numerically salient query tokens before they enter MaxSim, thereby improving numerically conditioned retrieval without changing the indexing format or the late-interaction serving stack (Fujimaki et al., 11 May 2026). Argus likewise preserves standard MaxSim but makes the document representation query-conditioned through a region-aware mixture-of-experts module, so that the final score is still computed by late interaction over a multi-vector page representation, now denoted $$\mathrm{MaxSim}(E^q,E^d)\neq \mathrm{MaxSim}(E^d,E^q)$$1 (Abdallah et al., 3 Jun 2026).

A third line of work compresses MaxSim into cheaper retrieval models. “Distilling Dense Representations for Ranking using Tightly-Coupled Teachers” uses ColBERT’s MaxSim as a teacher and distills its ranking behavior into a single-vector dot-product student, enabling single-step ANN search with only modest effectiveness loss (Lin et al., 2020). LEMUR learns a one-hidden-layer latent representation such that

$$\mathrm{MaxSim}(E^q,E^d)\neq \mathrm{MaxSim}(E^d,E^q)$$2

reducing approximate MaxSim retrieval to single-vector maximum inner-product search in a learned latent space (Jääsaari et al., 29 Jan 2026). These approaches do not preserve MaxSim exactly; they treat it as a target function to be approximated or distilled.

Across these variants, MaxSim remains the reference formulation against which smoother, symmetric, query-conditioned, compressed, or distilled alternatives are defined. Its continuing importance lies in that dual status: it is simultaneously a practical scoring rule, a source of late interaction’s strongest empirical results, and the baseline whose computational and statistical properties newer retrieval operators still seek either to preserve or to improve.