Vector-Adapted Retrieval Scoring (VARS)
- VARS is a framework that adapts retrieval scoring to the structure of vector representations by aligning scores with semantic topical coverage, verification utility, or user preferences.
- It employs diverse methodologies—set-based similarity, distance-aware verification, and fixed-dimensional encodings—to closely approximate task-specific retrieval objectives.
- Demonstrated across document retrieval, image search, and conversational agents, VARS emphasizes calibration challenges and effective score adaptation under various constraints.
Searching arXiv for the cited papers and VARS-related context. Vector-Adapted Retrieval Scoring (VARS) denotes a family of retrieval-scoring constructions in which the score used for ranking, filtering, or reranking is adapted to the structure of vector representations rather than treated as a generic nearest-neighbor surrogate. In the literature considered here, the term appears explicitly as a personalization framework for conversational agents, and it also functions as a broader organizing description for methods that score query–document pairs as set-to-set vector similarities, distance-conditioned verification utilities, or fixed-dimensional proxies for late-interaction objectives (Hao et al., 21 Mar 2026, Roy et al., 2016, Szilvasy et al., 2024, Dhulipala et al., 2024). Across these settings, the common aim is to align retrieval-time scoring with the quantity that matters downstream: semantic topical coverage, expected positive yield under a verification budget, fidelity to multi-vector interaction, or user-specific preference relevance.
1. Conceptual scope
In the broad sense suggested by these works, VARS replaces a generic retrieval score with a score explicitly matched to vector semantics or deployment constraints. The adapted target may be the similarity between sets of embedded words rather than between single document vectors, the expected utility of verifying a candidate at a given distance, the late-interaction score of a multi-vector retriever, or a user-conditioned residual added to a frozen reranker (Roy et al., 2016, Szilvasy et al., 2024, Dhulipala et al., 2024, Hao et al., 21 Mar 2026).
This usage is broader than a single algorithm. In the document-retrieval setting, queries and documents are represented as sets of word vectors, and scoring is defined over query vectors and document centroids rather than over a single document embedding (Roy et al., 2016). In small-radius vector search, scoring is tied to the probability that a candidate passes post-verification at a given distance, so ranking is budget-aware and distance-aware rather than based on recall@k alone (Szilvasy et al., 2024). In late-interaction retrieval, fixed-dimensional encodings are constructed so that a single inner product approximates the multi-vector Chamfer or MaxSim objective (Dhulipala et al., 2024). In conversational agents, the score is adapted to user state by adding a dot-product bonus between a learned per-user vector and retrieved preference memories (Hao et al., 21 Mar 2026).
A plausible implication is that VARS is best understood as a design principle rather than a single canonical formula. What remains invariant is the decision to make the retrieval score reflect vector structure that would otherwise be discarded by standard top- or single-vector scoring.
2. Core scoring forms
The papers instantiate VARS through several mathematically distinct scoring rules.
| Setting | Score | Adaptation target |
|---|---|---|
| Set-based IR | Query-to-document centroid similarity | |
| Small-radius search | and | Expected verification utility |
| Multi-vector retrieval | Proxy for late interaction | |
| Personalized reranking | User-aware preference relevance |
In the set-based IR formulation, the score is an average-link similarity between query word embeddings and document cluster centroids. In the small-radius formulation, the score is the sum of distance-conditioned utilities over a shortlist, with the default choice and total VARS equal to RSM when . In MUVERA, the score is adapted by constructing fixed-dimensional encodings whose inner product approximates the late-interaction objective. In the conversational-agent formulation, the score is adapted by a low-rank residual determined by long-term and short-term user vectors (Roy et al., 2016, Szilvasy et al., 2024, Dhulipala et al., 2024, Hao et al., 21 Mar 2026).
These forms differ in surface syntax but share two structural properties. First, the score is not merely a raw embedding similarity; it is calibrated to a task-specific object. Second, the adapted score is intended to be usable at retrieval or reranking time without requiring a full end-to-end evaluation for every candidate set.
3. Set-based document scoring in classical information retrieval
A canonical early instance of this broader VARS pattern is the method in “Representing Documents and Queries as Sets of Word Embedded Vectors for Information Retrieval” (Roy et al., 2016). The paper argues that “a vector addition for composition does not scale well for a larger unit of text, such as passages or full documents, because of the broad context present within a whole document,” and it therefore represents both document and query as sets of word vectors rather than compressing a document into one embedding.
Formally, if the bag-of-words representation of a document is , then the bag-of-vectors representation is with 0. The query is likewise represented as observed query points 1. Each document is viewed as a mixture distribution over topics, operationalized by K-means clustering. After pre-clustering the collection vocabulary, words in a document are grouped by global cluster id and averaged into document-specific centroids
2
The resulting document representation typically contains far fewer than 3 clusters, with the paper reporting typically 4–5 clusters per document.
The core score is the average-link similarity
6
This vector score is combined with a language-model score using
7
with 8 selected by grid search and 9 instantiated as a Jelinek–Mercer LLM with 0. The embeddings are 200-dimensional CBOW word2vec vectors with negative sampling, trained on a pre-processed TREC collection.
The reported experimental setting uses TREC News collections and evaluates MAP, GMAP, Recall, and 1. The paper’s abstract states that the method “improves MAP by up to 2, in comparison to standard text-based LLM similarity, on the TREC 6, 7, 8 and Robust ad-hoc test collections.” On TREC-8 initial retrieval, LM MAP is reported as 3, while LM+wvsim_kmeans reaches 4 and LM+wvsim_one_cluster reaches 5; on Robust, LM MAP is 6 and LM+wvsim_kmeans reaches 7 (Roy et al., 2016).
The method also illustrates a recurrent VARS trade-off. Its vector component preserves topical granularity better than a single-vector document embedding, but the paper notes a scale mismatch between LM probabilities and vector similarities, handled only implicitly through 8. This suggests that calibration is a central issue whenever heterogeneous retrieval signals are combined within a single score.
4. Distance-aware and budget-aware scoring for small-radius vector search
A second, more explicitly utility-theoretic VARS formulation appears in “Vector search with small radiuses” (Szilvasy et al., 2024). The paper begins from the observation that the dominant vector-search metric, recall of a fixed-size top-9 list, is “distantly related to the end-to-end accuracy of a full system that integrates vector search.” Its motivating use case is a two-stage pipeline in which vector search produces candidates and a costly post-verification stage determines whether a query image matches a database image.
The paper therefore frames retrieval as range search. For a query 0, a database 1, and a radius 2, the range set is
3
The key modeling move is to define utility by the probability that a pair passes post-verification given its distance:
4
with 5 monotonically decreasing. For a shortlist 6, the expected number of positives is
7
and over multiple queries the paper defines
8
with 9. The score is therefore distance-aware, and the optimal budgeted strategy is to select the globally 0 smallest distances across all queries, equivalently setting a global radius that yields 1 candidates in total.
The function 2 is estimated nonparametrically by isotonic regression on labeled pairs 3 with 4 and 5, solving
6
after sorting distances increasingly. The paper states that the pool-adjacent-violators algorithm solves this in linear time. Operationally, the VARS-style score is then taken as 7, and per-query VARS is
8
with total VARS equal to RSM when 9 (Szilvasy et al., 2024).
The empirical setting uses 85M filtered images derived from YFCC100M, with 0k queries, 1M database images, and 75M training images. Embeddings are 512-dimensional SSCD descriptors with squared 2 distance. Verification uses KAZE keypoints and a RANSAC similarity transform via OpenCV. Budgets are 3 in the strict setting and 4 in the relaxed setting, with throughput around 85 geometric verifications per second.
The paper reports that recall@k and RSM can favor different index designs. For IVF with 5 centroids, RSM in the strict setting saturates after visiting about 6 clusters, while in the relaxed setting there is no RSM improvement beyond 7, which is less than 8 of clusters. Fast-scan PQ centroid search achieves the best RSM, HNSW saturates in high-RSM regimes, and compact codes are comparatively effective because “RSM is dominated by very close neighbors that small codes already capture.” For example, with strict budget 9, residual PQ1608 reaches 1 (in units of 2), close to exact at 3; in the relaxed setting, PQ3248 reaches 5 versus exact 6 (Szilvasy et al., 2024).
This formulation directly challenges a common misconception that maximizing top-7 recall is equivalent to maximizing end-to-end retrieval quality. Here, the relevant objective is expected positives per verification budget, and the adapted score is built to estimate exactly that quantity.
5. Fixed-dimensional encodings for late-interaction retrieval
A third VARS instantiation arises in multi-vector retrieval, particularly in “MUVERA: MUlti-VEctor Retrieval Algorithm” (Dhulipala et al., 2024). The paper does not define VARS explicitly; however, it states that, interpreted as “choosing a retrieval-time scoring function adapted to the vector representation,” MUVERA is exactly a VARS method. Its target is the ColBERT-style late-interaction objective
8
with normalized form
9
MUVERA constructs Fixed Dimensional Encodings (FDEs) 0 and 1 such that
2
The construction has three steps. First, token vectors are partitioned by angular LSH using SimHash with 3 Gaussian hyperplanes, yielding 4 clusters. Second, within each cluster, query blocks are sums of query vectors, while document blocks are centroids of document vectors; if a document cluster is empty, the method fills it with the document token whose SimHash code has minimum Hamming distance to that cluster id. Third, within-block random projections reduce dimension from 5 to 6, and the whole process is repeated 7 times and concatenated, giving FDE dimension
8
Theoretical guarantees are stated for unit-normalized token embeddings. With no inner projection, the paper reports a one-sided estimator:
9
deterministically. Its main approximation theorem states that with 0, 1, and 2, the FDE inner product yields an 3-additive approximation to normalized Chamfer with probability at least 4 and also in expectation. A second theorem states that with 5, top-1 retrieval under FDE dot product returns an 6-approximate nearest neighbor under normalized Chamfer with high probability (Dhulipala et al., 2024).
Systemically, the consequence is that multi-vector retrieval can be reduced to a single-vector MIPS stage followed by exact reranking under the true Chamfer score. MUVERA indexes document FDEs in a standard MIPS or ANN system; the paper uses DiskANN, applies “ball carving” to reduce the number of query token groups for reranking, and compresses FDEs with PQ-256-8. The reported results state that FDEs achieve the same recall as prior state-of-the-art heuristics while retrieving 7–8 fewer candidates, and that MUVERA achieves an average of about 9 improved recall with about 0 lower latency than PLAID across six BEIR datasets. On MS MARCO, a 10240-dimensional FDE with PQ-256-8 matches PLAID within 1 Recall@2, while offline experiments report that FDE-10240 needs 3 candidates to reach 4 recall, compared with 5 for the deduplicated SV heuristic and 6 for the non-deduplicated SV heuristic (Dhulipala et al., 2024).
The conceptual significance for VARS is precise: the retrieval-time inner product is engineered to be a mathematically coupled proxy for the score that the late-interaction model actually cares about, rather than a heuristic prefilter only loosely related to that objective.
6. User-aware VARS in conversational retrieval-augmented agents
The term VARS is used explicitly in “User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction” (Hao et al., 21 Mar 2026). There, VARS is a “pipeline-agnostic, frozen-backbone framework” that learns a compact per-user representation as long-term and short-term vectors in a shared preference space and uses these vectors to bias retrieval scoring over structured preference memory.
The system stores preference “cards” containing a structured preference tuple, a short note, an is_global flag, source metadata, and embeddings from a frozen model, 7, specifically Qwen3-Embedding-8B with raw dimension 8. A PCA projection produces a shared item space of dimension 9, and each memory card obtains an item vector
00
For each user 01, the long-term vector 02 persists across sessions, while the short-term vector 03 is reset each session and decayed between turns. The effective user vector is
04
with 05 and 06.
Conditional preference cards are first retrieved by dense cosine retrieval. A frozen cross-encoder then assigns a base score
07
and VARS adds a user-conditioned bonus:
08
A softmax policy over candidates is defined with temperature 09, the top-10 cards are injected into the prompt with 11, and global cards bypass retrieval and are directly injected up to a cap of 12.
Online learning proceeds from weak scalar rewards 13 obtained from the next user turn. The reward combines keyword sentiment and topic coherence, is clipped to 14, and is further scaled by a heuristic retrieval-attribution gate 15. A per-user EMA baseline is updated as
16
with 17. Letting 18, the long-term and short-term vectors are updated by REINFORCE-style rules
19
with 20, 21, and short-term decay 22 (Hao et al., 21 Mar 2026).
Evaluation is conducted on MultiSessionCollab with 60 user profiles, 60 sessions per profile, up to 10 turns per session, and three task domains: math-hard, math-500, and bigcodebench. Across 3600 sessions per method, VARS reports Success 23, Timeout 24, and User tokens 25, compared with Reflection at 26, 27, and 28. Relative to Reflection, success improves by 29 percentage points with 30 and is not significant, while timeout decreases by 31 percentage points with 32 and user tokens decrease by 33 with 34. The paper therefore concludes that, under frozen backbones, the principal benefit of user-aware retrieval is improved interaction efficiency rather than large gains in raw task accuracy (Hao et al., 21 Mar 2026).
This explicit VARS formulation makes personalization a scoring problem rather than a backbone fine-tuning problem. The only learned state is two 256-dimensional per-user vectors plus a baseline scalar, and the added inference cost is a single 256-dimensional dot product per candidate.
7. Assumptions, limitations, and recurrent issues
Several assumptions recur across these VARS formulations. In the small-radius setting, RSM assumes stationarity of the learned 35, monotonic decrease of utility with distance, and comparability of distances across queries; it is also sensitive to sharp radius effects in high dimensions and to encoding distortion from PQ or ITQ (Szilvasy et al., 2024). In MUVERA, the theoretical guarantees assume unit-normalized token embeddings and depend on SimHash collisions, fill-empty behavior, and bounded random-projection error (Dhulipala et al., 2024). In the conversational-agent framework, reward quality and reward attribution are central, since removing gating causes “454% vector-norm inflation and directional drift in sensitivity tests” (Hao et al., 21 Mar 2026). In the set-based IR method, the mixed score combines a probability-like LM component with an inner-product similarity, and the paper identifies calibration as an unresolved issue (Roy et al., 2016).
These works also clarify several misconceptions. VARS is not identical to top-36 nearest-neighbor retrieval: the small-radius formulation shows directly that recall@37 may improve while RSM does not (Szilvasy et al., 2024). VARS is not synonymous with single-vector document embeddings: the set-based IR method was motivated precisely by the claim that single-vector composition does not scale well to larger text units (Roy et al., 2016). VARS is not restricted to learned end-to-end retrievers: MUVERA uses data-oblivious random partitions and off-the-shelf MIPS infrastructure, while the conversational framework keeps chat, embedding, and reranker backbones frozen (Dhulipala et al., 2024, Hao et al., 21 Mar 2026).
Open directions listed in the papers are correspondingly diverse. The small-radius work identifies joint training with losses that normalize distances, adaptive per-query budgets, and extensions incorporating additional signals or multi-stage utility (Szilvasy et al., 2024). The conversational work proposes richer feedback signals, improved gating, multi-objective preference modeling, and partial backbone adaptation or adapter layers (Hao et al., 21 Mar 2026). The set-based IR paper suggests alternative distance measures, multi-sense embeddings, and further work on feedback and expansion (Roy et al., 2016). MUVERA’s results suggest further exploration of dimensionality–approximation trade-offs and compression–latency trade-offs within a single-vector proxy regime (Dhulipala et al., 2024).
Taken together, these directions indicate that the central unresolved problem in VARS is not whether vector adaptation is useful, but how to calibrate adapted scores so that they remain faithful under approximation, budget constraints, personalization noise, and domain shift.