Residual Query Adapter (RQA) Overview

• Residual Query Adapter (RQA) is a learnable mapping that aligns heterogeneous embedding spaces while preserving the discriminative structure of the anchor manifold.
• It employs lightweight residual mappings via a shallow MLP or linear adapter to enable high-fidelity vector database upgrades and stable multimodal fusion.
• Empirical results demonstrate consistent improvements in retrieval accuracy with minimal latency overhead, making RQA highly scalable for large databases.

A Residual Query Adapter (RQA) is a class of learnable mapping modules that mediate between heterogeneous embedding spaces—such as those arising from adjacent model versions or distinct backbone modalities—while preserving the anchor manifold's discriminative structure. In recent literature, the RQA paradigm enables near-seamless embedding model upgrades in vector databases and provides stable, robust fusion strategies in multimodal visual representation systems. RQAs combine lightweight residual mapping (via a shallow MLP or linear adaptor) with explicit anchoring and, in some settings, residual query-based global aggregation. This architecture supports both high-fidelity backward compatibility and enhanced feature enrichment without destabilizing retrieval geometry or incurring significant inference latency (Vejendla, 27 Sep 2025, Zhu et al., 19 Jan 2026).

1. Mathematical Formalism and Core Architecture

The foundational RQA instantiates a learnable mapping $g_\theta: \mathbb{R}^d \to \mathbb{R}^d$ designed to align new embeddings from an updated encoder $f_\text{new}$ with the legacy embedding space $f_\text{old}$. The residual MLP variant, introduced in the context of vector database upgrades, is parameterized as:

$$g_\theta(x) = x + W_2 \, \sigma( W_1 x + b_1 ) + b_2$$

where $x \in \mathbb{R}^d$ is the current embedding, $W_1 \in \mathbb{R}^{h \times d}$, $b_1 \in \mathbb{R}^h$ (with $h = 256$), $W_2 \in \mathbb{R}^{d \times h}$, $b_2 \in \mathbb{R}^d$, and $f_\text{new}$0 is the GELU nonlinearity. An optional trainable diagonal scaling $f_\text{new}$1 (parameterized as $f_\text{new}$2 scalars) further post-processes the mapped vector.

In multimodal or multi-backbone tasks such as DC-VLAQ visual place recognition, the RQA paradigm leverages a linear residual adapter $f_\text{new}$3 applied to the difference of two sets of token embeddings (anchor and complementary):

$f_\text{new}$4

where $f_\text{new}$5 is a single linear layer. Both architectures are marked by explicit anchoring (the "additive identity" in $f_\text{new}$6 or $f_\text{new}$7) and the residual mapping as a correction mechanism (Vejendla, 27 Sep 2025, Zhu et al., 19 Jan 2026).

2. Training Regimes and Optimization

For upgrade-bridging in ANN-based vector retrieval, RQAs are supervised using mean-squared error over a dataset of $f_\text{new}$8 paired embeddings:

$f_\text{new}$9

where $f_\text{old}$0 and $f_\text{old}$1. AdamW is used for optimization (initial $f_\text{old}$2, weight decay $f_\text{old}$3), batch size $f_\text{old}$4, up to $f_\text{old}$5 epochs with early stopping on validation MSE. The typical training time is $f_\text{old}$6–$f_\text{old}$7 seconds on A100 hardware. No batch/layer normalization is incorporated. If the diagonal scaling module is enabled, its parameters are learned jointly with the residual MLP.

In RQA-based multimodal fusion, the residual adapter $f_\text{old}$8 is jointly trained with other model parameters against a retrieval (multi-similarity) loss on normalized global descriptors. The primary regularization is L2 anchoring, preserving proximity to the anchor backbone (Vejendla, 27 Sep 2025, Zhu et al., 19 Jan 2026).

3. Aggregation via Residual Queries

In the DC-VLAQ framework, RQAs are extended to the aggregation stage via Vector of Local Aggregated Queries (VLAQ). This module encodes learned query prototypes $f_\text{old}$9 and aggregates feature tokens by their residuals relative to these queries:

$$g_\theta(x) = x + W_2 \, \sigma( W_1 x + b_1 ) + b_2$$0

with the global embedding $$g_\theta(x) = x + W_2 \, \sigma( W_1 x + b_1 ) + b_2$$1 formed by concatenating $$g_\theta(x) = x + W_2 \, \sigma( W_1 x + b_1 ) + b_2$$2 and then L2-normalizing. Two stacked VLAQ blocks are used, each with its own learned queries ($$g_\theta(x) = x + W_2 \, \sigma( W_1 x + b_1 ) + b_2$$3 per block) (Zhu et al., 19 Jan 2026).

This approach confers robustness to distribution shift and preserves retrieval geometry, outperforming absolute-pooling variants (BoQ). Empirical ablations show the RQA+VLAQ combination yields a consistent $$g_\theta(x) = x + W_2 \, \sigma( W_1 x + b_1 ) + b_2$$4\,$$g_\theta(x) = x + W_2 \, \sigma( W_1 x + b_1 ) + b_2$$5 R@1 improvement on challenging VPR datasets.

4. Empirical Performance and Operational Overheads

On large-scale text corpora and CLIP-encoded vision datasets, the residual MLP RQA achieves 98–99% of the recall of a full corpus re-index at under $$g_\theta(x) = x + W_2 \, \sigma( W_1 x + b_1 ) + b_2$$6 per-query CPU overhead. In comparative metrics (Recall@10), the residual MLP exceeds linear adapters (e.g., Procrustes, low-rank affine) by 1–2% ARR, with added query latency of $$g_\theta(x) = x + W_2 \, \sigma( W_1 x + b_1 ) + b_2$$7–$$g_\theta(x) = x + W_2 \, \sigma( W_1 x + b_1 ) + b_2$$8 for $$g_\theta(x) = x + W_2 \, \sigma( W_1 x + b_1 ) + b_2$$9 and $x \in \mathbb{R}^d$0 MB total parameter memory. In scenarios of severe representation drift (e.g., GloVe → MPNet), the nonlinear RQA outperforms linear counterparts by a substantial margin (R@10: $x \in \mathbb{R}^d$1 vs. $x \in \mathbb{R}^d$2 for Procrustes).

In visual place recognition, DC-VLAQ employing RQA mechanisms achieves state-of-the-art results across benchmarks such as Pitts30k (R@1 = $x \in \mathbb{R}^d$3), Tokyo24/7 ($x \in \mathbb{R}^d$4), and MSLS-val ($x \in \mathbb{R}^d$5). Each main design component (anchored residual fusion, query-residual aggregation) individually contributes to these gains. Replacing residual schemes with naive addition or absolute query pooling consistently reduces retrieval accuracy (Vejendla, 27 Sep 2025, Zhu et al., 19 Jan 2026).

5. Scalability, Robustness, and Limitations

Training and deployment of RQAs scale with the embedding dimension $x \in \mathbb{R}^d$6 and the sample size $x \in \mathbb{R}^d$7, but are independent of corpus size ($x \in \mathbb{R}^d$8), rendering them suitable for billion-item vector databases. The constant per-query compute overhead ($x \in \mathbb{R}^d$98 $W_1 \in \mathbb{R}^{h \times d}$0) is negligible relative to typical ANN search times (e.g., $W_1 \in \mathbb{R}^{h \times d}$1 ms at $W_1 \in \mathbb{R}^{h \times d}$2 M items). In continual upgrade scenarios, retraining the residual MLP periodically (e.g., hourly) with fresh pairs maintains high retrieval accuracy (ARR $W_1 \in \mathbb{R}^{h \times d}$3 0.95).

Limitations include the requirement of access to both legacy and new embeddings for a subset of data, inability to fully obviate re-indexing for long-term optimality, and possible underperformance of a global adapter under highly heterogeneous drift. Semantic-boundary queries and rare entities are typical points of failure. Mixture-of-experts approaches can partially mitigate these risks in highly non-uniform settings. In multi-modal fusion, improper anchoring or removal of the residual design results in unstable geometry and degraded retrieval (Vejendla, 27 Sep 2025, Zhu et al., 19 Jan 2026).

6. Broader Applicability and Generalization

The RQA paradigm generalizes beyond immediate applications in embedding upgrades and VPR. It provides a template for fusing distinct representation manifolds in any retrieval-centric setting: select a stable anchor backbone, compute residuals from complementary modalities, learn residual adapters as local corrections, and aggregate via learned queries encoding deviations rather than absolute values. Applications encompass multimodal retrieval (e.g., image–text), point cloud search (geometric + intensity/color), and video retrieval (appearance + motion). This construction ensures retention of the original retrieval geometry while providing a principled integration path for new informational cues (Zhu et al., 19 Jan 2026).

Table: RQA Architectural Variants and Empirical Highlights

Setting	RQA Variant	Key Metrics (R@10)
Vector DB upgrade (Vejendla, 27 Sep 2025)	Residual MLP (256)	AG: 0.992; DB: 0.990
Visual Place Recognition (Zhu et al., 19 Jan 2026)	Linear fusion + VLAQ	Pitts30k: 98.3; MSLS-val: 97.6
Large-drift scenario (AG, MLP)	Residual MLP	0.715 (vs. 0.587 Procrustes)

These results codify the operational and empirical significance of the Residual Query Adapter approach in both high-throughput backend and multi-backbone representation fusion contexts.