Visual Query Tuning: Efficient Feature Adaptation

• Visual Query Tuning (VQT) is a method that inserts learnable query tokens into Vision Transformers to summarize and adapt intermediate features without full model fine-tuning.
• It introduces lightweight modules, including Q-Adapter and query-informed biases, to achieve robust performance with only a fraction of the parameters updated.
• VQT supports efficient transfer learning in data-scarce and multimodal environments by reducing computational overhead and memory usage while maintaining high accuracy.

Visual Query Tuning (VQT) refers to a class of methodologies designed to enable efficient extraction, modulation, or restoration of visual features in pretrained vision models—especially Vision Transformers (ViTs)—by leveraging learnable "query" mechanisms without requiring large-scale or full-dataset model adaptation. VQT arose as a parameter- and memory-efficient alternative to full fine-tuning, offering robust transfer learning, adaptable semantic grounding, and enhanced resilience to query imperfections in multimodal and vision-language tasks. The approach centers on augmenting standard ViT architectures with lightweight, task-aware modules, particularly through learnable query tokens or vectors, to summarize, adapt, or repair visual representations in alignment with downstream objectives (Tu et al., 2022, Chen et al., 11 Oct 2025, Le et al., 3 Apr 2025, Zhang et al., 13 Feb 2026).

1. Foundational Principles and Definitions

VQT fundamentally exploits the attention architecture of ViTs, where information aggregation is mediated via query-key-value interactions in self-attention or cross-attention modules. A distinguishing feature of VQT is the insertion of learnable query tokens or vectors into intermediate or final layers of pretrained vision backbones to "summarize" or modulate hidden features in a manner explicitly tailored to downstream tasks.

• In canonical VQT (Tu et al., 2022), query tokens are appended at each or selected layers' attention block, producing learned summaries of intermediate representations through attention pooling.
• The "query" in VQT can be either modality-agnostic (fixed or learned per-task tokens) or modality-aware (informed by external textual queries), as in Query-Informed ViTs (Le et al., 3 Apr 2025).
• Visual Query Pre-processing (Zhang et al., 13 Feb 2026) extends the concept to agentic repair, where visual queries are proactively transformed via perceptual tools to maximize retrieval or reasoning fidelity in multimodal systems.

By decoupling query-based feature extraction from backbone parameter updates, VQT achieves high accuracy and efficiency, permitting frozen-large-model reuse, and enabling robust adaptation in resource-limited or distributionally-shifted settings.

2. Architectural Design and Mathematical Formulation

The core architectural modifications characterizing VQT are:

Visual Query Token Insertion

For each pre-trained ViT layer $\ell$ with input features $F_\ell \in \mathbb{R}^{D \times N}$:

• $T$ learnable query tokens $P_{\ell-1} \in \mathbb{R}^{D \times T}$ are projected via $W_q$, appended to form queries:

$$Q'_\ell = W_q P_{\ell-1} \in \mathbb{R}^{D \times T}$$

• Standard keys and values $K_\ell=W_k F_{\ell-1}$, $V_\ell=W_v F_{\ell-1}$ remain frozen.

The query-augmented attention yields two outputs: $$V_\ell \text{Softmax} \left( \frac{K_\ell^\top Q_\ell}{\sqrt{D}} \right) \quad\text{(standard tokens)},\qquad Z'_\ell = V_\ell \text{Softmax} \left( \frac{K_\ell^\top Q'_\ell}{\sqrt{D}} \right)$$

$Z'_\ell$ is processed through the layer's (frozen) MLP block: $F_\ell \in \mathbb{R}^{D \times N}$0

This provides a succinct and learnable summary of layer $F_\ell \in \mathbb{R}^{D \times N}$1's features, decoupled from the full token set.

Query-Informed Feature Modulation

Within QID (Le et al., 3 Apr 2025), query vectors derived from external modalities (e.g., text encoder $F_\ell \in \mathbb{R}^{D \times N}$2) are linearly mapped and injected as additive biases into key and value projections at higher ViT layers: $F_\ell \in \mathbb{R}^{D \times N}$3 This enables precise, query-driven attention reweighting for cross-modal semantic grounding.

Adapter-based Extraction and Tuning

With Q-Adapter (Chen et al., 11 Oct 2025), adapters are inserted after MSA and in parallel to the MLP in each block, mediating interactions via learnable queries and gated cross-attention:

• Gating: $F_\ell \in \mathbb{R}^{D \times N}$4
• Query-guided cross-attention and feature refinement
• Only adapter parameters are updated, ensuring strict parameter efficiency.

3. Training Protocols and Efficiency

VQT achieves notable resource savings by restricting learning to lightweight modules:

• In (Tu et al., 2022), only $F_\ell \in \mathbb{R}^{D \times N}$5 query token parameters and the head are updated; original ViT layers remain entirely frozen.
• QID (Le et al., 3 Apr 2025) trainable parameters are $F_\ell \in \mathbb{R}^{D \times N}$6 of ViT; Q-Adapter (Chen et al., 11 Oct 2025) uses only $F_\ell \in \mathbb{R}^{D \times N}$7 of all parameters.
• Memory footprint is drastically reduced (e.g., VQT achieves $F_\ell \in \mathbb{R}^{D \times N}$876% lower peak GPU memory than VPT-Deep (Tu et al., 2022)).

Optimizers are typically AdamW, with cross-entropy loss for classification or cross-modal generation, sometimes augmented with entropy regularization or group-lasso penalties for sparsity or robustness.

Fine-tuning strategies depend on data regime:

• For abundant data, adapters and query vectors are trained across all tasks.
• For scarce-data settings (Le et al., 3 Apr 2025), only the last $F_\ell \in \mathbb{R}^{D \times N}$9–$T$0 layers are adapted to balance representation richness with overfitting risk.

4. Empirical Evaluations and Performance Benchmarks

VQT exhibits strong empirical results across diverse settings:

Method	FT Ratio	VTAB NAT	VTAB SPEC	VTAB STR	Overall Acc	MSR-VTT (B@4)	MSVD (B@4)
Linear Probing	–	63.2	77.1	39.7	52.7	–	–
Full Fine-tuning	100%	65.2	81.7	46.3	63.2	57.8	75.0
VQT (Tu et al., 2022)	~2–3%	72.7	84.5	49.3	65.3	–	–
Q-Adapter (Chen et al., 11 Oct 2025)	1.4%	–	–	–	–	49.84	64.84
AdaptFormer+VQT	~2–3%	71.1	83.3	59.2	–	–	–

Performance is consistently superior to competing parameter-efficient transfer methods and often approaches, matches, or surpasses full fine-tuning, but at orders-of-magnitude lower computational cost. Q-Adapter achieves BLEU@4 = 49.84 on MSR-VTT (vs. 57.8 for full fine-tuning) while tuning only 1.4% of parameters.

In data-scarce VDU, QID lifts accuracy (e.g., FUNSD F1 from 76.2 to 78.5 (Le et al., 3 Apr 2025)). In MRAG settings, agentic VQT remedies embedding drift from query imperfections, recovering retrieval and QA performance, especially after supervised fine-tuning (Zhang et al., 13 Feb 2026).

5. Robustness, Agentic Query Pre-processing, and MRAG Extensions

VQT is extended into agentic "Visual Query Pre-processing" (V-QPP) for MRAG (Zhang et al., 13 Feb 2026), where the system proactively diagnoses input imperfections (geometric, quality, semantic), then applies a sequence of perceptual tools (rotation, deblur, crop, etc.) to maximize the retriever's recall: $T$1 The agent is trained (e.g., LoRA on Qwen3-VL-4B) to select tools and parameters, optimizing tool selection accuracy (TSA), parameter score (PS: regression for continuous, IoU for bounding boxes), and ultimate retrieval efficacy (Recall@K).

Benchmarks (V-QPP-Bench, 46,700 queries) reveal:

• Retrieval Recall@5 drops up to 80–98% for severe corruption without preprocessing.
• Oracle tool sequences recover near-original performance; off-the-shelf MLLMs perform poorly (TSA 20–50%) unless fine-tuned (TSA to 90%+, Recall@5 up to 0.42 on RealWorld queries).
• Compact fine-tuned agents rival larger, proprietary models.

This agentic VQT reframes pre-retrieval robustness as a concrete, learnable decision-stage in multimodal pipelines.

6. Complementarity, Best Practices, and Limitations

VQT operates orthogonally to feature-adapting PETL strategies (e.g., AdaptFormer, VPT). When composited (AdaptFormer+VQT), additive gains are observed, with VQT enhancing the ability to aggregate relevant intermediate features (Tu et al., 2022). Best practices extracted from the literature:

• For classification/regression, concatenate VQT summaries from all layers; for generation/retrieval, modulate representation in deeper layers.
• Number of learnable queries should be minimal yet sufficient (e.g., $T$2, (Chen et al., 11 Oct 2025)).
• Place adapters in late ViT layers to target high-level semantic refinements.
• The gating mechanism stabilizes learning by selectively opening query adapter flow where backbone features misalign with the current task (Chen et al., 11 Oct 2025).
• For out-of-distribution or resource-limited settings, limit to a few top layers, consider larger entropy regularization (QID), or curriculum-based progressive deepening (Chen et al., 11 Oct 2025, Le et al., 3 Apr 2025).

Identified limitations include reduced marginal benefit for tasks far from pretraining domain (e.g., structured VTAB tasks), and potential for further gains via learned cross-layer fusion strategies.

7. Applicability and Future Directions

VQT is applicable across classification, retrieval, generation, and video captioning. Its modular, lightweight, and architecture-preserving nature supports easy transplantation to new tasks and edge-device constraints. The technique’s demonstrated compatibility with parameter-efficient adaptation, the agentic MRAG paradigm, and data-scarce domains positions it for broad adoption in vision-language modeling.

Ongoing research directions include:

• Agentic query repair in open-world MRAG.
• Hybrid fusion of cross-layer query summaries.
• Application to non-ViT or recurrent visual backbones.
• Exploration of specialized gating and entropy-based regularizers for even more robust feature selection and transfer.

VQT thus constitutes a principal building block for modern, efficient, and robust visual understanding and multimodal reasoning systems, with a strong empirical and methodological foundation across recent literature (Tu et al., 2022, Chen et al., 11 Oct 2025, Le et al., 3 Apr 2025, Zhang et al., 13 Feb 2026).