QTSplus: Query-Aware Visual Token Selector

• QTSplus is a visual token selection architecture that adaptively routes a minimal, query-relevant subset of tokens, reducing load from ~180,000 to ~20,000 tokens in long videos.
• It employs a cross-attention mechanism and an instance-specific retention budget to compute token relevance, ensuring semantic coverage and precise temporal ordering.
• Integrated within the Qwen2.5-VL pipeline, QTSplus delivers up to 28% speedup and notable improvements in temporal reasoning benchmarks, enhancing long-video MLLM efficiency.

QTSplus (Query-Aware Token Selector with Adaptive Budget) is a visual token selection architecture designed to address the scalability and efficiency bottlenecks in long-video multimodal LLMs (MLLMs). Integrated into the Qwen2.5-VL family, QTSplus dynamically selects a minimal, query-relevant subset of visual tokens between the vision encoder and LLM, dramatically reducing vision token load and computational resources while maintaining or improving accuracy on temporally demanding video reasoning tasks (Li et al., 14 Nov 2025).

1. Motivation and Context

Conventional MLLMs process every vision token extracted from long-form video, incurring computational, memory, and latency costs scaling linearly with video length and quadratically with the number of tokens during transformer attention. For video inputs reaching hundreds of frames and high spatial resolution, the resulting vision token count $M$ can easily surpass $10^5$, overwhelming the LLM’s cache and attention mechanisms.

QTSplus operates as an information gate: given a text query and the full set of video tokens $\mathcal X$, it adaptively routes only the most relevant $n \ll M$ to the LLM, with $n$ determined per instance and informed by the query and token statistics. This design aims to preserve video understanding accuracy while scaling MLLMs to longer and higher-frame-rate sequences.

2. Integration within Qwen2.5-VL Pipeline

In the Qwen2.5-VL stack, video input comprising $T$ frames is encoded by a frozen Vision Transformer (ViT), producing a flattened token sequence: $$\mathcal V = \{\mathbf f_t\}_{t=1}^T \rightarrow \mathcal X = [x_1, ..., x_M]^\top \in \mathbb{R}^{M \times d}$$ where $M = (T/\Delta t)(HW/P^2)$. Normally, $\mathcal X$ would be directly provided to the LLM. QTSplus is inserted between the ViT and the LLM, replacing $\mathcal X$ with a compressed, query-centric set $10^5$0, with $10^5$1, that maintains both semantic coverage and temporal information.

The end-to-end sequence consists of:

• Vision encoder → ViT tokens with absolute time encoding
• QTSplus (cross-attention scoring → adaptive budget prediction → token selection → lightweight re-encoding)
• Compressed tokens to LLM decoder

3. Architecture: Core Components and Algorithms

3.1 Cross-Attention Scoring

Relevance of each vision token $10^5$2 to the input query sequence $10^5$3 is computed by a single-layer, multi-head cross-attention module: $10^5$4 For each head $10^5$5 and text position $10^5$6, the attention is

$10^5$7

Token relevance $10^5$8 is obtained by: $10^5$9 yielding a relevance vector $\mathcal X$0 reflecting the peak attention incurred by any word-head combination.

3.2 Instance-Specific Retention Budget

QTSplus predicts the retention rate $\mathcal X$1 for each instance by aggregating:

• Mean query embedding $\mathcal X$2
• $\mathcal X$3 (token count scale)
• Max relevance $\mathcal X$4
• Relevance entropy $\mathcal X$5, with $\mathcal X$6

The input $\mathcal X$7 feeds a small MLP $\mathcal X$8. The logistic-transformed output predicts: $\mathcal X$9 with $n \ll M$0, and $n \ll M$1.

3.3 Token Selection

3.3.1 Training (Differentiable Top-n Gate)

During training, to maintain end-to-end differentiability, a soft Top-n gate is implemented using a threshold $n \ll M$2 given by: $n \ll M$3 solved by Newton iterations. For each token, logits $n \ll M$4 are fed through a Gumbel-Softmax sampler ($n \ll M$5, hard sampling and straight-through gradients) to yield binary selection indicators $n \ll M$6. Tokens with $n \ll M$7 are retained, and if none are selected, the token with maximal $n \ll M$8 is forced to be kept.

3.3.2 Inference (Hard Top-n Gate)

At inference, tokens are ranked by $n \ll M$9 and the top-$n$0 selected. Temporal ordering is restored post-selection.

3.4 Temporal Order Re-encoding

Absolute time encoding is added to all vision tokens. After selection, a shallow Transformer block (two layers: RMSNorm → MHA → RMSNorm → FFN, FlashAttention enabled, weights initialized from Qwen2.5-VL’s text model) re-encodes the token subset, yielding a stream that preserves both semantic relevance and precise timing, supporting second-level temporal localization.

4. Quantitative and Qualitative Performance

When incorporated into Qwen2.5-VL-3B-Instruct, QTSplus was evaluated on video inputs of up to 600 frames (e.g., 20 seconds at 30 fps):

• Vision-token load reduced from ≈180,000 to ≈20,000 ($n$1 compression)
• End-to-end latency (single NVIDIA A100 GPU) reduced from ≈83 seconds to ≈60 seconds ($n$2 speedup)

Across eight long-video understanding benchmarks (lmms-eval suite), QTSplus-3B matches the teacher model within $n$33 points on most metrics and improves:

• TempCompass direction accuracy by $n$4 points (from $n$5 to $n$6)
• TempCompass order accuracy by $n$7 points (from $n$8 to $n$9)
• Video-MMMU adaptation by $T$0 points (from $T$1 to $T$2)

A plausible implication is that attention-efficient routing of visual evidence enables temporally focused reasoning performance not accessible to uniform token downsampling approaches.

Fine-tuning the full LM (QTSplus-3B-FT) recovers or exceeds teacher performance on broad-coverage tasks, maintaining token and latency savings.

5. Training Regimes and Implementation Details

QTSplus variants include:

• QTSplus-3B (LLM frozen, $T$3129 M QTSplus parameters, trained on VSCQ distillation)
• QTSplus-3B-FT (LLM + QTSplus, $T$43.2 B parameters, VSCQ and VQA distillation)

Critical hyperparameters:

• Cross-attention: 1 layer; Re-encoder: 2 layers
• Gumbel-Softmax temperature $T$5
• Newton iterations for threshold $T$6 solving: $T$7
• Budget head MLP $T$8: Two hidden layers (width 512, GELU)
• Compute-aware loss with $T$9, $$\mathcal V = \{\mathbf f_t\}_{t=1}^T \rightarrow \mathcal X = [x_1, ..., x_M]^\top \in \mathbb{R}^{M \times d}$$0

Optimization uses AdamW, learning rate $$\mathcal V = \{\mathbf f_t\}_{t=1}^T \rightarrow \mathcal X = [x_1, ..., x_M]^\top \in \mathbb{R}^{M \times d}$$1, batch size 64, up to 3 epochs. Experiments are conducted on 8× NVIDIA RTX 5090 GPUs and a 200-core Intel Xeon CPU.

6. Significance, Extensions, and Release Plan

QTSplus provides a general, lightweight, and highly efficient mechanism for scaling MLLMs to long-video scenarios with strict compute and memory requirements. Its principal architectural innovation—dynamic, query-aware budget prediction and token selection—enables global context preservation and fine-grained temporal localization without compromising downstream accuracy.

Continuous release of code, data, and weights is planned to facilitate community reproducibility and further development. The modular nature of QTSplus suggests straightforward extension to other large-scale multimodal domains with similar input redundancy and temporal complexity.

(Li et al., 14 Nov 2025)