Grounded Visual Token Sampling

• Grounded Visual Token Sampling (GroundVTS) is a multimodal strategy that uses visual evidence cues to dynamically select and weight tokens for efficient reasoning.
• It integrates query-guided filtering, reinforcement learning, and region-based compression to enhance video temporal grounding and reduce computational redundancy.
• While GroundVTS methods improve accuracy and performance across benchmarks, they also incur extra processing costs and rely on extensive supervision data.

Grounded Visual Token Sampling (GroundVTS) denotes a family of multimodal modeling strategies in which token retention, weighting, routing, insertion, or credit assignment is driven by signals tied to visual evidence rather than by uniform processing or language-only heuristics. In its narrowest usage, the term refers to the Vid-LLM architecture "GroundVTS: Visual Token Sampling in Multimodal LLMs for Video Temporal Grounding," which performs query-guided token filtering before large-language-model reasoning for video temporal grounding (Fan et al., 2 Apr 2026). In a broader technical sense, closely related work has treated grounded token selection as a problem of query-conditioned visual pruning, visually anchored reinforcement-learning credit assignment, geometry- or trajectory-preserving compression, or explicit region/proxy-token generation during reasoning (Jin et al., 2 Jun 2026, Ye et al., 2 Apr 2026, You et al., 16 Jun 2026, Zheng et al., 29 May 2025, Hodemon et al., 22 Jun 2026).

1. Conceptual scope and design space

GroundVTS addresses a recurring inefficiency in multimodal systems: not all tokens carry the same amount of task-relevant visual evidence, yet many architectures still allocate computation uniformly. In video temporal grounding, uniform frame sampling can miss brief query-relevant events and dilute salient temporal evidence (Fan et al., 2 Apr 2026). In multimodal reinforcement learning, uniform token-level advantage assignment can underweight visually critical reasoning steps (Ye et al., 2 Apr 2026). In grounded generation and segmentation, global-attention-driven pruning can erase small objects and boundary-sensitive details required for faithful mask prediction (Bai et al., 31 Mar 2025).

A useful synthesis is that grounded token sampling can operate on at least four different objects: visual encoder patches, generated response tokens, region-level summaries, or explicit symbolic pointers to latent visual positions. The grounding signal can likewise be query relevance, counterfactual visual dependence, saliency and geometry proxies, temporal persistence, or explicit grounding annotations. This suggests that GroundVTS is better understood as a role-aware allocation principle than as a single algorithmic template.

Formulation	Selected or weighted unit	Grounding signal
GroundVTS	visual tokens before the LLM	query-guided relevance
VEPO, PGPO, VIG-TUQ	generated response tokens	visual sensitivity or visual grounding scores
RegimeVGGT, VTS, ALTP, TrajViT	patches, K/V support, local regions, or trajectories	geometry, saliency, temporal novelty, local density
Composer, GR3D, VGR, visually grounded thinking	proxy tokens, region tokens, replayed regions, grounding tags	explicit object or region grounding

Two conceptual distinctions recur across the literature. First, some methods are compute-facing: they reduce or reallocate visual-token computation, as in GroundVTS for VTG, RegimeVGGT for multi-view geometry, VTS for driving video, ALTP for grounded conversation generation, and TrajViT for long-video encoding (Fan et al., 2 Apr 2026, You et al., 16 Jun 2026, Ma et al., 2024, Bai et al., 31 Mar 2025, Zheng et al., 29 May 2025). Second, other methods are reasoning-facing: they identify which generated tokens or reasoning steps are truly image-dependent, then upweight, serialize, or supervise those steps, as in VEPO, PGPO, VIG-TUQ, Composer, GR3D, VGR, and "Thinking with Visual Grounding" (Jin et al., 2 Jun 2026, Ye et al., 2 Apr 2026, Hoche et al., 26 May 2026, Hodemon et al., 22 Jun 2026, Cheng et al., 28 May 2026, Wang et al., 13 Jun 2025, Zhang et al., 15 Jun 2026).

2. Query-guided token sampling for video temporal grounding

The named GroundVTS architecture targets video temporal grounding and highlight detection in Vid-LLMs. Its premise is that temporal localization fails not only because of weak temporal modeling, but because visual evidence is sampled uniformly before the LLM ever reasons over it. The method therefore performs fine-grained, query-guided filtering over visual tokens after visual encoding and multimodal projection, retaining only a top-$K$ subset before multimodal fusion (Fan et al., 2 Apr 2026).

Let the query embeddings be $Q \in \mathbb{R}^{N_t \times D}$ and the projected visual tokens be $V \in \mathbb{R}^{N_v \times D}$. GroundVTS projects both into a relevance space,

$$V' = W_vV,\qquad \mathbf{q'} = W_q \mathrm{Pool}(Q),$$

then scores each visual token by

$$\mathbf{w} = \operatorname{softmax}\left(V' \mathbf{q'}^{\top} / \tau \right),$$

and keeps

$K=\lceil \rho \cdot N_v \rceil$

tokens with differentiable top-$K$ selection based on a Gumbel-Softmax relaxation and Straight-Through Estimator. The retained token embeddings are reweighted by normalized relevance scores, and the original positional encodings are preserved for retained tokens to maintain temporal coherence under non-uniform sampling (Fan et al., 2 Apr 2026).

The model is optimized by a three-stage curriculum. Stage 1 warms up the VTS module alone on LLaVA-Video-178K. Stage 2 jointly adapts VTS, the multimodal projector, and the LLM via LoRA on the same dataset. Stage 3 fine-tunes the same trainable components on Grounding-FT, a 70K instruction-style temporal-grounding dataset constructed from Charades-STA, QVHighlights, and ActivityNet-Captions. The reported default sampling ratio is $\rho = 0.5$, with $D_r=512$ for the Qwen2.5VL-7B variant and $D_r=128$ for the InternVL3.5-8B variant (Fan et al., 2 Apr 2026).

The empirical argument is strong. A frame-rate sensitivity study on Charades-STA with Qwen2.5VL-7B shows performance peaking around $Q \in \mathbb{R}^{N_t \times D}$0–$Q \in \mathbb{R}^{N_t \times D}$1 FPS with $Q \in \mathbb{R}^{N_t \times D}$2 mIoU, then dropping as more frames are sampled, indicating that too many uniformly sampled tokens dilute salient temporal evidence. On Charades-STA, GroundVTS-Q reaches $Q \in \mathbb{R}^{N_t \times D}$3 mIoU versus $Q \in \mathbb{R}^{N_t \times D}$4 for a supervised fine-tuned Qwen2.5VL-7B baseline; on ActivityNet-Captions it reaches $Q \in \mathbb{R}^{N_t \times D}$5 mIoU versus $Q \in \mathbb{R}^{N_t \times D}$6; on QVHighlights highlight detection, GroundVTS-I reaches $Q \in \mathbb{R}^{N_t \times D}$7 mAP and $Q \in \mathbb{R}^{N_t \times D}$8 Hit@1 versus $Q \in \mathbb{R}^{N_t \times D}$9 mAP and $V \in \mathbb{R}^{N_v \times D}$0 Hit@1 for an InternVL3.5-8B grounding baseline. The abstract summarizes these gains as a $V \in \mathbb{R}^{N_v \times D}$1-point improvement in mIoU for moment retrieval and a $V \in \mathbb{R}^{N_v \times D}$2-point improvement in mAP for highlight detection (Fan et al., 2 Apr 2026).

Ablations clarify the method’s position in the GroundVTS landscape. Token-level query-guided selection outperforms frame-level query selection, uniform sampling, and random sampling. On Charades-STA, the token-level variant obtains $V \in \mathbb{R}^{N_v \times D}$3 mIoU, versus $V \in \mathbb{R}^{N_v \times D}$4 for frame-level selection, $V \in \mathbb{R}^{N_v \times D}$5 for uniform sampling, and $V \in \mathbb{R}^{N_v \times D}$6 for random sampling. Removing preserved positional encodings causes collapse, dropping performance to $V \in \mathbb{R}^{N_v \times D}$7 mIoU. The progressive optimization schedule is also decisive: the full three-stage curriculum reaches $V \in \mathbb{R}^{N_v \times D}$8 mIoU, compared with $V \in \mathbb{R}^{N_v \times D}$9 after Stage 1 only and $$V' = W_vV,\qquad \mathbf{q'} = W_q \mathrm{Pool}(Q),$$0 after Stages 1+2 (Fan et al., 2 Apr 2026).

3. Vision-anchored token weighting in reinforcement learning and uncertainty estimation

A second line of work relocates GroundVTS from input-token pruning to response-token selection. These methods do not primarily prune encoder-side visual patches; instead, they decide which generated tokens are visually grounded enough to deserve RL credit or uncertainty mass.

VEPO, introduced in "Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selection," starts from the observation that entropy-only token selection, which works in text-only RLVR, breaks in visual reasoning because many crucial vision-dependent tokens lie in low-entropy regions. The method measures visual sensitivity by comparing next-token distributions under an original image $$V' = W_vV,\qquad \mathbf{q'} = W_q \mathrm{Pool}(Q),$$1 and a perturbed image $$V' = W_vV,\qquad \mathbf{q'} = W_q \mathrm{Pool}(Q),$$2: $$V' = W_vV,\qquad \mathbf{q'} = W_q \mathrm{Pool}(Q),$$3 It then combines Jensen–Shannon divergence and absolute entropy gap into a fused visual-dependency score,

$$V' = W_vV,\qquad \mathbf{q'} = W_q \mathrm{Pool}(Q),$$4

and multiplies by normalized entropy to obtain the final vision-entropy selection score $$V' = W_vV,\qquad \mathbf{q'} = W_q \mathrm{Pool}(Q),$$5. Hard top-$$V' = W_vV,\qquad \mathbf{q'} = W_q \mathrm{Pool}(Q),$$6 token selection is then inserted into GRPO. VEPO reaches $$V' = W_vV,\qquad \mathbf{q'} = W_q \mathrm{Pool}(Q),$$7 average at 7B, versus $$V' = W_vV,\qquad \mathbf{q'} = W_q \mathrm{Pool}(Q),$$8 for the stronger entropy-only baseline and $$V' = W_vV,\qquad \mathbf{q'} = W_q \mathrm{Pool}(Q),$$9 for full-token GRPO, a $$\mathbf{w} = \operatorname{softmax}\left(V' \mathbf{q'}^{\top} / \tau \right),$$0 gain over entropy-only selection; at 3B it reaches $$\mathbf{w} = \operatorname{softmax}\left(V' \mathbf{q'}^{\top} / \tau \right),$$1 versus $$\mathbf{w} = \operatorname{softmax}\left(V' \mathbf{q'}^{\top} / \tau \right),$$2, a $$\mathbf{w} = \operatorname{softmax}\left(V' \mathbf{q'}^{\top} / \tau \right),$$3 gain. At retention ratio $$\mathbf{w} = \operatorname{softmax}\left(V' \mathbf{q'}^{\top} / \tau \right),$$4, top-entropy selection recovers only about $$\mathbf{w} = \operatorname{softmax}\left(V' \mathbf{q'}^{\top} / \tau \right),$$5–$$\mathbf{w} = \operatorname{softmax}\left(V' \mathbf{q'}^{\top} / \tau \right),$$6 of top visual-sensitive tokens, whereas VEPO reduces the miss rate for vision-sensitive tokens from about $$\mathbf{w} = \operatorname{softmax}\left(V' \mathbf{q'}^{\top} / \tau \right),$$7 to about $$\mathbf{w} = \operatorname{softmax}\left(V' \mathbf{q'}^{\top} / \tau \right),$$8 (Jin et al., 2 Jun 2026).

PGPO, or Perception-Grounded Policy Optimization, formulates an explicitly causal grounding score for generated tokens. Token Visual Dependency is defined as the KL divergence between the image-conditioned and text-only predictive distributions: $$\mathbf{w} = \operatorname{softmax}\left(V' \mathbf{q'}^{\top} / \tau \right),$$9 After log compression and sequence-wise min-max normalization, PGPO applies a threshold-gated, mass-conserving reweighting of the sequence-level advantage. Tokens below a threshold $K=\lceil \rho \cdot N_v \rceil$0 are suppressed; tokens above $K=\lceil \rho \cdot N_v \rceil$1 are amplified by a coefficient $K=\lceil \rho \cdot N_v \rceil$2; the normalized weights are constrained to preserve total advantage mass. The paper argues theoretically that this reduces nuisance gradient variance from visually irrelevant tokens, and empirically reports an average improvement of $K=\lceil \rho \cdot N_v \rceil$3 across seven multimodal reasoning benchmarks. The best setting is reported at $K=\lceil \rho \cdot N_v \rceil$4 and $K=\lceil \rho \cdot N_v \rceil$5, and training overhead from the auxiliary forward pass is about $K=\lceil \rho \cdot N_v \rceil$6 relative to DAPO (Ye et al., 2 Apr 2026).

VIG-TUQ, a training-free token-level uncertainty framework for LVLMs, provides a closely related perspective. It weights token entropy by two grounding signals: a distribution-based signal

$K=\lceil \rho \cdot N_v \rceil$7

and an attention-based score computed from attention mass over visual tokens. On OKVQA, for example, token entropy reaches AUROC $K=\lceil \rho \cdot N_v \rceil$8, while the fused VIG-TUQ score reaches $K=\lceil \rho \cdot N_v \rceil$9. The paper’s token-selection analysis further reports that selecting only the most visually grounded tokens often outperforms using all tokens for uncertainty estimation, which strongly supports the GroundVTS view that visually grounded response positions constitute a small but disproportionately informative subset (Hoche et al., 26 May 2026).

Taken together, these works suggest a broader GroundVTS principle: in multimodal RL and uncertainty estimation, the critical sampling problem may concern which reasoning tokens should be updated or trusted, not only which visual patches should be retained. A recurring limitation, however, is cost. VEPO requires an auxiliary forward pass on a perturbed image and is about $K$0 slower than top-entropy selection, though still about $K$1 faster than full-token GRPO (Jin et al., 2 Jun 2026). PGPO similarly relies on a second pass with visual information masked out (Ye et al., 2 Apr 2026).

4. Grounded visual compression in geometry, video, and grounded generation

A third body of work applies GroundVTS directly to the visual stream, but with grounding signals other than textual query relevance.

RegimeVGGT accelerates Visual Geometry Grounded Transformer by treating token reduction as a geometry-, saliency-, and cross-view-path-preserving sampling problem. Its analyses partition the 24-layer aggregator into three regimes: shallow layers $K$2–$K$3, middle layers $K$4–$K$5, and deep layers $K$6–$K$7. Compression is U-shaped across depth. On the token-count axis, Saliency-Guided Banded Merging uses merge ratios $K$8, $K$9, and $\rho = 0.5$0, while protecting a top-$\rho = 0.5$1 subset under a DINOv2 [CLS]-attention saliency score with $\rho = 0.5$2. On the K/V axis, Selectively Protected K/V Downsampling uses $\rho = 0.5$3, $\rho = 0.5$4, and $\rho = 0.5$5, together with a phase-shifted spatial grid, a frame-0 anchor, and uncompressed camera/register tokens. Training-free, the method reports a $\rho = 0.5$6 speedup over VGGT* at matched reconstruction quality on ScanNet-1000 (You et al., 16 Jun 2026).

TrajViT pushes the GroundVTS idea even further by redefining the token primitive itself. Instead of starting from space-time patches, it constructs one token per panoptic sub-object trajectory. If $\rho = 0.5$7 denotes the mask of trajectory $\rho = 0.5$8 at frame $\rho = 0.5$9, and $D_r=512$0 is a feature map, the framewise appearance feature is

$D_r=512$1

A Perceiver Resampler with one latent query then aggregates appearance and box-coordinate sequences into a single trajectory token. The paper reports a $D_r=512$2 token reduction relative to ViT3D, a $D_r=512$3 top-5 recall improvement on average for video-text retrieval, an average $D_r=512$4 improvement across six VideoQA benchmarks, $D_r=512$5 faster training time, and $D_r=512$6 less inference FLOPs when used as the video encoder for a VideoLLM (Zheng et al., 29 May 2025).

For autonomous driving MLLMs, VTS adopts a query-agnostic but temporally grounded sampler. A lightweight MobileOne-S0 proposal model computes saliency maps over the current and previous frames, selects the key frame by the highest average saliency, and ranks non-key-frame tokens by saliency plus dissimilarity to the corresponding key-frame token. Token count drops from $D_r=512$7 to $D_r=512$8, and the paper reports up to a $D_r=512$9 throughput improvement and a $D_r=128$0 memory reduction without compromising performance on DRAMA and LingoQA (Ma et al., 2024).

ALTP adapts grounded visual pruning to Grounded Conversation Generation. It argues that FastV and PyramidDrop fail on GCG because they preserve global semantics but discard local object-centric evidence critical for phrase-level masks. ALTP first partitions the image into SLIC superpixels (Detail Density Capture), then allocates per-region token budgets according to a density score

$D_r=128$1

followed by a normalized exponential weighting over regions (Dynamic Density Formation). On GLaMM at $D_r=128$2 token reduction, ALTP improves over PyramidDrop by $D_r=128$3 AP50 and $D_r=128$4 Recall; on OMG-LLaVA at the same reduction level, it improves AP by $D_r=128$5 and mIOU by $D_r=128$6 relative to PDrop (Bai et al., 31 Mar 2025).

These methods differ in unit and mechanism—patches, K/V sets, local superpixel groups, or object trajectories—but they share a central claim: token reduction must preserve the structural carriers of downstream correctness, whether these are geometry-critical boundaries, cross-view paths, dynamic agents, or small grounded entities.

5. Discrete grounded tokens, region insertion, and replayed evidence

Another strand of research does not primarily prune tokens; instead, it makes grounding explicit by creating symbolic or region-level tokens that can be generated, inserted, or replayed during reasoning. This is adjacent to GroundVTS because it converts latent evidence selection into observable token operations.

Composer replaces coordinate-string grounding with learned proxy-tokens. Using CLIP ViT-L/16, the model has $D_r=128$7 visual tokens and introduces a matching proxy-token vocabulary

$D_r=128$8

The multimodal input sequence is interleaved as

$D_r=128$9

and a visual entity $Q \in \mathbb{R}^{N_t \times D}$00 is represented by a subset $Q \in \mathbb{R}^{N_t \times D}$01. Composer achieves performance parity in final answer accuracy relative to a coordinate-based counterpart, while improving visual grounding accuracy by $Q \in \mathbb{R}^{N_t \times D}$02 points at IoU@0.95 on ComposerGCoT-val (Hodemon et al., 22 Jun 2026).

GR3D implements a region retrieval-and-insertion mechanism rather than patch sampling. During chain-of-thought generation, when an entity is mentioned, the model predicts a 2D box in text, extracts a region feature from the image, converts it into a region token, and inserts that token back into the text stream. The same grounded region then conditions monocular 3D box prediction. The paper explicitly characterizes this as dynamic region-token insertion rather than formal patch-token sampling, but the mechanism is closely aligned with GroundVTS because only task-relevant region embeddings are injected into the reasoning stream at the moment they are needed (Cheng et al., 28 May 2026).

VGR makes this selective replay interpretation explicit. It builds a high-resolution feature pool $Q \in \mathbb{R}^{N_t \times D}$03 from a base image and AnyRes local crops, compresses the global view, and allows the model to emit replay signals of the form

$Q \in \mathbb{R}^{N_t \times D}$04

When such a signal is generated, the system extracts the corresponding region from the cached feature map,

$Q \in \mathbb{R}^{N_t \times D}$05

applies $Q \in \mathbb{R}^{N_t \times D}$06 pooling, flattens the result, and appends it to the LLM context. In its compressed setting, VGR uses only $Q \in \mathbb{R}^{N_t \times D}$07 of the image token count relative to LLaVA-NeXT, while improving on detail-sensitive benchmarks such as MMStar, AI2D, and ChartQA (Wang et al., 13 Jun 2025).

"Thinking with Visual Grounding" moves the same idea into a reasoning-trace supervision framework. The model emits <obj> tags interleaved with natural-language thoughts, either as boxes,

$Q \in \mathbb{R}^{N_t \times D}$08

or as points,

$Q \in \mathbb{R}^{N_t \times D}$09

with coordinates normalized to $Q \in \mathbb{R}^{N_t \times D}$10. The pipeline synthesizes 19,909 reasoning traces, 107,613 grounding annotations, and 72,381 distinct grounded objects, then applies GRPO with grounding rewards based on box IoU or point F1 against SAM3-derived masks. Across counting and spatial reasoning benchmarks, grounded 4B models consistently outperform non-grounded thinking and in some cases surpass Gemma3-27B-IT from the same family (Zhang et al., 15 Jun 2026).

These methods imply a broader interpretation of GroundVTS: grounded visual tokens need not be only pruned encoder patches. They can also be symbolic references, region summaries, or replayed evidence carriers that make visual selection explicit and auditable.

6. Supervision resources, benchmarks, and recurring limitations

The recent GroundVTS ecosystem is supported by a growing set of data resources that make token selection or grounded reasoning directly supervisable. Ground-V is a large-scale instruction-following segmentation resource built from COCO and PACO, covering hallucinated references, multi-object scenarios, reasoning, multi-granularity, and part-level references. The main train statistics report 423,815 instructions over 50,000 images, with 57,591 human-validated test instructions over 5,000 images. Adding Ground-V during training yields an average gIoU improvement of $Q \in \mathbb{R}^{N_t \times D}$11 for LISA and $Q \in \mathbb{R}^{N_t \times D}$12 for PSALM across six benchmarks, and achieves N-Acc $Q \in \mathbb{R}^{N_t \times D}$13 on gRefCOCO, more than $Q \in \mathbb{R}^{N_t \times D}$14 points above the previous state of the art (Zong et al., 20 May 2025).

Grounding-FT, used by GroundVTS for VTG specialization, contains 70K instruction-style training pairs from Charades-STA, QVHighlights, and ActivityNet-Captions (Fan et al., 2 Apr 2026). ComposerGCoT contributes 163K reasoning chains over 55K images for step-wise grounded reasoning with proxy tokens (Hodemon et al., 22 Jun 2026). VGR-SFT supplies 158.1K reasoning traces with interleaved replay regions for OCR-, diagram-, chart-, and document-heavy tasks (Wang et al., 13 Jun 2025). "Thinking with Visual Grounding" contributes 19,909 grounded traces together with dense point/box supervision derived from SAM3 masks (Zhang et al., 15 Jun 2026). A plausible implication is that GroundVTS research is becoming increasingly data-centric: the quality of object extraction, grounding alignment, and negative examples may matter as much as the sampling mechanism itself.

Several limitations recur across otherwise different formulations. First, many methods incur extra cost from auxiliary passes or preprocessing: VEPO requires a counterfactual image pass; PGPO requires a visually masked second pass; RegimeVGGT relies on offline regime analysis and runtime saliency protection; TrajViT depends on segmentation-and-tracking quality; VGR maintains a high-resolution feature pool and dynamic replay logic (Jin et al., 2 Jun 2026, Ye et al., 2 Apr 2026, You et al., 16 Jun 2026, Zheng et al., 29 May 2025, Wang et al., 13 Jun 2025). Second, several systems ground text tokens or region summaries rather than directly pruning encoder-side visual patches; VEPO itself notes that it grounds text tokens via counterfactual vision dependence and does not sample or prune visual tokens inside the encoder (Jin et al., 2 Jun 2026). Third, some compression schemes are intentionally query-agnostic, as in autonomous-driving VTS, which is grounded in saliency and temporal novelty rather than the language query (Ma et al., 2024).

A final synthesis is that GroundVTS has bifurcated into two complementary research programs. One seeks compute-efficient visual token allocation, exemplified by query-guided filtering, geometry-aware compression, local-detail-aware pruning, and trajectory tokenization (Fan et al., 2 Apr 2026, You et al., 16 Jun 2026, Bai et al., 31 Mar 2025, Zheng et al., 29 May 2025). The other seeks faithful reasoning-time grounding, exemplified by visually anchored RL credit assignment, proxy-token vocabularies, dynamic region insertion, region replay, and explicit object-tagged thought traces (Ye et al., 2 Apr 2026, Hodemon et al., 22 Jun 2026, Cheng et al., 28 May 2026, Wang et al., 13 Jun 2025, Zhang et al., 15 Jun 2026). Taken together, these works suggest that the long-term technical agenda is not merely to keep fewer tokens, but to align token computation, token supervision, and reasoning structure with the specific visual evidence that makes a multimodal claim true.