Decomposed Spatio-Temporal Highlighting

• The paper introduces DSTH, a zero-shot framework that redistributes MLLM attention via query decomposition, achieving up to +12.1% improvement on [email protected] with strong backbones.
• DSTH is a method that splits text queries into attribute-related and action-related sub-queries, enabling accurate spatial and temporal grounding without fine-tuning the core model.
• The framework employs logit-guided re-attention and temporal-augmented assembling to optimize visual prompts at inference time, making it a plug-and-play solution for frozen MLLMs.

Searching arXiv for the specified paper and topic details. Decomposed Spatio-Temporal Highlighting (DSTH) is a zero-shot framework for spatio-temporal video grounding (STVG) built on multimodal LLMs (MLLMs). In the formulation considered here, STVG seeks the spatio-temporal tube of the object referred to by a text query in an untrimmed video, represented as a sequence of boxes $\{b_t\}_{t=t_s}^{t_e}$, where $b_t$ is the target box in frame $t$, and $t_s,t_e$ are the start and end times. DSTH was introduced together with temporal-augmented assembling (TAS) as the central mechanism of a MLLM-based zero-shot STVG system that aims to steer frozen MLLMs toward more reliable spatial and temporal evidence without grounding supervision or model fine-tuning (Yang et al., 18 Sep 2025).

1. Problem setting and motivation

STVG is characterized in the source work as harder than ordinary image grounding because it requires two coupled forms of inference: spatial grounding, namely identifying the correct object in each relevant frame among distractors, and temporal grounding, namely determining the correct start and end boundaries of the action or event span (Yang et al., 18 Sep 2025).

The same work situates DSTH against the annotation burden of traditional STVG pipelines. Traditional STVG methods are described as often fully supervised and dependent on costly frame-level tube annotations, while weakly supervised or zero-shot alternatives attempt to reduce this burden but are frequently constrained by the underlying vision-language backbone. In particular, CLIP-style methods are described as mostly aligning global image-text representations and as being weak for localization.

Within this context, MLLMs are treated as promising because of stronger cross-modal reasoning, yet difficult to use directly for zero-shot STVG. The stated difficulties are that they were not trained specifically for tube grounding, their attention is distributed over many token types rather than only over query words, and they may understand the overall query while still failing to integrate all discriminative cues needed for precise grounding, including attributes such as “in red clothes” and actions such as “walking” or “waving.” DSTH is designed specifically to address that mismatch by steering visual attention at test time while keeping the core MLLM frozen. This suggests that DSTH is not merely a prompt reformulation device, but a mechanism for redistributing the model’s internal attention budget across spatial and temporal evidence.

2. Empirical basis: grounding tokens and incomplete cue integration

A central empirical observation behind DSTH is that MLLMs dynamically assign some special tokens to behave as grounding tokens. These are tokens that exhibit unusually strong text-to-visual attention and are able to localize the target region effectively (Yang et al., 18 Sep 2025).

The input to the MLLM contains visual tokens, query tokens, and also system tokens and special role tokens associated with dialogue or instruction formatting. Among these special tokens, certain ones are reported to show high visual activation and strong localization ability. Their behavior is quantified through an attention ratio that compares the maximum attention inside the ground-truth box with the maximum attention outside it, using $f_{\mathrm{B2M}}$ to convert a bounding box into a binary mask. A higher ratio indicates better grounding. The paper also defines a token’s hit ratio as the fraction of samples for which it is the best grounding token.

The best grounding token is reported to be neither fixed across samples nor fixed across MLLMs. A token that grounds one sample well may fail on another, and the best token differs across model families. At the same time, the superior grounding token tends to have higher visual activation. This observation is methodologically important because it motivates sample-specific token selection rather than a static token heuristic.

The second observation is that even when a grounding token exists, the token may still omit critical textual cues. Two failure modes are emphasized. In spatial failures, the model ignores attributes that distinguish the target from similar distractors. In temporal failures, the model ignores action cues that are necessary to localize the relevant time span. DSTH is therefore built around two linked propositions: a grounding token must first be identified, and then that token must be induced to attend to the correct attribute and action evidence.

3. Query decomposition and logit-guided re-attention

DSTH is the core mechanism for adapting MLLMs to zero-shot STVG. Its first stage decomposes the original query $Q$ into two sub-queries: $Q_s$, an attribute-related description used for spatial reasoning, and $Q_t$, an action-related description used for temporal reasoning (Yang et al., 18 Sep 2025).

The decomposition is performed with an LLM, specified in implementation as GPT-4o, using instructions and in-context examples. The resulting descriptions are converted into interrogative questions with a fixed template. The rationale is explicit: attributes such as color, clothing, and left/right relations help identify the object instance spatially, whereas actions such as “running,” “turning,” and “raising hand” help identify when the target is active temporally.

The second stage introduces the logit-guided re-attention (LRA) module. For each sub-query, LRA creates a learnable latent variable that is added to the visual tokens and optimized at test time so that the frozen MLLM becomes more confident in answering “yes” rather than “no” to the corresponding binary question. For the spatial branch, the learnable visual prompt is denoted $\mathrm{V}_s$, with the same shape as the visual tokens $\mathrm{T}_v$. The model is run on $b_t$0, where $b_t$1 are the text tokens of the spatial sub-query. The predicted token probability is given as

$b_t$2

where $b_t$3 is the frozen LLM and $b_t$4 are previous generated tokens.

The optimization objective contrasts the logits of “yes” and “no”: $b_t$5 By minimizing this loss through backpropagation at inference time, the method learns the spatial prompt $b_t$6. The same procedure is applied to the temporal sub-query to learn the temporal prompt $b_t$7.

The intended functional split is explicit. The spatial branch $b_t$8 emphasizes attributes and other appearance cues, guiding the grounding token toward the correct object in each frame. The temporal branch $b_t$9 emphasizes action cues, supporting temporal grounding. Rather than directly supervising attention maps, LRA regularizes the model’s token prediction on a binary question. This suggests a form of self-referential inference-time adaptation in which the MLLM’s own generation logits become the supervisory signal for reconfiguring attention.

4. Attention steering, proposal scoring, and temporal-augmented assembling

After prompt tuning, the model produces attention maps from the selected grounding token: $t$0 for the spatial attention map and $t$1 for the temporal attention map (Yang et al., 18 Sep 2025). These attention maps are then used to score object track proposals and frame proposals.

For a track $t$2, the object-track score is defined from the attention map and the box-to-mask operator $t$3, and the track with the highest score becomes the spatial prediction. A per-frame temporal score $t$4 is computed analogously, and the top-$t$5 frames are selected as the final temporal segment. In the reported setup, a moderate number of temporal frames works best, and the system selects top-7 frames for temporal grounding.

DSTH is paired with temporal-augmented assembling (TAS) because the paper observes that spatial grounding from the attribute branch should be temporally consistent, whereas MLLMs can be unstable with respect to frame order. To quantify this, the paper defines a temporal consistency score $t$6 using the overlap between the spatial attention maps before and after reversing the input frames. Higher $t$7 indicates more consistent grounding under temporal perturbation.

TAS performs a frame-level reversal of the visual tokens and the spatial prompt simultaneously. The spatial prompt is then optimized on both the original frame order and the temporally augmented reversed order. At inference time, the framework assembles the spatial predictions from the original and temporally augmented inputs. The source text summarizes TAS as a consistency regularizer through temporal perturbation. A plausible implication is that TAS is specifically aimed at reducing order-sensitive instability in the attribute branch rather than replacing the temporal branch learned from action cues.

5. Zero-shot setup, model dependencies, and plug-and-play design

DSTH is explicitly defined as zero-shot and test-time tuning based. The core MLLM parameters remain frozen, and only the learnable visual prompts $t$8 and $t$9 are optimized per test sample. No grounding training labels are used (Yang et al., 18 Sep 2025).

The reported system uses GPT-4o for query decomposition, G-DINO for extracting object track proposals, and SAM2 for tracking tubelets. The MLLM backbones evaluated include LLaVA-Next-Video-7B, Qwen2-VL-7B, ShareGPT4Video-8B, and LLaVA-OneVision-7B. The method is described as plug-and-play across MLLMs because it depends only on internal token attention and generation logits, not on architectural modification or grounding supervision.

This design position is important for interpreting DSTH. It is not a retraining recipe for a specific STVG backbone; instead, it is an inference-time adaptation layer that operates over frozen video MLLMs and external proposal or tracking modules. At the same time, the source text notes that stronger trackers yield better final STVG performance, although the method remains effective with different trackers such as ByteTrack and BoTSort. This indicates that DSTH addresses one part of the STVG pipeline—cross-modal reasoning and attention steering—while still depending on upstream proposal quality.

6. Empirical performance, ablations, and limitations

The method is evaluated on three benchmarks: HC-STVG-v1, HC-STVG-v2, and VidSTG, using m_vIoU, [email protected], and [email protected] as metrics (Yang et al., 18 Sep 2025). The paper states that the method outperforms state-of-the-art methods on three common STVG benchmarks and reports improvements over zero-shot baselines such as ReCLIP, RedCircle, and especially E3M.

On HC-STVG-v1, the reported results for LLaVA-Next-Video-7B with the proposed method are $t_s,t_e$0 m_vIoU, $t_s,t_e$1 [email protected], and $t_s,t_e$2 [email protected]. For LLaVA-OneVision-7B, the reported results are $t_s,t_e$3 m_vIoU, $t_s,t_e$4 [email protected], and $t_s,t_e$5 [email protected]. Compared with E3M, the paper reports gains of $t_s,t_e$6 on [email protected] and $t_s,t_e$7 on [email protected]; with the stronger LLaVA-OneVision backbone, the gains become $t_s,t_e$8 on [email protected] and $t_s,t_e$9 on [email protected]. The method is also described as outperforming weakly supervised methods on most metrics and as competitive with fully supervised approaches.

The ablation results isolate the contributions of the main components. GTI alone improves over naive averaging of all special tokens. Adding spatial prompt learning improves performance further, adding temporal prompt learning also helps, using both spatial and temporal DSTH branches improves more, and adding TAS on top of DSTH gives the best results. For LLaVA-Next-Video, the sequence is reported as baseline without GTI/DSTH/TAS: $f_{\mathrm{B2M}}$0 m_vIoU; $f_{\mathrm{B2M}}$1: $f_{\mathrm{B2M}}$2; $f_{\mathrm{B2M}}$3 prompt: $f_{\mathrm{B2M}}$4; $f_{\mathrm{B2M}}$5 prompt: $f_{\mathrm{B2M}}$6; $f_{\mathrm{B2M}}$7 prompts: $f_{\mathrm{B2M}}$8; $f_{\mathrm{B2M}}$9: $Q$0. For LLaVA-OneVision, the sequence is baseline: $Q$1; $Q$2: $Q$3; $Q$4 prompts: $Q$5; $Q$6: $Q$7. More input frames generally help up to a point.

Qualitative behavior is summarized in terms of attribute sensitivity. Without DSTH, the model may attend to the wrong distractor because it ignores a discriminative attribute; with DSTH, the attribute cue is highlighted and the model localizes the correct person or object. The reported limitations are that long videos are expensive for MLLMs to process, temporal inconsistency can still hurt spatial grounding, the method depends on the quality of the underlying MLLM and the proposal or tracking components, and future work may require token pruning and key-frame selection for efficiency.

Taken together, the reported evidence supports a specific interpretation of DSTH. The source work’s overall takeaway is that zero-shot STVG becomes more effective when the model is not required to solve the entire query in a single undifferentiated pass. Instead, it decomposes the query into attribute and action sub-queries, identifies and exploits grounding tokens, uses logit-guided re-attention to learn test-time visual prompts, and employs temporal augmentation to stabilize spatial grounding. This suggests that DSTH’s significance lies less in introducing a new grounding head than in reorganizing inference so that frozen MLLMs can expose latent grounding ability already present in their token dynamics.