Tube-Mined Temporal Regularization

• TTReg is a temporal consistency mechanism that mines object tubes and applies explicit cross-frame losses to ensure coherent video detection.
• It integrates memory-based tube association with feature and geometric consistency losses to mitigate exposure bias and progressive localization drift.
• Experimental evaluations on benchmarks like HC-STVG demonstrate improved m_tIoU and m_vIoU, showcasing TTReg’s robust performance in video tracking.

Tube-Mined Temporal Regularization (TTReg) is a temporal consistency mechanism developed for open-vocabulary detectors (OVDs) in spatio-temporal video grounding tasks. It addresses the exposure bias and progressive localization drift that arise in prior large multimodal LLMs (MLLMs) utilizing autoregressive box-token decoding. By tube-mining across video frames and applying explicit cross-frame regularization losses, TTReg ensures temporally coherent detection and tracking within a video LLM architecture, such as DEViL, which unifies an MLLM and OVD via a reference-semantic token (RST) (Gao et al., 7 Dec 2025).

1. Motivation and Intuition

Autoregressive spatial decoding in existing MLLMs (e.g., LLaVA-ST) textualizes bounding box coordinates as tokens generated sequentially per frame. This process imports exposure bias: errors in bounding-box token prediction propagate along the output sequence, resulting in significant coordinate drift and jitter. Let $\epsilon$ denote the per-timestep box error probability; the probability of maintaining a correct box sequence of length $L$ approximates $(1-\epsilon)^L \approx 1-L\epsilon$, so longer videos and finer box discretizations exacerbate drift.

TTReg overcomes this by embedding tube mining within the OVD (e.g., Grounding-DINO backbone), which generates $N_q$ object queries per frame. "Tubes" are hypothesized spatio-temporal tracks, constructed by associating these object queries across $T$ frames (using similarity or motion cues). TTReg mines the tube best aligned with ground-truth annotations and imposes cross-frame losses: (a) feature consistency (sustaining similarity between object-query embeddings across frames) and (b) geometric consistency (enforcing high intersection-over-union of predicted boxes in adjacent frames). This approach transforms an OVD into an effective, lightweight temporal tracker by design (Gao et al., 7 Dec 2025).

2. Tube Mining Algorithms

2.1. Memory-Based Tube Association (Inference Time)

At inference, identity persistence is achieved by aligning each query index $i$ with a single tracked object. This employs an exponential moving average (EMA) memory combined with the Hungarian algorithm for frame-to-frame matching:

• $M_1[i] \leftarrow q_1^i$ for initialization,
• For $t>1$, pairwise cosine similarity between current queries $q_t^i$ and previous memory slots $M_{t-1}[j]$ yields an assignment via Hungarian matching,
• Memory is updated as $$M_t \leftarrow (1-\alpha) M_{t-1} + \alpha~(\text{reordered }q_t)$$, with EMA rate $\alpha = 0.1$.
• Final tube is selected by averaging classification confidence over the video.

2.2. Ground-Truth-Aligned Tube Mining (Training Time)

For supervision, TTReg evaluates $N_q$ candidate tubes against ground truth, defining a cost for each tube $i$ as follows:

• Classification cost: $C^i_{\text{cls}}$, mismatch between $q_t^i$ and phrase embedding,
• Box regression cost: $C^i_{\text{bbox}}$, $L_1$ or $\ell_2$ distance,
• GIoU cost: $$C^i_{\text{giou}} = 1-\mathrm{GIoU}(b_t^i, b^{\mathrm{GT}}_t)$$ (averaged over $t$),
• Temporal jitter cost: $$C^i_{\text{temp}} = \frac{1}{T-1}\sum_{t=1}^{T-1}(1 - \mathrm{GIoU}(b_t^i, b_{t+1}^i))$$.

Total cost:

$$C^i = \lambda_{\text{cls}} C^i_{\text{cls}} + \lambda_{\text{bbox}} C^i_{\text{bbox}} + \lambda_{\text{giou}} C^i_{\text{giou}} + \lambda_{\text{temp}} C^i_{\text{temp}}$$

Tube $i^* = \operatorname*{argmin}_i C^i$ is selected, yielding the supervision track for TTReg's temporal losses.

Cost Component	Description	Symbol
Classification	Phrase/Query mismatch (mean over frames)	$C_{\text{cls}}$
Box Regression	$L_1$ or $\ell_2$ on boxes	$C_{\text{bbox}}$
GIoU	Mean $1-\mathrm{GIoU}$ over time	$C_{\text{giou}}$
Temporal Jitter	Mean $1-\mathrm{GIoU}$ between consecutive boxes	$C_{\text{temp}}$

3. Mathematical Formulation

3.1. Loss Functions

Let $T$ be the number of frames, $N_q$ the number of object queries, and $(q_t^*, b_t^*)$ the tube mined per above.

Feature Consistency:

$$L_{\text{feat}} = \frac{1}{T-1}\sum_{t=1}^{T-1} \left[1 - \frac{q_t^* \cdot q_{t+1}^*}{\lVert q_t^*\rVert\lVert q_{t+1}^*\rVert}\right]$$

Geometric Consistency:

$$L_{\text{geom}} = \frac{1}{T-1}\sum_{t=1}^{T-1} [1-\mathrm{GIoU}(b_t^*, b_{t+1}^*)]$$

Total TTReg Loss:

$$L_{\text{TTReg}} = \lambda_{\text{feat}}L_{\text{feat}} + \lambda_{\text{geom}}L_{\text{geom}}$$

This regularization is incorporated into the full detection objective during Stage-3 joint training with the OVD:

$$L_{\text{total}} = L_{\text{DET}} + L_{\text{TTReg}}$$

where $L_{\text{DET}}$ is the classic DETR-type detection loss.

4. Implementation, Integration, and Hyperparameters

The OVD consumes the RST—an LLM-extracted embedding distilled from the user query—and produces queries steered toward the referent. TTReg is included only in Stage-3 spatio-temporal training, when both the LLM's RST projector and OVD backbone are unfrozen.

Key implementation hyperparameters include:

• Top $N_q=15$ queries per frame for tube mining and association,
• Loss component weights: $\lambda_{\text{cls}}=1$, $\lambda_{\text{bbox}}=5$, $\lambda_{\text{giou}}=3$, $\lambda_{\text{temp}}=2$,
• TTReg loss weights: $\lambda_\text{feat}=1$, $\lambda_\text{geom}=1$,
• EMA association rate: $\alpha=0.1$,
• Tube discarding threshold: at least $50\%$ overlap with ground-truth temporal span,
• Optimization: Detector LR $1\mathrm{e}{-4}$, LLM/projector LR $1\mathrm{e}{-5}$, batch sizes $(128, 32, 8)$ in Stages 1, 2, 3,
• Backbone/Detector: Grounding-DINO (Swin-B) (COCO pretrained), LLM: Qwen2.5-7B (VideoLLaMA3 init), vision: SigLIP.

5. Experimental Evaluation

TTReg was evaluated on spatio-temporal video grounding and reasoning datasets: HC-STVG v1/v2, VidSTG, V-STaR, NExT-GQA, and Charades-STA.

Key quantitative improvements on HC-STVG:

Model Configuration	HC-STVG v1 (m_tIoU / m_vIoU)	HC-STVG v2 (m_tIoU / m_vIoU)
Baseline (no TTReg)	53.3 / 35.5	57.4 / 35.8
+GTM only	54.4 / 35.8	57.6 / 36.1
+CFR only	54.9 / 35.6	57.9 / 35.5
+GTM + CFR (full TTReg)	54.7 / 36.2	58.0 / 36.5

An ablation of memory-based tube association (MTA) at inference showed that enabling MTA with TTReg improves m_vIoU (e.g., 35.8→36.2 (v1) and 35.9→36.5 (v2)), with m_tIoU unchanged.

6. Analysis and Limitations

By mining and supervising directly on tubes that best match ground-truth, TTReg avoids the noise present in distractor tubes, and its feature/geometric consistency losses deliver precise cross-frame feedback, which is not available in autoregressive token decoders or per-frame detectors. TTReg’s explicit temporal constraints yield more temporally consistent and accurate spatio-temporal localization, effectively transforming the OVD into a video-level tracker.

However, current TTReg and RST methods operate in a single-target setting; extensions to multi-entity queries would require multi-tube mining and multiple RSTs. Scaling to longer videos may necessitate hierarchical or windowed approaches, or curriculum learning to expand temporal context. Explicit motion cues (e.g., optical flow) in the tube-mining process are a potential direction for increasing robustness in fast-motion scenarios (Gao et al., 7 Dec 2025).

7. Context and Significance

TTReg represents a lightweight, generalizable augmentation for query-based video detectors within LLM pipelines. By ensuring that OVD query embeddings and detections are temporally stabilized, it mitigates limitations of autoregressive video LLM localization. As a plug-in module, it opens a path to more accurate, drift-free, and temporally coherent video grounding and reasoning without the overhead of explicit tracking models or sequence generation, thus setting a new state-of-the-art in fine-grained video understanding benchmarks (Gao et al., 7 Dec 2025).