Grounded VideoDiT: Diffusion-Grounded Video LLM

• The paper introduces a diffusion-grounded VideoLLM architecture that integrates first-class temporal and entity grounding to improve long-video understanding.
• It employs a novel Diffusion Temporal Latent encoder that captures early denoising features, preserving appearance and motion for precise temporal localization.
• Empirical results on benchmarks demonstrate improvements in temporal sentence grounding and video QA, highlighting advancements in object-centric video tokenization.

Grounded VideoDiT most directly denotes the “Diffusion-Grounded VideoLLM” architecture introduced for long-video understanding, where temporal localization and entity grounding are treated as first-class components of video reasoning rather than as post hoc annotations (Fang et al., 21 Aug 2025). In that formulation, the central problem is to answer both when an event occurs and what or who is involved by combining a diffusion-based temporal encoder, query-driven object grounding and tracking, and a mixed token layout that exposes temporal information to the LLM. More broadly, the term also sits within a larger research program on grounded video models, spanning grounded generation, grounded editing, explicit temporal interfaces, and object-centric video tokenization, although several neighboring systems are not literal VideoDiT backbones.

1. Conceptual scope and problem formulation

Grounded VideoDiT addresses three closely related settings: temporal video grounding or temporal sentence grounding, grounded VideoQA, and open-ended VideoQA on long videos where temporal and entity reasoning remain coupled (Fang et al., 21 Aug 2025). The model is motivated by a specific diagnosis of earlier Video LLMs: timestamps are often represented only implicitly, frame or clip features are weak in capturing continuity, language-vision alignment drifts away from the queried entities, and segmentation or tracking is frequently applied only after language inference. In this view, long-video understanding requires both “when” reasoning, which localizes boundaries, order, and spans, and “what” reasoning, which preserves object and entity identity across time.

This framing separates Grounded VideoDiT from two common baselines. The first is the standard sampled-frame Video LLM, which concatenates visual tokens and text and relies on the LLM to infer temporal structure. The second is the classical grounding model, which may localize objects or time spans but does not integrate that grounded evidence into language reasoning. The Grounded VideoDiT formulation instead makes grounding part of the internal representation. A similar unification pressure appears in grounded text-to-video generation and grounded video editing, where control conditions must bind semantics to time and space rather than remain external prompts (Dou et al., 2024, Yu et al., 18 Mar 2025).

A recurring misconception is that any grounded video system is a Grounded VideoDiT. The literature is more heterogeneous. Some methods are Video LLMs with diffusion-style encoders, some are latent video diffusion U-Nets with grounding modules, some are true DiT-based generators, and some are training-free reasoning interfaces over existing MLLMs. The term is therefore most precise when reserved either for the specific “Diffusion-Grounded VideoLLM” architecture or for architectures that combine transformer-based video diffusion or diffusion-derived temporal latents with explicit spatial-temporal grounding.

2. Core architecture of Grounded VideoDiT

Grounded VideoDiT is organized into four functional stages: query parsing with object grounding and tracking, object-conditioned diffusion video encoding, temporal-text fusion with mixed tokens, and LLM reasoning with temporal localization heads (Fang et al., 21 Aug 2025). The implementation couples a frozen Grounded-SAM2 module for segmentation and tracking, a frozen WAN/VACE-style inpainting-capable video diffusion backbone, and a frozen Phi-3.5-Vision-Instruct-3.8B LLM adapted with LoRA.

Its main architectural novelty is the Diffusion Temporal Latent (DTL) encoder. Instead of using diffusion for generation, the model uses an intermediate denoising state as a temporal feature extractor. The forward corruption is written as

$$q(\mathbf{x}_{\tau}\mid \mathbf{x}_{0})=\mathcal{N}\!\big(\alpha_{\tau}\mathbf{x}_{0},\,\sigma_{\tau}^{2}\mathbf{I}\big),$$

and an early denoising feature is taken as

$\mathbf{h}_{\tau_0}=E_\theta(\mathbf{X},c,\tau_0),$

with a pooled and projected embedding

$$\mathbf{z}=g_\phi(\mathrm{Pool}(\mathbf{h}_{\tau_0})).$$

The stated rationale is that a small $\tau_0$ preserves detailed appearance and short-range motion before the denoising trajectory becomes dominated by the generative prior. The implementation section describes DTL as following the WAN design by injecting Gaussian noise into frame-wise features and performing conditional denoising to derive differentiable temporal latent tokens.

Object grounding is performed before language reasoning. Given a query $q$, a parser extracts noun phrases

$\mathcal{N}=\{n_i\}_{i=1}^{M}=\mathcal{E}(q),$

and Grounded-SAM2 is used frame-wise to propose regions and scores for each noun. To avoid starting from a partial scene, the model defines an AND-gated co-occurrence variable

$$g_t=\prod_{i=1}^{M}\mathbb{1}[\hat{s}_{i,t}\ge \tau_i],$$

and then enforces persistence across $K$ frames:

$\Gamma_t^{(K)}=\prod_{u=0}^{K-1}g_{t+u}.$

The earliest frame satisfying $\Gamma_t^{(K)}=1$ becomes the tracking start. One binary mask per noun is then propagated, and the diffusion encoder conditions on the per-frame union mask

$\mathbf{h}_{\tau_0}=E_\theta(\mathbf{X},c,\tau_0),$0

The paper stresses that this union mask remains raw and binary because it is used as a hard inpainting constraint.

The final reasoning stage feeds the LLM a mixed multimodal sequence. Framewise visual embeddings, a query embedding, and time information are fused before projection to the LLM space:

$\mathbf{h}_{\tau_0}=E_\theta(\mathbf{X},c,\tau_0),$1

Temporal localization is then decoded with token-level start and end classifiers:

$\mathbf{h}_{\tau_0}=E_\theta(\mathbf{X},c,\tau_0),$2

followed by joint maximization over valid spans.

3. Representation of time, entities, and grounding evidence

Time is central to the model’s identity, but the paper presents two partially different descriptions of how time enters the sequence (Fang et al., 21 Aug 2025). The title and abstract emphasize a mixed token scheme with discrete temporal tokens, and teaser examples use forms such as \<24> and \<96>-\<120>. By contrast, the Method section gives more detail on continuous normalized timestamp embeddings. For frame index $\mathbf{h}_{\tau_0}=E_\theta(\mathbf{X},c,\tau_0),$3 in a video of length $\mathbf{h}_{\tau_0}=E_\theta(\mathbf{X},c,\tau_0),$4,

$\mathbf{h}_{\tau_0}=E_\theta(\mathbf{X},c,\tau_0),$5

and the default time embedding is sinusoidal rather than discrete. An MLP-based time embedding is also proposed. This creates a documented ambiguity: the paper repeatedly claims explicit discrete temporal tokens, but the mathematical specification centers on continuous time embeddings added to fused frame tokens.

A similar under-specification appears in the object pathway. The paper states that temporally consistent object track embeddings are fused as explicit object tokens, and a token-budget ablation reports the best trade-off at 4 object tokens / 8 time tokens, with higher allocations degrading mIoU from 34.5 to 34.0 and 33.7. Yet the exact object-token construction is not formally defined. No mask pooling equation, track aggregation formula, or token serialization order is provided. The exact parser used for noun extraction, the thresholds $\mathbf{h}_{\tau_0}=E_\theta(\mathbf{X},c,\tau_0),$6, the persistence window $\mathbf{h}_{\tau_0}=E_\theta(\mathbf{X},c,\tau_0),$7, and the minimum span length $\mathbf{h}_{\tau_0}=E_\theta(\mathbf{X},c,\tau_0),$8 are also not specified.

The DTL branch is conceptually clear but mathematically incomplete in a second way. The diffusion subsection ends with a single compact embedding $\mathbf{h}_{\tau_0}=E_\theta(\mathbf{X},c,\tau_0),$9, while later multimodal fusion assumes framewise embeddings $$\mathbf{z}=g_\phi(\mathrm{Pool}(\mathbf{h}_{\tau_0})).$$0. This indicates a reproducibility gap rather than a contradiction in the overall design. The intended semantics remain clear: diffusion-derived temporal features are meant to be boundary-sensitive, object-aware, and temporally consistent because they are conditioned on masks and extracted from an early denoising stage rather than from static frame encoders.

Grounding evidence is therefore distributed across three levels. First, noun-conditioned masks and tracks bind queried entities to localized regions. Second, the diffusion encoder turns those masked video observations into temporal latents. Third, the mixed token layout exposes time and entity structure to the LLM. The article’s “when and what” slogan is thus literal: time and entity identity are meant to be represented before and during language reasoning, not only after it.

4. Optimization, ablations, and empirical profile

Grounded VideoDiT is trained with frozen pretrained modules plus lightweight adaptation rather than full end-to-end optimization through segmentation and diffusion backbones (Fang et al., 21 Aug 2025). Grounded-SAM2, the diffusion backbone, and the base LLM are frozen. Trainable components include projection heads, fusion layers, the grounding head, and LoRA adapters. Reported optimization details are AdamW, cosine schedule, base learning rate $$\mathbf{z}=g_\phi(\mathrm{Pool}(\mathbf{h}_{\tau_0})).$$1, warm-up ratio 5%, 3 epochs, global batch size 128, and LoRA parameters $$\mathbf{z}=g_\phi(\mathrm{Pool}(\mathbf{h}_{\tau_0})).$$2 and $$\mathbf{z}=g_\phi(\mathrm{Pool}(\mathbf{h}_{\tau_0})).$$3 on 8 × H800 GPUs. Each video is uniformly sampled into $$\mathbf{z}=g_\phi(\mathrm{Pool}(\mathbf{h}_{\tau_0})).$$4 frames and divided into $$\mathbf{z}=g_\phi(\mathrm{Pool}(\mathbf{h}_{\tau_0})).$$5 segments for temporal noise injection.

The paper presents two explicit loss-related elements. One is the start/end grounding head described above. The other is a KL regularization term aligning diffusion features with an auxiliary encoder:

$$\mathbf{z}=g_\phi(\mathrm{Pool}(\mathbf{h}_{\tau_0})).$$6

The exact language-modeling loss for open-ended QA and the weighting among possible losses are not specified.

On Charades-STA, Grounded VideoDiT reports [email protected] = 58.7, [email protected] = 41.2, [email protected] = 21.0, and mIoU = 39.5. On DiDeMo, it reports [email protected] = 53.0, [email protected] = 37.0, [email protected] = 18.5, and mIoU = 35.2. Relative to Grounded-VideoLLM, the gains shown in the table are +2.7 mIoU and +4.8 [email protected] on Charades-STA, and +3.2 mIoU and +3.6 [email protected] on DiDeMo. However, the same table shows LLaVA-ST above Grounded VideoDiT on both datasets, with Charades-STA mIoU 42.4 and [email protected] 44.8, and DiDeMo mIoU 37.6 and [email protected] 39.8. The abstract’s stronger “state-of-the-art” framing is therefore not uniformly supported by the shown temporal grounding results.

The grounded QA results are stronger relative to the cited baseline. On NExT-GQA, the model reports Acc@GQA = 28.4, mIoP = 36.8, [email protected] = 35.9, mIoU = 23.2, and [email protected] = 19.9, improving over Grounded-VideoLLM by +1.7 Acc@GQA, +2.3 mIoP, and +2.1 mIoU. On open-ended QA, the paper reports 56.9 Acc / 3.6 Score on NExT-QA, 78.0 / 4.3 on MSVD-QA, 62.1 / 3.7 on MSRVTT-QA, and 58.4 / 3.6 on ANet-QA.

The ablation profile supports the intended decomposition. On Charades-STA, an adapted Grounded-VideoLLM baseline gives 54.2 / 36.4 / 19.7 / 36.8 for [email protected], [email protected], [email protected], and mIoU. Adding DTL only yields 55.6 / 37.8 / 21.0 / 38.2. The full DTL + Obj + Time model reaches 58.7 / 41.2 / 21.0 / 39.5. A token-budget ablation gives mIoU 33.6 for 2 object / 4 time tokens, 34.5 for 4 / 8, 34.0 for 8 / 16, and 33.7 for 16 / 32, suggesting that too many auxiliary tokens dilute useful context. DTL hyperparameter ablation reports the best trade-off at $$\mathbf{z}=g_\phi(\mathrm{Pool}(\mathbf{h}_{\tau_0})).$$7, cosine schedule, and GS = 1.0, with [email protected] 36.9 and mIoU 34.5; $$\mathbf{z}=g_\phi(\mathrm{Pool}(\mathbf{h}_{\tau_0})).$$8 slightly improves these values to 37.0 and 34.7 but at higher cost.

5. Position within the broader grounded-video literature

The surrounding literature shows that “Grounded VideoDiT” is not a single method family but a spectrum of architectures, interfaces, and control schemes. Some neighboring systems are close in spirit but not in backbone; others are genuine DiT systems but domain-specific or task-specific.

System	Core formulation	Relation to Grounded VideoDiT
VTimeCoT (Zhang et al., 16 Oct 2025)	Training-free visuotemporal CoT over MLLMs using a progress bar, highlighting, and video cutting	Not a generative VideoDiT; a grounding-and-reasoning interface
VEGGIE (Yu et al., 18 Mar 2025)	MLLM-generated frame-wise grounded task queries rendered by a video diffusion UNet adapted from SD1.5 and AnimateDiff	Conceptually close to grounded generation, but not a DiT backbone
GVDIFF (Dou et al., 2024)	Grounded latent video diffusion U-Net with uncertainty-based attention bias, STGL, and DGN	A grounded diffusion blueprint portable to DiT-style token processing
DiVE (Jiang et al., 2024)	OpenSora-based latent rectified-flow DiT with joint text-layout conditioning, road-sketch ControlNet-Transformer, and view-inflated attention	A domain-specific grounded VideoDiT for autonomous driving
MotionGrounder (Teodoro et al., 1 Apr 2026)	Inference-time control of a pretrained DiT using Flow-based Motion Signal and Object-Caption Alignment Loss	A grounded multi-object motion-transfer framework rather than a universal generator

Two additional lines of work matter even though they are not direct Grounded VideoDiT models. GROVE builds large-scale phrase-to-track supervision for grounded video caption generation and introduces HowToGround1M plus iGround, showing that temporally dense phrase-grounded box supervision can be constructed at scale and substantially improves grounded video modeling (Kazakos et al., 13 Mar 2025). TrajViT argues for “one trajectory, one token,” replacing patch tokenization with panoptic sub-object trajectories; this is not a diffusion model, but it offers an object-centric tokenization strategy whose scaling behavior depends more on scene complexity than clip duration (Zheng et al., 29 May 2025).

These comparisons clarify the conceptual boundaries of the term. VTimeCoT is adjacent because it externalizes temporal grounding for reasoning, but it is explicitly “not a generative video transformer method in the ‘VideoDiT’ sense.” VEGGIE is close to what one might want from a grounded video diffusion editor, yet the paper explicitly states that its backbone is a latent diffusion UNet rather than a VideoDiT. DiVE and MotionGrounder are the most literal DiT-side neighbors: one is a BEV-grounded multi-view autonomous-driving generator, and the other is a mask-grounded multi-object motion-transfer controller. Grounded VideoDiT, by contrast, uses diffusion primarily as a temporal representation inside a Video LLM for understanding rather than as a pure generative denoiser.

6. Limitations, reproducibility gaps, and plausible trajectories

The most immediate limitations are those already visible inside the Grounded VideoDiT paper itself (Fang et al., 21 Aug 2025). The model depends on the quality of Grounded-SAM2 segmentation and tracking, and any failure there propagates into the temporal latent representation. On temporal grounding benchmarks, its reported numbers remain below LLaVA-ST in the table shown for Charades-STA and DiDeMo. More denoising steps and more object or time tokens increase cost without guaranteed gains. Because the segmentation and diffusion backbones are frozen, the system may also inherit bottlenecks from pretrained modules rather than adapting deeply to new distributions.

The paper is also conceptually clearer than it is fully reproducible. The exact noun parser, grounding thresholds, persistence window, discrete time-token vocabulary, object-token construction, auxiliary encoder identity, and language-modeling objective are all absent from the description. The method emphasizes discrete temporal tokens, but its mathematical account is more explicit about continuous normalized timestamp embeddings. It claims object tokens, but formalizes only visual-text-time fusion at the frame-token level. Those omissions do not obscure the architectural intent, but they do matter for exact reimplementation.

Across the broader literature, several converging design directions emerge. DiVE shows that a true DiT backbone can absorb structured grounding such as BEV layouts, road sketches, and multi-condition classifier-free guidance in a transformer-native manner (Jiang et al., 2024). MotionGrounder shows that object-caption attention supervision and external motion priors can make a pretrained DiT more explicitly grounded at inference time (Teodoro et al., 1 Apr 2026). VEGGIE suggests that frame-wise latent task queries produced by an MLLM can unify editing, grounding, and reasoning under a single diffusion loss, even without a DiT backbone (Yu et al., 18 Mar 2025). TrajViT suggests that future grounded video transformers may benefit from replacing grid-centric tokenization with trajectory-centric tokenization (Zheng et al., 29 May 2025). GROVE suggests that large-scale phrase-to-track pre-training is a practical substrate for grounded video supervision (Kazakos et al., 13 Mar 2025).

A plausible implication is that future Grounded VideoDiT systems will combine several of these strands rather than follow a single lineage. One can read the current literature as moving toward architectures in which temporal structure is explicit, entities are grounded before or during reasoning, motion can be supervised as a first-class signal, and video tokens themselves become object-centric rather than purely patch-centric. What remains open is how to realize all of this simultaneously in a scalable, fully specified, and broadly applicable VideoDiT.