Dynamic Vision Aligner (DyVA)

• The paper introduces DyVA as a vision-language model variant that repurposes a video diffusion model to extract dynamics-aware features alongside a static image encoder.
• DyVA fuses latent features from a frozen SigLIP encoder and a dynamics-aware Stable Video Diffusion branch via lightweight projection, improving spatial and multi-frame reasoning.
• Empirical results demonstrate that leveraging world-model priors boosts performance on spatial tasks and multi-frame inference, surpassing traditional static-only models.

Searching arXiv for the specified DyVA paper and closely related work to ground the article in current papers. Dynamic Vision Aligner (DyVA) is a vision-LLM architecture introduced as the best-performing variant within a broader family of World-LLMs (WorldLMs), whose central premise is that a video diffusion world model can be repurposed as a generative visual encoder for multimodal understanding rather than only for video synthesis (Zhang et al., 1 Oct 2025). DyVA combines a strong static image encoder, SigLIP, with dynamics-aware latent features extracted from Stable Video Diffusion (SVD) after a single denoising step, then projects both streams into the token space of a LLM. In the reported formulation, the design is intended to improve spatial reasoning and to enable single-image models to perform multi-frame reasoning by transferring priors internalized during video pre-training, especially motion consistency, structure, and viewpoint-related regularities (Zhang et al., 1 Oct 2025).

1. Definition and conceptual scope

DyVA is defined as a WorldLM variant that leverages a video diffusion model as a generative encoder and uses the resulting latents as visual embeddings for downstream vision-language modeling (Zhang et al., 1 Oct 2025). The underlying research question is whether world model priors, acquired from internet-scale video data, can benefit Vision-LLMs on generic multimodal tasks rather than only on narrowly defined visual tasks. The reported answer is affirmative in the specific configuration studied: the best-performing system, named Dynamic Vision Aligner, uses a single denoising step from a video diffusion model to generate dynamics-rich features and then fuses them with static semantic features from a conventional image encoder (Zhang et al., 1 Oct 2025).

The conceptual distinction from standard VLM design is that DyVA does not rely exclusively on a static vision encoder. Instead, it treats generative video-model representations as a complementary source of information. The paper attributes the resulting gains to the inherited motion-consistency internalization of the world model, arguing that such priors support spatial reasoning, cross-view understanding, and multi-frame inference (Zhang et al., 1 Oct 2025). This suggests that the relevant transfer is not merely additional feature capacity, but a change in representational bias induced by video pre-training.

A frequent source of confusion is the assumption that DyVA is primarily a video generation system. In the reported architecture, the video model is not used to synthesize video at inference time for downstream reasoning. Rather, hidden activations from the denoiser are extracted and treated as embeddings. The generative model is therefore repurposed as an encoder, not deployed in its original role as a decoder-driven synthesis pipeline (Zhang et al., 1 Oct 2025).

2. Architectural composition

The DyVA architecture has two visual branches and one language branch. The static branch uses a frozen SigLIP image encoder to produce patch embeddings from the input image. These embeddings are projected into the LLM space through a lightweight projector $P_{\text{siglip}}$, yielding

$V_s = P_{\text{siglip}}(S) = \mathrm{MLP}_s(S).$

In parallel, the dynamic branch uses Stable Video Diffusion, a VAE+U-Net based diffusion model trained on large-scale video data, as a generative encoder rather than as a video generator (Blattmann et al., 2023). Its output features are projected through a second lightweight projector $P_{\text{svd}}$ to obtain dynamics-aware tokens $V_d$ (Zhang et al., 1 Oct 2025).

The two token streams are concatenated,

$V = [V_s; V_d],$

and the resulting visual context is fed to a frozen LLM, such as LLaMA-2 or Qwen2.5, together with the prompt embeddings. The model then generates the textual answer from the combined sequence (Zhang et al., 1 Oct 2025). In the notation reported for the system,

$u_{out} = \text{LM}_\theta([V; E_{prompt}]).$

A defining implementation choice is that the visual encoders remain frozen during training. Only the lightweight projection MLPs and the language backbone are trainable. Training is performed in a single stage, and the reported configuration is explicitly characterized as efficient: 10.3 hours on $16\times$A800 GPUs (Zhang et al., 1 Oct 2025). This matters methodologically because it separates the claimed benefit of world-model priors from gains that might otherwise be attributed to expensive end-to-end adaptation.

3. Generative encoding with a single denoising step

The dynamic branch begins by encoding the image into a latent code $z_0$ with the SVD VAE encoder $\phi$. For single-image input, $z_0$ is tiled across $V_s = P_{\text{siglip}}(S) = \mathrm{MLP}_s(S).$0 frames, producing

$V_s = P_{\text{siglip}}(S) = \mathrm{MLP}_s(S).$1

For multi-image input, encoded keyframes are inserted at evenly spaced positions in the $V_s = P_{\text{siglip}}(S) = \mathrm{MLP}_s(S).$2-frame latent tensor (Zhang et al., 1 Oct 2025). The default inference setting reported for DyVA uses image size $V_s = P_{\text{siglip}}(S) = \mathrm{MLP}_s(S).$3 and $V_s = P_{\text{siglip}}(S) = \mathrm{MLP}_s(S).$4 frames (Zhang et al., 1 Oct 2025).

The central operation is a single Euler ODE denoising step at initial noise scale $V_s = P_{\text{siglip}}(S) = \mathrm{MLP}_s(S).$5, performed without classifier-free guidance: $V_s = P_{\text{siglip}}(S) = \mathrm{MLP}_s(S).$6 where $V_s = P_{\text{siglip}}(S) = \mathrm{MLP}_s(S).$7 is the SVD U-Net denoiser and $V_s = P_{\text{siglip}}(S) = \mathrm{MLP}_s(S).$8 denotes conditioning (Zhang et al., 1 Oct 2025). Rather than decoding $V_s = P_{\text{siglip}}(S) = \mathrm{MLP}_s(S).$9 into video, DyVA extracts internal U-Net activations at the lowest spatial resolution, specifically from a middle or pre-mid-layer representation. The reported formulation emphasizes pre-mid hidden activations: $P_{\text{svd}}$0

These activations are flattened into

$P_{\text{svd}}$1

then projected into the language-model token space: $P_{\text{svd}}$2 The paper reports that predicted dynamics matter: replacing this denoising-based representation with the VAE encoder alone leads to substantially worse results, with decreases of 10%–12% on all spatial tasks (Zhang et al., 1 Oct 2025). It also reports that fusing temporal latents after the U-Net’s middle block marginally improves spatial reasoning metrics, although pre-mid extraction is preferred for efficiency (Zhang et al., 1 Oct 2025).

This design is explicitly meant to expose a dynamics-aware latent space to the LLM. The reported interpretation is that the denoiser’s hidden activations encode spatiotemporal priors learned during video training, including object permanence, 3D structure, physics, and motion consistency (Zhang et al., 1 Oct 2025, Blattmann et al., 2023).

4. Training protocol and evaluation regime

DyVA is trained with a mixture of multimodal instruction data and established benchmark datasets. The reported training corpus includes LLaVA-1.5, GQA, TextCaps, and ShareGPT (Zhang et al., 1 Oct 2025). Training is single-stage multitask tuning in which both static and dynamics-aware visual features are aligned to the LLM token space (Zhang et al., 1 Oct 2025). Because the encoders are frozen, the optimization target is concentrated on the projectors and the language backbone.

The evaluation suite is organized around two task families. Single-image spatial reasoning is assessed on VSR, TallyQA, SpatialMM-Obj, and 3DSR-Bench-real. Multi-image and multi-frame reasoning is evaluated on MMSI-Bench, SAT-Bench, and MindCube (Zhang et al., 1 Oct 2025). The paper emphasizes that DyVA is often trained only with single-image data while still exhibiting strong performance on multi-image or multi-frame reasoning tasks (Zhang et al., 1 Oct 2025).

The multi-image formulation is notable because it does not require a separate video training stage for downstream tasks. Instead, multiple images are inserted as keyframes into the temporal slots of the latent tensor used by the SVD branch (Zhang et al., 1 Oct 2025). This suggests a direct reuse of the world model’s temporal inductive structure as a reasoning substrate for non-video multimodal tasks.

5. Empirical performance and ablation findings

The reported headline result is that DyVA attains state-of-the-art or comparable performance on a curated suite of visual reasoning tasks, especially those that stress spatial reasoning and multi-frame reasoning (Zhang et al., 1 Oct 2025). On MindCube (Avg), DyVA-Qwen2.5-7B achieves 49.80%, compared with GPT-4o at 38.81%. On MindCube (Around), it records 51.70%, with a stated baseline around 44%. On SAT Synthetic (Avg), it reaches 55.24%, compared with Qwen2.5-VL at 53%. On VSR it achieves 67.1%, on TallyQA 68.1%, and on 3DSR-Bench “Multiple” 49.6%, with the reported margins over leading baselines ranging from roughly 2% to 11% depending on the benchmark (Zhang et al., 1 Oct 2025).

The empirical profile is not uniform across all task types. The paper states that DyVA is particularly strong on spatial and 3D reasoning, including mental rotation, adjacency, orientation, and multi-object spatial layouts, as well as on zero-shot multi-frame reasoning (Zhang et al., 1 Oct 2025). At the same time, the reported summary identifies OCR, language-heavy settings, and unusual scenes as weaker areas (Zhang et al., 1 Oct 2025). This distribution is consistent with the claim that the world-model prior contributes most where spatial-temporal regularities matter.

The ablation results clarify which components are necessary. First, the dynamic branch is not sufficient on its own: SigLIP+SVD or CLIP+SVD outperforms DINOv2+SVD, and the paper states that text-loss supervision is critical for aligning dynamic tokens with semantic space (Zhang et al., 1 Oct 2025). Second, the world-model denoising step is not replaceable by the VAE encoder alone, because removing the prediction step causes large degradations on spatial tasks (Zhang et al., 1 Oct 2025). Third, temporal capacity matters: increasing the number of SVD frames improves results on VQAv2, VSR, and TallyQA, with reported values progressing from 59.4/52.1/51.0 at 1 frame to 60.8/52.2/51.5 at 8 frames and 61.7/53.4/52.2 at 14 frames (Zhang et al., 1 Oct 2025).

The study also reports a negative result on backbone adaptation. Text-only tuning of the SVD backbone does not improve performance and can degrade it (Zhang et al., 1 Oct 2025). The stated implication is that the generative prior should remain fixed, and that extending text loss to the generative backbone does not straightforwardly adapt low-level visual dynamics to the VLM setting.

6. Interpretation, limitations, and relation to adjacent work

The paper interprets DyVA’s gains through the notion of motion-consistency internalization. Because SVD is pre-trained on temporally coherent video-text pairs, its latent representations are said to encode viewpoint invariance, motion regularity, and other spatiotemporal constraints that benefit downstream multimodal reasoning (Zhang et al., 1 Oct 2025). In this reading, DyVA’s contribution is not only architectural fusion but a transfer mechanism from generative world modeling into discriminative multimodal understanding.

A second interpretive point concerns what kind of reasoning is being improved. The qualitative analysis reported in the paper contrasts conventional VLMs, which describe static content, with DyVA or WorldLM variants that “envision” dynamic possibilities and exhibit structured temporal foresight (Zhang et al., 1 Oct 2025). This suggests that the relevant benefit is partly counterfactual or predictive: the model appears better able to reason about possible state transitions and multiple viewpoints even from limited visual evidence.

DyVA should also be distinguished from a different line of work on dynamic vision-language alignment in 3D scenes. In "Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Control" (Armitage et al., 31 Dec 2025), the central problem is online camera control for adapting 2D-trained VLMs to 3D multi-object scenes through regret-minimizing multivariate mutual information estimation. That system improves viewpoint selection without pretraining or finetuning the VLM, whereas DyVA changes the visual encoding regime by importing world-model priors into the encoder stack (Armitage et al., 31 Dec 2025, Zhang et al., 1 Oct 2025). The two approaches therefore address related multimodal alignment difficulties from different levels of the pipeline: one through adaptive control of observations, the other through dynamics-aware representation learning.

Within the scope of the reported experiments, DyVA’s limitations remain substantive. The model’s weaknesses on OCR, language-heavy tasks, and unusual scenes indicate that world-model priors do not uniformly improve all forms of multimodal competence (Zhang et al., 1 Oct 2025). The ablations further show that the generative branch cannot replace a text-aligned static encoder, and that naive fine-tuning of the world model is counterproductive (Zhang et al., 1 Oct 2025). A plausible implication is that DyVA’s main value lies in complementing, rather than supplanting, conventional vision encoders.

7. Significance for vision-language modeling

DyVA is presented as a systematic repurposing of world models for general-purpose vision-language reasoning, using a video diffusion model as a generative encoder and a single denoising step as the mechanism for extracting dynamics-aware tokens (Zhang et al., 1 Oct 2025). Its reported gains on MindCube, SAT-Bench, VSR, TallyQA, and 3DSR-Bench-real support the claim that video-pretrained generative representations can enhance spatial and multi-frame reasoning even when downstream supervision is predominantly single-image (Zhang et al., 1 Oct 2025).

The broader significance of DyVA lies in the type of prior it operationalizes. Standard VLM pipelines are usually dominated by static patch-level semantics. DyVA adds a second stream that is explicitly shaped by video-world regularities. This suggests that multimodal reasoning performance can benefit from latent spaces that encode not only what is visible, but also how visible structure would evolve across time or across viewpoints. In that sense, DyVA occupies an intermediate position between classical image-language modeling and explicit world-model-based reasoning.

As reported, the architecture does not establish that world models should replace conventional visual encoders. Rather, its main finding is that a frozen, video-pretrained generative encoder can be productively fused with a frozen text-aligned image encoder, and that the resulting combination improves exactly those benchmarks where structure, spatial layout, and temporal consistency are central (Zhang et al., 1 Oct 2025).