DynaVieW: Schema-Guided World Modeling for Understanding Hierarchical Visual Dynamics

Abstract: Multimodal LLMs struggle to systematically model the temporal evolution of visual scenes in videos or multi-image sequences. Such inputs require models to predict or simulate multiple levels of dynamic constituents, such as actions taken in the visual sequence, and the associated changes to the visual environment that result. To address this challenge, we propose a dynamic schema-guided world model, DynaVieW, optimized for visual dynamic prediction and simulation. DynaVieW achieves an in-depth understanding of visual dynamics by learning interleaved state-transition sequences, where states cover broad visual scenes from video keyframes, and transitions capture comprehensive dynamic constituents within a hierarchical schema. DynaVieW jointly models transition prediction and state simulation under a mixture-of-experts architecture, with a cross-expert selective attention and a schema token re-weighted loss, to ensure effective and robust learning. DynaVieW's understanding of visual dynamics boosts its downstream performance in visual narrative creation and world simulation, showing improved consistency, controllability, and instruction-following.

• The paper introduces DynaVieW, a schema-guided architecture that models hierarchical visual dynamics by decomposing transitions into high-level activities and fine-grained actions.
• It employs a Mixture-of-Transformer-Experts with selective attention and a structured JSON schema to robustly predict state transitions and simulate dynamic visual scenes.
• Evaluations demonstrate improved cross-scene consistency, controllability, and style fidelity in visual narrative and instruction-following tasks compared to prior models.

Schema-Guided Hierarchical Visual World Modeling with DynaVieW

Motivation and Problem Formulation

DynaVieW addresses the systematic deficiency of multimodal LLMs and VLMs in modeling the hierarchical temporal evolution of visual scenes across sequential data, such as videos or multi-image narratives. Existing models primarily focus on static vision-language fusion or coarse-grained interleaved vision-text modeling, failing to incorporate structured, multi-level understanding and simulation of dynamic constituents (e.g., activities, fine-grained spatial transformations, low-level actions). This deficiency results in limited consistency, controllability, and instruction-following in generated visual narratives and simulated worlds.

Data Construction and Hierarchical Transition Schema

DynaVieW's pre-training utilizes a novel interleaved state-transition dataset constructed from diverse source videos (Ego4D, AgiBotWorld-Alpha, ShareGPT4Video). Keyframes are extracted using adaptive intervals targeting maximal image sharpness (Laplacian variance) and minimal clip embedding similarity, ensuring both high visual quality and temporal distinctness.

Figure 1: Keyframe selection and transition annotation pipeline for DynaVieW data construction, maximizing scene variation and annotation accuracy.

Transitions between keyframes are annotated through an oracle VLM (InternVL-78B-Instruct) using a hierarchical JSON schema. This schema decomposes transitions into high-level activities, constituent sub-activities, atomic actions, and associated transformations (object introduction/removal/persistence, state changes, spatial relation shifts, actions, motion, camera/background dynamics). Contributions of transformations are further analyzed at the atomic, sub-activity, and temporal levels, enabling a structured representation of causal and temporal dynamics. This formulation supports comprehensive and granular modeling for both prediction and simulation tasks.

Figure 2: Example annotated hierarchical JSON schema transition capturing high-level, mid-level, and fine-grained dynamic elements.

Model Architecture and Training Objectives

DynaVieW adopts a Mixture-of-Transformer-Experts (MoT) foundation, extending BAGEL with explicit joint modeling of transition prediction and state simulation. Visual states are encoded both via ViT (SigLIP2 with NaViT preservation) for contextual understanding and via VAE (FLUX) for diffusion-based simulation. Shared multimodal self-attention unites world understanding and generation experts at every transformer layer.

Figure 3: Architecture of DynaVieW with MoT, multimodal selective attention, and dual objectives for state simulation and structured transition prediction.

Selective attention masking is applied to restrict attention flows in interleaved sequences—transitions only attend to preceding states, while states attend to the last transition and prior states, with dropout for historical contexts to avoid overfitting. This prevents erroneous copying and promotes robust causal modeling.

Transition prediction utilizes a schema token re-weighted CE loss, reducing overfitting on the static schema structure and enhancing slot-value learning. State simulation employs velocity prediction with diffusion MSE loss on VAE encodings. Both objectives are alternated with equal weighting during training.

Quantitative Validation

Evaluation with GPT-4o as a judge on 900 validation samples demonstrates high reliability—transition rejection rates for DynaVieW do not exceed 2% across activities, transformations, and contributions, comparable to gold references. Style consistency scores of simulated states average above 8.5/10, with transformation rejection rates below 5%. DynaVieW produces richer transformation diversity at slightly higher recall, confirming effective schema learning and robust visual dynamic generation.

Downstream Task Performance

Visual Narrative Generation

On VinaBench, DynaVieW outperforms Emu2, BAGEL, and Story2Board in zero-shot generalization and achieves highest or second-best scores post fine-tuning. Major gains are observed in cross-scene consistency (average score: 0.630 zero-shot, 0.610 SFT) and location fidelity, attributed to schema-driven modeling of spatial dynamics. Human preference evaluation corroborates these findings, favoring DynaVieW for storyboard coherence and fidelity.

Controllability via Transition Steerability

Experiments with auxiliary JSON-schema perturbations show DynaVieW is highly controllable, achieving average scores of 0.604 versus 0.283 for BAGEL under model-generated schema prompts. The hierarchical modeling enables robust translation of high-level constraints to image sequences, maintaining low performance variance even under noisy conditions.

Instruction-Following in World Simulation

On LEGO benchmark (Ego4D, Epic-Kitchens), DynaVieW exhibits robust instruction-following across all metrics: FID reduction (20.96 post-SFT, a 19.8% improvement), CLIP score advantages, and superior EgoVLP alignment. Image-to-text evaluations (BLIP-B and BLIP-L) further validate semantic fidelity. DynaVieW shows consistent gains over BAGEL and prior baselines across both generalization and supervised settings.

Figure 4: Comparative world simulation outputs from DynaVieW and BAGEL against LEGO ground truth.

Ablation Study

Ablation experiments substantiate the impact of major components. Removing multimodal selective attention, schema re-weighting, or hierarchical JSON-schema results in significant drops in image-text alignment and cross-scene consistency (up to -0.064 average score for coarse-grained transitions). Fine-grained schema granularity is critical for style and character consistency across sequences.

Figure 5: Performance degradation from removing key DynaVieW components, indicating necessity of hierarchical schema and selective attention.

Practical and Theoretical Implications

DynaVieW's schema-guided modeling demonstrates that explicit hierarchical representations of visual dynamics are essential for reliable, consistent, and controllable multimodal world modeling. Practically, this framework enables agentic simulation, visual narrative creation, instruction-followed state transitions, and planning via hierarchical visual reasoning. Theoretically, DynaVieW supports new directions in world modeling—incorporating fine-grained schema-driven causal structures, scalable dynamic sequence modeling, and transferability to downstream tasks (robotics, embodied agents, creative AI, simulation).

Future Directions

Further developments may include scaling DynaVieW with extended token budgets for longer sequence modeling, integration with agentic LLMs for planning/interaction, broader cross-domain pre-training (games, social, medical), and expansion to multi-agent simulation environments. The explicit compositional schema also opens avenues for disentangling latent factors of dynamic worlds and interpretable planning.

Conclusion

DynaVieW introduces a schema-guided architecture for hierarchical world modeling of visual dynamics. The structured approach yields improved consistency, controllability, and instruction-following across narrative and simulation tasks, demonstrating the necessity of explicit hierarchical dynamic modeling for multimodal LM foundations. These findings suggest future multimodal world models should adopt comprehensive, schema-driven training regimes to achieve reliable and agentic simulation capabilities.

Figure 6: DynaVieW's schema-guided world modeling task formulation via interleaved state-transition sequences.