- The paper presents a novel framework, R-DMesh, that rectifies pose misalignment in video-guided 3D animation with a hierarchical VAE and explicit jump offset modeling.
- It introduces a triflow attention mechanism that couples motion representation with mesh topology, ensuring smooth and coherent dynamic sequences.
- Empirical evaluations show enhanced rendering alignment, temporal consistency, and geometric accuracy with significantly faster inference than prior methods.
R-DMesh: Rectified Dynamic Mesh Flow for Video-Guided 3D Animation
Problem Statement and Motivation
Video-guided 3D animation enables precise and intuitive manipulation of 3D assets by leveraging reference videos to control the motion dynamics of static meshes. However, deployment of such systems in open-world scenarios is fundamentally hampered by the pose misalignment dilemma: the initial pose of a user-provided mesh typically does not coincide with the starting frame of a reference video. Forcing direct motion transfer in these unaligned cases results in severe geometric distortions and animation failures. Existing frameworks either ignore this issue or address it through costly, ad hoc optimization that severely limits scalability and versatility.
The authors introduce "Rectified Dynamic Mesh" (R-DMesh), a unified framework explicitly designed to generate high-fidelity, video-aligned 4D mesh sequences by resolving pose misalignment in a principled and scalable manner (2605.13838). The approach is complemented by a new large-scale dataset, Video-RDMesh, designed to rigorously simulate pose misalignment scenarios.
Methodological Contributions
Disentangled Latent Representation and VAE
R-DMesh leverages a hierarchical VAE architecture that decomposes the mesh sequence into four independent modalities: mesh faces, initial vertex positions, a jump offset (the pose rectification vector), and relative motion trajectories. The jump offset is critical—it encodes the residual displacement required to align the initial pose of the static mesh to the first frame of the driving video. This design ensures that large-scale temporal discontinuities due to initial pose mismatch do not contaminate the learning and generation of smooth motion trajectories. The VAE employs separate centering/norm strategies for the mesh and the sequence, further stabilizing training.
Triflow Attention Mechanism
The Triflow Attention mechanism is devised to coordinate information flow between geometry, pose rectification, and motion. Rather than stacking separate attentions per modality, a shared vertex-wise attention map is constructed, tightly coupling the motion representation to mesh topology and facilitating efficient encoding of local rigidity and motion coherence. This mechanism supports robust disentanglement and enables high-fidelity dynamic mesh reconstruction even for sequences with significant pose jumps.
For generative modeling, R-DMesh introduces a Rectified Flow (RF) Transformer network. During training, the RF operates on stochastic interpolations between clean and noisy latent trajectories, conditioned on both the static mesh latent and semantic features distilled from a frozen, large-scale video diffusion model (VDM). The video features are extracted mid-transformer-stack rather than from final layers, balancing global temporal context and low-level spatial cues. Temporal control is injected via AdaLN-Zero modulators, stabilizing residual learning and enabling flexible classifier-free guidance during sampling. This design ensures efficient, physically consistent animation driven directly by video, with reduced inference time compared to optimization- or template-based baselines.
Large-Scale Dataset: Video-RDMesh
The training procedure leverages a novel dataset containing over 500,000 dynamic sequences curated from Objaverse, with explicit simulation of pose misalignment. Each data point is a paired sample of a reference video and a dynamic mesh sequence, facilitating supervised training of both pose rectification and motion transfer.
Experimental Results
Empirical evaluation is performed on the Video-RDMesh test sets containing both ground truth and generated reference videos, spanning diverse categories (humanoids, animals, inanimate objects).
- Rendering Alignment (PSNR): R-DMesh attains 25.8, outperforming prior state-of-the-art video-to-4D and mesh animation methods.
- Temporal Consistency (Subject Consistency/SC., Motion Smoothness/SM.): Achieves 0.949 and 0.995, respectively, the highest among all baselines.
- Geometric Accuracy (Euclidean Distance): Error of 0.012 versus 0.035 from the strongest mesh-based prior, enabling more precise shape and motion transfer.
- Inference Time: Approximately 10 seconds per sample, dramatically faster than optimization-based approaches (often ≫30 minutes) and comparable with feedforward foundational models.
Qualitative analysis demonstrates that R-DMesh maintains subject identity and motion coherence even under severe pose misalignment, where other methods fail to initialize or exhibit severe deformations.
Ablation and Analysis
Ablations confirm the necessity of each architectural component. Removing the jump decomposition causes initial frame pose to dominate the latent space, preventing proper motion learning. Disabling Triflow Attention results in entangled and less structured flows, degrading fidelity. Feature extraction ablations show mid-layer VDM latents yield optimal conditioning; concatenating multiple layers is suboptimal, likely due to mixing inconsistent abstraction levels.
Applications and Generalization
R-DMesh enables several advanced downstream tasks:
- Pose Retargeting: Robustly transfers arbitrary poses, even from real images, to static input meshes, exploiting explicit jump offset modeling.
- Motion Retargeting: Animates any target mesh given video-driven motion, requiring no fine-tuning or skeletal correspondence.
- Holistic Video-to-4D Generation: When combined with modern 3D asset generation models, enables end-to-end pipelines turning real-world videos into temporally consistent, animatable 4D assets across domains.
Despite these strengths, limitations remain in handling mesh interpenetration (heritage of noisy training data) and rare category generalization due to training set bias.
Theoretical and Practical Implications
The rectification paradigm established here—explicitly disentangling pose alignment from motion—improves generalization and transfer across unconstrained asset categories, going beyond parametric or skeletal supervision. By incorporating pre-trained video diffusion priors and a flow-based sequence generator, the framework bridges video-driven and mesh-based animation at scale. This advance opens a path toward generalized, feedforward 4D asset creation that can be robustly controlled by arbitrary video, lowering manual curation barriers for high-fidelity digital content pipelines.
Future avenues include learning explicit collision avoidance for self-intersecting meshes, improved generalization to rare or composite objects via data augmentation and self-supervised pretraining, and extension to fully open-world, multi-object scenarios with dynamic topologies. The Video-RDMesh dataset further provides a benchmark for subsequent work on misalignment-resilient 4D generative models.
Conclusion
R-DMesh establishes a scalable, efficient, and versatile architecture for video-guided 3D animation under realistic, open-world pose distribution shift (2605.13838). Its contribution resides in a principled rectification pipeline—combining explicit jump modeling, topology-aware attention, and video-driven flow-based diffusion generation—validated both quantitatively and qualitatively. This framework significantly enhances the generality, alignment, and speed of 4D mesh animation, setting a new reference point for video-driven content generation research and its deployment in production.