Fast 4D Mesh Generation by Spatio-Temporal Attention Chains

Abstract: 4D mesh generation has recently emerged as a powerful paradigm for recovering dynamic 3D structure from videos, but existing methods remain slow, computationally expensive, and difficult to scale to longer sequences. We introduce a training-free approach that accelerates 4D mesh generation while improving temporal correspondence quality. Our key observation is that temporal correspondences emerge inside a 4D backbone long before its generated meshes become visually accurate. We exploit this with a general framework we call Spatio-Temporal Attention Chain which propagates information across space and time. Starting from vertices on an anchor mesh, the chain maps vertices to latent tokens. It then follows temporal correspondences in latent space, and recovers frame-specific vertices through latent-to-vertex attention. This design avoids expensive explicit matching while preserving anchor mesh details and thereby improving dynamic mesh geometry and temporal consistency. Compared to state-of-the-art, our method generates a 4D mesh in 9 seconds, achieving a $13\times$ speedup while producing higher-quality results. Moreover, our approach scales to videos up to $16\times$ longer without degrading mesh quality. Beyond generation, the improved correspondences enable competitive zero-shot performance on two downstream tasks: 2D object tracking and 4D tracking. We further show that our framework enables reliable camera estimation, a capability not supported by prior 4D mesh generation methods.

• The paper introduces a training-free spatio-temporal attention chain that establishes dense vertex correspondences across frames for rapid 4D mesh reconstruction.
• The approach achieves a 13× speedup over prior methods and superior geometric metrics, ensuring stable, high-quality meshes over extended video sequences.
• The method enables zero-shot 2D and 4D tracking along with reliable camera pose estimation, bridging pixel, latent, and mesh domains for dynamic scene understanding.

Fast 4D Mesh Generation via Spatio-Temporal Attention Chains

Introduction and Motivation

The paper "Fast 4D Mesh Generation by Spatio-Temporal Attention Chains" (2605.19786) presents a novel methodology for 4D mesh reconstruction from video, addressing the fundamental challenge of generating temporally consistent, high-fidelity 3D meshes efficiently over extended video sequences. Previous approaches rely on staged pipelines with optimization or learned deformation networks, which are computationally expensive and often restricted to short temporal windows. These pipelines suffer from drift, require significant 4D supervision, and lack explicit correspondences linking output meshes to input video pixels, thereby limiting practical downstream tasks such as tracking and camera pose estimation.

Methodology

The central innovation is the Spatio-Temporal Attention Chain, a training-free framework that leverages emergent correspondences within a frozen 4D generative backbone. The attention chain composes a sequence of spatial and temporal attention maps to establish dense correspondences from anchor-frame mesh vertices to target-frame surface points:

• Vertex-to-Token Attention: $V_a \rightarrow Z_a$, where anchor mesh vertices query latent tokens explaining geometry.
• Token-to-Token Temporal Attention: $Z_a \rightarrow Z_f$, where anchor-frame tokens are mapped to target-frame tokens across time.
• Token-to-Surface Attention: $Z_f \rightarrow V_f$, where target-frame tokens are decoded to candidate surface points.

This composition enables efficient transport of geometry across frames, directly animating the anchor mesh without explicit deformation networks or optimization.

Figure 1: Method overview illustrating the attention chain path from anchor mesh vertices to latent tokens, temporal correspondence, and recovery of target mesh vertices; endpoints allow extension to 2D tracking, camera estimation, and 4D tracking.

Temporal coherence is further reinforced during denoising, boosting stable correspondences to mitigate drift during long-sequence autoregressive rollout. Mesh animation is accomplished by tracking sparse control landmarks, filtering and smoothing their trajectories, and propagating motion to the dense mesh via geodesic-rigid skinning. This strictly preserves anchor topology, avoiding artefacts common in linear blend skinning.

Scaling and Stability in Long Sequences

The attention chain framework supports autoregressive extension to videos up to $16\times$ longer than prior methods, with robust prevention of accumulated drift. Correspondence reinforcement during denoising maintains strong latent correlations, preserving mesh quality over hundreds of frames. Compared to naive autoregressive rollout that leads to rapid degradation, the proposed reinforcement mechanism delivers stable mesh geometry and pose throughout extended sequences.

Figure 2: Long-sequence rollout; naive autoregressive methods accumulate errors, whereas correspondence reinforcement stabilizes mesh quality across time.

Quantitative and Qualitative Evaluation

4D Mesh Generation

On the ActionBench dataset, the method achieves a generation time of 9 seconds per 16-frame clip, representing a $13\times$ speedup over ActionMesh, and surpasses state-of-the-art methods in CD-3D ($0.048$), CD-4D ($0.077$), and Normal Consistency ($0.97$). The approach is competitive on CD-Motion, and camera-aligned renderings outperform all non-aligned baselines on Consistent4D, especially when augmented with pose estimation.

Figure 3: 4D mesh generation; the method produces sharp, temporally consistent, and camera-aligned meshes with high foreground overlap and minimal mismatch.

Side-by-side qualitative comparisons confirm sharper geometry, reduced temporal distortion, and improved silhouette alignment relative to ActionMesh. The method integrates generated meshes into reconstructed 3D scenes using dense 2D-to-3D correspondences, supporting novel-view rendering.

Figure 4: Video-to-4D scene alignment; generated meshes are placed accurately in reconstructed environments via dense correspondences.

Additional visual results demonstrate consistently higher mesh quality on diverse sequences and robustness in long-sequence generation.

Figure 5: Additional mesh generation results; the method yields temporally consistent geometry across a range of inputs.

Figure 6: Long-sequence mesh generation; quality remains stable over 240 frames, with ActionMesh losing recognizable structure in later frames.

2D and 4D Point Tracking

Attention chain correspondences naturally extend to point tracking. Zero-shot 2D tracking using composed image patch attention matches or exceeds supervised benchmarks on foreground objects, notably outperforming DiffTrack and Denoise-to-Track. On articulated motion benchmarks such as BADJA, the method approaches or surpasses the best supervised trackers.

Figure 7: Zero-shot 2D point tracking; attention-chain tracks outperform state-of-the-art zero-shot trackers, especially for articulated objects.

World-coordinate 4D point tracking utilizes dense camera-aligned correspondences established via robust PnP solvers; the approach closes the gap with supervised trackers and decisively improves over ActionMesh baselines.

Component Analysis and Ablations

Ablation studies show robust operation with only four denoising steps, yielding state-of-the-art metrics at a fraction of the computational cost. The attention-chain correspondence outperforms learned Stage II networks on CD-3D and CD-4D, although learned methods have slightly smoother frame-to-frame motion (CD-M). Speedup arises both from reduced denoising steps and replacement of networks with analytic operations. Component analysis on long sequences confirms that each design, particularly camera pose estimation, significantly reduces reconstruction and motion errors.

Figure 8: CD-3D, CD-4D, and CD-Motion vs. denoising steps; the method saturates performance rapidly and outpaces baselines across metrics.

Practical and Theoretical Implications

The proposed attention-chain mechanism delivers efficient, topology-consistent, and temporally stable 4D mesh generation from videos, eliminating the need for additional training, deformation networks, or optimization. Direct extraction of correspondences enables competitive zero-shot 2D/4D tracking and reliable camera pose recovery, transforming 4D mesh outputs into practical assets for downstream multi-modal tasks. The methodology bridges pixel, latent, and mesh domains, facilitating robust world-space integration.

On the theoretical side, the framework demonstrates that emergent spatial-temporal correspondences in diffusion backbones can be formalized as compositional Markov-like chains, unlocking latent transport for tracking, animation, and alignment. This advances the understanding of attention flow within transformers for 4D generation and can inform models in geometric vision, video editing, and embodied AI.

Limitations remain: mesh quality is bounded by the underlying frozen generators; rigid skinning may damp fine motion; and extended autoregressive rollouts still accumulate subtle alignment errors. Further research can explore adaptive temporal regularization and integrate advanced geometric priors.

Conclusion

The Spatio-Temporal Attention Chain delivers substantial advances in 4D mesh generation, achieving order-of-magnitude acceleration, superior geometric metrics, and unlocking zero-shot tracking and camera alignment capabilities—all without additional training or networks. The framework directly leverages attention-based correspondence signals within frozen generative backbones, paving the way for scalable, practical dynamic scene understanding and mesh generation.