ActionMesh: Temporal 3D Mesh Generation

• ActionMesh is a generative framework for animated 3D mesh generation that leverages temporal diffusion techniques and autoencoding for smooth, topology-consistent animations.
• It employs an inflated self-attention mechanism and multi-modal conditioning (video, text, static meshes) to synchronously generate coherent mesh sequences without relying on traditional rigging.
• Extensive evaluations on Objaverse and Consistent4D benchmarks demonstrate improved runtime and lower Chamfer distances, offering a robust alternative to auto-rigged methods.

ActionMesh is a generative modeling framework for animated 3D mesh generation that employs temporal 3D diffusion techniques to deliver high-quality, production-ready, topology-consistent mesh animations in a fully feed-forward manner. Designed to address limitations of prior 4D synthesis methods—such as slow runtime, rig-dependency, or compromised quality—ActionMesh advances the field by synchronously generating temporally coherent animated mesh sequences from diverse input modalities including monocular video, text prompts, or static 3D meshes. The core architecture combines a temporally-extended 3D latent diffusion model with a temporal autoencoder for explicit animation synthesis, enabling rapid, rig-free, and artifact-minimized motion creation (Sabathier et al., 22 Jan 2026).

1. Temporal 3D Diffusion: Architectural Overview

ActionMesh extends single-frame 3D latent diffusion to video by adding a temporal axis to the denoising and representation process. The foundational model—based on the latent diffusion approach in 3DShape2VecSet or TripoSG—leverages a pretrained VAE (encoder $E$, decoder $D$) that projects 3D meshes into compact latent spaces $z$. Rather than denoising one frame at a time, ActionMesh utilizes a diffusion transformer $G$ to jointly denoise an entire sequence of $N$ frame-latents $\{z_1, \ldots, z_N\}$ in a single forward pass.

The key innovation is the "inflated self-attention" mechanism, which flattens the latent representation across both temporal (sequence) and spatial (token) dimensions. Let $X \in \mathbb{R}^{N \times T \times D}$ denote the latent tokens of $N$ frames. The process

• reshapes $X$ to $\mathbb{R}^{1 \times (N\cdot T) \times D}$,
• applies standard (pretrained) self-attention (“sattn”),
• unflattens to restore the original structure:

$D$0

where rotary positional embeddings on the frame axis encode relative timing, enforcing temporal smoothness without explicit rigging. During training, a random subset $D$1 of the sequence is provided as noise-free latent context (“masked generation”), facilitating conditioning on arbitrary reference frames or multi-modal input.

The diffusion loss is a rectified-flow-matching objective, structurally identical to those used in rectified-flow-based models, e.g.,

$D$2

with $D$3 being the continuous flow time.

2. Temporal 3D Autoencoder and Latent-to-Mesh Decoding

To translate time-varying latent sequences into vertex-level mesh deformation, ActionMesh employs a temporal 3D autoencoder as a second stage. Each mesh in the generated sequence is encoded to a point-cloud latent $D$4, and a reference mesh $D$5 is established (typically from the first frame).

The autoencoder’s decoder operates as follows:

• Accepts the entire latent sequence $D$6, plus two time tokens $D$7 with Fourier encoding.
• Inflated self-attention is used to provide temporal context across frames.
• For each vertex $D$8 of $D$9, forms a query $z$0 based on position, normal, and time parameters.
• Cross-attends these queries to the sequence context to predict $z$1 (per-vertex displacement from source to target frame).
• At inference, reconstructs full animation by sweeping $z$2 through the timeline.

Training minimizes the $z$3 displacement regression loss:

$z$4

This formulation ensures frame-to-frame topological consistency and smooth deformation across the animation.

3. Conditioning, Representations, and Multi-Modal Inputs

The pipeline supports various conditioning modalities, all leveraging cross-attention in the diffusion transformer $z$5:

• Video to 4D: Each video frame $z$6 is mapped by a frozen DINOv2 encoder to per-frame features $z$7.
• {3D+video} $z$8 4D: Known mesh $z$9 is encoded as $G$0 and provided as static, noise-free context at any position in the sequence.
• {Image+text} $G$1 4D: Text is used to generate a short video via off-the-shelf text-to-video models (e.g., Make-A-Video), then processed as above.
• {3D+text} $G$2 4D: The 3D mesh is rendered, paired with a text-driven video, and the pipeline proceeds via mesh conditioning and injected video context.

Relative frame positions are encoded via rotary embeddings. During training, $G$3 frames are used; longer sequences can be generated autoregressively.

4. Implementation: Architectures, Training, and Efficiency

ActionMesh uses a TripoSG latent DiT backbone with approximately 12 transformer blocks. Training and inference employ T=1024 or 2048 tokens, hidden dimensionality $G$4, and FlashAttention2 to improve efficiency of inflated attention. The autoencoder architecture utilizes the same (frozen) VecSet encoder and a decoder comprised of inflated self-attention and cross-attention layers.

Training uses AdamW ($G$5, $G$6, bfloat16) on clusters of A100 or V100 GPUs (∼300 GPU hours total) with batch size 96 across 170k steps per stage. The dataset comprises 13,200 sequences from Objaverse, Objaverse-XL, and internal sources, each providing 16 RGB renders and dense aligned point clouds (up to 500k points with normals).

Voxelized occupancy prediction is used for mesh extraction, with marching cubes on $G$7 grids typically yielding $G$8–$G$9 vertices per mesh.

5. Evaluation: Benchmarks, Metrics, and Results

Evaluation leverages the Objaverse (quantitative) and Consistent4D (qualitative) benchmarks. Metrics include:

• CD-3D: Frame-wise Chamfer distance post-ICP alignment.
• CD-4D: Sequence-wise Chamfer distance with consistent alignment.
• CD-M: Framewise motion Chamfer with propagated correspondences. Lower values indicate superior geometric and temporal consistency. ActionMesh demonstrates substantial gains over prior methods on all metrics, as summarized below:

Method	Inference Time	CD-3D ↓	CD-4D ↓	CD-M ↓
LIM	15 min	0.095	0.127	0.258
DM4D	35 min	0.095	0.140	0.247
V2M4	35 min	0.063	0.223	0.500
Ours	3 min	0.050	0.069	0.137

On Consistent4D and DAVIS, ActionMesh preserves fine details and stable topology, handles challenging motions (e.g., animal locomotion, object deformation), and sustains performance on in-the-wild data. Multi-modal demos include both text-conditioned animation (e.g., “an octopus playing maracas”) and motion retargeting (e.g., transferring bird motion to a dragon).

6. Limitations and Areas for Future Research

Despite its capabilities, ActionMesh exhibits constraints:

• Topology preservation: Fixed mesh connectivity is assumed throughout the sequence; the model cannot synthesize topological changes such as emerging/disappearing limbs or punctured surfaces.
• Severe occlusion sensitivity: If regions are persistently invisible or occluded in video/context, reconstructions in those areas may be inaccurate or exhibit hallucinations.

Potential directions include topology-aware latent modeling, occlusion-robust training via multi-view or generative priors, and step-distillation for real-time animated mesh generation (Sabathier et al., 22 Jan 2026).

7. Context and Relationship to Other Methods

ActionMesh operates in contrast to auto-rigged or skeleton-based approaches (e.g., ViPS, Puppeteer), as its denoising and deformation are topology- and rig-free. Methods such as ViPS build pose spaces by distilling video-diffusion priors onto kinematic rigs, enabling semantic constraints and differentiable geometric validation (Chen et al., 19 Apr 2026). ActionMesh, by comparison, enforces geometric and temporal consistency via latent-space diffusion and per-vertex decoding into mesh displacement, optimizing for production readiness and minimal manual intervention. This highlights a distinction between latent mesh-geometry diffusion (ActionMesh) and pose-space diffusion over rigged models (ViPS). Both frameworks, however, leverage advances in video diffusion for animation, feature feed-forward synthesis, and prioritize scalability to broad asset domains and input types.