CharacterShot: 4D Character Animation

• CharacterShot is a unified framework for creating continuous, spatially-consistent 4D character animations using a single reference image and a 2D pose sequence.
• It integrates advances from DiT-based diffusion, multi-view learning with dual-attention, and neighbor-constrained 4D Gaussian splatting to ensure high-quality results.
• The approach delivers state-of-the-art performance across static 3D, view-consistent video, and full 4D tasks, validated on large-scale benchmarks like Character4D and CharacterBench.

CharacterShot is a unified framework for controllable and consistent 4D character animation that enables the synthesis of dynamic, spatially-coherent 3D characters (i.e., 4D representations: 3D+time) from only a single reference image and a 2D pose sequence. It integrates advances from DiT-based diffusion models, multi-view learning, and structure-preserving 4D Gaussian splatting within a cohesive pipeline designed for high-quality character animation and view-consistent representation. The approach achieves state-of-the-art results across static 3D, view-consistent video, and spatiotemporal 4D tasks, as validated on the large-scale Character4D and challenging CharacterBench benchmarks (Gao et al., 10 Aug 2025).

1. Problem Definition and Motivation

Character-centric animation requires generating temporally and spatially consistent models of characters across arbitrary motion sequences and viewpoints, starting from minimal user input. CharacterShot addresses the challenge of generating 4D character animations—continuous, camera-view coherent, time-resolved 3D character representations—using only a single image as appearance anchor and a 2D pose sequence as motion control, with no requirement for explicit 3D assets or full video supervision (Gao et al., 10 Aug 2025). This paradigm is crucial for animation, immersive graphics, gaming, and AR/VR, where rapid character creation, control, and rendering consistency are required.

2. Pipeline Architecture

CharacterShot employs a three-stage design:

1. 2D Character Animation Pretraining

A DiT-based image-to-video model (adapted from CogVideoX) builds the foundation for appearance- and pose-driven motion. The architecture uses a 3D VAE to compress both reference image and pose/driven video into latent spaces. Pose control is mediated via DWpose-extracted pose images, which are also encoded into latents and concatenated with appearance features. The resulting latent is patchified and processed by a denoising diffusion transformer (ε_θ). The 2D pretraining loss is the standard Gaussian noise reconstruction loss:

$$\mathcal{L}_{2D} = \mathbb{E}_{x_t, \varepsilon \sim \mathcal{N}(0, I), c, t} \left\| \epsilon_\theta(x_t, c, t) - \varepsilon \right\|^2$$

where $c$ aggregates appearance and driving pose information.

2. Lifting from 2D to 3D: Multi-View Video Generation

The model is extended for multi-view synthesis by expanding latent inputs across $V$ camera viewpoints, with each view encoding both extrinsic and intrinsic camera priors (Plücker embeddings) as tokens merged into the transformer input stream. A key innovation is the dual-attention module:

• Spatial–Temporal stream: Full-attention along the temporal-spatial axis within each view
• Spatial–View stream: Full-attention over viewpoint–spatial axis at each time step

Dual-attention outputs are fused pre-FFN, allowing direct propagation of information both temporally and across views, enforcing spatial-temporal and spatial-view consistency in outputs. The multi-view loss remains a diffusion noise-prediction objective:

$$\mathcal{L}_{MV} = \mathbb{E}_{x_t, \varepsilon, c, \pi, t} \left\| \epsilon_\theta(x_t + x_v, c, \pi, t) - \varepsilon \right\|^2$$

3. Neighbor-Constrained 4D Gaussian Splatting Optimization

4D Gaussian Splatting (4DGS) is used to consolidate and regularize the multi-view, time-resolved outputs into a continuous spatiotemporal 4D representation. Each animated point is a Gaussian parameterized by time-dependent mean, covariance, color, and opacity ($\mu_i^t, \Sigma_i^t, c_i^t, \alpha_i^t$). The field is decomposed into canonical Gaussians plus time-varying deformation modeled by a small MLP over Fourier-encoded coordinates.

A neighbor-consistency loss is introduced to maintain local structure and prevent flicker:

$$\mathcal{L}_{neighbor} = \sum_{(i,j)} w_{ij}\,m_{ij}\,\|L_i^t - L_i^{t-1}\|_2^2$$

with outlier gating based on Gaussian displacement and spatial adjacency. Further loss terms include photometric ($\ell_1$), perceptual (LPIPS), and TV regularization.

3. Data Resources: Character4D and CharacterBench

The Character4D dataset is introduced as a large-scale, character-centric 4D corpus comprising:

• 13,115 unique 3D character meshes (VRoidHub, OBJ format)
• 40 retargeted motion types (Mixamo/Rokoko)
• 21 multi-view renders per character in A-pose for static view synthesis
• Motion videos rendered at 480×720 for 4D tasks

CharacterBench augments the test split with "out-of-Character4D" examples (OOC: 2D-anime sprites, Internet images, and synthetic avatars) to probe generalization and robustness (Gao et al., 10 Aug 2025).

4. Evaluation and Quantitative Comparison

CharacterShot is evaluated on static 3D, multi-view video, and full 4D animation synthesis:

Task	Metric	Best Baseline	CharacterShot
3D Multi-view (A-pose)	SSIM	0.922 (Hi3D)	0.945
	LPIPS	0.073 (Hi3D)	0.054
	FID	77.35 (Hi3D)	71.66
Multi-view video (pose)	SSIM	0.891 (SV4D)	0.967
	LPIPS	0.135 (Diff²)	0.021
	FV4D	1392.3	490.5
Full 4D anim. (post-4DGS)	SSIM	0.915 (STAG4D)	0.971
	LPIPS	0.082 (STAG4D)	0.025
	FV4D	970.2 (STAG4D)	406.6

Ablation confirms that camera priors and dual attention are critical to cross-view/time consistency (adding them raises SSIM from 0.956 to 0.967 and cuts FV4D by >400 points) (Gao et al., 10 Aug 2025).

A user study on OOC-character animations (30 raters) shows CharacterShot superior to prior art in appearance, pose accuracy, temporal coherence, and view consistency (e.g., scores of 39.24/29.01/30.33/37.05 for key axes vs SC4D’s 21.8/20.0/21.0/20.8).

5. Key Innovations

• Dual-attention transformer: Enables coordinated propagation of features both temporally and across spatial views.
• Camera-aware fusion: Plücker embeddings encode per-view geometry for optimal multi-view conditioning.
• Neighbor-constrained 4DGS: Local structure-preserving regularization guarantees temporal and spatial consistency in the evolving mesh/point cloud, addressing flicker/outlier modes in Gaussian splatting-based representations.
• Minimal input regime: Achieves high-fidelity 4D character animation from only a single reference image and pose sequence, removing the reliance on multi-view, multi-pose real world captures or artist-drawn view sheets.

6. Limitations and Prospects

Failure modes include dependency on the expressiveness of the base DiT model and pose detector, occasional artifacts in extreme motion or ambiguous pose frames, and computational overhead in 4DGS step for high frame/point densities. Extensions under consideration include generalizing to multi-character scenes, data-driven or user-guided control of camera path, and integration with interactive storytelling engines.

Applications of CharacterShot encompass animation production, virtual avatars, immersive telepresence, gaming, and AR/VR pipelines requiring swift, consistent, and controllable character generation without large-scale artist-driven asset creation.

---

References:

• "CharacterShot: Controllable and Consistent 4D Character Animation" (Gao et al., 10 Aug 2025)