CtrlAni3D: Synthetic Data & 3D Animation

• CtrlAni3D comprises a cluster of artifacts including a synthetic quadruped dataset and distinct 3D-aware animation systems.
• The dataset leverages SMAL annotations and diffusion-generated imagery to provide photorealistic, pixel-aligned training data across varied species.
• The animation frameworks decouple geometric control from appearance synthesis to enable precise, interactive, and temporally consistent motion rendering.

Searching arXiv for the cited papers to ground the article. CtrlAni3D is a term used in recent arXiv literature for multiple, technically distinct artifacts. Most commonly, it denotes a synthetic, photorealistic, SMAL-annotated quadruped dataset introduced to support animal pose and shape estimation (Lyu et al., 2024, Lyu et al., 1 Aug 2025). In a separate graphics context, it denotes a single-image framework for precise, interactive, 3D-aware animation via a 2D–3D aligned proxy embedding (Zhu et al., 17 Dec 2025). The provided material also applies the name to a scene-adaptive human image animation system with camera movement, termed 3STC-HIA in the paper (Liu et al., 29 Jun 2026). The shared label therefore refers not to a single canonical benchmark or method, but to a small cluster of works centered on controllable animation and 3D-aware reconstruction.

1. Referential scope and nomenclature

Across the cited works, the same name is attached to three different research objects. The dataset sense is the one that recurs across animal pose and mesh recovery papers, while the graphics-framework senses arise in single-image animation and human image animation.

Referent	Technical role	Source
CtrlAni3D dataset	Synthetic benchmark for monocular 3D animal pose estimation and quadruped mesh recovery	(Lyu et al., 2024, Lyu et al., 1 Aug 2025, Wang et al., 5 Feb 2026, Yu et al., 1 Jun 2026)
CtrlAni3D framework in "3DProxyImg"	Precise, interactive 3D-aware animation from a single image via 2D–3D aligned proxy embedding	(Zhu et al., 17 Dec 2025)
CtrlAni3D in the provided summary of 3STC-HIA	Scene-adaptive, trajectory-controllable human image animation with camera movement	(Liu et al., 29 Jun 2026)

This naming overlap matters because the dataset and the animation systems solve different tasks, use different representations, and are evaluated with different metrics. A common misconception is to treat CtrlAni3D as a single benchmark-method package. The cited literature does not support that reading: one line of work uses CtrlAni3D as a data resource for SMAL-based animal reconstruction, while another uses it as a method name for controllable animation.

2. CtrlAni3D as single-image 3D-aware animation

In "3DProxyImg: Controllable 3D-Aware Animation Synthesis from Single Image via 2D-3D Aligned Proxy Embedding," CtrlAni3D is defined as “precise, interactive 3D-aware animation from a single image via 2D–3D aligned proxy embedding” (Zhu et al., 17 Dec 2025). The task is to synthesize a temporally coherent sequence $\{\hat{I}_t\}$ from a single input image $I$, user controls $C$, and a camera trajectory $\{K, R(t), t(t)\}$ such that the output preserves identity and appearance, exhibits 3D-consistent parallax and occlusions, and obeys geometric control signals with high fidelity.

The method is organized around a quality–control trade-off. Classical 3D pipelines provide precise, interactive control, but require accurate geometry, rigging, and expert labor; image-to-video diffusion can render high-quality frames, but usually treats motion as latent dynamics and limits explicit 3D control. CtrlAni3D addresses this by decoupling geometric control from appearance synthesis. Its central representation is a 2D–3D aligned proxy: a coarse, aligned 3D structural carrier formed from a pixel-aligned point cloud from VGGT and a full mesh from Hunyuan3D, aligned by ICP and further refined by mask projection and Laplacian regularization.

The structural carrier is converted into a sparse triangulated proxy mesh with vertices $V=\{v_i\}$, faces $F$, and learnable per-vertex texture features $f_i \in \mathbb{R}^d$ with $d=16$. A lightweight decoder $D_\theta$ is an 8-layer MLP of width 128 that maps encoded features to RGB. The projection model is explicit: for intrinsics $K$, extrinsics $I$0, and homogeneous 3D point $I$1,

$I$2

written compactly as $I$3. Because features are attached to 3D vertices and projected into 2D, 3D manipulations of the proxy induce view-consistent appearance even when the geometry is only coarse.

Foreground and background are explicitly disentangled. The foreground is rendered from the driven proxy geometry, while a separate background proxy layer covers the image and fills motion-revealed regions through a local latent code. Compositing uses

$I$4

with coherent occlusion ordering because both layers are rendered in the same camera.

3. Proxy control, optimization, and rendering behavior

CtrlAni3D supports two geometric control modes. The first is interactive manipulation through position-based dynamics (PBD), where handle vertices are constrained to user-specified positions or trajectories and constraints such as edge-length preservation, local rigidity, and handle attachment are enforced by iterative projections. The second is rig-driven motion through automatic rigging and weighting, exemplified by Puppeteer, followed by linear blend skinning:

$I$5

Motion may be user-provided, retargeted from a motion library, or generated from text, with AnyTop mentioned as an example source in the provided details (Zhu et al., 17 Dec 2025).

Rendering is proxy-based and differentiable. Faces are depth-sorted by a Z-buffer; per-pixel features are barycentrically interpolated; positional encoding $I$6 with $I$7 bands is applied to those features; and the MLP decoder produces color. The appearance model is learned with a reference-view photometric loss and a novel-view Score Distillation Sampling objective using SD3 as the 2D prior. The total loss is

$I$8

with $I$9 and $C$0 in the reported experiments. Optimization uses Adam with learning rate $C$1 for 1000 iterations; texture and consistency training is reported on $C$2RTX 3090, after which per-frame synthesis is lightweight because it reduces to rasterization, feature interpolation, and compact decoding (Zhu et al., 17 Dec 2025).

The reported comparisons emphasize controllability rather than only perceptual realism. Against 3DIT, DragDiffusion, and Image Sculpting in 3D editing, CtrlAni3D attains SSIM $C$3 versus $C$4, LPIPS $C$5 versus $C$6, and CLIP-I $C$7 versus $C$8, while also supporting free-form 3D rotation. Qualitative comparisons on articulated subjects such as dolls and birds and deformables such as fish and snakes indicate improved identity preservation, temporal stability, and control over joints or parts. Ablations attribute performance to shape alignment, positional encoding, and the reference-view loss; removing alignment degrades geometric fidelity, removing positional encoding blurs high-frequency textures, and removing $C$9 harms identity and consistency. The reported limitations are geometry inaccuracies from imperfect single-view reconstruction, weaker semantics in large disocclusions, and the need for denser proxies under extreme fast motion.

4. CtrlAni3D as a synthetic quadruped dataset

In the animal reconstruction literature, CtrlAni3D is a large-scale synthetic dataset created to provide image-realistic quadruped training data with pixel-aligned SMAL annotations (Lyu et al., 2024, Lyu et al., 1 Aug 2025). AniMer describes it as “a novel large-scale synthetic dataset created through a new diffusion-based conditional image generation pipeline” and states that it consists of 9,711 synthetic images, described in the main paper as “about 10k,” each at resolution $\{K, R(t), t(t)\}$0 (Lyu et al., 2024). AniMer+ presents the same dataset as a diffusion-generated, pixel-aligned quadruped set comprising 9,711 images with SMAL labels (Lyu et al., 1 Aug 2025).

The dataset spans 10 species in 5 families. The species distribution reported by AniMer is: cat 80, lion 630, cheetah 299, tiger 280, dog 2,976, wolf 413, horse 2,228, zebra 1,460, cow 890, and hippo 455 (Lyu et al., 2024). This distribution is strongly imbalanced, with Canidae and Equidae dominating. A plausible implication is that the dataset is useful both as supervision and as a testbed for long-tail robustness, which is exactly how later works motivate their methods.

Its generation pipeline couples SMAL geometry to diffusion-based image synthesis. Shape $\{K, R(t), t(t)\}$1 is sampled from SMAL’s Gaussian shape space; pose $\{K, R(t), t(t)\}$2 is sampled from a combined quadruped pose library built from BITE and WLDO dog poses together with PFERD horse motion. Structural conditioning is obtained by rendering a SMAL mask and depth map from randomly sampled viewpoints. In AniMer and AniMer+, ControlNet is used to condition a latent diffusion model on those rendered signals together with text prompts containing species keywords and pose or behavior keywords, with ChatGPT used to complete descriptive prompts (Lyu et al., 2024, Lyu et al., 1 Aug 2025). Global rotation components are sampled uniformly in $\{K, R(t), t(t)\}$3, camera position is sampled uniformly between $\{K, R(t), t(t)\}$4 and $\{K, R(t), t(t)\}$5, and the viewpoint sampling intentionally yields truncated fields of view.

Backgrounds are diversified but controlled. For $\{K, R(t), t(t)\}$6 of images, backgrounds are sampled from COCO; the remainder use AI-generated backgrounds. Quality control combines SAM2-based cycle consistency between the extracted foreground and the conditioning mask with manual verification that removes misaligned poses or poor body detail. The cited papers stress that ControlNet can drift from the intended pose or species, so filtering is part of the dataset construction rather than a post hoc cleanup only.

The annotation schema is SMAL-centric. Each sample provides $\{K, R(t), t(t)\}$7, $\{K, R(t), t(t)\}$8, and $\{K, R(t), t(t)\}$9, together with SMAL-derived geometry, 26 3D keypoints, corresponding 2D projections, and per-joint visibility (Lyu et al., 2024). Visibility is computed by depth consistency: visibility is $V=\{v_i\}$0 if the projected joint depth $V=\{v_i\}$1 at the joint pixel, otherwise $V=\{v_i\}$2. AniMer further specifies the underlying SMAL mesh as $V=\{v_i\}$3 and faces $V=\{v_i\}$4, with joints linearly regressed from vertices (Lyu et al., 2024). The training camera in AniMer and AniMer+ is weak perspective, written as

$V=\{v_i\}$5

with fixed focal length $V=\{v_i\}$6 in AniMer and fixed intrinsic matrix $V=\{v_i\}$7 (Lyu et al., 2024, Lyu et al., 1 Aug 2025). The papers do not specify file formats, explicit metadata schemas, or licensing terms in the provided text.

5. Benchmark function in animal pose and mesh recovery

CtrlAni3D functions as both training data and evaluation benchmark for monocular animal pose and mesh recovery. In AniMer, it is used in a two-stage regime: Stage 1 trains on 3D-only datasets, namely Animal3D and CtrlAni3D; Stage 2 extends to mixed 3D and 2D datasets. CtrlAni3D contributes 8,277 images to training in the aggregated corpus described by AniMer, and AniMer+ reports a sampling weight of $V=\{v_i\}$8 for CtrlAni3D during training (Lyu et al., 2024, Lyu et al., 1 Aug 2025). The direct 3D supervision on $V=\{v_i\}$9, $F$0, and 3D keypoints is reported to accelerate convergence and improve out-of-distribution generalization.

On the synthetic CtrlAni3D benchmark, AniMer reports AUC $F$1, PCK@HTH $F$2, PA-MPJPE $F$3, and PA-MPVPE $F$4, compared with WLDO at AUC $F$5, PCK@HTH $F$6, PA-MPJPE $F$7, and PA-MPVPE $F$8 (Lyu et al., 2024). AniMer also reports that including CtrlAni3D in training improves 2D metrics on Animal Kingdom and Animal Pose relative to training without CtrlAni3D. In the pretraining ablation, HMR-CtrlAni3D attains PCK@HTH $F$9 and PA-MPJPE $f_i \in \mathbb{R}^d$0, outperforming HMR-Synthetic based on CG assets at $f_i \in \mathbb{R}^d$1 and $f_i \in \mathbb{R}^d$2, respectively.

FMPose3D uses CtrlAni3D as a synthetic benchmark for monocular 3D animal pose estimation, describing it as photorealistic single-view images paired with pixel-aligned SMAL meshes and using a unified $f_i \in \mathbb{R}^d$3 SMAL-derived joint set (Wang et al., 5 Feb 2026). In that paper, the reported split is 8k training frames and 1.4k test frames, and the evaluation metric is Procrustes-aligned mean per-joint position error. FMPose3D attains P-MPJPE $f_i \in \mathbb{R}^d$4, compared with AniMer at $f_i \in \mathbb{R}^d$5, WLDO at $f_i \in \mathbb{R}^d$6, and HMR2.0 at $f_i \in \mathbb{R}^d$7. The same paper reports strong sensitivity to training set size on CtrlAni3D: $f_i \in \mathbb{R}^d$8 at 10% data, $f_i \in \mathbb{R}^d$9 at 20%, $d=16$0 at 40%, and $d=16$1 at 80%. The reported animal results use single-hypothesis prediction with $d=16$2, even though the method supports multi-hypothesis generation and a Reprojection-based Posterior Expectation Aggregation module; this is important because it distinguishes the method’s general formulation from the specific CtrlAni3D evaluation protocol.

PRIMA treats CtrlAni3D as a 3D animal mesh benchmark for unified, multi-species quadruped reconstruction and explicitly keeps evaluation feed-forward, without test-time adaptation, to preserve fairness (Yu et al., 1 Jun 2026). On CtrlAni3D, PRIMA Stage-1 reports AUC $d=16$3, PAJ $d=16$4, and PAV $d=16$5, while PRIMA Stage-3 reports AUC $d=16$6, PAJ $d=16$7, and PAV $d=16$8. Relative to AniMer, this corresponds to PAJ $d=16$9 and PAV $D_\theta$0. PRIMA’s ablations show that removing BioCLIP features or keypoint tokens yields PAJ $D_\theta$1 and PAV $D_\theta$2, and removing the bio-informed $D_\theta$3 yields PAJ $D_\theta$4 and PAV $D_\theta$5. These results frame CtrlAni3D not merely as auxiliary synthetic data, but as a stable benchmark where long-tail species imbalance, family-aware modeling, and biological priors can be quantitatively assessed.

6. Human-animation usage and cross-cutting limitations

The provided material also uses CtrlAni3D to describe a scene-adaptive human image animation framework with camera movement, although the paper itself names the method 3STC-HIA (Liu et al., 29 Jun 2026). Its task differs from the animal dataset line: given a single reference image, a target SMPL-X action sequence $D_\theta$6, a user-drawn trajectory on the $D_\theta$7–$D_\theta$8 plane, and a camera trajectory $D_\theta$9, the system synthesizes a photorealistic video in which the subject follows both human-motion and camera-motion controls. The method reconstructs a metric 3D scene point cloud, retargets the action to a ground-adaptive world-space trajectory, renders human and scene conditions per frame, and injects viewpoint-visible scene priors into a pretrained video generator through a visibility-masked latent fusion mechanism. The reported controllability metrics on Trajectory100 are Translation Error $K$0 m and Rotation Error $K$1, compared with RealisMotion at $K$2 m and $K$3, RealisDance-DiT at $K$4 m and $K$5, and Tora at $K$6 m and $K$7 (Liu et al., 29 Jun 2026).

Seen together, these works reveal several recurrent limitations. For the animal dataset, the annotation space is bounded by SMAL; AniMer explicitly notes that species far from SMAL’s space, such as pigs or goats, may not be well represented, and AniMer+ likewise notes that species SMAL cannot represent well are excluded (Lyu et al., 2024, Lyu et al., 1 Aug 2025). The species distribution is long-tailed, which later work such as PRIMA makes central to its design (Yu et al., 1 Jun 2026). A synthetic-to-real gap remains even though the diffusion pipeline improves texture and lighting realism over classical CG. For the single-image animation framework, coarse geometry can still yield texture stretch or minor mis-occlusions under severe reconstruction errors, and drastic disocclusions or extreme fast motions may require denser proxies (Zhu et al., 17 Dec 2025). For FMPose3D on animals, the camera model is not explicitly specified in the CtrlAni3D evaluation protocol, and the reported animal results do not exploit the method’s multi-hypothesis aggregation (Wang et al., 5 Feb 2026). For the human animation system, terrain extremes, sparse or erroneous point clouds, aggressive camera motion, and the absence of explicit IK, contact, or collision handling remain failure modes (Liu et al., 29 Jun 2026).

The broader significance of CtrlAni3D therefore lies less in a single unified object than in a recurring research agenda: controllable animation and 3D-aware reasoning from sparse visual input. In the dataset papers, CtrlAni3D supplies pixel-aligned synthetic supervision for quadruped reconstruction. In the graphics papers, it names or characterizes frameworks that separate geometric control from appearance synthesis or couple motion planning with scene-aware camera control. This suggests that the term has become associated with a particular design objective—explicit control under partial observation—even when the concrete technical instantiations differ substantially.