AniMer+: Unified Animal Mesh Recovery

• The paper introduces AniMer+ as a unified animal mesh recovery framework that estimates pose and shape from monocular images across mammals and birds.
• It employs a high-capacity, family-aware Vision Transformer with Mixture-of-Experts and a diffusion-based pipeline to generate synthetic 3D supervision.
• Empirical results demonstrate improved performance on challenging benchmarks by overcoming data scarcity and architectural bottlenecks in multi-species reconstruction.

AniMer+ is a unified animal mesh recovery framework for monocular pose and shape estimation across Mammalia and Aves, designed to reconstruct quadruped mammals with SMAL and birds with AVES within a single network. It is presented as an extended version of the earlier AniMer framework, with two central interventions: a high-capacity, family-aware Vision Transformer incorporating a Mixture-of-Experts design, and a diffusion-based conditional image generation pipeline that supplies large-scale synthetic 3D supervision through CtrlAni3D for quadrupeds and CtrlAVES3D for birds. The stated objective is to overcome both the limited capacity of previous multi-species architectures and the scarcity of comprehensive 3D animal datasets, especially for birds, while improving out-of-domain generalization on challenging benchmarks such as Animal Kingdom (Lyu et al., 1 Aug 2025).

1. Problem setting and scope

AniMer+ addresses unified animal pose and shape estimation from a single image. The problem is framed as substantially harder than transformer-based human mesh recovery because animal anatomy varies sharply not only across families within mammals but also across taxa such as mammals and birds. The paper identifies two bottlenecks: an architectural bottleneck, in which existing backbones lack sufficient capacity to model large anatomical diversity within one network, and a supervision bottleneck, in which large-scale 3D annotations are scarce, particularly for birds (Lyu et al., 1 Aug 2025).

The framework is explicitly not a video generator, motion generator, or animation synthesis system. A common misconception is to read the name “AniMer+” as referring to animation production. In fact, the system is a parametric reconstruction method: it predicts pose, shape, and camera parameters for animal body models from monocular imagery. Its mammalian branch targets SMAL-based quadrupeds, while its avian branch targets AVES-based birds. The paper positions this as the first unified animal mesh recovery method spanning both mammalia and aves within one model (Lyu et al., 1 Aug 2025).

This design goal has practical relevance for biological and ecological analysis, where accurate pose and shape reconstruction across many species is needed for quantitative study. A plausible implication is that AniMer+ aims to supply a single backbone for cross-species spatial intelligence rather than maintaining isolated species-specific models.

2. Family-aware architecture and representation learning

AniMer+ inherits the earlier AniMer design principle of family-aware representation learning and extends it with a ViT-MoE encoder. The overall system comprises a ViT-MoE encoder, a Transformer decoder, a predictor head for family-aware contrastive learning, and a regression head for parametric outputs (Lyu et al., 1 Aug 2025).

In the original AniMer formulation, an input image $I \in \mathbb{R}^{H \times W \times 3}$ together with a learnable class token is encoded into image tokens $\mathbf{F} \in \mathbb{R}^{192 \times 1280}$. The class token carries family information, and a decoded feature vector $\boldsymbol{f} \in \mathbb{R}^{1 \times 1024}$ is mapped by MLP heads to pose, shape, and camera parameters. AniMer+ modifies the encoder by partitioning the second feed-forward layer of each ViT block into a taxa-shared branch and taxa-specific branches. The shared branch is trained across all taxa, while the specific branch is selected according to whether the input belongs to mammals or birds. The outputs are concatenated to restore the original hidden size, with the dimensions reported as:

• $$\mathbf{F}_{\text{taxa-shared}} \in \mathbb{R}^{193 \times 960}$$
• $$\mathbf{F}_{\text{taxa-specific}} \in \mathbb{R}^{193 \times 320}$$
• $$\mathbf{F}_{\text{block}} \in \mathbb{R}^{193 \times 1280}$$

The paper is explicit that this is not a sparse-gating MoE with learned top-$k$ routing. Instead, it is a structured split in which taxa identity determines the specific branch, and the output is fused by concatenation. This architectural distinction matters because it clarifies that AniMer+ uses expert partitioning to reduce feature interference between mammalian and avian anatomy rather than learned token-level routing (Lyu et al., 1 Aug 2025).

A second representation-learning mechanism is the family-aware supervised contrastive loss applied to the class token. The objective is written as:

$$\mathbf{F}_{\text{taxa-specific}} \in \mathbb{R}^{193 \times 320}$$3

Here, $P(i)$ denotes same-family positives, $O(i)$ the other samples in the minibatch, and $\tau$ the temperature. The stated purpose is to encourage intra-family compactness and inter-family separability in feature space, which the paper argues is more beneficial for geometric regression than a plain classification loss (Lyu et al., 1 Aug 2025).

3. Synthetic data generation: CtrlAni3D and CtrlAVES3D

A major contribution of AniMer+ is its attempt to address the shortage of 3D annotations through diffusion-based conditional image generation. The method constructs two synthetic datasets: CtrlAni3D for quadrupeds and CtrlAVES3D for birds (Lyu et al., 1 Aug 2025).

For CtrlAni3D, the pipeline samples a posed SMAL mesh and a camera viewpoint, renders mask and depth maps, and uses these as ControlNet conditions. Shape parameters $\mathbf{F} \in \mathbb{R}^{192 \times 1280}$0 are sampled from the Gaussian SMAL shape space, and pose parameters $\mathbf{F} \in \mathbb{R}^{192 \times 1280}$1 are sampled from a combined dog-and-horse pose prior. Global rotation is sampled uniformly from $\mathbf{F} \in \mathbb{R}^{192 \times 1280}$2 in each dimension, and camera position is sampled uniformly between $\mathbf{F} \in \mathbb{R}^{192 \times 1280}$3 and $\mathbf{F} \in \mathbb{R}^{192 \times 1280}$4. The paper states that this deliberate viewpoint randomness can create some truncation and thereby improve robustness. Text prompts are formed from a common species name and a pose description such as “stands,” after which ChatGPT expands them into richer behavioral sentences. The final images are filtered with SAM2.0 and manual verification (Lyu et al., 1 Aug 2025).

For CtrlAVES3D, the process is adapted to the bird domain. The paper identifies three differences relative to CtrlAni3D: it uses Flux-Union-Pro-V2 ControlNet instead of the ControlNet variant used for quadrupeds; it samples the AVES bone-length parameter $\mathbf{F} \in \mathbb{R}^{192 \times 1280}$5 in $\mathbf{F} \in \mathbb{R}^{192 \times 1280}$6; and because the Flux-based ControlNet does not support mask conditioning, it uses Canny edges and depth maps as structural conditions instead of a mask (Lyu et al., 1 Aug 2025).

Each synthetic sample includes both parametric labels and keypoints:

• SMAL: $\mathbf{F} \in \mathbb{R}^{192 \times 1280}$7, $\mathbf{F} \in \mathbb{R}^{192 \times 1280}$8, $\mathbf{F} \in \mathbb{R}^{192 \times 1280}$9, plus 26 3D keypoints
• AVES: $\boldsymbol{f} \in \mathbb{R}^{1 \times 1024}$0, $\boldsymbol{f} \in \mathbb{R}^{1 \times 1024}$1, $\boldsymbol{f} \in \mathbb{R}^{1 \times 1024}$2, $\boldsymbol{f} \in \mathbb{R}^{1 \times 1024}$3, plus 18 3D keypoints

Visible 2D keypoints are derived by projecting 3D joints and comparing joint depth $\boldsymbol{f} \in \mathbb{R}^{1 \times 1024}$4 to the rendered depth $\boldsymbol{f} \in \mathbb{R}^{1 \times 1024}$5, with a keypoint marked visible when $\boldsymbol{f} \in \mathbb{R}^{1 \times 1024}$6. This makes the synthetic data usable for both 3D supervision and 2D keypoint training (Lyu et al., 1 Aug 2025).

The dataset aggregation reported for training is summarized below.

Taxon	Dataset pool	Images
Mammals	Animal3D, CtrlAni3D, Animal Pose, AwA-Pose, Zebra Synthetic, Stanford Extra, APT-36K	41,332
Birds	CtrlAVES3D, CUB	12,428

The paper emphasizes that CtrlAVES3D is the first large-scale, 3D-annotated dataset for birds, and treats this as crucial for resolving single-view depth ambiguities in avian reconstruction (Lyu et al., 1 Aug 2025).

4. Objective functions, supervision, and training recipe

AniMer+ combines 3D supervision, 2D supervision, priors, and contrastive learning in a single objective:

$$\mathbf{F}_{\text{taxa-specific}} \in \mathbb{R}^{193 \times 320}$$4

The reported weights are:

• $\boldsymbol{f} \in \mathbb{R}^{1 \times 1024}$7
• $\boldsymbol{f} \in \mathbb{R}^{1 \times 1024}$8
• $\boldsymbol{f} \in \mathbb{R}^{1 \times 1024}$9
• $$\mathbf{F}_{\text{taxa-shared}} \in \mathbb{R}^{193 \times 960}$$0
• $$\mathbf{F}_{\text{taxa-shared}} \in \mathbb{R}^{193 \times 960}$$1

The 3D term is given as:

$$\mathbf{F}_{\text{taxa-specific}} \in \mathbb{R}^{193 \times 320}$$5

with $$\mathbf{F}_{\text{taxa-shared}} \in \mathbb{R}^{193 \times 960}$$2, $$\mathbf{F}_{\text{taxa-shared}} \in \mathbb{R}^{193 \times 960}$$3, and for birds $$\mathbf{F}_{\text{taxa-shared}} \in \mathbb{R}^{193 \times 960}$$4.

The 2D term is:

$$\mathbf{F}_{\text{taxa-specific}} \in \mathbb{R}^{193 \times 320}$$6

with $$\mathbf{F}_{\text{taxa-shared}} \in \mathbb{R}^{193 \times 960}$$5.

The SMAL prior uses a Mahalanobis distance to the SMAL mean and covariance:

$$\mathbf{F}_{\text{taxa-specific}} \in \mathbb{R}^{193 \times 320}$$7

and the AVES prior is:

$$\mathbf{F}_{\text{taxa-specific}} \in \mathbb{R}^{193 \times 320}$$8

The training procedure is stage-wise for AniMer: a first stage uses only 3D data to establish feasible pose and shape prediction, and a second stage adds all 3D and 2D data. The avian-only model AniMer-A is instead trained in a single stage because bird pose and shape are described as less diverse and faster to converge (Lyu et al., 1 Aug 2025).

The reported implementation details include PyTorch, AdamW, linear learning rate decay, and an initial learning rate of $$\mathbf{F}_{\text{taxa-shared}} \in \mathbb{R}^{193 \times 960}$$6. All experiments are said to run on a single NVIDIA RTX 4090. The ViT encoder is initialized from ViTPose++ pretrained weights, which the paper identifies as important for performance. Sampling across datasets is explicitly non-uniform: for example, Animal3D receives weight $$\mathbf{F}_{\text{taxa-shared}} \in \mathbb{R}^{193 \times 960}$$7, CtrlAni3D $$\mathbf{F}_{\text{taxa-shared}} \in \mathbb{R}^{193 \times 960}$$8, CtrlAVES3D and CUB $$\mathbf{F}_{\text{taxa-shared}} \in \mathbb{R}^{193 \times 960}$$9, and smaller 2D datasets often $$\mathbf{F}_{\text{taxa-specific}} \in \mathbb{R}^{193 \times 320}$$0 or $$\mathbf{F}_{\text{taxa-specific}} \in \mathbb{R}^{193 \times 320}$$1 (Lyu et al., 1 Aug 2025).

5. Empirical performance and ablation evidence

The paper reports strong results on both mammalian and avian benchmarks, including in-domain and out-of-domain settings (Lyu et al., 1 Aug 2025).

On Animal3D, the unified AniMer+ model achieves:

• 88.9 AUC
• 93.0 PCK@HTH
• 78.0 PA-MPJPE
• 82.7 PA-MPVPE

On CtrlAni3D, AniMer+ reports:

• 93.6 AUC
• 95.9 PCK@HTH
• 47.8 PA-MPJPE
• 50.8 PA-MPVPE

On the out-of-domain Animal Kingdom benchmark, AniMer+ reaches:

• 82.6 AUC
• 92.9 PCK@HTH
• 34.2 [email protected]
• 53.0 [email protected]

The paper further states that, relative to 3D Fauna on Animal3D, AniMer-M and AniMer+ improve PA-CD by 68.2% and 67.4% respectively, even where [email protected] is comparable. This is used to argue that template-based regression achieves better geometric fidelity even when keypoint accuracy appears similar (Lyu et al., 1 Aug 2025).

On birds, AniMer+ also surpasses prior baselines on CtrlAVES3D, CUB, and Cow Bird. The reported AniMer+ results are:

Benchmark	AUC	PCK@HTH	[email protected]	[email protected]
CtrlAVES3D	93.0	96.3	—	—
CUB	89.5	97.7	59.4	80.5
Cow Bird	59.3	47.2	16.4	28.4

The paper notes an important nuance: AniMer-A can slightly exceed AniMer+ on some real bird benchmarks because its full capacity is dedicated to avian reconstruction. This qualification is significant because it shows that the unified model is not claimed to dominate every taxon-specific specialist in every regime; rather, its contribution lies in strong joint performance across taxa (Lyu et al., 1 Aug 2025).

The ablation studies support three claims. First, synthetic data matter: adding CtrlAni3D improves Animal Kingdom AUC from 78.5 to 82.7 and [email protected] from 24.5 to 33.8 in the reported setups, and also lowers Animal3D errors. For birds, adding CtrlAVES3D improves Cow Bird AUC from 53.4 to 62.7 and [email protected] from 11.3 to 16.6 (Lyu et al., 1 Aug 2025).

Second, the architecture matters: replacing the ViT with ResNet-152, removing pretraining, or replacing the Transformer decoder with an MLP all reduce performance. Third, the family-aware contrastive loss improves both mammal and bird results, and the paper reports tighter family clusters in t-SNE visualizations when $$\mathbf{F}_{\text{taxa-specific}} \in \mathbb{R}^{193 \times 320}$$2 is used. The MoE extension also yields gains over a fully shared mixed-taxa model (AniMer-AM) at the same parameter count, suggesting that taxa-specific branches reduce destructive interference in multi-taxa training (Lyu et al., 1 Aug 2025).

6. Position within adjacent animal modeling research

AniMer+ belongs to the animal modeling literature but occupies a distinct niche. It is a single-image reconstruction method focused on pose and shape estimation, not text-driven motion generation, not image animation, and not avatar appearance editing (Lyu et al., 1 Aug 2025).

This makes it complementary to several neighboring lines of work. SMAL-pets combines SMAL with 3D Gaussian Splatting to generate editable dog avatars from a single image, emphasizing animatability and text-guided appearance refinement rather than unified multi-taxa reconstruction. Its hybrid design suggests a possible downstream use case for accurate parametric predictions of the kind AniMer+ produces, especially in pipelines where structured pose and shape estimates must support later avatar animation (Borycki et al., 17 Mar 2026).

At a broader motion level, X-MoGen studies text-driven motion generation across humans and animals through a unified latent representation and a morphology-aware generation module. Although its task is generative rather than reconstructive, it shares with AniMer+ the premise that cross-species modeling requires explicit treatment of morphology rather than naïve pooling across species (Wang et al., 7 Aug 2025).

A plausible implication is that AniMer+ can be understood as a perception-side counterpart to these generative systems: it focuses on recovering anatomically grounded state variables from images, whereas neighboring work addresses motion synthesis, avatar rendering, or cross-species latent generation. The common technical theme across these works is that animal diversity cannot be handled robustly without structured modeling of anatomy, morphology, or taxonomic variation.

7. Limitations, interpretation, and significance

The paper is explicit that AniMer+ still struggles with heavy occlusion, strong motion blur, and extreme poses. For birds, it further notes that the current AVES model is not expressive enough to represent flight poses well, so bird-in-flight reconstruction remains a failure mode. The synthetic pipeline is also imperfect: ControlNet may sometimes generate samples misaligned with the prompt or the intended 3D structure, so the utility of synthetic data still depends on filtering (Lyu et al., 1 Aug 2025).

These limitations clarify the scope of the contribution. AniMer+ does not solve general animal reconstruction under arbitrary dynamics; rather, it advances the field by showing that a unified multi-species approach can be made practical if three conditions are met: a sufficiently strong pretrained transformer backbone, explicit family-aware or taxa-aware representation structuring, and synthetic 3D supervision that narrows the gap left by real-world data scarcity (Lyu et al., 1 Aug 2025).

Its significance therefore lies less in any single module than in the joint design of architecture and data. The shared branch of the ViT-MoE learns cross-species commonalities, the taxa-specific branches preserve anatomy-specific detail, the contrastive objective regularizes feature geometry at the family level, and the diffusion-generated synthetic datasets expand 3D supervision where real annotations are sparse. This suggests a broader design pattern for animal vision: unified modeling is feasible, but only when representation capacity, supervision scale, and anatomical structure are co-designed rather than treated independently.