---
title: 'AniMer+: Unified Animal Mesh Recovery'
url: https://www.emergentmind.com/topics/animer
type: topic
---

# AniMer+: Unified Animal Mesh Recovery

AniMer+ is a unified animal mesh recovery framework for monocular pose and shape estimation across **Mammalia** and **Aves**, designed to reconstruct quadruped mammals with **SMAL** and birds with **AVES** within a single network. It is presented as an extended version of the earlier AniMer framework, with two central interventions: a **high-capacity, family-aware Vision Transformer** incorporating a **Mixture-of-Experts** design, and a **diffusion-based conditional image generation pipeline** that supplies large-scale synthetic 3D supervision through **CtrlAni3D** for quadrupeds and **CtrlAVES3D** for birds. The stated objective is to overcome both the limited capacity of previous multi-species architectures and the scarcity of comprehensive 3D animal datasets, especially for birds, while improving out-of-domain generalization on challenging benchmarks such as **Animal Kingdom** [2508.00298].

## 1. Problem setting and scope

AniMer+ addresses **unified animal pose and shape estimation** from a single image. The problem is framed as substantially harder than transformer-based human mesh recovery because animal anatomy varies sharply not only across families within mammals but also across taxa such as mammals and birds. The paper identifies two bottlenecks: an **architectural bottleneck**, in which existing backbones lack sufficient capacity to model large anatomical diversity within one network, and a **supervision bottleneck**, in which large-scale 3D annotations are scarce, particularly for birds [2508.00298].

The framework is explicitly **not** a video generator, motion generator, or animation synthesis system. A common misconception is to read the name “AniMer+” as referring to animation production. In fact, the system is a **parametric reconstruction method**: it predicts pose, shape, and camera parameters for animal body models from monocular imagery. Its mammalian branch targets **SMAL-based quadrupeds**, while its avian branch targets **AVES-based birds**. The paper positions this as the first unified animal mesh recovery method spanning both mammalia and aves within one model [2508.00298].

This design goal has practical relevance for biological and ecological analysis, where accurate pose and shape reconstruction across many species is needed for quantitative study. A plausible implication is that AniMer+ aims to supply a single backbone for cross-species spatial intelligence rather than maintaining isolated species-specific models.

## 2. Family-aware architecture and representation learning

AniMer+ inherits the earlier AniMer design principle of **family-aware representation learning** and extends it with a **ViT-MoE encoder**. The overall system comprises a **ViT-MoE encoder**, a **Transformer decoder**, a **predictor head** for family-aware contrastive learning, and a **regression head** for parametric outputs [2508.00298].

In the original AniMer formulation, an input image \(I \in \mathbb{R}^{H \times W \times 3}\) together with a learnable class token is encoded into image tokens \(\mathbf{F} \in \mathbb{R}^{192 \times 1280}\). The class token carries family information, and a decoded feature vector \(\boldsymbol{f} \in \mathbb{R}^{1 \times 1024}\) is mapped by MLP heads to pose, shape, and camera parameters. AniMer+ modifies the encoder by partitioning the second feed-forward layer of each ViT block into a **taxa-shared** branch and **taxa-specific** branches. The shared branch is trained across all taxa, while the specific branch is selected according to whether the input belongs to mammals or birds. The outputs are concatenated to restore the original hidden size, with the dimensions reported as:

- \(\mathbf{F}_{\text{taxa-shared}} \in \mathbb{R}^{193 \times 960}\)
- \(\mathbf{F}_{\text{taxa-specific}} \in \mathbb{R}^{193 \times 320}\)
- \(\mathbf{F}_{\text{block}} \in \mathbb{R}^{193 \times 1280}\)

The paper is explicit that this is **not** a sparse-gating MoE with learned top-\(k\) routing. Instead, it is a structured split in which taxa identity determines the specific branch, and the output is fused by concatenation. This architectural distinction matters because it clarifies that AniMer+ uses expert partitioning to reduce feature interference between mammalian and avian anatomy rather than learned token-level routing [2508.00298].

A second representation-learning mechanism is the **family-aware supervised contrastive loss** applied to the class token. The objective is written as:

```latex
\mathcal{L}_{\text{con} =
  \sum_{i \in I} \frac{-1}{|P(i)|} \sum_{p \in P(i)} \frac{\exp{(\boldsymbol{z_i} \cdot \boldsymbol{z_p} / \tau) } {\sum_{o \in O(i)} \exp{(\boldsymbol{z_i} \cdot \boldsymbol{z_o} / \tau)}.
```

Here, \(P(i)\) denotes same-family positives, \(O(i)\) the other samples in the minibatch, and \(\tau\) the temperature. The stated purpose is to encourage intra-family compactness and inter-family separability in feature space, which the paper argues is more beneficial for geometric regression than a plain classification loss [2508.00298].

## 3. Synthetic data generation: CtrlAni3D and CtrlAVES3D

A major contribution of AniMer+ is its attempt to address the shortage of 3D annotations through **diffusion-based conditional image generation**. The method constructs two synthetic datasets: **CtrlAni3D** for quadrupeds and **CtrlAVES3D** for birds [2508.00298].

For **CtrlAni3D**, the pipeline samples a posed **SMAL** mesh and a camera viewpoint, renders **mask** and **depth** maps, and uses these as **ControlNet** conditions. Shape parameters \(\beta\) are sampled from the Gaussian SMAL shape space, and pose parameters \(\theta\) are sampled from a combined dog-and-horse pose prior. Global rotation is sampled uniformly from \((-\pi,\pi)\) in each dimension, and camera position is sampled uniformly between \([-0.5,-0.5,4]\) and \([0.5,0.5,8]\). The paper states that this deliberate viewpoint randomness can create some truncation and thereby improve robustness. Text prompts are formed from a common species name and a pose description such as “stands,” after which ChatGPT expands them into richer behavioral sentences. The final images are filtered with **SAM2.0** and manual verification [2508.00298].

For **CtrlAVES3D**, the process is adapted to the bird domain. The paper identifies three differences relative to CtrlAni3D: it uses **Flux-Union-Pro-V2 ControlNet** instead of the ControlNet variant used for quadrupeds; it samples the AVES bone-length parameter \(\alpha\) in \([-0.5, 3.5]\); and because the Flux-based ControlNet does not support mask conditioning, it uses **Canny edges** and **depth maps** as structural conditions instead of a mask [2508.00298].

Each synthetic sample includes both parametric labels and keypoints:

- **SMAL**: \(\beta\), \(\theta\), \(\gamma\), plus **26 3D keypoints**
- **AVES**: \(\beta\), \(\theta\), \(\alpha\), \(\gamma\), plus **18 3D keypoints**

Visible 2D keypoints are derived by projecting 3D joints and comparing joint depth \(d_k\) to the rendered depth \(d_p\), with a keypoint marked visible when \(d_k \le d_p\). This makes the synthetic data usable for both 3D supervision and 2D keypoint training [2508.00298].

The dataset aggregation reported for training is summarized below.

| Taxon | Dataset pool | Images |
|---|---|---:|
| Mammals | Animal3D, CtrlAni3D, Animal Pose, AwA-Pose, Zebra Synthetic, Stanford Extra, APT-36K | 41,332 |
| Birds | CtrlAVES3D, CUB | 12,428 |

The paper emphasizes that **CtrlAVES3D** is the **first large-scale, 3D-annotated dataset for birds**, and treats this as crucial for resolving single-view depth ambiguities in avian reconstruction [2508.00298].

## 4. Objective functions, supervision, and training recipe

AniMer+ combines **3D supervision**, **2D supervision**, **priors**, and **contrastive learning** in a single objective:

```latex
\begin{aligned}
\mathcal{L}_{\text{total} =& \lambda_{\text{3D} \mathcal{L}_{\text{3D} + \lambda_{\text{2D} \mathcal{L}_{\text{2D} + \lambda_{\text{smal\_prior} \mathcal{L}_{\text{smal\_prior} \\
+& \lambda_{\text{con} \mathcal{L}_{\text{con} + \lambda_{\text{aves\_prior}\mathcal{L}_{\text{aves\_prior} .
\end{aligned}
```

The reported weights are:

- \(\lambda_{\text{3D}}=0.05\)
- \(\lambda_{\text{2D}}=0.01\)
- \(\lambda_{\text{smal\_prior}}=0.001\)
- \(\lambda_{\text{con}}=0.0005\)
- \(\lambda_{\text{aves\_prior}}=0.002\)

The 3D term is given as:

```latex
\begin{aligned}
\mathcal{L}_{\text{3D} =& \lambda_{\beta} ||\hat{\beta} - \beta||_{2}^{2} + \lambda_{\theta} ||\hat{\theta} - \theta||_{2}^{2} \\
+& ||\hat{K}_{3D} - K_{3D}||_{1} + \lambda_{\alpha}||\hat{\alpha}-\alpha||_{1},
\end{aligned}
```

with \(\lambda_{\beta}=0.01\), \(\lambda_{\theta}=0.2\), and for birds \(\lambda_{\alpha}=0.04\).

The 2D term is:

```latex
\mathcal{L}_{\text{2D} = ||\pi(\hat{K}_{3D}) - K_{2D}||_{1} + \lambda_{M}||\hat{\mathcal{M}-\mathcal{M}||_2^2,
```

with \(\lambda_M=2\).

The SMAL prior uses a Mahalanobis distance to the SMAL mean and covariance:

```latex
\mathcal{L}_{\text{smal\_prior} = \lambda_{\beta} (\hat{\beta} - \mu_{\beta})^{T} \Sigma^{-1}_{\beta} (\hat{\beta} - \mu_{\beta}) + (\hat{\theta} - \mu_{\theta})^{T} \Sigma^{-1}_{\theta} (\hat{\theta} - \mu_{\theta}),
```

and the AVES prior is:

```latex
\mathcal{L}_{\text{aves\_prior}=\lambda_{\beta}||\beta||_2^2+\lambda_{\theta}||\hat{\theta}-\overline{\theta}||_2^2+||\hat{\alpha}-\overline{\alpha}||_2^2.
```

The training procedure is **stage-wise** for AniMer: a first stage uses only 3D data to establish feasible pose and shape prediction, and a second stage adds all 3D and 2D data. The avian-only model **AniMer-A** is instead trained in a single stage because bird pose and shape are described as less diverse and faster to converge [2508.00298].

The reported implementation details include **PyTorch**, **AdamW**, **linear learning rate decay**, and an initial learning rate of \(1.25 \times 10^{-6}\). All experiments are said to run on a **single NVIDIA RTX 4090**. The ViT encoder is initialized from **ViTPose++** pretrained weights, which the paper identifies as important for performance. Sampling across datasets is explicitly non-uniform: for example, **Animal3D** receives weight \(1\), **CtrlAni3D** \(0.6\), **CtrlAVES3D** and **CUB** \(0.45\), and smaller 2D datasets often \(0.15\) or \(0.05\) [2508.00298].

## 5. Empirical performance and ablation evidence

The paper reports strong results on both mammalian and avian benchmarks, including in-domain and out-of-domain settings [2508.00298].

On **Animal3D**, the unified AniMer+ model achieves:

- **88.9 AUC**
- **93.0 PCK@HTH**
- **78.0 PA-MPJPE**
- **82.7 PA-MPVPE**

On **CtrlAni3D**, AniMer+ reports:

- **93.6 AUC**
- **95.9 PCK@HTH**
- **47.8 PA-MPJPE**
- **50.8 PA-MPVPE**

On the out-of-domain **Animal Kingdom** benchmark, AniMer+ reaches:

- **82.6 AUC**
- **92.9 PCK@HTH**
- **34.2 PCK@0.1**
- **53.0 PCK@0.15**

The paper further states that, relative to **3D Fauna** on Animal3D, **AniMer-M** and **AniMer+** improve **PA-CD** by **68.2%** and **67.4%** respectively, even where PCK@0.1 is comparable. This is used to argue that template-based regression achieves better geometric fidelity even when keypoint accuracy appears similar [2508.00298].

On birds, AniMer+ also surpasses prior baselines on **CtrlAVES3D**, **CUB**, and **Cow Bird**. The reported AniMer+ results are:

| Benchmark | AUC | PCK@HTH | PCK@0.1 | PCK@0.15 |
|---|---:|---:|---:|---:|
| CtrlAVES3D | 93.0 | 96.3 | — | — |
| CUB | 89.5 | 97.7 | 59.4 | 80.5 |
| Cow Bird | 59.3 | 47.2 | 16.4 | 28.4 |

The paper notes an important nuance: **AniMer-A** can slightly exceed AniMer+ on some real bird benchmarks because its full capacity is dedicated to avian reconstruction. This qualification is significant because it shows that the unified model is not claimed to dominate every taxon-specific specialist in every regime; rather, its contribution lies in strong **joint** performance across taxa [2508.00298].

The ablation studies support three claims. First, synthetic data matter: adding **CtrlAni3D** improves **Animal Kingdom** AUC from **78.5** to **82.7** and PCK@0.1 from **24.5** to **33.8** in the reported setups, and also lowers Animal3D errors. For birds, adding **CtrlAVES3D** improves **Cow Bird** AUC from **53.4** to **62.7** and PCK@0.1 from **11.3** to **16.6** [2508.00298].

Second, the architecture matters: replacing the **ViT** with **ResNet-152**, removing **pretraining**, or replacing the **Transformer decoder** with an **MLP** all reduce performance. Third, the **family-aware contrastive loss** improves both mammal and bird results, and the paper reports tighter family clusters in t-SNE visualizations when \(\mathcal{L}_{\text{con}}\) is used. The **MoE extension** also yields gains over a fully shared mixed-taxa model (**AniMer-AM**) at the same parameter count, suggesting that taxa-specific branches reduce destructive interference in multi-taxa training [2508.00298].

## 6. Position within adjacent animal modeling research

AniMer+ belongs to the animal modeling literature but occupies a distinct niche. It is a **single-image reconstruction** method focused on **pose and shape estimation**, not text-driven motion generation, not image animation, and not avatar appearance editing [2508.00298].

This makes it complementary to several neighboring lines of work. **SMAL-pets** combines **SMAL** with **3D Gaussian Splatting** to generate editable dog avatars from a single image, emphasizing animatability and text-guided appearance refinement rather than unified multi-taxa reconstruction. Its hybrid design suggests a possible downstream use case for accurate parametric predictions of the kind AniMer+ produces, especially in pipelines where structured pose and shape estimates must support later avatar animation [2603.17131].

At a broader motion level, **X-MoGen** studies **text-driven motion generation across humans and animals** through a unified latent representation and a morphology-aware generation module. Although its task is generative rather than reconstructive, it shares with AniMer+ the premise that cross-species modeling requires explicit treatment of morphology rather than naïve pooling across species [2508.05162].

A plausible implication is that AniMer+ can be understood as a **perception-side counterpart** to these generative systems: it focuses on recovering anatomically grounded state variables from images, whereas neighboring work addresses motion synthesis, avatar rendering, or cross-species latent generation. The common technical theme across these works is that animal diversity cannot be handled robustly without structured modeling of anatomy, morphology, or taxonomic variation.

## 7. Limitations, interpretation, and significance

The paper is explicit that AniMer+ still struggles with **heavy occlusion**, **strong motion blur**, and **extreme poses**. For birds, it further notes that the current **AVES** model is not expressive enough to represent **flight poses** well, so bird-in-flight reconstruction remains a failure mode. The synthetic pipeline is also imperfect: **ControlNet** may sometimes generate samples misaligned with the prompt or the intended 3D structure, so the utility of synthetic data still depends on filtering [2508.00298].

These limitations clarify the scope of the contribution. AniMer+ does not solve general animal reconstruction under arbitrary dynamics; rather, it advances the field by showing that a **unified multi-species** approach can be made practical if three conditions are met: a sufficiently strong pretrained transformer backbone, explicit **family-aware** or **taxa-aware** representation structuring, and synthetic 3D supervision that narrows the gap left by real-world data scarcity [2508.00298].

Its significance therefore lies less in any single module than in the joint design of architecture and data. The shared branch of the ViT-MoE learns cross-species commonalities, the taxa-specific branches preserve anatomy-specific detail, the contrastive objective regularizes feature geometry at the family level, and the diffusion-generated synthetic datasets expand 3D supervision where real annotations are sparse. This suggests a broader design pattern for animal vision: unified modeling is feasible, but only when representation capacity, supervision scale, and anatomical structure are co-designed rather than treated independently.

Source: https://www.emergentmind.com/topics/animer