CtrlAVES3D: Synthetic Avian 3D Dataset

• CtrlAVES3D is a large-scale synthetic bird dataset with 3D annotations designed to support accurate avian mesh recovery.
• It leverages the AVES parametric bird model and employs a robust generation pipeline with text prompt conditioning and advanced post-processing.
• CtrlAVES3D mitigates single-view depth ambiguity in monocular reconstruction, enhancing unified training across mammalian and avian models.

Searching arXiv for the provided topic and closely related systems to ground the article in current papers. CtrlAVES3D is a large-scale synthetic bird dataset with 3D annotations introduced in the AniMer+ framework for unified mammal-and-bird pose-and-shape estimation (Lyu et al., 1 Aug 2025). In that context, it is built around the AVES parametric bird model and is intended to mitigate the scarcity of avian 3D supervision, especially for monocular reconstruction, where the paper identifies single-view depth ambiguity as a central difficulty (Lyu et al., 1 Aug 2025). The term also invites potential confusion with earlier immersive-visualization work on CAVE/AVE systems, but the 2016 CAVE2 spectral-cube paper explicitly does not use “CtrlAVES3D” anywhere in its text; it instead describes a CAVE2-based visualization framework with PRD nodes, a server node, and a web client (Vohl et al., 2016). As a result, CtrlAVES3D is best understood primarily as the avian dataset introduced in AniMer+, while related AVE/CAVE papers remain relevant chiefly for terminological disambiguation and conceptual contrast (Lyu et al., 1 Aug 2025, Vohl et al., 2016).

1. Terminology, scope, and disambiguation

In AniMer+, CtrlAVES3D is defined as a new synthetic, diffusion-generated, 3D-annotated bird dataset for avian mesh recovery (Lyu et al., 1 Aug 2025). The paper describes it as “the first large-scale avian dataset with 3D annotations” and reports 6,773 images in total, with 6,464 images used as the training subset in the aggregated training setting (Lyu et al., 1 Aug 2025). Its role is parallel to CtrlAni3D for quadrupeds, but specialized to birds through the AVES model rather than SMAL (Lyu et al., 1 Aug 2025).

The term should not be conflated with earlier CAVE/AVE visualization systems. The CAVE2 paper, "An interactive, comparative and quantitative 3D visualization system for large-scale spectral-cube surveys using CAVE2" (Vohl et al., 2016), describes a comparative spectral-cube environment with a web-based controller, but the paper text explicitly does not confirm “CtrlAVES3D” as the formal name of either the controller or the whole framework. A related but distinct precedent is Multiverse, an application launcher and three-dimensional desktop environment for a CAVE-type immersive VR system (Kageyama et al., 2013). Another distinct line of work uses Augmented Virtual Environment (AVE) to denote geospatially grounded immersive environments assembled from mobile imagery and open-source spatial data; that paper likewise does not use CtrlAVES3D as a system name (Beale et al., 17 Sep 2025).

This suggests that CtrlAVES3D is not a generic label for AVE/CAVE control platforms. Within the supplied literature, the only explicit use of the term is the avian dataset in AniMer+ (Lyu et al., 1 Aug 2025).

2. Dataset definition and scientific motivation

CtrlAVES3D was introduced because bird reconstruction lacked strong 3D supervision. The AniMer+ paper states that existing avian datasets such as CUB provide only 2D annotations, and that there had been no large-scale 3D-annotated bird dataset before CtrlAVES3D (Lyu et al., 1 Aug 2025). This matters because the AVES model requires supervision not only for pose and shape, but also for bone-length parameters (Lyu et al., 1 Aug 2025).

The paper ties the dataset directly to the monocular reconstruction problem. It states that CtrlAVES3D is crucial for “mitigating the depth ambiguity issue [in] single-view reconstruction tasks” (Lyu et al., 1 Aug 2025). The need is intensified by bird-specific structure: the AVES model contains articulated geometry and explicit bone scaling, and bird limbs, necks, tails, and body proportions vary strongly with viewpoint (Lyu et al., 1 Aug 2025).

Within AniMer+, CtrlAVES3D functions as the avian counterpart of CtrlAni3D. The paper reports the following pair: CtrlAni3D: 9,711 images and CtrlAVES3D: 6,773 images (Lyu et al., 1 Aug 2025). Together with real datasets, these synthetic resources support training on an aggregated collection of 41.3k mammalian and 12.4k avian images, combining real + synthetic data (Lyu et al., 1 Aug 2025). A plausible implication is that CtrlAVES3D is not merely an auxiliary corpus, but part of the data foundation that makes unified cross-taxa training operational.

3. Parametric basis, annotations, and generation pipeline

CtrlAVES3D is built on the AVES parametric model, which the paper defines as

$\mathcal{M}(\beta, \theta, \alpha, \gamma),$

with $\beta \in \mathbb{R}^{15}$ for bird shape, $\theta \in \mathbb{R}^{25 \times 3}$ for pose, $\alpha \in \mathbb{R}^{24}$ for bone-length scaling, and $\gamma \in \mathbb{R}^{3}$ for translation (Lyu et al., 1 Aug 2025). Given $(\beta,\theta,\alpha)$, AVES outputs vertices $V \in \mathbb{R}^{8210 \times 3}$, faces $F \in \mathbb{N}^{12468 \times 3}$, and joints $J \in \mathbb{R}^{25 \times 3}$ (Lyu et al., 1 Aug 2025). The inclusion of $\alpha$ distinguishes AVES from SMAL and is central to the dataset’s bird-specific supervision regime.

The generation process is described as a three-part pipeline: text prompt generation, condition image generation, and image generation and post-processing (Lyu et al., 1 Aug 2025). For birds, the paper specifies three major differences from CtrlAni3D. First, instead of the early ControlNet version used by CtrlAni3D, CtrlAVES3D employs Flux-Union-Pro-V2, described as a modern version ControlNet that generates more realistic bird textures (Lyu et al., 1 Aug 2025). Second, the pipeline samples AVES-specific parameters, including the bone-length parameter, and states: “We sample $\beta \in \mathbb{R}^{15}$0 within $\beta \in \mathbb{R}^{15}$1 according to AVES's practice.” (Lyu et al., 1 Aug 2025). Third, because Flux-Union-Pro-V2 does not support mask as a condition, the structural controls are Canny edge and depth map rather than mask-plus-depth (Lyu et al., 1 Aug 2025).

Text conditioning is also explicit. The prompts use common names rather than scientific names, a species keyword, a pose or behavior keyword, and a completed natural-language sentence generated by ChatGPT (Lyu et al., 1 Aug 2025). The paper denotes structural prompt input by $\beta \in \mathbb{R}^{15}$2 and text prompt by $\beta \in \mathbb{R}^{15}$3, which act jointly as conditions to the controllable generator (Lyu et al., 1 Aug 2025).

The post-processing stage is semi-automated. The authors use SAM2 to extract the foreground mask from each generated image, enabling cycle-consistency checking, and then manually filter out images that do not match the mesh poses (Lyu et al., 1 Aug 2025). Two failure modes are explicitly identified: the generator may fail to align the pose with the mesh, or may fail to render intricate animal-body details well (Lyu et al., 1 Aug 2025). This establishes CtrlAVES3D as a curated synthetic dataset rather than an unfiltered generative dump.

4. Dataset contents, taxonomy, and supervision

CtrlAVES3D contains 17 bird categories/species and 6,773 images in total (Lyu et al., 1 Aug 2025). The paper enumerates the per-category counts exactly.

Species/category	Images
Laysan albatross	501
Cardinal	447
Northern flicker	441
Boat tailed grackle	334
California gull	394
Green kingfisher	487
Mallard	287
Geococcyx	305
Pileated woodpecker	374
Painted bunting	494
American crow	414
Scissor tailed flycatcher	264
Evening grosbeak	363
Blue jay	370
White breasted kingfisher	501
Horned puffin	330
Cedar waxwing	467

The paper states that the training subset used in the aggregated training mix contains 6,464 images (Lyu et al., 1 Aug 2025). It does not provide a full benchmark-style train/val/test partition for CtrlAVES3D (Lyu et al., 1 Aug 2025).

Each sample is paired with pixel-aligned AVES model labels and derived geometric annotations (Lyu et al., 1 Aug 2025). The annotations include the parametric variables $\beta \in \mathbb{R}^{15}$4, 18 keypoints for AVES, and visible 2D keypoints obtained by projection and depth-based visibility reasoning (Lyu et al., 1 Aug 2025). The visibility rule is given explicitly: if $\beta \in \mathbb{R}^{15}$5 is the depth of a projected keypoint and $\beta \in \mathbb{R}^{15}$6 is the rendered depth at that pixel, then visibility is $\beta \in \mathbb{R}^{15}$7 if $\beta \in \mathbb{R}^{15}$8, else $\beta \in \mathbb{R}^{15}$9 (Lyu et al., 1 Aug 2025). The dataset is also described as being annotated “in the same style as Animal3D” (Lyu et al., 1 Aug 2025).

The camera model used downstream is a weak-perspective projection function,

$\theta \in \mathbb{R}^{25 \times 3}$0

where $\theta \in \mathbb{R}^{25 \times 3}$1 is a 3D point, $\theta \in \mathbb{R}^{25 \times 3}$2 is translation, $\theta \in \mathbb{R}^{25 \times 3}$3 is a fixed intrinsic matrix, and $\theta \in \mathbb{R}^{25 \times 3}$4 converts homogeneous coordinates to pixel coordinates (Lyu et al., 1 Aug 2025). In this setting, CtrlAVES3D provides a consistent bridge between AVES-space supervision and 2D image-space constraints.

5. Role within AniMer+ training and architecture

AniMer+ extends AniMer to jointly handle mammalia and aves through a family-aware Vision Transformer (ViT) with a Mixture-of-Experts (MoE) design (Lyu et al., 1 Aug 2025). Within that system, CtrlAVES3D supplies the bird-side 3D labels required for training the avian branch and for stabilizing joint learning across anatomically distinct taxa (Lyu et al., 1 Aug 2025).

The paper reports that the full training dataset contains 53,760 images and includes both CtrlAVES3D and CUB as avian components (Lyu et al., 1 Aug 2025). CtrlAVES3D contributes 6,464 images, 12.0% of the total, with a sample weight of 0.45, while CUB contributes 5,964 images, 11.1%, also with sample weight 0.45 (Lyu et al., 1 Aug 2025). The authors state that only CtrlAVES3D has 3D annotations among the avian training sources (Lyu et al., 1 Aug 2025).

CtrlAVES3D serves two roles that are explicitly distinguished in the paper: direct 3D supervision for bird reconstruction and taxon balancing in unified training (Lyu et al., 1 Aug 2025). This second role is important because, without enough bird 3D data, the mammal side could dominate the shared representation (Lyu et al., 1 Aug 2025).

The dataset also interacts directly with the model’s family-aware representation learning. AniMer/AniMer+ use a learnable class token for animal-family supervised contrastive learning, with the loss

$\theta \in \mathbb{R}^{25 \times 3}$5

Here $\theta \in \mathbb{R}^{25 \times 3}$6 are samples sharing the same family label, $\theta \in \mathbb{R}^{25 \times 3}$7 are other samples, and $\theta \in \mathbb{R}^{25 \times 3}$8 is the temperature (Lyu et al., 1 Aug 2025). Because CtrlAVES3D adds coherent avian geometry at scale, it provides the bird-side support for this family-aware signal.

Its role is even more explicit in the MoE encoder. The paper partitions the second FC layer in each ViT block into one taxa-shared layer and two taxa-specific layers, one for mammalia and one for aves, and gives the block equations: $\theta \in \mathbb{R}^{25 \times 3}$9 with $\alpha \in \mathbb{R}^{24}$0, $\alpha \in \mathbb{R}^{24}$1, and $\alpha \in \mathbb{R}^{24}$2 (Lyu et al., 1 Aug 2025). A plausible implication is that the aves-specific expert pathway would be far weaker without CtrlAVES3D’s 3D supervision.

6. Losses, empirical effects, and limitations

The AniMer+ paper defines the total loss as

$\alpha \in \mathbb{R}^{24}$3

with weights $\alpha \in \mathbb{R}^{24}$4, $\alpha \in \mathbb{R}^{24}$5, $\alpha \in \mathbb{R}^{24}$6, $\alpha \in \mathbb{R}^{24}$7, and $\alpha \in \mathbb{R}^{24}$8 (Lyu et al., 1 Aug 2025). For bird data such as CtrlAVES3D, the 3D loss includes direct supervision of $\alpha \in \mathbb{R}^{24}$9, $\gamma \in \mathbb{R}^{3}$0, 3D keypoints, and $\gamma \in \mathbb{R}^{3}$1: $\gamma \in \mathbb{R}^{3}$2 with $\gamma \in \mathbb{R}^{3}$3, $\gamma \in \mathbb{R}^{3}$4, and for birds $\gamma \in \mathbb{R}^{3}$5 (Lyu et al., 1 Aug 2025).

Empirically, CtrlAVES3D has a substantial reported effect. On the in-domain CtrlAVES3D benchmark, the main comparison table gives the following values: AVES achieves AUC 85.9, PCK@HTH 88.3, PA-MPJPE 86.5, PA-MPVPE 93.9; AniMer-A reaches AUC 92.9, PCK@HTH 96.4, PA-MPJPE 65.5, PA-MPVPE 90.2; and AniMer+ reaches AUC 93.0, PCK@HTH 96.3, PA-MPJPE 65.6, PA-MPVPE 70.9 (Lyu et al., 1 Aug 2025). The large drop in PA-MPVPE from 93.9 to 70.9 is presented in the source summary as especially indicative of improved mesh recovery (Lyu et al., 1 Aug 2025).

The ablation isolating CtrlAVES3D is more direct. On Cow Bird, training with CUB only gives AUC 53.4, PCK@HTH 39.3, [email protected] 11.3, [email protected] 19.9; training with CtrlAVES3D only gives AUC 62.7, PCK@HTH 51.3, [email protected] 16.6, [email protected] 30.9; and training with CUB + CtrlAVES3D gives AUC 65.7, PCK@HTH 56.7, [email protected] 19.4, [email protected] 31.3 (Lyu et al., 1 Aug 2025). The paper states that adding CtrlAVES3D to CUB boosts AUC from 53.4 to 65.7, an absolute increase of 12.3, reported as 18.7% (Lyu et al., 1 Aug 2025). This is evidence that the dataset is complementary to real 2D bird data rather than a substitute for it.

The paper also reports limitations. ControlNet’s species-specific performance remains uneven, and some generated birds may not correspond to the intended prompt (Lyu et al., 1 Aug 2025). The dataset therefore depends on SAM2-based filtering and manual verification (Lyu et al., 1 Aug 2025). Taxonomic coverage is restricted to 17 bird categories, and the AVES model itself “lacks the expressiveness to accurately represent complex avian articulations, such as those during flight.” (Lyu et al., 1 Aug 2025). These caveats indicate that CtrlAVES3D is a strong enabling resource within its design envelope, but not a complete representation of avian diversity or kinematics.

7. Relationship to immersive visualization and AVE/CAVE literature

The name CtrlAVES3D may appear superficially aligned with AVE/CAVE interface systems, but the supplied literature supports a stricter distinction. The 2016 CAVE2 paper presents a comparative 3D spectral-cube framework for astronomy, capable of simultaneous visualization of $\gamma \in \mathbb{R}^{3}$6 spectral-cubes, using 80 stereo-capable displays, 84 million pixels, and $\gamma \in \mathbb{R}^{3}$7 TFLOPS of integrated GPU power (Vohl et al., 2016). Its architecture consists of PRD nodes, a server node, and a web client, with S2PLOT used for volumetric rendering and quantitative products such as moment maps and histograms (Vohl et al., 2016). Yet that paper explicitly does not use the term CtrlAVES3D (Vohl et al., 2016).

Similarly, Multiverse is an immersive application launcher and three-dimensional desktop environment for a CAVE-type immersive VR system, composed of World and Universes, and allowing the user to jump back to World and switch to another Universe at any time from any Universe (Kageyama et al., 2013). This makes Multiverse conceptually close to a 3D control shell, but it remains terminologically and functionally distinct from CtrlAVES3D as used in AniMer+ (Kageyama et al., 2013, Lyu et al., 1 Aug 2025).

A third, separate AVE lineage is represented by the 2025 paper on constructing Augmented Virtual Environments (AVEs) from mobile phone images, OpenStreetMap (OSM), and Digital Terrain Models (DTM), implemented via Python, Unity, and UDP-based two-way communication (Beale et al., 17 Sep 2025). That system addresses geospatial AVE assembly, projector calibration, and object detection, rather than 3D-annotated avian training data (Beale et al., 17 Sep 2025).

Taken together, these papers show that “CtrlAVES3D” should not be generalized into a broad AVE/CAVE systems label. Within the supplied record, its precise referent is the synthetic avian dataset introduced for AniMer+ (Lyu et al., 1 Aug 2025). The AVE/CAVE papers are best read as adjacent literature on immersive environments, visualization frameworks, and control metaphors rather than as sources defining CtrlAVES3D itself (Vohl et al., 2016, Kageyama et al., 2013, Beale et al., 17 Sep 2025).