Adding Another Dimension to Image-based Animal Detection

Abstract: Monocular imaging of animals inherently reduces 3D structures to 2D projections. Detection algorithms lead to 2D bounding boxes that lack information about animal's orientation relative to the camera. To build 3D detection methods for RGB animal images, there is a lack of labeled datasets; such labeling processes require 3D input streams along with RGB data. We present a pipeline that utilises Skinned Multi Animal Linear models to estimate 3D bounding boxes and to project them as robust labels into 2D image space using a dedicated camera pose refinement algorithm. To assess which sides of the animal are captured, cuboid face visibility metrics are computed. These 3D bounding boxes and metrics form a crucial step toward developing and benchmarking future monocular 3D animal detection algorithms. We evaluate our method on the Animal3D dataset, demonstrating accurate performance across species and settings.

• The paper develops a pipeline that uses SMAL models and anatomical keypoints to produce accurate, orientation-aware 3D bounding boxes for animals.
• The method achieves a significant drop in rotation variance and reprojection error compared to PCA-based approaches, ensuring robust detection.
• The pipeline computes profile visibility metrics and refines camera pose estimates, supporting practical wildlife monitoring and ecological studies.

Adding Another Dimension to Image-based Animal Detection: Summary and Analysis

Introduction and Motivation

This work addresses a major limitation in visual animal biometrics (VAB): the inability of image-based detection pipelines to extract and leverage 3D geometric context from monocular (RGB) imagery. Conventional 2D animal detectors output bounding boxes in image space, which occlude essential geometric information, especially the orientation of an animal relative to the camera. This lack of orientation-aware detection impairs downstream ecological analyses, individual identification, and robust behavioral tracking in both in situ conservation and livestock management.

The critical research gap stems from the absence of labeled datasets with ground truth 3D boxes for animals, as manual annotation is infeasible without auxiliary 3D modalities. This bottleneck motivates the development of a new pipeline for generating accurate, orientation-aware 3D bounding boxes and associated profile visibility metrics—using only monocular images in conjunction with Skinned Multi-Animal Linear (SMAL) models.

Figure 1: Transformation from 2D detections to oriented 3D bounding boxes, highlighting the loss of geometric context in conventional VAB pipelines.

Methodology

The proposed pipeline utilizes SMAL models as geometric priors for representing animal shapes and poses, extracting not only tight 3D bounding boxes but also consistent, semantically meaningful orientation assignments. The central steps are as follows:

1. SMAL Fitting and Keypoint Extraction: The pipeline takes an RGB image and pre-aligned SMAL mesh, leveraging 2D keypoint annotations to establish 2D–3D correspondences. The SMAL fitting inherently generates 3D keypoints through model optimization.
2. Oriented 3D Bounding Box Estimation: Rather than relying on PCA—which exhibits instability due to non-representative anatomical features and fails on symmetric or elongated bodies—the method establishes axes using reliable anatomical keypoints (anterior-posterior, left-right, dorsal-ventral), yielding consistently oriented bounding boxes resilient to keypoint noise.
3. Camera Pose Refinement: Initial pose estimates are computed with EPnP and RANSAC, then refined by modeling keypoint uncertainty through covariance and enforcing global alignment with segmentation mask constraints in a joint optimization framework. This dual-stage optimization improves robustness to noisy or partially visible keypoints.
4. Profile Visibility Computation: The final pipeline computes the visibility of each cuboid face relative to the estimated camera position, quantifying which anatomical profiles are visible in the image and at what proportion.

   Figure 3: The end-to-end pipeline, in which SMAL model fitting with 2D keypoints serves as the core for generating accurately oriented 3D bounding boxes and visibility metrics.

Comparative Evaluation

The work conducts extensive qualitative and quantitative evaluation on the Animal3D dataset and UAV-acquired imagery. Key methodological improvements are substantiated by ablation and stability tests:

• Orientation Consistency: The proposed landmark-based axis assignment achieves dramatic improvements in orientation robustness to keypoint noise. Specifically, rotation variation under Gaussian perturbations is reduced from a mean of 120.68° for PCA-based boxes to only 0.0046° with the new method. Alignment stability metrics also outperform PCA by a wide margin, with the mean deviation dropping from 0.5034 (PCA) to near zero, enabling reliable interpretation of animal body orientation even with typical detector errors.

  Figure 2: Top: bounding box projection with simple 2D–3D correspondences; middle: using PCA-based orientation; bottom: using the proposed anatomically informed method, which produces superior reprojection accuracy and orientation semantic alignment.

  

  Figure 4: Oriented 3D bounding boxes as fitted to SMAL meshes, projected back onto the image and underlying mesh to validate geometric fidelity.

• Reprojection Accuracy: With a stringent degeneracy criterion (area of reprojected box <1% of animal mask), failure rates are reduced from 13.81% (basic method) to 0% with the full pipeline. Mean reprojection errors drop by 89.96%, from 75.28 to just 7.56 pixels, demonstrating a high degree of geometric correctness for the labels.
• Profile Visibility Metric: For UAV perspectives and novel camera positions, the system yields plausible and accurate visibility metrics across varying animal poses and complex backgrounds.

  Figure 5: For varied aerial viewpoints, the 3D bounding box and corresponding visibility metrics enable precise quantification of visible anatomical profiles, supporting downstream ecological and behavioral analysis.

Implications and Future Directions

This work provides a principled framework for generating high-quality 3D bounding box labels from monocular animal imagery. Critically, it overcomes practical obstacles that have previously prevented the derivation of robust 3D annotations in the absence of depth sensors or LiDAR, such as alignment errors, lack of orientation invariance, and instability to noisy keypoints.

Numerical evidence from the evaluation demonstrates that the use of anatomical keypoints and probabilistic pose refinement yields orientation and projection stability several orders of magnitude better than PCA-based baselines, thus making these labels viable for training and benchmarking learning-based monocular 3D animal detectors.

The pipeline enables the large-scale generation of orientation-aware 3D labels from existing curated RGB datasets, which will be instrumental for advancing monocular 3D animal detection networks, view-specific feature extraction, and open-vocabulary detection transfer—core current challenges in VAB and ecological monitoring.

Pragmatically, the approach is readily extensible to UAV-based animal survey data, supporting real-time ecological field robotics and optimal viewpoint selection for autonomous systems. The profile visibility metric paves the way for more granular behavioral analysis and targeted phenotypic profiling.

The primary limitation of the method is its reliance on SMAL models, restricting applicability to species contained within the SMAL feature space. Extending morphological priors to broader taxa remains an open direction. Additionally, the release of datasets with these new labels and metrics, as planned by the authors, will catalyze progress in learning-based monocular 3D animal detection and multi-view geometric learning for fine-grained animal biometrics and conservation.

Conclusion

This work presents an effective, highly robust pipeline for creating 3D animal bounding box and profile visibility annotations from monocular imagery using SMAL model priors and camera pose optimization. Empirical results reveal substantial improvements in orientation stability and reprojection accuracy over prior methods predicated on PCA or simple correspondences. The generation of accurate, orientation-aware 3D labels facilitates the development and benchmarking of future monocular 3D animal detection algorithms, with immediate applications in wildlife monitoring, autonomous data acquisition, and animal biometrics. This contribution forms a foundational step for advancing rich 3D contextual understanding in computational animal ecology.

Reference: "Adding Another Dimension to Image-based Animal Detection" (2604.09210)