WildLIFT: 3D Wildlife Monitoring Framework

• WildLIFT is a computational framework that converts monocular drone video into structured 3D tracks with identity consistency and rich metadata for wildlife monitoring.
• It integrates 3D scene reconstruction with open-vocabulary 2D segmentation to produce KITTI-format oriented bounding boxes and detailed viewpoint analyses.
• The modular pipeline—comprising tracking, keyframe-based annotation, and viewpoint analysis—significantly reduces manual correction and enhances ecological research efficiency.

WildLIFT is a computational framework for transforming monocular drone video into structured, identity-consistent 3D tracks and rich viewpoint-aware metadata for species-agnostic wildlife monitoring. It integrates 3D scene geometry recovered from uncalibrated video with open-vocabulary 2D instance segmentation, enabling oriented 3D detection, tracking, and coverage assessment without the need for intrinsics, special sensors, or species-specific training. Outputs include KITTI-format 3D bounding boxes with semantic face tags and quantitative metadata suitable for downstream ecological and behavioural analyses (Shukla et al., 27 Apr 2026).

1. Modular Pipeline Structure

WildLIFT follows a sequential modular design comprising three principal stages:

1. Reconstruct & Track (WildLIFT-RT): Performs dense 3D reconstruction and open-vocabulary 2D instance segmentation on monocular video frames, lifting instance-level masks to the reconstructed 3D point cloud. Kalman-filtered 3D tracking ensures temporal and identity consistency.
2. Annotate (WildLIFT-A): Fits oriented 3D bounding boxes (OBBs) to each 3D point cluster using PCA, optionally constrained by gimbal telemetry. Human-in-the-loop correction is accomplished using keyframe-based interpolation (LERP for positions, SLERP for rotations) and semantic face labelling.
3. Viewpoint Analysis (WildLIFT-V): Computes per-face semantic visibility, occlusion via ray–OBB intersection, effective coverage statistics, coverage diversity (Shannon entropy), and assigns viewpoint grade labels.

Input requirements:

• Raw, uncalibrated monocular drone video $I_1, \dots, I_T$ (3.75–15 FPS)
• Text prompt specifying the target species (e.g., “zebra”)
• Optionally, gimbal telemetry $(\theta_p, \theta_r)$ for enhanced vertical orientation stabilization

Principal outputs:

• Identity-consistent 3D object tracks
• KITTI-format OBB labels with semantic face tags
• Per-track viewpoint coverage, occlusion, and quality statistics

2. Monocular 3D Reconstruction and Segmentation

The core of WildLIFT-RT is online feed-forward geometric reconstruction using the CUT3R transformer, which regresses dense 3D pointmaps $\mathcal{P}_t \subset \mathbb{R}^3$ and camera poses $T_t \in SE(3)$ from RGB frames. CUT3R, trained on large multi-view video corpora, requires neither known intrinsics nor standard bundle adjustment, instead relying on persistent state encoding via sequence learning.

Mask segmentation is achieved using the open-vocabulary Grounded-SAM model, which leverages CLIP-based text–image alignment for zero-shot instance segmentation. Masks $M_t^i$ are directly associated with prompts (species) without retraining. Lifting to 3D is executed as follows: $$\mathcal{Q}_t^i = \left\{\mathbf{p} \in \mathcal{P}_t : \pi(\mathbf{p}) \in M_t^i \right\}$$ where $\pi$ denotes camera projection.

Outlier removal within each cluster follows a 1.5× interquartile range rule on point centroids.

3. Fitting, Annotation, and Keyframe-Based Refinement

Each 3D point cluster $\mathcal{Q}_t^i$ is fitted with a PCA-based oriented bounding box $b=(\mathbf{c}, \mathbf{d}, R)$, where $\mathbf{c}$ is the centroid, $(\theta_p, \theta_r)$0 are box extents, and $(\theta_p, \theta_r)$1 is the rotation (principal axes). Gimbal telemetry, when available, constrains the vertical axis via ground-normal calculation: $(\theta_p, \theta_r)$2 The principal axis most aligned with $(\theta_p, \theta_r)$3 determines the box’s $(\theta_p, \theta_r)$4–axis; the remaining axes are orthogonalized. Unconstrained PCA is applied if telemetry is absent.

Manual geometric correction is reduced by keyframe-based interpolation: only a sparse set of frames (“keyframes”) are hand-annotated, with centers and dimensions interpolated linearly (LERP) and box orientations interpolated on $(\theta_p, \theta_r)$5 (SLERP). Semantic face labels are propagated independently, allowing flexible correction of headings without revisiting the geometry.

On empirical datasets, only 2.3% of frames require manual correction, yielding a mean annotation time of 3.6 s/frame—orders of magnitude lower than manual LiDAR or frame-by-frame annotation.

4. Viewpoint Coverage, Occlusion, and Quality Metrics

WildLIFT-V generates per-face (front, back, left, right, top) statistics for every 3D OBB instance. For frame $(\theta_p, \theta_r)$6 and face $(\theta_p, \theta_r)$7, semantic visibility is

$(\theta_p, \theta_r)$8

Quality scoring combines viewing angle, projected area, centrality, and foreshortening: $(\theta_p, \theta_r)$9 where $\mathcal{P}_t \subset \mathbb{R}^3$0 is the absolute dot product of the face normal and viewing vector, $\mathcal{P}_t \subset \mathbb{R}^3$1 the normalized area, $\mathcal{P}_t \subset \mathbb{R}^3$2 centrality, and $\mathcal{P}_t \subset \mathbb{R}^3$3 foreshortening.

Inter-animal occlusion is computed by casting rays from the camera through grid points on face $\mathcal{P}_t \subset \mathbb{R}^3$4 and testing for intersections with other OBBs via the slab algorithm. Effective visibility is calculated as

$\mathcal{P}_t \subset \mathbb{R}^3$5

where $\mathcal{P}_t \subset \mathbb{R}^3$6 is the occlusion fraction.

Aggregate statistics include per-face coverage vectors, normalized Shannon diversity $\mathcal{P}_t \subset \mathbb{R}^3$7, and assignment of letter grades (A/B/C/F) based on face-coverage thresholds.

5. Quantitative Evaluation and Performance

WildLIFT was validated on 27 drone sequences (Ol Pejeta, KABR, Bristol Zoo), including 77 individuals from four large mammal species (rhino, elephant, zebra, giraffe), comprising 2,581 manually curated frames and 6,799 ground-truth 3D detections.

Tracking (WildLIFT-RT):

• Recall: 1.000
• IDF$\mathcal{P}_t \subset \mathbb{R}^3$8: 0.982 (±2.2 percentage points over next-best 2D)
• Identity over-count: 7% (vs 17–38% for 2D methods)
• Largest gains observed under prolonged occlusion/vegetation

Annotation efficiency (WildLIFT-A):

• 93% of frames required no geometric OBB correction (vs 58.5% in image pipelines)
• Heading flip corrections: 10% of frames (handled semantically)
• Annotation time: 3.6 s/frame (vs 60–120 s/frame manual)

Viewpoint classification (WildLIFT-V):

• Frame×face classification accuracy: rhino 0.95, elephant 0.91, zebra 0.86; recall≈1.0
• Effective occlusion: 15–40% of geometrically visible frames occluded in multi-animal scenes

6. Applications and Significance

WildLIFT closes the loop between data acquisition, geometric modeling, and ecological analytics by producing structured, viewpoint-aware 3D representations from ordinary consumer-grade drone video. The framework’s ability to deliver identity-consistent tracks, viewpoint-resolved coverage, and occlusion statistics enables:

• Improved multi-animal tracking, particularly in occluded, dynamic environments (50% fewer identity switches than 2D methods)
• Relieved annotation workload through keyframe-based interpolation (20× faster than manual LiDAR point cloud labelling)
• Quantitative assessment of viewpoint biases for coverage planning and behaviour/demography studies

KITTI-format 3D OBBs with semantic face tags facilitate the generation of large-scale training data for monocular 3D detectors. WildLIFT’s species-agnostic architecture, low GPU memory footprint (~6 GB VRAM), and compatibility with open-vocabulary segmentation models support rapid deployment across taxa and hardware platforms.

Downstream applications include behavioural research, individual re-identification (facilitated by specific flank views), population density estimation, and quantification of occlusion effects in habitat analyses.

7. Limitations and Future Directions

WildLIFT leverages consumer hardware and monocular reconstruction, without requiring special-purpose sensors or pre-calibrated intrinsics. A plausible implication is that generalization to additional taxa depends on the segmentation model's performance. Semantic heading flips and multi-animal occlusion remain the chief sources of annotation and tracking errors, though the modular keyframe-based workflow minimizes associated burden. The structured output format and explicit coverage/occlusion metrics position the framework as a foundation for supervised learning, longitudinal studies, and automated behaviour recognition in wildlife monitoring pipelines (Shukla et al., 27 Apr 2026).