T-LEAP Temporal Pose Estimation

Updated 3 July 2026

The paper introduces T-LEAP, an end-to-end spatiotemporal model that leverages 3D convolutions to improve keypoint localization under occlusion.
It integrates temporal context by processing sequences of RGB frames, significantly outperforming static frame-by-frame methods in challenging farm conditions.
T-LEAP supports automated cow gait and lameness detection by generating biologically meaningful keypoint trajectories for downstream behavioral analysis.

T-LEAP Pose Estimation Model

The T-LEAP (Temporal LEAP) pose estimation model is a video-based deep learning method designed to robustly track anatomical landmarks on walking dairy cows, with a primary application in automated gait and lameness analysis under real-world farm conditions. T-LEAP extends the LEAP architecture, a fully convolutional animal pose estimator, by incorporating temporal information via 3D convolutional operations, thus enabling the joint exploitation of spatial and temporal context for improved keypoint localization, especially under occlusion. The model is unique among animal pose estimators for integrating end-to-end temporal reasoning through spatial-temporal convolutions, in contrast to earlier systems that process frames independently. T-LEAP has been shown to outperform static approaches under artificially induced occlusions, generalize to previously unseen cows with moderate calibration, and provide high-fidelity, biologically meaningful coordinate trajectories suitable for downstream behavioral interpretation (Russello et al., 2021).

1. Model Definition and Motivation

T-LEAP is designed to address the limitations of static frame-by-frame animal pose estimation models, which are prone to failure in complex, unconstrained settings common on dairy farms—such as changing illumination, environmental occlusions (e.g., fences, rails), and visual clutter. Traditional marker-based gait analysis is labor-intensive and unreliable under these circumstances. T-LEAP offers a markerless, learned alternative by predicting the locations of anatomical landmarks directly from short sequences of RGB images, leveraging temporal continuity in animal motion. The primary hypothesis underlying T-LEAP is that the inclusion of past frames allows the network to learn motion dynamics and infer the configuration of occluded parts more robustly than single-frame models, a need that is acute in video-based livestock monitoring (Russello et al., 2021).

2. Architectural Details

T-LEAP is an encoder-decoder convolutional neural network with all spatial operations extended from 2D to 3D to support spatiotemporal processing. The static LEAP baseline comprises four encoder groups (each with three 2D conv layers, interleaved by max pooling half-resolution steps), two decoder groups (each with two 2D conv layers and preceding 2D transposed convolutions for upsampling and resolution recovery), and a final (transposed) 2D convolution and softmax to produce per-keypoint confidence maps. In T-LEAP, every convolution, pooling, and transposed convolution becomes a 3D operation, so that kernels span both spatial and temporal axes. The model accepts a sequence of $T$ consecutive RGB frames (cropped and resized to $200 \times 200$ ), aggregates temporal features primarily through max pooling (collapsing the temporal axis after the second pooling layer), and outputs keypoint confidence maps for only the last frame in the sequence. Ground-truth for training consists of per-keypoint Gaussian heatmaps centered at manually annotated locations, with standard deviation $\sigma=5$ pixels, one channel per landmark (Russello et al., 2021).

3. Training Procedure, Evaluation Protocol, and Dataset

T-LEAP models are trained using a mean squared error loss per pixel, per heatmap, comparing predicted and ground-truth confidence maps. The CoWalk-30 dataset—constructed for benchmarking—comprises videos of 30 different Holstein-Frisian dairy cows, each sequence carefully selected to contain straight, uninterrupted, left-to-right walking under outdoor natural illumination, filmed at 30 FPS. In total, 4275 frames were annotated and grouped into 1059 four-frame samples. Augmentation includes random rotation (±10°), brightness perturbation (±100 units), and random contrast gain scaling (±3.0), all applied per sample during training; no test-time augmentation is performed. Split protocol varies by experiment: random train/test splits for occlusion benchmarking, and a grouped split by individual for generalization analysis (i.e., evaluation with both known and previously unseen cows) (Russello et al., 2021).

Key datasets and train/test schemes

Dataset	Train Samples	Test Samples	Grouping	Usage
CoWalk-30	847	212	Random split	Occlusion analysis
CoWalk-10	168	891 (172+719)	Individual-based	Generalization (known/unknown)

4. Keypoint Target Definition and Losses

T-LEAP predicts 17 anatomical landmarks: four per leg (hoof, fetlock, carpal/tarsal), and five for the condensed torso and head (nose, forehead, withers, sacrum, caudal thoracic vertebrae). For each keypoint, the output is a confidence map, with the predicted coordinate taken as the argmax location. The loss function is pixelwise MSE across heatmaps and batch samples. No auxiliary supervision or regularization is utilized. All training hyperparameters (e.g., optimizer AMSGrad, initial LR 0.001 with 0.1 decay every 10 epochs, batch size 8, 50 epochs) are selected via grid search with Weights & Biases sweeps (Russello et al., 2021).

5. Empirical Robustness: Occlusion, Generalization, Depth

On non-occluded data, T-LEAP and its static variant achieve indistinguishable results ( $\sim99\%$ [email protected]). Under increasing levels of artificial occlusion—vertical bars obscuring hind/front legs and head—the temporal model dramatically exceeds the static baseline, with a maximum observed gain of 32.9% (i.e., 59.4% to 88.4% [email protected] in hardest occlusion). The improvement is most pronounced for short temporal windows ( $T=2$ ); longer ( $T=4$ ) context does not help for occlusions spanning only two frames, but might for more persistent occlusions. In cross-individual experiments, T-LEAP achieves 93.8% [email protected] on known cows and 87.6% on fully unseen cows, with higher error for cows with coat patterns not represented in training data. Model depth significantly boosts accuracy, with an 11.7% [email protected] gain over the original architecture in triple-occlusion cases, highlighting the necessity of both temporal context and adequate receptive field (Russello et al., 2021).

6. Role in Automated Cow Gait and Lameness Analysis

T-LEAP has been integrated as the core pose estimator in multiple downstream pipelines for automatic lameness detection (Russello et al., 2024, Russello et al., 14 Aug 2025). In such systems, T-LEAP extracts temporally coherent keypoint trajectories, which are then (a) used to derive locomotion traits (back posture, head bobbing, tracking distance, stride length/symmetry, stance/swing) for interpretable feature-based classification (Russello et al., 2024), or (b) directly fed as raw coordinate sequences into temporal classifiers such as bidirectional LSTMs for end-to-end data-driven gait assessment (Russello et al., 14 Aug 2025). These studies highlight that T-LEAP achieves 99.6% [email protected] on outdoor farm videos (postprocessed with MAD and Savitzky-Golay filtering), and that its outputs suffice for automatic lameness detection with short (1–3 s) video windows, outperforming prior pipelines reliant on manual feature engineering (Russello et al., 2024, Russello et al., 14 Aug 2025).

T-LEAP should be distinguished from related temporal or animal pose estimation models that process video differently. For example, STEP (Verma et al., 17 Mar 2025) is an integrated tracking and pose estimation method for animals, emphasizing joint instance-tracking and pose within a transformer-based online pipeline. TePose (Wang et al., 2022) is a temporally embedded SMPL regressor for live-stream human 3D pose and shape recovery, focusing on human mesh recovery with recurrent architectures and adversarial motion regularization. LEAP (Mihajlovic et al., 2021), the predecessor of T-LEAP, is a static, per-frame keypoint heatmap model for laboratory animals. T-LEAP’s unique contribution lies in spatiotemporal end-to-end learning for markerless animal pose (video inputs, 3D convolutions), with demonstrated gains under occlusion, and established as a practical backbone for animal welfare monitoring in real-world conditions.

References:

T-LEAP: Occlusion-robust pose estimation of walking cows using temporal information (Russello et al., 2021) Video-based Automatic Lameness Detection of Dairy Cows using Pose Estimation and Multiple Locomotion Traits (Russello et al., 2024) Lameness detection in dairy cows using pose estimation and bidirectional LSTMs (Russello et al., 14 Aug 2025) STEP: Simultaneous Tracking and Estimation of Pose for Animals and Humans (Verma et al., 17 Mar 2025) Live Stream Temporally Embedded 3D Human Body Pose and Shape Estimation (Wang et al., 2022) LEAP: Learning Articulated Occupancy of People (Mihajlovic et al., 2021)