LeafTrackNet: A Deep Learning Framework for Robust Leaf Tracking in Top-Down Plant Phenotyping

Abstract: High resolution phenotyping at the level of individual leaves offers fine-grained insights into plant development and stress responses. However, the full potential of accurate leaf tracking over time remains largely unexplored due to the absence of robust tracking methods-particularly for structurally complex crops such as canola. Existing plant-specific tracking methods are typically limited to small-scale species or rely on constrained imaging conditions. In contrast, generic multi-object tracking (MOT) methods are not designed for dynamic biological scenes. Progress in the development of accurate leaf tracking models has also been hindered by a lack of large-scale datasets captured under realistic conditions. In this work, we introduce CanolaTrack, a new benchmark dataset comprising 5,704 RGB images with 31,840 annotated leaf instances spanning the early growth stages of 184 canola plants. To enable accurate leaf tracking over time, we introduce LeafTrackNet, an efficient framework that combines a YOLOv10-based leaf detector with a MobileNetV3-based embedding network. During inference, leaf identities are maintained over time through an embedding-based memory association strategy. LeafTrackNet outperforms both plant-specific trackers and state-of-the-art MOT baselines, achieving a 9% HOTA improvement on CanolaTrack. With our work we provide a new standard for leaf-level tracking under realistic conditions and we provide CanolaTrack - the largest dataset for leaf tracking in agriculture crops, which will contribute to future research in plant phenotyping. Our code and dataset are publicly available at https://github.com/shl-shawn/LeafTrackNet.

• The paper presents LeafTrackNet, a framework that decouples high-fidelity leaf detection from identity association using YOLOv10 and MobileNetV3-based embeddings.
• It leverages the extensive CanolaTrack dataset with 5,704 images and over 31,800 annotated leaf instances to address challenges like occlusion, rotation, and emerging leaves.
• Experimental results show significant improvements in HOTA, MOTA, and IDF1 metrics, demonstrating the framework’s superior performance over state-of-the-art MOT approaches.

LeafTrackNet: A Deep Learning Framework for Robust Leaf Tracking in Top-Down Plant Phenotyping

Introduction and Motivation

Leaf-level phenotyping in crops such as canola requires persistent temporal tracking of individual leaves, a challenge exacerbated by dense canopies, occlusion, rotational artifacts, and dynamic morphological changes. While general multi-object tracking (MOT) and existing plant-specific methods—such as those relying on IoU-based mask matching or off-the-shelf appearance embeddings—fail to generalize or maintain identity consistency over realistic plant growth sequences, the progress in this domain has been limited by the lack of a comprehensive, large-scale annotated dataset.

The paper introduces two central contributions: (1) CanolaTrack, the largest and most challenging dataset to date for leaf identity tracking in complex crop settings; and (2) LeafTrackNet, a novel MOT pipeline that decouples high-fidelity leaf detection (via YOLOv10) from leaf identity association (via MobileNetV3-based embedding and memory matching), trained with a triplet margin loss. The proposed framework specifically targets the key domain challenges in plant phenotyping and achieves strong, cross-metric improvements compared to state-of-the-art MOT and plant-specific trackers.

Methodology

The design of LeafTrackNet directly addresses three observed failure points in the leaf MOT setting: (1) the inadequacy of geometric/motion priors due to nonlinear biological motion and occlusion, (2) the limited discriminative power of generic embedding networks for visually similar plant leaves, and (3) the infeasibility of end-to-end transformer frameworks designed for dense, rigid-object MOT in temporally sparse plant imagery.

LeafTrackNet consists of a two-phase pipeline:

1. Training Phase: Utilizes annotated leaf crops from CanolaTrack to train a MobileNetV3-based embedding network with a triplet margin loss, enforcing intra-leaf temporal consistency and inter-leaf separability.
2. Inference & Tracking Phase: Sequential RGB images are first processed with a fine-tuned YOLOv10 detector to localize leaves. Each detected region is projected into the embedding space, and temporal tracklets are established via a cosine similarity-based Hungarian matching against a memory bank of prototype leaf embeddings, using exponential moving averaging for prototype updates. Tracks are initialized, extended, or pruned with explicit age management and similarity thresholding.

   Figure 1: Overview of the two-phase LeafTrackNet pipeline, depicting separate phases for discriminative embedding learning and inference-time memory-based identity matching across plant growth sequences.

This architecture is explicitly invariant to rotation, occlusion, and emergent/disappearing leaves, as it eschews any strong geometric or smoothness prior.

CanolaTrack Dataset

CanolaTrack consists of 5,704 high-resolution ($1200\times1200$) RGB images, collected daily over 31 days from 184 canola plants, yielding over 31,800 annotated leaf instances. Ground-truths include bounding boxes and persistent identity assignment for each leaf throughout the observation window. This dataset uniquely captures occlusion, rotational, and emergence phenomena inherent to complex field-like conditions and facilitates unbiased benchmarking for long-term plant organ tracking.

Figure 2: Temporal progression of a representative canola plant, displaying color-coded, identity-assigned leaf bounding boxes from day 1 to day 31.

Experimental Results

Quantitative Comparison

LeafTrackNet exhibits superior performance relative to both general-purpose (BoT-SORT, ByteTrack, MOTRv2) and contemporary plant-specific (LeTra, Plant-Doctor) trackers across all MOT metrics, including HOTA, MOTA, AssA, DetA, and IDF1.

• HOTA: LeafTrackNet achieves 88.03, representing a 9.7 point gain over the best baseline (MOTRv2).
• MOTA/IDF1: Margins of improvement reach +11.55 (MOTA) and +9.12 (IDF1).
• Temporal Association: 84.07 AssA, indicating robust identity maintenance and minimal drifting or switching over extended plant development windows.

The detection backbone ablation underscores that MobileNetV3, despite its low parameter count (~3M), matches or exceeds both ResNet and ViT alternatives in tracking accuracy, confirming that scale is less relevant than embedding discriminativity in this application.

Qualitative and Embedding Analysis

Qualitative visualizations reveal that baseline MOT approaches consistently fail under scenarios of high leaf overlap, occlusion, and pot rotation, exhibiting ID switches and fragmented tracks. Plant-specific methods relying on geometric or non-specialized appearance features show improved but still insufficient resilience. LeafTrackNet successfully avoids identity fragments even in extreme cases, using its learned embedding-based association that demonstrates strong invariance and stability.

Figure 3: Side-by-side tracking results for Plant-158, showing LeafTrackNet’s resilience to both (a) high-overlap and (b) large pot rotation events.

The t-SNE visualization of embedding trajectories over 31 days demonstrates tight self-clustering of individual leaf representations, despite significant appearance variation. Separability between different leaf identities remains high, showing embedding space structure optimized for the identity persistence objective.

Figure 4: (a) Individual leaf area trajectories in Plant-158 illustrating appearance emergence/disappearance; (b) t-SNE plot of per-instance embeddings visualizing cluster tightness and identity separability over time.

Ablation Studies

• Triplet Sampling: Cross-plant negative sampling slightly outperforms intra-plant sampling, hinting that increased negative diversity further regularizes the learned embedding space. Temporal-window triplet sampling performs much worse, confirming that hard negatives within the plant/growth stage boost discriminative learning.
• Memory Bank Hyperparameters: HOTA is insensitive in the range $\tau_s\in[0.4,0.6]$ and moderate smoothing $(\alpha=0.5)$ yields best results, indicating algorithm robustness to matching/EMA hyperparameters.

  Figure 5: Influence of similarity threshold $(\tau_s)$ and smoothing coefficient $(\alpha)$ on HOTA; results are stable across reasonable parameter choices.

• Long-Term Temporal Performance: Both daily frame-level and per-leaf heatmap accuracy visualizations demonstrate that LeafTrackNet maintains high identity consistency across the entire observation horizon, whereas others fail after occlusion, rotation, or during emergence/disappearance transitions.

  Figure 6: (a) Per-day frame-level accuracy and (b) per-leaf binary tracking matrix on Plant-158 show LeafTrackNet’s sustained tracking success versus severe decay for all baselines.

Implications and Future Directions

The proposed method simultaneously advances the practical state-of-the-art for automated, scalable leaf-level plant phenotyping and provides a methodological foundation for further research in organ-level tracking of other crops, cross-species generalization, and large-scale field deployment. Its embedding-based, memory-augmented association suits temporally sparse, dynamic biological MOT tasks that break conventional motion/model assumptions. The large-scale, diverse CanolaTrack dataset sets a new benchmark and resource for reproducible evaluation in the domain.

Immediate follow-on directions include transfer learning for other plant species with variable morphologies, extension to segmentation-based instance tracking, integration with physiological or stress phenotyping analytics, and eventual deployment in field-scale or robotic phenotyping systems. Efficient online and real-time adaptation modules could further extend applicability to unconstrained agricultural environments.

Conclusion

LeafTrackNet establishes a new standard for fine-grained, robust, and efficient leaf identity tracking in realistic, dynamically structured crop scenarios. By decoupling detection and identity association, leveraging carefully constructed temporal embeddings and memory-based matching, it overcomes critical weaknesses of both MOT and previous organ-level plant trackers. This approach directly supports more informative and actionable plant phenotyping pipelines and forms the basis for future research in data-driven crop science and precision agriculture.