A pipeline for multiple orange detection and tracking with 3-D fruit relocalization and neural-net based yield regression in commercial citrus orchards (2312.16724v1)

Abstract: Traditionally, sweet orange crop forecasting has involved manually counting fruits from numerous trees, which is a labor-intensive process. Automatic systems for fruit counting, based on proximal imaging, computer vision, and machine learning, have been considered a promising alternative or complement to manual counting. These systems require data association components that prevent multiple counting of the same fruit observed in different images. However, there is a lack of work evaluating the accuracy of multiple fruit counting, especially considering (i) occluded and re-entering green fruits on leafy trees, and (ii) counting ground-truth data measured in the crop field. We propose a non-invasive alternative that utilizes fruit counting from videos, implemented as a pipeline. Firstly, we employ CNNs for the detection of visible fruits. Inter-frame association techniques are then applied to track the fruits across frames. To handle occluded and re-appeared fruit, we introduce a relocalization component that employs 3-D estimation of fruit locations. Finally, a neural network regressor is utilized to estimate the total number of fruit, integrating image-based fruit counting with other tree data such as crop variety and tree size. The results demonstrate that the performance of our approach is closely tied to the quality of the field-collected videos. By ensuring that at least 30% of the fruit is accurately detected, tracked, and counted, our yield regressor achieves an impressive coefficient of determination of 0.85. To the best of our knowledge, this study represents one of the few endeavors in fruit estimation that incorporates manual fruit counting as a reference point for evaluation. We also introduce annotated datasets for multiple orange tracking (MOrangeT) and detection (OranDet), publicly available to foster the development of novel methods for image-based fruit counting.

• The paper introduces a video-based pipeline that combines CNN detection with 3-D fruit relocalization and neural-net yield regression.
• It employs tracking algorithms, including the Hungarian algorithm, to accurately detect and track oranges under occlusion and variable lighting.
• Empirical results show a high prediction accuracy (R²=0.85) when at least 30% of fruits are tracked, marking a notable advance in automated citrus yield estimation.

Introduction to the Problem

The task of yield prediction in citrus orchards involves estimating the number of fruits produced by trees. Traditional methods require labor-intensive manual counting, which can be highly demanding. Automation through imaging, computer vision, and machine learning offers a promising alternative. Despite the advances in automated fruit detection, accurate multiple fruit counting, particularly where fruits are hidden or re-appear in cluttered tree canopies, remains a scientific and technical challenge.

Pipeline Overview

A new approach proposed comprises a pipeline that uses video footage for fruit detection and incorporates tracking algorithms to handle challenges like fruits obscured by leaves or falling outside the camera view. This pipeline includes a step for relocalizing the fruits based on their three-dimensional locations and culminates with a neural network-based yield regressor that estimates the total fruit count, taking into account additional tree data like variety and size.

Fruit Detection and Tracking

The first phase of the pipeline involves using convolutional neural networks (CNNs) to detect oranges from video inputs, which are then tracked across frames. When fruits become occluded, a relocalization mechanism based on 3-D projection comes into play, ensuring fruits are accounted for even if they temporarily disappear from view. The tracking methodology incorporates Hungarian algorithm for frame-to-frame association and adjusts fruit locations by referencing camera motion data. The results show that accurate tracking is achievable even under less-than-ideal conditions such as varying light and motion blur.

Yield Estimation

The final phase uses the output from the tracking phase - the number of detected fruits on individual trees - as input for a neural network regressor. This tool integrates the counts with other attributes, including tree variety, height, and age, to produce a total yield estimate. Empirical results indicate that the regressor can achieve a high coefficient of determination (0.85) when at least 30% of the fruits are correctly detected and tracked. The comprehensive evaluation demonstrates that while perfect fruit tracking eludes, the pipeline can still yield accurate counting results, offering a significant advancement for yield prediction in commercial citrus orchards.

Conclusion and Future Work

This paper presents a robust solution for the challenging task of citrus fruit yield estimation using a video-based detection and tracking pipeline. Future research directions involve refining the pipeline for even better accuracy and adapting it for other agricultural environments. Such advancement represents a major step forward in digital agriculture, promoting efficiency and accuracy in citrus orchard management.