FruitNeRF: A Unified Neural Radiance Field based Fruit Counting Framework (2408.06190v2)

Abstract: We introduce FruitNeRF, a unified novel fruit counting framework that leverages state-of-the-art view synthesis methods to count any fruit type directly in 3D. Our framework takes an unordered set of posed images captured by a monocular camera and segments fruit in each image. To make our system independent of the fruit type, we employ a foundation model that generates binary segmentation masks for any fruit. Utilizing both modalities, RGB and semantic, we train a semantic neural radiance field. Through uniform volume sampling of the implicit Fruit Field, we obtain fruit-only point clouds. By applying cascaded clustering on the extracted point cloud, our approach achieves precise fruit count.The use of neural radiance fields provides significant advantages over conventional methods such as object tracking or optical flow, as the counting itself is lifted into 3D. Our method prevents double counting fruit and avoids counting irrelevant fruit.We evaluate our methodology using both real-world and synthetic datasets. The real-world dataset consists of three apple trees with manually counted ground truths, a benchmark apple dataset with one row and ground truth fruit location, while the synthetic dataset comprises various fruit types including apple, plum, lemon, pear, peach, and mango.Additionally, we assess the performance of fruit counting using the foundation model compared to a U-Net.

• The paper introduces FruitNeRF, a unified framework that leverages Neural Radiance Fields for precise 3D fruit counting in agricultural settings.
• It employs Structure from Motion for camera calibration and segmentation techniques like Grounded-SAM and U-Net to generate accurate fruit masks.
• Experiments report F1-scores up to 0.95 on synthetic data and over 89% detection on real-world datasets, highlighting its scalability and precision.

FruitNeRF: A Unified Neural Radiance Field Based Fruit Counting Framework

The paper presents "FruitNeRF," a novel framework for fruit counting that leverages Neural Radiance Fields (NeRF) to achieve accurate counting directly in 3D. The authors implemented this framework to address inherent challenges in fruit counting, particularly within the field of Precision Agriculture (PA). The paper meticulously outlines the methodology and examines the efficacy of the approach through various experiments on synthetic and real-world datasets.

Overview

The FruitNeRF framework capitalizes on the capabilities of NeRF to perform volumetric rendering and semantic rendering jointly. This approach entails the use of an unordered set of posed monocular camera images, wherein each image undergoes segmentation to generate binary fruit masks. These masks, along with RGB images, are employed to train a semantic neural radiance field.

Key components of the FruitNeRF framework include:

1. Data Preparation: Utilizing Structure from Motion (SfM) to recover intrinsic and extrinsic camera parameters.
2. Fruit Segmentation: Termed as a unified fruit model, employing Grounded-SAM and comparing its performance with a U-Net specifically trained for apples.
3. FruitNeRF Training: Involves separate fields for density, appearance, and fruit semantics, enabling the encoding of spatial and visual information.
4. Point Cloud Export: Extracting 3D fruit points using a combination of density and semantic fields.
5. Fruit Counting: Employing a cascaded clustering methodology to ensure precise fruit counts, mitigating issues like double-counting and irrelevant fruit counting.

Quantitative Evaluation

The framework is evaluated on both synthetic and real-world datasets. Key results include:

• Synthetic Dataset: FruitNeRF achieved an F1-score of 0.95 with ground truth masks and 0.88 using Grounded-SAM-generated masks across various fruit types, including apples, lemons, pears, and peaches.
• Real-World Dataset: Demonstrated an average detection rate exceeding 89% for apple trees when masks were generated using Grounded-SAM and U-Net. For the Fuji-SfM dataset, the larger FruitNeRF model attained an F1-score of 0.79.

A significant finding from the experiments is that high-quality counting can be accomplished with approximately 40 images per tree, demonstrating FruitNeRF’s scalability and efficiency.

Implications and Future Directions

Theoretical Implications

FruitNeRF introduces a new dimension to fruit counting in PA by leveraging advanced view synthesis methods. The incorporation of NeRF, combined with semantic rendering, enhances the accuracy and reliability of 3D reconstructions for counting purposes. This methodological innovation has potential implications for various 3D object counting and tracking tasks beyond agriculture.

Practical Implications

Practically, FruitNeRF addresses key challenges in fruit counting, such as occlusions, double counting, and handling various lighting conditions. The framework's fruit-agnostic nature, facilitated by Grounded-SAM, makes it versatile across different fruit types, streamlining the process of deploying the framework in diverse orchard environments.

Future developments could focus on optimizing training times and reducing the computational resources necessary for real-time applications. Exploring the integration of SLAM for online pose estimation and the inclusion of orthophoto generation could enhance the framework’s applicability to larger-scale orchard settings. Additionally, incorporating other imaging modalities such as near-infrared or thermal could provide further robustness against varying environmental conditions.

Conclusion

The FruitNeRF framework represents a significant step forward in the precision agriculture domain by introducing a robust, scalable, and versatile solution for fruit counting. Despite current limitations in computational demands and the need for parameter tuning, the methodology showcases promising results that hold potential for broader applications in 3D fruit counting and beyond. With ongoing advancements in neural rendering and computational photography, the future iterations of FruitNeRF could become increasingly efficient and widely applicable, driving further innovation in agricultural technology.