FruitNet: 4D Orchard Monitoring System

• FruitNet is a 4D spatio-temporal metric-semantic mapping system that fuses multi-modal sensor data for persistent, fruit-level tracking in orchards.
• It employs LiDAR, RGB cameras, IMU, and GPS with advanced techniques like YOLOv8 and Faster-LIO to achieve real-time 3D localization and temporal association.
• FruitNet supports detailed per-fruit growth curve analysis and yield estimation, facilitating improved plant phenotyping and optimized resource deployment.

FruitNet is a 4D spatio-temporal metric-semantic mapping system for persistent orchard monitoring that integrates multi-session measurements to track fruit growth over time. It is designed for fine-grained orchard monitoring at the individual tree or fruit level across an entire growth season, with the stated aims of supporting plant phenotyping and horticultural resource optimization, including chemical use and yield estimation. The system fuses LiDAR, monocular RGB cameras, IMU, and additional GPS for georeferencing, combines LiDAR-RGB fusion for 3D fruit localization with 4D fruit association, and synthesizes the resulting measurements into a spatio-temporal map that supports analytic queries such as per-fruit development curves and yield estimation (Lei et al., 2024).

1. Scope and problem setting

FruitNet addresses persistent, fruit-level monitoring under natural orchard conditions. In the formulation used by the system, “4D” denotes 3D space plus time: the objective is not only to localize fruits in metric 3D coordinates during a single session, but also to associate those fruit instances across sessions separated by weeks or months. This permits the construction of temporal trajectories for individual fruits and supports measurements of growth in size, 3D location, and appearance over changing environmental scenes (Lei et al., 2024).

The problem setting differs from single-session fruit counting or static orchard mapping. Prior work is described as having mostly focused on single-session 3D mapping and counting, or on expensive, high-precision point clouds not viable for large, natural orchards. FruitNet is presented as the first system to demonstrate practical, persistent 4D fruit instance tracking in uncontrolled orchard conditions across a season with multi-modal data. A plausible implication is that the system’s central contribution lies less in isolated detection accuracy than in long-term instance persistence under seasonal change (Lei et al., 2024).

2. Modular architecture

The framework is explicitly modular. Sensor streams are tightly synchronized, and the architecture is organized into five modules.

Module	Function
Module 1: Sensor Data Input	LiDAR, RGB camera, and IMU data are tightly synchronized and recorded
Module 2: Object Detection and Odometry	Faster-LIO for LiDAR-inertial odometry; YOLOv8 instance segmentation for fruit detection and masks
Module 3: LiDAR-RGB Fusion for 3D Metric-Semantic Mapping	Fuses image detections with LiDAR points, tracks fruit instances, and optimizes 3D positions and sizes
Module 4: 4D Data Association	Aligns orchard maps from multiple sessions and associates fruit instances over time
Module 5: 4D Metric-Semantic Map Generation	Produces a spatio-temporal map for analytics such as per-fruit development curves and yield estimation

Within this architecture, odometry is provided by Faster-LIO, described as a LiDAR-inertial odometry algorithm that yields motion-compensated, accurately registered point clouds. Detection is performed with a YOLOv8 instance segmentation neural network fine-tuned on orchard fruit images. The architecture therefore couples metric mapping and instance-level semantics at the fruit scale rather than treating them as separate post hoc stages (Lei et al., 2024).

The modular decomposition also clarifies the system boundary. FruitNet is not only a detector, and it is not only a mapper; it is a pipeline in which sensor synchronization, odometry, instance segmentation, 3D fusion, cross-session registration, and temporal association are all necessary to produce a persistent orchard representation.

3. LiDAR-RGB fusion and 3D fruit localization

The LiDAR-RGB fusion stage begins with fruit instance point cloud extraction. Each RGB image is pre-processed, for example with CLAHE for contrast normalization, and then passed through YOLOv8 to produce instance segmentation masks. Sensor poses are obtained from Faster-LIO, while RGB-LiDAR extrinsic and intrinsic calibration is performed pre-deployment through manual or board-based procedures and via Kalibr. Synchronized LiDAR point clouds are projected onto the image plane, and 3D points that fall within each fruit mask are retained as per-fruit instance point clouds. To improve spatial completeness, point clouds are aggregated over a sliding window of recent frames, typically 1.5 seconds, balancing density and computational burden (Lei et al., 2024).

Fruit tracking across frames is formulated as an assignment problem. If detections at frame $l$ are $I_l=\{o_{i,l}\}$ and detections at frame $l+1$ are $I_{l+1}=\{o_{j,l+1}\}$, the expected mask in the next frame is projected as

$mask_{i,l+1}=proj(T_l^{l+1}, p_{i,l}).$

The assignment cost is based on mask overlap: $$c(o_{i,l}, o_{j,l+1}) = 1 - \text{IoU}(mask_{i,l+1}, mask_{j,l+1}),$$ with low-IoU candidates optionally prevented from assignment. The cost matrix combines IoU costs with unassigned costs,

$C(I_l, I_{l+1}) = [C_{IoU}\quad C_u],$

and the Hungarian algorithm is then used for optimal data association. To reduce false positives, only fruits observed for more than three consecutive frames are retained; additional logic merges split tracks for re-appearing fruits (Lei et al., 2024).

Geometric refinement is performed by reprojection error minimization over all image frames and camera poses: $$\min_{X, R, t} \sum_{i=1}^{n} \sum_{j=1}^{m} a_{ij}(x_{ij} - proj(R_i, t_i, X_j))^2$$ subject to

$X_{j,0}-d \leq X_j \leq X_{j,0}+d.$

Here $a_{ij}$ denotes fruit presence in frame $I_l=\{o_{i,l}\}$0, and $I_l=\{o_{i,l}\}$1 is the 3D position of fruit $I_l=\{o_{i,l}\}$2. After optimization, tracks are merged further if their final 3D positions lie within a tight threshold. In practical terms, this stage turns repeated partial observations into more stable fruit centroids and size estimates.

4. Cross-session 4D fruit association

Once accurate 3D fruit positions are available for each time session, FruitNet performs cross-session alignment and fruit association. Map alignment is done with Iterative Closest Point (ICP) using an initial pose prior; the paper gives the example of session starts at a fixed ground mark to provide coarse alignment. This registration places point clouds from distinct growth stages into a common frame (Lei et al., 2024).

Fruit instance matching between adjacent sessions is then based on 3D Euclidean distance: $I_l=\{o_{i,l}\}$3 Matching is confined to a neighborhood,

$I_l=\{o_{i,l}\}$4

with an additional height check,

$I_l=\{o_{i,l}\}$5

motivated by the tendency of fruits to grow heavier and sink lower. Assignment is again solved with the Hungarian algorithm. The output is a set of temporal tracks linking fruit instances across weeks-to-months, enabling per-fruit growth curves and persistent identity across the season (Lei et al., 2024).

This combination of geometric registration and constrained assignment is central to the system’s “persistent” character. A plausible interpretation is that positional, visual, and topology information are used to stabilize identity over time in settings where appearance and foliage structure change substantially across the season.

5. Dataset, benchmarks, and reported performance

FruitNet is accompanied by a publicly released multimodal dataset hosted at https://4d-metric-semantic-mapping.org/. The dataset contains LiDAR point clouds, RGB and RGB-D images, and IMU/GPS measurements totaling over 4 TB. The sensing platform includes an Ouster OS0-128 LiDAR, a FLIR Chameleon global-shutter RGB camera, an Intel RealSense D435i RGB-D sensor, a VectorNav VN-100 IMU, and a Ublox ZED-F9P GPS (Lei et al., 2024).

The dataset covers 284 trees: 120 apple, 63 pear, 50 peach, 34 cherry, and 17 pistachio. Data were collected from April to August 2024 across orchards in Pennsylvania, Maryland, and California, spanning complete fruit growth cycles. Released assets include segmentation annotations, pre-trained models, and per-fruit ground truth for counts and sizes. Benchmarks are provided for fruit counting accuracy, size estimation, and cross-session instance association (Lei et al., 2024).

Reported performance is concentrated on three tasks. For fruit counting on 60 apple trees containing approximately 1,790 apples, the system achieved 96.9% fruit counting accuracy; the estimated count was 1,846 against a ground truth of 1,790, corresponding to an absolute error of 56 fruits, or 3.1%. On 22 batches of 2–3 trees each, the mean absolute error was 6 fruits (8.0%), with a standard deviation of 5.0 (6.4%). For fruit size estimation, controlled experiments with real apples manually attached and measured by caliper on orchard-like trees yielded a mean absolute error of 1.1 cm and a standard deviation of 0.45 cm. For 4D data association, evaluated over three time spans with manual annotation of 150 apples per session, precisions were 76.0%, 88.9%, and 75.2%; recalls were 87.8%, 86.0%, and 84.3%; and F1 scores were 81.4%, 87.4%, and 79.5%. The abstract additionally reports a 23.7% improvement in 4D data association precision over baselines (Lei et al., 2024).

The dataset is also described as covering more species, modalities, and time granularity than prior releases such as Fuji-air, Fuji-SfM, and MinneApple. That claim situates FruitNet simultaneously as a method paper and as an infrastructure contribution for orchard robotics, phenotyping, persistent mapping, long-term data association, and yield prediction.

6. Relation to adjacent fruit-vision research

FruitNet belongs to a broader family of fruit-sensing systems, but its technical target is distinct from both ripeness classification and generic fruit identification. In hyperspectral ripeness estimation, for example, one system measures the ripeness of avocados and kiwis with hyperspectral cameras and a compact HS-CNN, using 224-channel and 252-channel sensors, destructive ground-truth labeling, and a split of $I_l=\{o_{i,l}\}$6 training, $I_l=\{o_{i,l}\}$7 validation, and $I_l=\{o_{i,l}\}$8 test data. That work emphasizes non-destructive ripeness, firmness, and sweetness prediction, as well as false-color visualization of ripening, rather than persistent 3D or 4D orchard mapping (Varga et al., 2021).

Similarly, a separate fruit identification study uses YOLO-NAS on 2,909 images from 15 categories—Avocado, Banana, Cherry, Apple Braeburn, Apple Golden 1, Apricot, Grape, Kiwi, Mango, Orange, Papaya, Peach, Pineapple, Pomegranate, and Strawberry—and reports 99.98% mAP. Its emphasis is object detection, NAS-based architecture refinement, quantization-aware design, and deployment efficiency for real-time and edge-device applications (Dewi et al., 2024). FruitNet, by contrast, is organized around synchronized field robotics sensing, LiDAR-RGB fusion, metric-semantic map generation, and cross-session instance identity.

A source of terminological confusion is that the fruit-identification literature can use “FruitNet” more loosely. One comparative study states that “FruitNet systems (as referenced in the literature) typically achieve mAP up to 93.78%” (Dewi et al., 2024). This suggests that the label is not always restricted to the 4D orchard-monitoring framework. In the technical usage associated with persistent orchard monitoring, however, FruitNet denotes the modular, sensor-fused system for 4D fruit-level metric-semantic mapping and season-long individual-fruit tracking (Lei et al., 2024).

This distinction matters methodologically. A 2D classifier or detector can attain high mAP on curated category datasets without solving orchard-scale registration, instance persistence, or per-fruit longitudinal measurement. Conversely, FruitNet’s contribution is not ripeness-state inference or broad taxonomic recognition, but persistent association of the same physical fruit across changing scenes and timepoints.