FruitNet Dataset: Fruit Detection & 3D Completion

• FruitNet Dataset is a large-scale, annotated multimodal dataset offering fruit detection, instance localization, and 3D shape completion tasks for agricultural vision research.
• It includes dense annotations across RGB, NIR, and RGB-D imagery with multiple export formats and robust benchmark protocols to evaluate model performance.
• The dataset supports cutting-edge applications in precision agriculture by enabling assessment of detection mAP and 3D shape F1-scores, advancing robotic perception and synthetic data generation.

FruitNet refers to a set of large-scale, annotated datasets targeting fruit detection, instance localization, and—in some releases—three-dimensional shape completion for agricultural vision research. Distinct FruitNet collections have been adopted in multiple domains, including visible-spectrum object detection tasks (Sa et al., 2022) and 3D shape estimation under occlusions (Magistri et al., 2024). These datasets are characterized by dense annotation, multimodal imagery (including RGB, NIR, and RGB-D), and public benchmarks aimed at advancing robotic perception, machine learning, and synthetic data generation in precision agriculture.

1. Dataset Composition and Annotation Scope

1.1. FruitNet for Object Detection (deepNIR Project)

• Comprises 4,411 images, annotated with approximately 162,000 axis-aligned bounding boxes across 11 fruit classes: apple, avocado, capsicum, mango, orange, rockmelon, strawberry, blueberry, cherry, kiwi, and wheat.
• Images per class and bounding box frequencies vary widely, e.g., wheat has 3,373 images and 147,793 boxes, while apple has 63 images and 354 boxes.
• The dataset includes NIR+RGB image pairs from field, controlled, and satellite settings. Annotation splits are fixed at 80% training, 10% validation, and 10% test.
• Labels are stored in COCO JSON, Pascal VOC XML, YOLO TXT, and are accessible publicly under a CC-BY license (Sa et al., 2022).

1.2. FruitNet for 3D Shape Completion

• Provides 6,967 RGB-D frames of 129 unique sweet pepper fruits, sampled in both laboratory (66 fruits) and commercial greenhouse (63 fruits) conditions.
• The dataset supplies high-precision LiDAR point clouds as ground-truth 3D shapes for each fruit, alongside canonical segmentation masks and detailed sensor pose metadata.
• Split into training (4,580 frames), validation (1,387 frames), and a hidden test set (980 frames) for benchmarking, with scenario coverage varying by split (Magistri et al., 2024).

2. Imaging Modalities, Sensors, and Acquisition Protocols

2.1. RGB and NIR Data (deepNIR)

• Field-scale data employs the JAI AD-130GE dual-prism camera system, synchronously capturing RGB (400–700 nm) and NIR (750–790 nm) at 1280×960 px under controlled, night-time illumination.
• A separate “nirscene1” subset is acquired with dual off-the-shelf DSLRs (one with a 750 nm long-pass filter), processed to 256×256 px crops post-registration.
• Satellite collection uses ESA Sentinel-2’s high-resolution bands for both RGB and NIR at 10 m GSD, further standardized and tiled (Sa et al., 2022).

2.2. RGB-D and Ground-truth 3D for Shape Completion

• Intel RealSense D435i provides synchronized 640×480 px RGB and depth, with per-camera intrinsics (fx, fy, cx, cy, distortion) given.
• Ground truth 3D is gathered with a Perceptron V5 handheld LiDAR and precisely registered using the Hexagon ROMER Infinite 2.0 arm (≤0.1 mm positional accuracy). Registration between sensor and ground-truth reference is performed by TSDF fusion, RANSAC for plane detection, and iterative closest point (ICP) refinement (Magistri et al., 2024).

3. Benchmark Protocols and Performance Metrics

3.1. Object Detection Tasks

• Baseline models (YOLOv5s/x) are trained using both RGB-only and early-fusion RGB+synthetic NIR (4-channel). Evaluation metrics include mean Average Precision at Intersection-over-Union thresholds 0.5 and [0.5:0.95], along with per-class precision, recall, and F1 (Sa et al., 2022).
• Reported mAP@[0.5:0.95] ranges from 0.45 to 0.85, with synthetic NIR input conferring modest gains for some classes (e.g., apple, avocado), but minor reductions in others (e.g., cherry, kiwi). Synthetic NIR enhances detection in scenarios with occlusions, such as capsicum behind foliage.

3.2. 3D Shape Completion Challenge (FruitNet)

• Hidden test set hosted on CodaLab, requiring mesh submissions for each fruit instance in the canonical pose (peduncle aligned with +Z).
• Primary metric is $F_1$-score computed over multiple distance thresholds (0.01–0.1 m) between predicted ($R$) and ground-truth ($G$) point sets:

$F = \frac{2\bar p\,\bar r}{\bar p + \bar r}$

with area-under-curve precision ($\bar p$) and recall ($\bar r$), as defined by the dataset protocol.

• Chamfer distance $D_C(\mathcal{G},\mathcal{R})$ is used to break ties:

$$D_{C}(\mathcal{G},\mathcal{R}) = \tfrac12\,\bar d^2(\mathcal{G},\mathcal{R}) + \tfrac12\,\bar d^2(\mathcal{R},\mathcal{G})$$

where $$\bar d^2(A,B)=\frac1{|A|}\sum_{x\in A}\min_{y\in B}\|x-y\|^2$$ (Magistri et al., 2024).

3.3. Synthetic NIR Benchmarking

• Frechet Inception Distance (FID) is used for evaluating synthetic NIR generation:

$$d_{\mathrm{FID}}(A,B) = \|\mu_A-\mu_B\|_2^2 + \mathrm{Tr}\bigl(\Sigma_A + \Sigma_B - 2\,(\Sigma_A\Sigma_B)^{\tfrac12}\bigr)$$

with domain-specific results (e.g., SEN12MS: 11.36, nirscene1×400: 26.53, capsicum: 40.15) (Sa et al., 2022).

4. Data Organization, Annotation, and Licensing

• The FruitNet bounding-box dataset is organized to support major annotation export schemas (COCO, Pascal VOC, YOLO) and is accessible through a cloud annotation service as well as direct download; licensing is CC-BY (Sa et al., 2022).
• The 3D shape completion dataset follows a hierarchical folder structure by scenario (“lab”/“greenhouse”), split, and fruit id. Data for each instance includes spatially aligned RGB, depth, segmentation mask, registration pose, partial point clouds, and ground-truth meshes (Magistri et al., 2024).
• All dataset releases are accompanied by detailed documentation, canonical split protocols, and, where applicable, public leaderboard evaluation and instructions.

5. Recommended Applications and Known Limitations

Applications

• Training and benchmarking of single-stage and two-stage agricultural object detectors, with proven value for in-domain transfer learning and pre-training.
• Synthetic NIR generation as input for deep fusion models, NDVI computation, and semantic segmentation in precision farming scenarios.
• Benchmarking 3D shape completion techniques, particularly under real-world occlusions relevant to autonomous fruit picking and agrorobotics.
• Advancing multimodal domain translation (e.g., RGB-to-NIR via CycleGAN), semi-supervised learning via pseudo-labeling, and sensor fusion with thermal or depth channels (Sa et al., 2022, Magistri et al., 2024).

Limitations

• Generalization is limited for synthetic NIR when transferring to Internet-sourced or markedly different real-world imagery; models struggle to extrapolate beyond the training reflectance and illumination domains.
• The 3D dataset is restricted to sweet pepper, and while environmental variation is modeled (lab vs. greenhouse), the class scope is narrow.
• For bounding-box detection, domain gaps persist when applying models to cluttered or highly variable scenes not represented in the current corpus.
• Label split protocols are fixed, precluding standard k-fold or cross-validation workflows.

6. Relations to Existing Datasets and Future Directions

• FruitNet is complementary to the Fruits-360 dataset (Mureşan et al., 2017)—which focuses on 100×100 px, single-object, white-background images for classification—by extending to bounding-box detection in cluttered field images, multimodal fusion, and full 3D completion.
• A plausible implication is that convergence of high-precision 3D ground-truth and dense object-level annotation in FruitNet can catalyze new research in agricultural scene understanding, freeing models from reliance on synthetic or simulated data.
• The benchmark protocols, comprehensive sensor metadata, and focus on domain fidelity position FruitNet collections as standard resources for evaluating both image-based and geometry-based perception tasks in agricultural automation.

---

References:

• "deepNIR: Datasets for generating synthetic NIR images and improved fruit detection system using deep learning techniques" (Sa et al., 2022)
• "A Dataset and Benchmark for Shape Completion of Fruits for Agricultural Robotics" (Magistri et al., 2024)
• "Fruit recognition from images using deep learning" (Mureşan et al., 2017)