Annotated Open-Source Lychee Dataset

• The Annotated Open-Source Lychee Dataset is a comprehensive RGB-D image corpus designed for lychee detection and ripeness classification under natural orchard conditions.
• It contains 11,414 images across raw, augmented, and depth modalities, with detailed human-in-the-loop annotations for both spatial localization and maturity staging.
• Benchmark evaluations using models like YOLO demonstrate that data augmentation and environmental variability boost performance metrics by 10%–20%, supporting enhanced robotic harvesting.

The Annotated Open-Source Lychee Dataset is a specialized RGB-D image corpus constructed to advance computer vision technologies for lychee detection and maturity classification in natural orchard environments. Designed to meet the needs of robotic harvesting research, it addresses the previous absence of comprehensive, consistently annotated open-source lychee datasets. The dataset comprises multimodal observations, meticulous human-in-the-loop annotation protocols, rigorous statistical analyses, and extensive benchmark experiments, thereby establishing a new standard for agricultural vision datasets.

1. Dataset Composition and Image Modalities

The dataset consists of 11,414 high-resolution images sourced from lychee orchards. Its structure includes three principal modalities:

• Raw RGB Images (878): Captured directly from a sensor module in situ, these form the foundational corpus.
• Augmented RGB Images (8,780): Generated via data augmentation techniques such as geometric transformations, noise injection, and pixel value shifts, these images extend intra-class and inter-class variability for robust algorithmic generalization.
• Depth Images (1,756): Derived by applying the monocular depth estimation network “DepthAnything” to RGB images, depth modalities enhance 3D scene understanding essential for precise fruit localization.

Annotation is performed for two tasks: spatial localization (rectangular bounding boxes) and categorical maturity classification (unripe, semi-ripe, ripe). In total, 9,658 pairs of detection and maturity labels are provided, facilitating multitask learning contexts. Standard annotation tools utilized include LabelImg (v1.8.6) and X-AnyLabel (v3.0.3).

Modality	Count	Annotation Type
Raw RGB	878	Location + Maturity
Augmented RGB	8,780	Location + Maturity
Depth (RGB-D)	1,756	Location + Maturity

2. Lychee Varieties and Environmental Conditions

The dataset encompasses images from four common cultivars: Nuomici, Feizixiao, Heiye, and Huaizhi. This diversity supports broad intra-class variability and augments cross-variety generalization in model training and evaluation.

Collection was conducted exclusively in natural orchard settings, incorporating variation across:

• Weather: Images span sunny and lightly rainy conditions (e.g., rainy conditions recorded on June 11).
• Temporal Diversity: Sampling at morning, noon, and evening times ensures image sets reflect diurnal lighting and fruit presentation dynamics.

The combination of varietal breadth and environmental variability maximizes the ecological validity and robustness of unitary or ensemble learning approaches for robotic harvesting.

3. Ripeness Staging and Statistical Formulation

Lychee maturity is annotated at three levels: unripe, semi-ripe, and ripe. Each annotation captures both spatial extent and categorical stage. To assess inter-date and inter-annotator consistency, the following summary statistics are defined:

• Average Annotation Area ($MAC$):

$$MAC = \frac{1}{N_c}\sum_{i=1}^{N_e} (W_i \cdot H_i)$$

where $N_e$ is the number of annotation boxes for a category on a specific date, $W_i$ and $H_i$ the normalized bounding box width and height, and $N_c$ the count of boxes for that category.

• Total Annotation Area ($TMA$):

$TMA = \sum_{c} MAC$

The application of these metrics provides quantitative measures for regional annotation homogeneity and supports subsequent model evaluation, particularly regarding balanced representation of ripeness stages across dates and varieties.

4. Annotation Methodology and Quality Assurance

Annotation was conducted independently by three trained individuals, each responsible for bounding box localization and assignment of one of the three maturity categories. Discrepancies were resolved through aggregation and validation by a fourth senior reviewer, ensuring high-quality, consistent labeling via human-in-the-loop protocols. Only vertical rectangular bounding boxes were employed, with future support planned for rotated rectangles (Cornell, Jacquard formats) to better characterize fruit pose and orientation.

Standardized annotation tools—LabelImg and X-AnyLabel—were selected for their compatibility with object detection pipelines and adherence to best practices in data labeling. This infrastructure establishes interoperability with existing agricultural vision datasets and eases downstream adaptation for domain-specific benchmarking.

5. Experimental Evaluation and Benchmarking

Extensive experiments were undertaken using three representative deep learning models:

• RT-DETR-ResNet50 (RT-D-Res)
• YOLOv8n
• YOLOv12n

Models were benchmarked on both raw-only and raw+augmented image splits (training/validation/test ratio of 8:1:1). Key findings include:

• Augmented data improved performance metrics (mAP50, mAP50-95, F1 score) by approximately 10%–20%.
• Precision-Recall curves showed a shift to the optimal region when using data augmentation.
• Confusion matrices demonstrated reduced misclassification, notably in differentiating semi-ripe and ripe stages.
• Reports include number of parameters, memory footprint, and performance trade-offs, supporting suitability assessments for embedded/robotic platforms.

A plausible implication is that extensive data augmentation combined with variety/environmental heterogeneity fosters robust model generalization on real-world harvesting tasks.

6. Public Release and Research Utilization

The dataset is publicly available for academic research at https://github.com/SeiriosLab/Lychee. The repository offers not only image and annotation files but also Python scripts for data augmentation, similarity analysis, and annotation management. Intended exclusively for research applications, its availability supports standardization in evaluation protocols and reproducibility in precision agriculture and robotic harvesting investigations.

The comprehensive structure and rigor inherent in the Annotated Open-Source Lychee Dataset set a benchmark for future agricultural datasets, facilitating the development and deployment of vision-based robotic harvesters under realistic field conditions.