Measuring the Ripeness of Fruit with Hyperspectral Imaging and Deep Learning

Abstract: We present a system to measure the ripeness of fruit with a hyperspectral camera and a suitable deep neural network architecture. This architecture did outperform competitive baseline models on the prediction of the ripeness state of fruit. For this, we recorded a data set of ripening avocados and kiwis, which we make public. We also describe the process of data collection in a manner that the adaption for other fruit is easy. The trained network is validated empirically, and we investigate the trained features. Furthermore, a technique is introduced to visualize the ripening process.

• The paper introduces a custom hyperspectral convolutional neural network (HS-CNN) that achieves over 93% accuracy in predicting avocado firmness.
• It applies hyperspectral imaging to capture detailed spectral profiles reflecting chemical properties essential for determining the ripeness of avocados and kiwis.
• The study's ablation analysis confirms that augmentation techniques, focal loss, and Adabound optimizers are critical for enhancing network performance.

Measuring the Ripeness of Fruit with Hyperspectral Imaging and Deep Learning

Introduction

The determination of fruit ripeness is crucial in the agricultural and retail sectors. Traditional methods often involve destructive testing, limiting the scope to random samples. This paper explores a non-destructive approach to ascertain ripeness using hyperspectral imaging and deep neural networks. The authors present a dataset comprising hyperspectral images of avocados and kiwis, validated through empirical analysis demonstrating superior performance of a customized neural network architecture in predicting ripeness.

Figure 1: Visualization of the ripening process of an avocado.

Background and Related Work

Previous studies have employed hyperspectral imaging for ripeness assessment, generally utilizing classical machine learning algorithms. Notable works include Pinto et al. and Olarewaju et al., who assessed avocado ripeness, and Zhu et al., who analyzed kiwi firmness via hyperspectral techniques [Pinto2019], [Olarewaju2016], [Zhu2017]. Unlike these efforts, this study harnesses deep learning, which has shown promising results in hyperspectral data processing for remote sensing [Chen2014]. The deep learning approach in this study addresses the unique demands of fruit classification as opposed to established methods in remote sensing.

Figure 2: Two of the fruit crates at day 1 of the first test series.

Hyperspectral Imaging

Hyperspectral imaging captures beyond visible light spectra, with applications spanning various industries including medical technology and recycling [Lu2014], [Serranti2015]. The technique involves multiple channels, often exceeding 100, each representing intensity across specific wavelengths. In fruit ripening prediction, it identifies chemical properties indicative of ripeness, such as hydroxyl groups which are crucial for organic chemical transformations during ripening [Mitsui2008]. This methodology proves essential for examining the ripeness of avocados and kiwis.

Figure 3: The recording system. With the object holder and linear axis, the light source and the camera.

Data Collection and Processing

The authors describe an elaborate measurement setup involving hyperspectral cameras, object holders, and diffuse light sources to record high-definition images necessary for ripeness assessment. The dataset consists of 2,560 images of avocados and kiwis across stages from unripe to overripe. Labels are assigned based on firmness, sweetness, and overall appearance, with destructive testing used for validation of 442 samples.

Neural Network Architecture

A hyperspectral convolutional neural network (HS-CNN) tailored for fruit classification is proposed. It comprises key design elements like depth-wise separable convolutions to reduce parameters and prevent overfitting [Guo2019]. The network utilizes global average pooling layers for stable predictions and batch normalization to expedite the training process [Lin2014], [Ioffe2015].

Figure 4: Architecture of our hyperspectral convolutional neural network. The image of the input cube is a adapted version of \cite{Arbeck2013}.

Experiments and Results

Testing involved several models, including SVM, kNN, ResNet-18, AlexNet, and the proposed HS-CNN. The latter demonstrated superior accuracy, achieving over 93% for avocado firmness prediction and stability in ripeness classification for both avocados and kiwis. The study confirmed that access to full hyperspectral data led to significantly better outcomes than RGB or PCA preprocessed images.

Figure 5: The impact of the input on the decision of the class for an avocado recorded with the Specim FX 10.

Ablation Study

An ablation study investigated individual components of the HS-CNN, revealing the importance of augmentation techniques and specific architectural choices such as the use of focal loss and Adabound optimizers, which collectively contributed to the network's enhanced accuracy.

Visualization of Ripening Process

The authors introduce a technique involving pretrained autoencoders to generate false-color images that visually track ripening progression, aiding in non-destructive monitoring of fruit ripeness.

Figure 6: The architecture of the Pretrained approach. The image of the input cube is a adapted version of \cite{Arbeck2013}.

Conclusion

This paper presents a robust framework combining hyperspectral imaging with a specialized convolutional neural network to effectively predict fruit ripeness. The research not only contributes valuable datasets but offers a methodological approach that enhances predictive performance in avocados and kiwis. Future work could explore semi-supervised learning avenues to leverage unlabeled data, further enriching ripeness prediction systems.