NBV-SC: Next Best View Planning based on Shape Completion for Fruit Mapping and Reconstruction

Abstract: Active perception for fruit mapping and harvesting is a difficult task since occlusions occur frequently and the location as well as size of fruits change over time. State-of-the-art viewpoint planning approaches utilize computationally expensive ray casting operations to find good viewpoints aiming at maximizing information gain and covering the fruits in the scene. In this paper, we present a novel viewpoint planning approach that explicitly uses information about the predicted fruit shapes to compute targeted viewpoints that observe as yet unobserved parts of the fruits. Furthermore, we formulate the concept of viewpoint dissimilarity to reduce the sampling space for more efficient selection of useful, dissimilar viewpoints. Our simulation experiments with a UR5e arm equipped with an RGB-D sensor provide a quantitative demonstration of the efficacy of our iterative next best view planning method based on shape completion. In comparative experiments with a state-of-the-art viewpoint planner, we demonstrate improvement not only in the estimation of the fruit sizes, but also in their reconstruction, while significantly reducing the planning time. Finally, we show the viability of our approach for mapping sweet peppers plants with a real robotic system in a commercial glasshouse.

• The paper presents a novel approach where shape completion enhances next best view planning, significantly reducing computational overhead for fruit mapping.
• It integrates HSV-based fruit detection with 3D point cloud mapping to achieve improved accuracy and efficiency in reconstructing fruit geometry.
• Experimental results demonstrate an 80-90% reduction in planning time and validate NBV-SC's practical application in commercial glasshouse environments.

NBV-SC: Next Best View Planning based on Shape Completion for Fruit Mapping and Reconstruction

The paper "NBV-SC: Next Best View Planning based on Shape Completion for Fruit Mapping and Reconstruction" presents a novel approach in precision agriculture, focusing on the challenges associated with fruit mapping and reconstruction. The authors propose NBV-SC, an advanced viewpoint planning strategy that utilizes shape completion to enhance the efficiency and accuracy of fruit detection and mapping in agricultural environments.

Introduction

Fruit mapping in precision agriculture is vital for tasks such as yield estimation and automated harvesting. However, frequent occlusions and dynamic growth patterns present substantial challenges to accurate fruit perception and mapping. Traditional viewpoint planning strategies rely heavily on ray casting to maximize information gain, which is computationally expensive. In contrast, NBV-SC employs the predicted shapes of fruits to guide the next best view (NBV) selection, reducing computational overhead and improving information coverage. The system leverages real-time shape completion, enabling targeted viewpoint planning that focuses on unobserved fruit regions.

Figure 1: Shape completion based viewpoint planning. Completed shapes for partially detected fruits are estimated in real time for new viewpoint predictions.

System Overview

NBV-SC integrates several modules to achieve its objectives: fruit detection, mapping and shape completion, and NBV planning. The fruit detection module identifies fruits using an HSV-based segmentation method within RGB-D images. This segmentation process informs the surface mapping and shape completion module, which integrates observed point clouds to estimate the 3D shape of fruits. Subsequently, the NBV planning module utilizes these completed shape models to accurately synthesize and prioritize viewpoints aimed at minimizing occlusion and maximizing surface coverage.

Figure 2: Overview of our system, highlighting fruit detection, mapping, and viewpoint planning.

Missing Surface Viewpoint Synthesis

A key innovation in NBV-SC is the missing surface viewpoint synthesis strategy, which prioritizes viewpoints based on the geometry of unobserved fruit surfaces rather than random sampling. This method involves delineating predicted fruit nodes from Octomap occupancy data and synthesizing viewpoints that ensure coverage of previously occluded fruit areas. The synthesis process involves calculating prospective viewpoints that align with the contour of missing surfaces and optimizing for both utility and novelty relative to previous viewpoints.

Figure 3: Missing Surface Viewpoint Synthesis emphasizes occlusion-aware viewpoint planning.

Experimental Evaluation

The researchers undertook comprehensive simulations and real-world experiments to validate NBV-SC's effectiveness. Comparative evaluations with state-of-the-art RoI and viewpoint planners demonstrated marked improvements in both mapping accuracy and planning time. NBV-SC notably reduced the average viewpoint planning time by 80-90% compared to baselines in more complex scenarios. Fruit detection accuracy and reconstruction quality were significantly enhanced through targeted coverage facilitated by shape completion.

Moreover, NBV-SC's application in a commercial glasshouse setting confirmed its practical efficacy under real-world conditions, leveraging robotic systems to conduct mapping activities effectively.

Figure 4: NBV-SC deployed in real-world glasshouse environments, highlighting practical application capabilities.

Conclusion

The paper introduces NBV-SC as a robust and computationally efficient approach for next best view planning in fruit mapping applications. By incorporating shape completion, NBV-SC significantly improves information gain and reduces planning overhead. The implications for precision agriculture are substantial, promising advancements in automated fruit detection, mapping, and harvesting. Future work could explore the extension of NBV-SC to more diverse agricultural settings, optimizing algorithms for varying crop types and growing conditions, as well as incorporating machine learning methods to further refine the shape completion and viewpoint synthesis processes.