DexFruit: Robotic Gentle Fruit Manipulation

• DexFruit is a robotic manipulation framework integrating tactile-informed diffusion and optical tactile sensors for gentle, autonomous fruit handling.
• It combines visual and tactile inputs to dynamically adjust grasping, achieving up to 31% improvement in success rate and reducing visual bruising by 20% compared to baselines.
• The system features FruitSplat, a 3D Gaussian Splatting pipeline that quantifies surface bruising from pre- and post-manipulation inspections to provide rigorous damage metrics.

DexFruit is a robotic manipulation framework for repeated, autonomous pick-and-place of very fragile fruits while minimizing bruising and internal damage and then quantitatively measuring that damage. It combines a tactile-informed diffusion policy for gentle grasping with a standard parallel-jaw gripper and optical tactile sensors, and a 3D Gaussian Splatting pipeline, FruitSplat, for pre/post inspection of visual damage from webcam video. In reported experiments on strawberries, cherry tomatoes, and blackberries, the system achieves a 92% grasping policy success rate, up to a 20% reduction in visual bruising, and up to an 31% improvement in grasp success rate on challenging fruit compared to baselines, evaluated over more than 630 trials (Swann et al., 9 Aug 2025).

1. Problem setting and motivation

DexFruit addresses a narrow but difficult problem in agricultural robotics: repeated, autonomous pick-and-place of very fragile fruits while minimizing bruising and internal damage, followed by quantitative damage assessment. The targeted fruits are strawberries, cherry tomatoes, and blackberries, all of which are easily bruised or crushed, variable in size and shape, and associated with short post-harvest shelf life. The framework is explicitly motivated by the gap between fresh-market quality standards and the limitations of current mechanized or robotic harvesters, which are usually tailored to a single fruit, rely heavily on soft grippers, often damage fruit more than human pickers, and still depend on humans for sorting and quality control (Swann et al., 9 Aug 2025).

The evaluation problem is treated as equally important. Human visual grading is described as subjective and low-resolution, while higher-precision methods such as MRI, X-ray, OCT, and spectroscopy are expensive, slow, and impractical for field robotics. Existing deep learning approaches to bruise detection are characterized as predominantly 2D, lacking a proper 3D metric of damage on the fruit surface. DexFruit therefore couples manipulation and inspection rather than treating them as separate subsystems (Swann et al., 9 Aug 2025).

This dual emphasis distinguishes the framework from fruit-perception pipelines focused on detection or segmentation alone. Related work on annotation-free fruit perception has used SAM2 and OpenCLIP to produce pseudo labels for object detection, semantic segmentation, and instance segmentation without manual annotation, and has introduced the MegaFruits dataset with over 25,000 images (Wang et al., 2024). A separate orchard study evaluated YOLOv12, YOLO11, YOLOv10, YOLOv9, and YOLOv8 for fruitlet detection and counting in complex green-on-green environments (Sapkota et al., 2024). DexFruit operates in a different regime: dexterous handling of fragile fruit and post-manipulation damage quantification.

2. Robotic platform and task formulation

The hardware platform comprises a UFactory xArm7 in position control mode and a standard parallel-jaw gripper whose finger pads are two DenseTact optical tactile sensors. The policy also receives RGB input from a wrist-mounted Intel RealSense D435i, while FruitSplat uses a simple webcam for 3D reconstructions. During each grasp-lift-place cycle, the control inputs are a wrist-camera RGB image at $224 \times 224$, two DenseTact RGB images at $224 \times 224$ each, and robot state consisting of end-effector 3D position, 6D rotation, and gripper width. The policy outputs end-effector pose and gripper width in $[0,1]$, and runs at approximately 10 Hz (Swann et al., 9 Aug 2025).

The task is deliberately constrained to isolate handling quality. The robot begins approximately 20 cm from a fruit placed on a small pedestal. The policy must approach, grasp, lift, and place the fruit on a goal region defined as a green square. Success is defined as the fruit ending in the goal region without being dropped. Missed grasps and slips or drops before reaching the goal count as failures. The fruit’s initial location is fixed so that localization is not the focus; the experiments are intended to isolate handling quality and tactile effects (Swann et al., 9 Aug 2025).

Training data are collected by teleoperation with a 3DConnexion space mouse for arm motion and a slider for gripper width. For the full method, the teleoperator observes both RGB and DenseTact images. The dataset covers strawberries, cherry tomatoes, and blackberries, with 100–200 demonstrations per fruit type, and the same demonstration sets are used for all policy variants. All image streams are resized to $224 \times 224$, and training is performed for 1000 epochs with a DDIM scheduler (Swann et al., 9 Aug 2025).

3. Tactile-informed diffusion policy

DenseTact is used as an optical tactile sensor: a camera observes deformation of a soft gel when the fruit contacts the gripper. In DexFruit, each DenseTact sensor produces RGB images whose raw visual changes can be subtle for soft berries. The framework therefore maintains a reference undeformed image for each sensor, computes image differences at each time step, converts them to grayscale, and derives both per-pixel differences and an aggregate energy defined as the sum of squared differences. Policies trained on raw DenseTact RGB are reported to perform poorly, whereas difference images yield stronger and more informative contact patterns (Swann et al., 9 Aug 2025).

The manipulation policy is a diffusion-based imitation learning policy using the Diffusion Policy framework. The action $A_t$ is the end-effector pose, represented as 3D position plus a 6D rotation representation, together with gripper width. The observation $\mathcal{O}_t$ contains wrist-camera RGB, two DenseTact difference images, and robot state. Rather than learning a direct map $\pi(\mathcal{O}) \rightarrow A$, the system models action generation as a denoising process in action space: $$\hat{\varepsilon}_k = \varepsilon_{\theta}(A^k_t, \mathcal{O}_t, k),$$ with training objective

$$\mathcal{L} = \mathrm{MSE}\Bigl(\epsilon^k, \varepsilon_{\theta}(A^k_t, \mathcal{O}_t, k)\Bigr).$$

At inference, a noisy action sample is iteratively denoised with a DDIM scheduler to produce the action conditioned on the current observation (Swann et al., 9 Aug 2025).

The observation encoder is multimodal. Wrist RGB is encoded by a ResNet-34 trained from scratch, each DenseTact difference image is encoded by a ResNet-18 trained from scratch, and the robot state vector is concatenated directly. A central contribution is adaptive modality switching based on tactile contact energy. Before contact, the policy relies on wrist vision to localize and approach the fruit. Once the DenseTact difference energy exceeds a threshold $\tau$, vision is switched off and tactile sensing is switched on for contact-rich control. This design is explicitly contrasted with a naive Vision+Touch* variant that concatenates both modalities at all times. That naive fusion is reported to underperform, because the network tends to overfit to vision and ignore tactile cues, leading to more damage and lower success (Swann et al., 9 Aug 2025).

The ablations define four policies: Ours (DexFruit), Vision, Touch, and Vision+Touch*. Their comparison makes two points. First, tactile sensing is necessary for regulating grip geometry and avoiding damage once contact occurs. Second, tactile sensing alone does not provide sufficient global guidance for approach. This helps explain why Touch can succeed after contact but has lower overall success, while Vision can approach accurately yet causes substantial damage when grasp geometry is not contact-aware (Swann et al., 9 Aug 2025).

4. FruitSplat and 3D bruise quantification

FruitSplat is a 3D Gaussian Splatting-based pipeline for quantitative bruise analysis of strawberries. For each experimental strawberry, two 360° handheld webcam videos are captured: one before manipulation and one after repeated grasps. These videos are processed with COLMAP to estimate camera poses and a sparse 3D point cloud, providing calibration for 3D Gaussian Splatting. No additional hardware is required beyond a webcam (Swann et al., 9 Aug 2025).

The underlying rendering model is standard 3DGS. A camera ray is

$224 \times 224$0

and the rendered color along the ray is

$224 \times 224$1

RGB supervision uses a weighted blend of $224 \times 224$2 and D-SSIM: $224 \times 224$3 DexFruit builds on Nerfstudio’s Splatfacto but adds two scalar parameters per Gaussian: $224 \times 224$4, the probability that the Gaussian belongs to a strawberry, and $224 \times 224$5, the probability that it belongs to a bruise region. These channels are rendered analogously to color, and their 2D mask predictions are supervised with binary cross-entropy using pseudo ground-truth masks. A stop-grad trick is applied during rasterization of the mask channels so that gradients flow only into $224 \times 224$6 and $224 \times 224$7, which is reported to improve training stability (Swann et al., 9 Aug 2025).

The 2D supervisory masks come from two models: a custom fine-tuned YOLOv12 model for bruise segmentation and Grounded SAM 2 for strawberry instance masks. The YOLOv12 model is trained on approximately 2000 hand-labeled images of strawberries with bruises. After training the 3DGS model, Gaussian means, strawberry scores, and bruise scores are exported. Non-strawberry Gaussians are filtered out using $224 \times 224$8, yielding a high-resolution strawberry surface point cloud. Each retained point carries a bruise probability from $224 \times 224$9. For a bruise confidence threshold $[0,1]$0, points with $[0,1]$1 are treated as bruised, and the percentage of bruised points over all strawberry points is computed. The difference between the pre-manipulation and post-manipulation reconstructions defines the 3D quantitative surface bruising metric (Swann et al., 9 Aug 2025).

FruitSplat is explicitly modular with respect to the 2D models. Any model that outputs pixel-wise fruit and damage masks can be used without retraining the 3DGS architecture itself; the only requirements are images with camera poses and per-image 2D masks. This suggests a direct connection to broader agricultural perception pipelines. For example, SDM-D uses SAM2 and OpenCLIP to generate pseudo labels for fruit detection and segmentation without manual annotation, and emphasizes that such foundation-model pipelines can be distilled into compact models for edge deployment (Wang et al., 2024). In DexFruit, the modularity claim is narrower: YOLOv12 and Grounded SAM 2 are examples, not mandatory choices.

A frequent misunderstanding is that FruitSplat measures all forms of fruit damage. It does not. Because only surface Gaussians are retained, the reported quantity is visual surface bruising, which the authors identify as the relevant quantity for visual quality grading. Internal damage is handled separately through stiffness testing (Swann et al., 9 Aug 2025).

5. Experimental design and quantitative findings

The evaluation spans more than 630 trials across three fruit categories. Strawberries are divided into small ($[0,1]$2 cm), medium ($[0,1]$3–$[0,1]$4 cm), and large ($[0,1]$5 cm), with 25 repeated pick-and-place trials per strawberry. Cherry tomatoes are divided into small ($[0,1]$6 cm) and medium ($[0,1]$7 cm), with 50 trials per tomato. Blackberries are divided into small ($[0,1]$8 cm), medium ($[0,1]$9–$224 \times 224$0 cm), and large ($224 \times 224$1 cm), with 20 trials per blackberry. Fruit are sourced from a single lot on the same day to control ripeness and quality (Swann et al., 9 Aug 2025).

Damage is quantified in two ways. External bruising is measured only for strawberries, using FruitSplat to compare pre- and post-manipulation percentages of bruised surface points. Internal structural damage is measured for strawberries and tomatoes with a stiffness metric: an ATI Nano17 force–torque sensor and a 3D-printed circular indenter apply a 1.25 mm indentation at several points around the fruit circumference, the force in the $224 \times 224$2-direction is recorded, approximate spring constant $224 \times 224$3 is computed per point, and values are averaged across four points per fruit. Percent stiffness is defined relative to pre-experiment stiffness, with 100% corresponding to no internal damage. Blackberries are excluded from stiffness testing because of their extreme susceptibility to damage and the difficulty of consistent indentation (Swann et al., 9 Aug 2025).

The main results show clear separation among the policies. The full DexFruit method achieves 95% success on strawberries, 100% on tomatoes, and 80% on blackberries. The Vision baseline achieves 91%, 69%, and 58% respectively, while Touch achieves 67%, 100%, and 55%. On strawberries, retained stiffness is 80% for DexFruit, 78% for Touch, and 29% for Vision. On tomatoes, retained stiffness is 95% for DexFruit, 93% for Touch, and 77% for Vision. For strawberry visual bruising, DexFruit yields a 0.003% increase, Vision yields 20.68%, and Touch yields $224 \times 224$4, which the authors attribute likely to noise and slight inconsistencies in YOLO bruise segmentation and interpret as effectively no increase. The paper summarizes these findings as an average 92% grasping policy success rate, up to a 31% improvement in grasp success rate on challenging fruit, and up to a 20% reduction in visual bruising relative to baselines (Swann et al., 9 Aug 2025).

The size-dependent analysis is important. Vision-only shows difficulty for small and medium strawberries, which the authors attribute to likely overfitting to an average fruit size, producing grasps that are too loose for small fruits and too tight for large ones. Touch-only can often recover once contact occurs but suffers lower overall success because it lacks global guidance. Vision+Touch* underperforms all other methods across fruit sizes, reinforcing the claim that naive multimodal concatenation causes the policy to ignore tactile information. The core empirical point is therefore not merely that tactile sensing helps, but that its use must be conditioned on the contact regime (Swann et al., 9 Aug 2025).

Qualitative findings align with the metrics. FruitSplat reconstructions show minimal or no visible bruising after 25 grasps for DexFruit, while Vision produces large bruise regions around contact zones. For blackberries, the full method preserves many berries intact across 20 grasps, whereas the vision and touch baselines show more crushed berries and deformations. For tomatoes, visible bruising is minimal, and the stiffness test carries most of the evidential weight (Swann et al., 9 Aug 2025).

6. Limitations, interpretation, and significance

Several limitations are explicit. FruitSplat bruise detection is implemented only for strawberries, where bruises are clearly visible. Extension to tomatoes, blackberries, and other fruits requires dedicated bruise segmentation models, and external bruising may be much less visually apparent, as in tomatoes. The experiments are conducted in controlled laboratory settings rather than in the field, so variable lighting, occlusions, and moving foliage are outside the evaluated scope. Ripeness estimation is not incorporated into the tactile signal pathway. The visual-tactile switch is based on a fixed threshold $224 \times 224$5 over DenseTact difference energy rather than a learned gating mechanism. Finally, FruitSplat remains an offline per-fruit reconstruction step even though it is positioned as relatively fast for robotics (Swann et al., 9 Aug 2025).

These limitations clarify what DexFruit does and does not establish. It does not solve orchard-scale localization or field robustness, and it does not claim a general-purpose 3D defect-inspection framework for all crops. A plausible implication is that the system is best understood as a tightly integrated laboratory benchmark for gentle handling and rigorous post-manipulation evaluation. In that respect, its design is complementary to other agricultural vision research. Annotation-free perception frameworks such as SDM-D are aimed at scaling fruit detection and segmentation across datasets and deployment scenarios (Wang et al., 2024), while orchard evaluations of YOLOv12 and related families target real-time detection and counting in complex field environments (Sapkota et al., 2024). DexFruit instead contributes a manipulation-centric stack whose main novelty lies in contact-aware control and 3D damage quantification.

The significance claimed by the authors is threefold. First, a standard parallel-jaw gripper, combined with DenseTact sensing and a tactile-informed diffusion policy, can achieve soft-gripper-like gentleness without specialized mechanical designs. Second, the framework provides rigorous experimental evidence that tactile-aware manipulation reduces both external bruising and internal structural damage while improving task success. Third, FruitSplat offers a modular, quantitative 3D surface-bruising metric that can be used to compare manipulation strategies, policies, or gripper designs objectively (Swann et al., 9 Aug 2025). In aggregate, DexFruit defines a research program in which dexterous handling and damage measurement are co-designed rather than evaluated in isolation.