PlantSAM: Plant-Adaptive Segmentation Model

• PlantSAM is a plant-adapted segmentation approach that integrates YOLO (detection) with SAM (segmentation) to accurately isolate plant regions from complex backgrounds.
• The pipeline employs patching, specialized prompting, and multi-phase segmentation to enhance trait recognition, achieving metrics such as a mean IoU of 0.94 and Dice of 0.97.
• Its modular design leverages frozen foundation models with plant-specific priors and prompt refinements, enabling effective application across herbarium, crop, and 3D phenotyping tasks.

PlantSAM denotes a plant-adapted use of the Segment Anything Model family in which generic promptable segmentation is constrained by plant-specific priors, auxiliary detectors, or domain-specific representations. In the narrow sense, it is the name of an object-detection–driven segmentation pipeline for digitized herbarium specimens that integrates YOLOv10 with SAM1 or SAM2, with the best-performing variant, PlantSAM2, combining YOLOv10 and SAM2 for automated plant–background segmentation in herbarium sheets (Sklab et al., 22 Jul 2025). In a broader research sense, the label has also been used to describe SAM-centered systems for zero-shot leaf segmentation, crop–weed panoptic segmentation, field delineation, tree-crown extraction, orchard counting, and NeRF-based 3D phenotyping, all of which couple SAM-like segmentation with plant-specific prompting, post-processing, or multimodal cues (Williams et al., 2023, Nguyen et al., 2023, Scribano et al., 19 Jun 2025, Teng et al., 26 Mar 2025, Jmal et al., 28 Aug 2025, Teng et al., 5 Jun 2025, Song et al., 19 May 2025). A plausible implication is that PlantSAM now refers both to a specific herbarium pipeline and to a methodological family of plant-aware SAM adaptations.

1. PlantSAM as a herbarium-specimen segmentation pipeline

In the specific sense established by "PlantSAM: An Object Detection-Driven Segmentation Pipeline for Herbarium Specimens" (Sklab et al., 22 Jul 2025), PlantSAM addresses a central problem in herbarium image analysis: the foreground specimen typically occupies a small fraction of the sheet, while the remainder contains paper, labels, pins, barcodes, mounting strips, and other artifacts. The paper reports that many taxa have less than 15–20% plant coverage, with examples including Magnolia at 19.68% plant and 80.32% background, and Convolvulaceae at 7.12% plant and 92.88% background. The stated motivation is that heterogeneous background can mislead deep learning classifiers and reduce trait-classification accuracy.

The architecture is organized into four phases: patching, plant region detection with YOLOv10, segmentation with SAM1 or SAM2, and unpatching. Herbarium sheets of approximately 4000×6000 pixels are preprocessed with erosion and dilation, then partitioned into patches whose size depends on image width: 1024, 512, or 256 pixels. Each patch is normalized and processed by a herbarium-specific YOLOv10 detector that predicts bounding boxes for plant regions. These boxes are then used as prompts for SAM. Two prompting strategies are described: a single-box strategy and a multi-region strategy. The latter became the default because a controlled experiment on 19 images increased the average plant-area / bounding-box-area ratio from 42.01% to 51.96%, thereby reducing background within prompts (Sklab et al., 22 Jul 2025).

PlantSAM distinguishes two model variants. PlantSAM1 combines YOLOv10 with SAM1, whereas PlantSAM2 combines YOLOv10 with SAM2. Unless otherwise specified, PlantSAM refers to the general YOLOv10 + SAM design. Both SAM variants are fine-tuned on herbarium masks, and the final masks from multiple prompts within a patch are merged by logical OR or union. After reconstruction at full resolution, the output is a binary plant mask covering leaves, stems, fruits, and flowers while excluding the sheet background. The authors also describe a semi-automatic annotation tool in which PlantSAM2 can be refined with point prompts, turning unusable masks into usable ones and enlarging training data (Sklab et al., 22 Jul 2025).

2. Training data, optimization, and segmentation performance

The herbarium PlantSAM study uses three distinct data resources. For segmentation, 1,476 images with high-quality masks were selected from previously published herbarium segmentation datasets; these images span 9 genera and 2 families, including Magnolia, Castanea, Rubus, Laurus, Litsea, Eugenia, Ulmus, Monimiaceae, Amborella, Desmodium, and Convolvulaceae. Evaluation uses a separate curated test set of 333 images from the Digitised Herbarium Image Segmentation Dataset. For plant-region detection, YOLOv10 is trained on 19,078 annotated patches, split into 14,307 training, 3,815 validation, and 956 test patches. For out-of-distribution analysis, the paper also uses 171 digitized herbarium images with challenging backgrounds such as blue, grey, yellow, brown, orange, pink paper, newspaper, long thin stems, long leaves, and pins (Sklab et al., 22 Jul 2025).

The segmentation models are trained with Dice loss,

$$D(P,G) = \frac{2 |P \cap G|}{|P| + |G|}, \qquad L_{\text{Dice}} = 1 - D(P,G),$$

using Adam with initial learning rate $10^{-5}$ and cosine annealing. SAM1 and SAM2 are trained for up to 80 epochs with batch size 1. A UNet with ResNet101 backbone is trained for up to 100 epochs with early stopping. YOLOv10 is trained for 250 epochs on 640×640 inputs and achieves mAP@0.5 = 0.959 on the plant-region detection test split (Sklab et al., 22 Jul 2025).

On the curated herbarium test set, PlantSAM2 achieves mean IoU = 0.94 and Dice = 0.97, outperforming both UNet and PlantSAM1. The paper reports taxon-level IoU gains over UNet, including Convolvulaceae from 0.8222 to 0.8789, Desmodium from 0.8337 to 0.9018, Ulmus from 0.8995 to 0.9386, Rubus from 0.9185 to 0.9504, and Magnolia from 0.9497 to 0.9656. Dice improvements are smaller but remain positive for many taxa, including Amborella from 0.9619 to 0.9750, Desmodium from 0.9410 to 0.9528, Rubus from 0.9714 to 0.9798, and Ulmus from 0.9654 to 0.9729 (Sklab et al., 22 Jul 2025).

The same paper links segmentation quality to downstream trait recognition. ResNet101 classifiers trained on segmented and segmented-cropped images outperform classifiers trained on raw sheets across five botanical traits: armatures, fruits, leaves with acuminate tips, infructescence, and leaves with an acute base. The largest reported gains for segmented-cropped versus unsegmented inputs are +4.36 accuracy for armatures and +4.15 F1 for infructescence. This suggests that, within herbarium analysis, PlantSAM is not only a segmentation module but also a preprocessing stage that reallocates image resolution from background to diagnostically relevant plant structure (Sklab et al., 22 Jul 2025).

3. Zero-shot and prompt-driven predecessors

Several earlier SAM-based plant pipelines established the design patterns later associated with PlantSAM. "Leaf Only SAM: A Segment Anything Pipeline for Zero-Shot Automated Leaf Segmentation" (Williams et al., 2023) uses SAM in automatic mask generation mode with a 32×32 grid of points and multi-level crops, then applies four post-processing stages: a green-pixel filter in HSV space with $35 \leq \bar{H} \leq 75$ and $\bar{S} > 35$, removal of whole-plant masks via $\text{IoU}(M_j,U)>0.9$, a shape filter based on $r_j = A_j / A^\text{circ}_j$ with threshold 0.1, and overlap-based pruning of multi-leaf composites using $\bar{S}_j > 1.5$ and 90% containment rules. On a potato leaf dataset, Leaf Only SAM attains AR$_{0.75}$ = 63.2, AP$_{0.75}$ = 60.3, AR = 58.4, AP = 56.4, and DSC = 0.700, while a fine-tuned Mask R-CNN reaches AR$_{0.75}$ = 78.7, AP$10^{-5}$0 = 74.7, AR = 69.8, AP = 65.4, and DSC = 0.871. The paper explicitly presents the SAM-based system as zero-shot with respect to the target potato dataset.

A later development is "Unlocking Zero-Shot Plant Segmentation with Pl@ntNet Intelligence" (Ravé et al., 14 Oct 2025), which combines PlantNet, its DinoV2 backbone, and SAM for agricultural imagery. The abstract states that specialized plant representations are used to identify plant regions and produce coarse segmentation masks, which are then refined by SAM into detailed segmentations. The method is evaluated on four publicly available datasets and reports consistent performance gains when using Plantnet-fine-tuned DinoV2 over base DinoV2, measured by the Jaccard Index (IoU). Because the provided details note that the actual paper text was unavailable, no architecture, dataset names, or numerical IoU values beyond the abstract can be reconstructed faithfully. Even so, the abstract clearly situates PlantSAM-like work at the intersection of foundation segmentation and plant-specialized representation learning (Ravé et al., 14 Oct 2025).

Together, these studies show two recurrent mechanisms. One is prompt-free or weakly prompted zero-shot segmentation followed by plant-specific filtering, as in Leaf Only SAM. The other is plant-aware localization followed by SAM refinement, as in the Pl@ntNet-based zero-shot approach. This suggests that PlantSAM, in the broad methodological sense, is less a single architecture than a prompt-and-prior paradigm for converting generic segmentation models into plant-centric systems.

4. Expansion to agriculture, remote sensing, and 3D phenotyping

The broader literature repeatedly reuses the PlantSAM pattern in different domains. In "A SAM-based Solution for Hierarchical Panoptic Segmentation of Crops and Weeds Competition" (Nguyen et al., 2023), HQ-SAM is paired with DINO and YOLOv8 detectors to generate box prompts for crop, weed, and leaf masks on PhenoBench. The best system, using HQ-SAM ViT-H with LoRA and DINO-enhanced detection, reaches PQ+ = 81.33, with IoU(soil) = 99.18, IoU(weed) = 70.66, PQ(leaf) = 73.81, PQ(crop) = 81.66, and overall PQ = 77.73. The design is explicitly detector-guided and hierarchical: semantic segmentation, plant instance segmentation, and leaf instance segmentation are all generated from prompt-conditioned SAM outputs.

In remote sensing, "Segment Anything for Satellite Imagery: A Strong Baseline and a Regional Dataset for Automatic Field Delineation" (Scribano et al., 19 Jun 2025) adapts SAM to field-boundary extraction using LoRA on the image encoder and optionally the decoder. The best configuration applies LoRA to both encoder and decoder with rank $10^{-5}$1 and single-point prompts, achieving mAP50 = 27.0 and mAR150 = 43.3 on AI4Boundaries. The same work introduces the ERAS dataset for Emilia-Romagna and reports that direct zero-shot SAM performs poorly on field delineation, while fine-tuned SAM+LoRA outperforms Mask R-CNN across AI4B, ERAS S2, and ERAS HR. In this setting, PlantSAM refers not to whole-plant objects but to an agricultural SAM specialized for parcel delineation.

Tree-level forestry work extends the same logic with elevation data. "Assessing SAM for Tree Crown Instance Segmentation from Drone Imagery" (Teng et al., 26 Mar 2025) shows that out-of-the-box SAM variants do not outperform a custom Mask R-CNN for young plantations, but RSPrompter and DSMPrompter, which keep SAM frozen while learning prompt generators and DSM encoders, surpass the baseline. The best DSMPrompter model reaches single-class mAP 65.03, mIoU 83.24, multi-class mAP 54.40, and weighted mAP 64.84. "Bringing SAM to new heights: Leveraging elevation data for tree crown segmentation from drone imagery" (Teng et al., 5 Jun 2025) generalizes this across boreal plantations, temperate forest, and tropical forest, and introduces BalSAM, which fuses RGB and DSM embeddings as

$10^{-5}$2

On Quebec plantations, BalSAM achieves single-class mAP 65.03, mIoU 83.24, multi-class mAP 54.40, and weighted mAP 64.84, outperforming Mask R-CNN+DSM on multi-class metrics. The same paper emphasizes that zero-shot SAM remains inadequate for automatic crown mapping, even when DSM-based point prompts are used (Teng et al., 5 Jun 2025).

In orchards, "Olive Tree Satellite Image Segmentation Based On SAM and Multi-Phase Refinement" (Jmal et al., 28 Aug 2025) uses SAM Automatic Mask Generator, row-and-column inference, SAM Predictor prompts at grid intersections, IQR-based area filtering, and cosine-similarity shape constraints. It reports an increase from 82.928% phase-1 count accuracy to 98.564% overall accuracy across five Tunisian olive plots. Here the central prior is agronomic regularity: rows and columns are inferred from initial SAM detections, and missing trees are then segmented with point or point+box prompts.

A 3D phenotyping extension appears in "IPENS: Interactive Unsupervised Framework for Rapid Plant Phenotyping Extraction via NeRF-SAM2 Fusion" (Song et al., 19 May 2025). IPENS combines SAM2 video segmentation with NeRF-based inverse rendering to obtain organ-level 3D masks and point clouds for rice and wheat. It reports grain-level segmentation mIoU of 63.72% on rice and 89.68% on wheat, while supporting downstream trait estimation such as rice grain volume with $10^{-5}$3 and wheat spike volume with $10^{-5}$4. The formulation introduces a 3D voxel mask field rendered along rays as

$10^{-5}$5

followed by optimization against SAM2 masks. This is a different modality from 2D herbarium segmentation, but it preserves the same PlantSAM logic: promptable foundation segmentation coupled to plant-specific geometry (Song et al., 19 May 2025).

5. Design principles across PlantSAM systems

Across these studies, several architectural principles recur. The first is detector- or representation-guided prompting. In herbarium PlantSAM, YOLOv10 provides SAM2 box prompts (Sklab et al., 22 Jul 2025). In crop–weed panoptic segmentation, DINO and YOLOv8 serve as box generators for HQ-SAM (Nguyen et al., 2023). In tree-crown studies, RSPrompter, DSMPrompter, and BalSAM learn prompts in embedding space, often while SAM itself remains frozen (Teng et al., 26 Mar 2025, Teng et al., 5 Jun 2025). In olive orchards, grid intersections act as point prompts for the SAM Predictor (Jmal et al., 28 Aug 2025). In IPENS, prompts are manual or YOLOv11-derived, then propagated by SAM2 across a video sequence (Song et al., 19 May 2025).

The second principle is multimodal or prior-driven refinement. PlantNet-fine-tuned DinoV2 provides plant-specialized representations before SAM refinement in zero-shot agricultural segmentation (Ravé et al., 14 Oct 2025). DSM or elevation data are incorporated either as point-prompt generators, fourth-channel inputs to R-CNN backbones, or dedicated encoders fused with SAM embeddings (Teng et al., 26 Mar 2025, Teng et al., 5 Jun 2025). In orchards, spatial alignment and shape consistency act as agronomic priors (Jmal et al., 28 Aug 2025). In Leaf Only SAM, color, extent, shape, and overlap filters replace learned priors with hand-crafted ones (Williams et al., 2023).

A third principle is modularity rather than end-to-end retraining. Herbarium PlantSAM fine-tunes YOLOv10 and SAM1/SAM2 directly (Sklab et al., 22 Jul 2025), but much of the broader PlantSAM literature keeps SAM frozen and learns only LoRA modules, prompt generators, side encoders, or task-specific detection heads (Scribano et al., 19 Jun 2025, Teng et al., 26 Mar 2025, Teng et al., 5 Jun 2025). This suggests that the appeal of PlantSAM lies partly in parameter-efficient adaptation under scarce annotation.

The fourth principle is that segmentation quality is often judged not only by IoU or Dice but by downstream plant-science utility. Herbarium PlantSAM improves five trait-classification tasks (Sklab et al., 22 Jul 2025). IPENS turns SAM2-derived masks into surface area, volume, length, and width estimates (Song et al., 19 May 2025). Olive-tree segmentation is evaluated by count accuracy rather than pixel overlap because tree counting is the stated operational goal (Jmal et al., 28 Aug 2025). This suggests that PlantSAM should be understood as an intermediate representation layer in larger phenotyping, biodiversity, or agronomic pipelines.

6. Limitations, misconceptions, and future directions

A common misconception is that SAM can simply be deployed in automatic mode for plant imagery. Multiple studies contradict this. Leaf Only SAM remains below a fine-tuned Mask R-CNN on potato leaf phenotyping (Williams et al., 2023). Out-of-the-box SAM for tree crowns yields single-class mAP values around 8–11 and does not outperform custom Mask R-CNN baselines (Teng et al., 26 Mar 2025, Teng et al., 5 Jun 2025). Field delineation papers likewise report poor zero-shot performance before LoRA adaptation (Scribano et al., 19 Jun 2025). The evidence therefore favors the narrower claim that SAM is a strong substrate for plant-specific adaptation, not a universally sufficient off-the-shelf plant segmenter.

Another misconception is that PlantSAM denotes a single universally accepted architecture. The literature instead shows at least three meanings. One is the named herbarium system YOLOv10 + SAM2 (Sklab et al., 22 Jul 2025). Another is a broad class of prompt-driven plant segmentation pipelines, often described explicitly in the supplementary details of other papers (Williams et al., 2023, Nguyen et al., 2023, Scribano et al., 19 Jun 2025). A third is a conceptual direction toward plant-specific foundation segmentation models, suggested by work on Pl@ntNet-specialized DinoV2, SAM fine-tuning, DSM fusion, and interactive 3D mask lifting (Ravé et al., 14 Oct 2025, Teng et al., 5 Jun 2025, Song et al., 19 May 2025).

The limitations are also domain-specific. Herbarium PlantSAM still encounters difficult out-of-distribution backgrounds, extremely fine structures, and imperfect ground-truth masks (Sklab et al., 22 Jul 2025). Leaf Only SAM depends on glasshouse imagery and hand-tuned HSV thresholds that may not transfer to field scenes (Williams et al., 2023). Tree-crown and field-boundary systems require DSM quality, large memory budgets, and domain-specific tuning (Teng et al., 26 Mar 2025, Scribano et al., 19 Jun 2025, Teng et al., 5 Jun 2025). Olive-tree refinement assumes approximately regular orchard grids (Jmal et al., 28 Aug 2025). IPENS currently depends on controlled, wind-free acquisition for NeRF reconstruction (Song et al., 19 May 2025).

The current research trajectory points in several directions. One is foundation-model specialization through plant-centric pretraining or fine-tuning, as exemplified by Pl@ntNet-based DinoV2 representations (Ravé et al., 14 Oct 2025). Another is multimodal PlantSAM, where RGB is augmented with DSM, radiance fields, or possibly other sensing modalities, though only DSM and NeRF-based representations are concretely documented in the provided sources (Teng et al., 5 Jun 2025, Song et al., 19 May 2025). A third is prompt automation through detectors, learned prompters, and grid or geometric priors (Sklab et al., 22 Jul 2025, Teng et al., 26 Mar 2025, Jmal et al., 28 Aug 2025). A plausible implication is that future PlantSAM systems will be less about replacing plant-specific modeling with a single general foundation model and more about integrating foundation segmentation with plant-specific structure, sensing, and task objectives.