ZeroPlantSeg: Zero-Shot Plant Segmentation

• The paper introduces ZeroPlantSeg, a zero-shot method combining SAM-based leaf extraction with vision-language attention to group leaves into plant instances.
• It employs greedy clustering with Mahalanobis distance for unsupervised plant grouping, overcoming annotation burdens inherent to supervised approaches.
• Experimental results on PhenoBench, GrowliFlower, and SB20 demonstrate competitive performance across species and environmental conditions.

ZeroPlantSeg is a zero-shot hierarchical plant segmentation method for rosette-shaped plant individuals in top-view images. It combines a foundation segmentation model, which extracts leaf instances, with a vision-LLM, which reasons about plant structure to extract plant individuals without additional training. The method targets the hierarchical segmentation problem in dense crop scenes, where overlapping and mutually occluding leaves make it insufficient to segment leaves alone; the central objective is to recover both leaf instances and the plant individual to which each leaf belongs, while avoiding the species-specific annotation burden typical of supervised approaches (Xing et al., 11 Sep 2025).

1. Problem formulation and operating assumptions

ZeroPlantSeg is defined around top-view RGB imagery of rosette-shaped plants, a setting represented by datasets such as PhenoBench, GrowliFlower, and SB20. In this geometry, leaves radiate from a plant center, and that structural prior is central to the method. The paper frames hierarchical segmentation as a two-level task: leaf instance segmentation separates each leaf, while plant instance segmentation groups leaves into the correct plant individual. The difficulty arises because neighboring plants overlap heavily, so a generic zero-shot segmenter can produce plausible masks without resolving plant membership (Xing et al., 11 Sep 2025).

The method is motivated by the limitations of supervised hierarchical segmentation systems such as Weyler et al. and HAPT. Those methods can address the task when trained on annotated datasets, but the annotations are labor-intensive and often species-specific. ZeroPlantSeg replaces task-specific training with pre-trained foundation components and unsupervised grouping. A common misconception is to equate “zero-shot” with the absence of learned priors; in this case, the method is explicitly based on pre-trained segmentation and vision-LLMs, but it does not require additional training on the target segmentation task or domain (Xing et al., 11 Sep 2025).

A further operating assumption is morphological specificity. The approach is designed for rosette-shaped plants viewed from above, because the clustering logic depends on leaf bases concentrating near a plant center. The paper explicitly notes that this heuristic basis makes the method less directly applicable to crops with very different structures, such as wheat (Xing et al., 11 Sep 2025).

2. End-to-end architecture

The input image is an RGB top-view image

$\mathbf{I} \in \mathbb{R}^{3 \times H \times W}.$

Rather than applying Segment Anything Model (SAM) once on the full image, ZeroPlantSeg crops the image using a sliding window and upsamples the crop to improve detection of small leaves. Each crop is processed by SAM’s Automatic Mask Generator, which produces candidate masks from evenly distributed point prompts. Because crops overlap and SAM may emit multiple overlapping masks for the same object, duplicate proposals are removed by non-maximum suppression, following the idea used in Leaf Only SAM (Xing et al., 11 Sep 2025).

Raw SAM outputs are class-agnostic, so ZeroPlantSeg adds a mask filtering stage using OVSeg as a mask-adapted image classifier. The text prompts are "green leaf" and "soil". A candidate mask is discarded if the masked region is more similar to soil than to green leaf. The surviving leaf-mask set is written as

$$\mathcal{M} = \{\mathbf{M}^i\}_{i=1}^{N}, \qquad \mathbf{M}^i \in \{0,1\}^{H \times W}.$$

This stage defines the leaf instance segmentation output of the system (Xing et al., 11 Sep 2025).

For each binary mask, a masked leaf image is constructed by element-wise multiplication with the input image:

$$\mathcal{L} = \left\{\mathbf{L}^i \in \mathbb{R}^{3 \times H \times W} \mid \mathbf{L}^i = \mathbf{I} \otimes \mathbf{M}^i,\ \mathbf{M}^i \in \mathcal{M}\right\}_{i=1}^{N}.$$

Each masked leaf image is then cropped and resized to a square image

$$\hat{\mathbf{L}}^i \in \mathbb{R}^{3 \times D \times D},$$

with implementation setting $D=800$. This normalization isolates each candidate leaf for the subsequent structural reasoning stage (Xing et al., 11 Sep 2025).

3. Text-to-image attention and leaf-base inference

The key innovation of ZeroPlantSeg is that it does not group leaves by mask geometry alone. Instead, it uses text-to-image cross-attention from a pre-trained vision-LLM to infer stem- or petiole-related structure within each leaf crop. The specific source is the text-to-image cross-attention module of Grounding DINO’s SwinT OGC module, queried with prompts such as "stem" and "petiole". The model produces features at four resolutions, from $8\times 8$ to $64\times 64$, which are resized and averaged into a final attention map

$\mathbf{F}^i \in \mathbb{R}^{100 \times 100}_{+}.$

This map highlights regions in the cropped leaf that correspond to stem-like or petiole-like parts (Xing et al., 11 Sep 2025).

From $\mathbf{F}^i$, the method fits a straight line using weighted least squares. The fitted line segment inside the leaf mask is interpreted as a stem line, and its endpoints become the leaf keypoints

$$\mathcal{Q} = \{q^i = (\mathbf{e}_1^i,\mathbf{e}_2^i)\}_{i=1}^{N}.$$

These two endpoints are candidate base and tip points. To decide which endpoint is the base, the method uses the gravity center of the feature map: the endpoint closer to the gravity center is treated as the base point, while the other is treated as the tip point (Xing et al., 11 Sep 2025).

This design encodes a structural prior specific to rosette plants. Plant grouping is driven not by the center of the leaf mask, but by a point intended to approximate where the leaf attaches to the plant. The paper’s ablation shows that clustering based on attention-derived stem/base keypoints performs better than clustering based only on the gravity centers of leaf masks. This indicates that the vision-language attention is not used merely as a semantic detector, but as a mechanism for extracting a grouping-relevant geometric cue (Xing et al., 11 Sep 2025).

4. Greedy clustering and construction of plant instances

After keypoint extraction, ZeroPlantSeg infers plant individuals through an unsupervised greedy clustering procedure. A naive application of DBSCAN to all keypoints is described as unreliable, because the point set includes both base points, which cluster near plant centers, and tip points, which may lie far from the center and induce invalid clusters. ZeroPlantSeg therefore first selects base-like keypoints, denoted

$$\mathcal{M} = \{\mathbf{M}^i\}_{i=1}^{N}, \qquad \mathbf{M}^i \in \{0,1\}^{H \times W}.$$0

using endpoint proximity to the feature-map gravity center (Xing et al., 11 Sep 2025).

DBSCAN is then applied to $$\mathcal{M} = \{\mathbf{M}^i\}_{i=1}^{N}, \qquad \mathbf{M}^i \in \{0,1\}^{H \times W}.$$1, and the largest cluster $$\mathcal{M} = \{\mathbf{M}^i\}_{i=1}^{N}, \qquad \mathbf{M}^i \in \{0,1\}^{H \times W}.$$2 is interpreted as the center of a single plant individual. Once such a cluster has been found, all leaf instances whose keypoints belong to $$\mathcal{M} = \{\mathbf{M}^i\}_{i=1}^{N}, \qquad \mathbf{M}^i \in \{0,1\}^{H \times W}.$$3 are assigned to that plant and removed from further consideration:

$$\mathcal{M} = \{\mathbf{M}^i\}_{i=1}^{N}, \qquad \mathbf{M}^i \in \{0,1\}^{H \times W}.$$4

where $$\mathcal{M} = \{\mathbf{M}^i\}_{i=1}^{N}, \qquad \mathbf{M}^i \in \{0,1\}^{H \times W}.$$5 returns the set of leaf instances containing keypoints in $$\mathcal{M} = \{\mathbf{M}^i\}_{i=1}^{N}, \qquad \mathbf{M}^i \in \{0,1\}^{H \times W}.$$6, and $$\mathcal{M} = \{\mathbf{M}^i\}_{i=1}^{N}, \qquad \mathbf{M}^i \in \{0,1\}^{H \times W}.$$7 returns the keypoints of those leaf instances. This process is repeated until no more clusters are formed or a predefined number of clusters is reached (Xing et al., 11 Sep 2025).

Outliers are handled explicitly. Let

$$\mathcal{M} = \{\mathbf{M}^i\}_{i=1}^{N}, \qquad \mathbf{M}^i \in \{0,1\}^{H \times W}.$$8

be the remaining outlier points. Each outlier is assigned to the nearest cluster using Mahalanobis distance,

$$\mathcal{M} = \{\mathbf{M}^i\}_{i=1}^{N}, \qquad \mathbf{M}^i \in \{0,1\}^{H \times W}.$$9

and the assignment rule is

$$\mathcal{L} = \left\{\mathbf{L}^i \in \mathbb{R}^{3 \times H \times W} \mid \mathbf{L}^i = \mathbf{I} \otimes \mathbf{M}^i,\ \mathbf{M}^i \in \mathcal{M}\right\}_{i=1}^{N}.$$0

with $$\mathcal{L} = \left\{\mathbf{L}^i \in \mathbb{R}^{3 \times H \times W} \mid \mathbf{L}^i = \mathbf{I} \otimes \mathbf{M}^i,\ \mathbf{M}^i \in \mathcal{M}\right\}_{i=1}^{N}.$$1. If the distance is below threshold $$\mathcal{L} = \left\{\mathbf{L}^i \in \mathbb{R}^{3 \times H \times W} \mid \mathbf{L}^i = \mathbf{I} \otimes \mathbf{M}^i,\ \mathbf{M}^i \in \mathcal{M}\right\}_{i=1}^{N}.$$2, the leaf is assigned to an existing plant; otherwise it becomes a singleton cluster, corresponding to a small plant with a single leaf (Xing et al., 11 Sep 2025).

Final plant masks are constructed by union over the leaf masks assigned to each cluster:

$$\mathcal{L} = \left\{\mathbf{L}^i \in \mathbb{R}^{3 \times H \times W} \mid \mathbf{L}^i = \mathbf{I} \otimes \mathbf{M}^i,\ \mathbf{M}^i \in \mathcal{M}\right\}_{i=1}^{N}.$$3

If a pixel appears in multiple plant masks, the method resolves the ambiguity by majority voting. The output is therefore hierarchical: a set of leaf masks and a set of plant-instance masks assembled from them (Xing et al., 11 Sep 2025).

5. Evaluation protocol, comparative performance, and ablations

ZeroPlantSeg is evaluated on three datasets spanning different species, growth stages, and capture environments. PhenoBench contains sugar beet field images captured by UAV and is split into 1407 training, 386 validation, and 386 test images because test annotations are unavailable. GrowliFlower consists of cauliflower field images captured by UAV, using images from August 12 and August 19, with 254 validation and 261 test images. SB20 is a sugar beet field dataset from Campus Klein Altendorf in Bonn, Germany, captured using an agricultural field robot and split randomly into 72 validation and 72 test images. The reported metrics are $$\mathcal{L} = \left\{\mathbf{L}^i \in \mathbb{R}^{3 \times H \times W} \mid \mathbf{L}^i = \mathbf{I} \otimes \mathbf{M}^i,\ \mathbf{M}^i \in \mathcal{M}\right\}_{i=1}^{N}.$$4, $$\mathcal{L} = \left\{\mathbf{L}^i \in \mathbb{R}^{3 \times H \times W} \mid \mathbf{L}^i = \mathbf{I} \otimes \mathbf{M}^i,\ \mathbf{M}^i \in \mathcal{M}\right\}_{i=1}^{N}.$$5, $$\mathcal{L} = \left\{\mathbf{L}^i \in \mathbb{R}^{3 \times H \times W} \mid \mathbf{L}^i = \mathbf{I} \otimes \mathbf{M}^i,\ \mathbf{M}^i \in \mathcal{M}\right\}_{i=1}^{N}.$$6, $$\mathcal{L} = \left\{\mathbf{L}^i \in \mathbb{R}^{3 \times H \times W} \mid \mathbf{L}^i = \mathbf{I} \otimes \mathbf{M}^i,\ \mathbf{M}^i \in \mathcal{M}\right\}_{i=1}^{N}.$$7, and $$\mathcal{L} = \left\{\mathbf{L}^i \in \mathbb{R}^{3 \times H \times W} \mid \mathbf{L}^i = \mathbf{I} \otimes \mathbf{M}^i,\ \mathbf{M}^i \in \mathcal{M}\right\}_{i=1}^{N}.$$8 (Xing et al., 11 Sep 2025).

Dataset	Setting	Reported ZeroPlantSeg results
PhenoBench	In-domain sugar beet, UAV	$$\mathcal{L} = \left\{\mathbf{L}^i \in \mathbb{R}^{3 \times H \times W} \mid \mathbf{L}^i = \mathbf{I} \otimes \mathbf{M}^i,\ \mathbf{M}^i \in \mathcal{M}\right\}_{i=1}^{N}.$$9, $$\hat{\mathbf{L}}^i \in \mathbb{R}^{3 \times D \times D},$$0, $$\hat{\mathbf{L}}^i \in \mathbb{R}^{3 \times D \times D},$$1, $$\hat{\mathbf{L}}^i \in \mathbb{R}^{3 \times D \times D},$$2, $$\hat{\mathbf{L}}^i \in \mathbb{R}^{3 \times D \times D},$$3
GrowliFlower	Cross-species cauliflower, UAV	$$\hat{\mathbf{L}}^i \in \mathbb{R}^{3 \times D \times D},$$4, $$\hat{\mathbf{L}}^i \in \mathbb{R}^{3 \times D \times D},$$5, $$\hat{\mathbf{L}}^i \in \mathbb{R}^{3 \times D \times D},$$6, $$\hat{\mathbf{L}}^i \in \mathbb{R}^{3 \times D \times D},$$7, $$\hat{\mathbf{L}}^i \in \mathbb{R}^{3 \times D \times D},$$8
SB20	Cross-environment sugar beet, field robot	$$\hat{\mathbf{L}}^i \in \mathbb{R}^{3 \times D \times D},$$9, $D=800$0, $D=800$1, $D=800$2

The baselines include supervised hierarchical segmentation methods trained on PhenoBench—Weyler et al. and HAPT—and zero-shot methods LO-SAM, G-SAM, and G-SAM2. On PhenoBench, ZeroPlantSeg significantly outperforms zero-shot baselines and is competitive with supervised methods; the supervised Weyler baseline has slightly better or comparable plant-level performance on some metrics. On GrowliFlower, ZeroPlantSeg outperforms G-SAM, G-SAM2, LO-SAM, and the supervised baselines trained on PhenoBench, which degrade sharply under species shift. On SB20, it outperforms G-SAM2 and is better than the supervised methods, which generalize poorly to this shooting environment (Xing et al., 11 Sep 2025).

The ablations clarify where performance comes from. First, text-to-image attention for stem/base estimation is better than clustering on leaf-mask gravity centers, validating the use of attention-derived structural cues. Second, the proposed greedy clustering outperforms plain DBSCAN in all major metrics except slightly in precision, supporting the claim that clustering all keypoints jointly is unstable when tips and bases are mixed. Third, when predicted leaf masks are replaced by ground-truth leaf masks, performance rises sharply: on PhenoBench, $D=800$3 increases to 84.79 and $D=800$4 to 65.77; on GrowliFlower, $D=800$5 increases to 95.80, $D=800$6 to 62.13, and $D=800$7 to 64.37. This indicates that the plant-grouping stage is strong and that further gains are likely to depend on improved leaf-mask quality rather than a complete redesign of the clustering stage (Xing et al., 11 Sep 2025).

6. Scope, limitations, and relation to adjacent zero-shot plant segmentation

The method’s main limitation is explicit in its design: it is heavily based on heuristics of rosette-shaped plants. The plant-center prior, the interpretation of petiole attention, and the clustering of base points all assume that leaves radiate from a compact center. The paper therefore states that the method is less directly applicable to crops with different architectures, such as wheat, and suggests that analogous structural priors should be developed for other crop types (Xing et al., 11 Sep 2025).

ZeroPlantSeg also occupies a distinct position relative to earlier zero-shot SAM-based plant segmentation. Leaf Only SAM uses SAM in fully automatic mode with deterministic post-processing rules to retain likely potato leaf masks. On the test set, Leaf Only SAM reports average recall 63.2 and average precision 60.3, compared with recall 78.7 and precision 74.7 for a fine-tuned Mask R-CNN; its contribution is a viable annotation-free leaf segmentation workflow, but it remains a leaf-only system and does not solve plant-level grouping (Williams et al., 2023). ZeroPlantSeg can be understood as extending the zero-shot plant segmentation agenda from object extraction to hierarchical assembly.

A second adjacent direction uses specialized plant representations rather than structural grouping. “Unlocking Zero-Shot Plant Segmentation with Pl@ntNet Intelligence” combines Plantnet, a DinoV2 backbone, and SAM to identify plant regions, produce coarse masks, and refine them with SAM. It evaluates on four publicly available datasets and reports consistent performance gains for Plantnet-fine-tuned DinoV2 over base DinoV2 as measured by the Jaccard Index (IoU) (Ravé et al., 14 Oct 2025). This suggests a different strategy from ZeroPlantSeg: leveraging plant-centric feature specialization to improve mask quality, rather than using text-conditioned attention to infer plant membership from leaf structure.

Beyond 2D top-view imagery, related work on annotation reduction in plant segmentation includes self-supervised pre-training for 3D leaf instance segmentation, which reports that similar or better mAP can be achieved using only 45% of the labeled data or more, and benchmarking work on organ-level 3D plant segmentation that emphasizes sparse convolutional backbones, transformer-based instance segmentation, sim-to-real learning, and the Plant Segmentation Studio framework (Roggiolani et al., 2024, Du et al., 8 Sep 2025). These lines do not solve the same hierarchical rosette grouping problem, but they locate ZeroPlantSeg within a broader research program aimed at reducing plant-annotation demands through foundation models, self-supervision, synthetic data, and reproducible benchmarking.

Taken together, these comparisons show that ZeroPlantSeg is neither a generic segmentation model nor a universal solution for plant imagery. Its specific contribution is a training-free hierarchical pipeline for rosette plants that integrates SAM-based leaf extraction, OVSeg mask filtering, Grounding DINO cross-attention for stem/petiole localization, and greedy clustering with Mahalanobis-based outlier assignment. Within that scope, it demonstrates that plant-level reasoning can be added to zero-shot leaf segmentation without introducing task-specific supervision (Xing et al., 11 Sep 2025).