Zero-Shot Tree Detection and Segmentation from Aerial Forest Imagery (2506.03114v1)

Abstract: Large-scale delineation of individual trees from remote sensing imagery is crucial to the advancement of ecological research, particularly as climate change and other environmental factors rapidly transform forest landscapes across the world. Current RGB tree segmentation methods rely on training specialized machine learning models with labeled tree datasets. While these learning-based approaches can outperform manual data collection when accurate, the existing models still depend on training data that's hard to scale. In this paper, we investigate the efficacy of using a state-of-the-art image segmentation model, Segment Anything Model 2 (SAM2), in a zero-shot manner for individual tree detection and segmentation. We evaluate a pretrained SAM2 model on two tasks in this domain: (1) zero-shot segmentation and (2) zero-shot transfer by using predictions from an existing tree detection model as prompts. Our results suggest that SAM2 not only has impressive generalization capabilities, but also can form a natural synergy with specialized methods trained on in-domain labeled data. We find that applying large pretrained models to problems in remote sensing is a promising avenue for future progress. We make our code available at: https://github.com/open-forest-observatory/tree-detection-framework.

• The paper introduces a zero-shot approach using SAM2 to detect and segment trees without the need for domain-specific training.
• It employs a two-part methodology with automatic mask generation and bounding box prompts to indirectly capitalize on existing tree detectors.
• Experimental results reveal competitive recall and enhanced segmentation of intricate tree structures, enabling scalable ecological monitoring.

Zero-Shot Tree Detection and Segmentation from Aerial Forest Imagery

The paper "Zero-Shot Tree Detection and Segmentation from Aerial Forest Imagery" explores the application of the Segment Anything Model 2 (SAM2) for ecological research by addressing the challenge of large-scale delineation of individual trees from aerial imagery. This investigation marks a significant step in the use of advanced machine learning for remote sensing and ecological monitoring.

Summary of Research

The central objective of this paper is to leverage the capabilities of SAM2 to perform tree detection and segmentation in a zero-shot context, thus bypassing the bottleneck of assembling large, labeled training datasets. The research centers around two primary tasks: zero-shot segmentation and zero-shot transfer of knowledge from existing tree detection models to SAM2. The model demonstrates impressive generalization capabilities, indicating its efficacy across different tree species and forest conditions without the need for retraining or domain-specific adaptation.

Methodological Framework

SAM2 is a next-generation image segmentation foundation model characterized by its prompt-based segmentation approach and a robust vision transformer architecture. The model was evaluated using images from the NEON TreeEvaluation dataset, the Detectree2 dataset, and the Emerald Point dataset, representing a diverse array of geographies and forest environments.

The methodology involves two main components:

1. Zero-shot segmentation which uses SAM2's capabilities with automatic mask generators across sampled image grids, enabling tree segmentation without prior domain-specific training.
2. Zero-shot transfer which involves utilizing bounding box outputs from specialized tree detectors as prompts for SAM2, effectively enhancing its segmentation capability by leveraging existing domain knowledge indirectly through model output.

Experimental Findings

Quantitative results indicate that SAM2, without any direct training in the domain, can achieve recall rates competitive with existing CNN-based models like DeepForest and Detectree2. Although precision levels were lower due to a tendency to over-segment, SAM2's recall performance underscores its potential for large-scale remote sensing tasks.

Qualitatively, SAM2 outperformed traditional models, especially in capturing smaller and more intricate tree structures. This suggests broader applicability for varied ecological systems and forest types without requiring additional data labeling efforts.

Implications and Future Directions

This research highlights the potential of large, pretrained models in remote sensing for ecological applications, offering a new vista in areas such as biodiversity assessment, forest inventory, and climate change studies. The success of SAM2 in zero-shot tasks invites further exploration into the integration of foundation models in remote sensing workflows, possibly revolutionizing the scale and scope of ecological data analysis.

Future research could focus on refining prompt engineering methods to mitigate over-segmentation and improve precision. Additionally, exploring multimodal datasets, integrating different sensor modalities, and leveraging other foundation models could broaden the deployment scenarios and enhance the accuracy and applicability of results.

Overall, the paper presents a compelling case for the adoption of foundation models like SAM2 in ecological and environmental remote sensing, showcasing how advanced machine learning techniques can address ongoing challenges in the field and contribute to more efficient ecological monitoring and management practices.