- The paper presents DiffuSAM, a hybrid pipeline that integrates generative diffusion and segmentation models for zero-shot object grounding in remote sensing imagery.
- The method uses preprocessing and adaptive segmentation to enhance image clarity and dynamically select models, achieving a 14% absolute improvement in [email protected] on NWPU-VHR-10.
- The approach demonstrates practical gains in scalability and data efficiency while addressing challenges such as diffusion model hallucinations.
DiffuSAM: Diffusion Guided Zero-Shot Object Grounding in Remote Sensing Imagery
Introduction
DiffuSAM presents a hybrid pipeline designed to address the challenge of zero-shot object grounding in remote sensing imagery. Traditional approaches in remote sensing object localization rely heavily on large, annotated datasets and task-specific training, which limits their scalability and generalization across diverse geographic regions and imaging conditions. DiffuSAM circumvents these limitations by integrating generative diffusion models with foundational segmentation models, specifically focusing on text-guided localization without extensive training or domain-specific data.
Methodology
Preprocessing
Remote sensing imagery commonly suffers from atmospheric artifacts such as haze and low contrast. DiffuSAM implements a preprocessing module that converts images to CIELAB color space for luminance-chrominance separation. Local contrast enhancement is performed using CLAHE, followed by unsharp masking. This procedure increases image clarity and edge definition, optimizing conditions for downstream segmentation and localization. Additionally, natural language prompts are processed via Qwen3-8b, an LLM, to strip directionally ambiguous or extraneous instructions that degrade model effectiveness.
Diffusion-Based Localization
Textual object descriptions are transformed into coarse spatial cues using Qwen-Image-Edit, a diffusion-based image editing model. It highlights candidate objects within the input image with red bounding boxes based purely on the prompt, providing rapid and generalizable localization without task-specific supervision.
Segmentation and Adaptive Refinement
The edited image is input to SAM3, a foundation segmentation model, which produces initial masks constrained by the diffusion-generated bounding boxes. Subsequent adaptive refinement involves cropping regions of interest and dynamically selecting the most appropriate segmentation model: RemoteSAM is used for large regions due to its domain specialization in remote sensing, while SAM3 is used for smaller, more precise crops. The choice of model is governed by the proportion of the cropped region relative to the original image, empirically optimized at a 10% threshold. A fallback mechanism ensures robust prediction—if one segmentation model fails to output a mask, the pipeline switches to the other; if both fail, the diffusion-generated bounding box is used.
Final bounding boxes are derived from the segmentation mask output, constituting the pipeline’s ground truth for object localization.
Experimental Results
DiffuSAM was evaluated on VRSBench and NWPU-VHR-10 datasets, with performance compared against RemoteSAM, SAM3, EarthMind, and Falcon. The pipeline achieves a 14% absolute increase in [email protected] on NWPU-VHR-10 relative to RemoteSAM, the previous state-of-the-art. On VRSBench, DiffuSAM delivers competitive gains, with marginal improvements over RemoteSAM. The superior performance on NWPU-VHR-10 is attributed to the axis-aligned annotation format that aligns more closely with DiffuSAM’s bounding box extraction approach.
| Model |
VRSBench [email protected] |
NWPU-VHR-10 [email protected] |
| RemoteSAM |
0.7812 |
0.6624 |
| SAM3 |
0.2051 |
0.6352 |
| EarthMind |
0.7414 |
0.6088 |
| Falcon |
0.7221 |
0.6271 |
| DiffuSAM |
0.7913 |
0.8049 |
DiffuSAM achieves statistically significant improvements in both accuracy and mean IoU, confirming the advantage of model hybridization and adaptive segmentation.
Limitations
The primary constraint in DiffuSAM arises from the generative characteristics of diffusion models. In cases where the object is absent or difficult to discern, Qwen-Image-Edit may hallucinate features, leading to erroneous bounding boxes or the inclusion of non-target artifacts. Furthermore, the axis-alignment enforced by the diffusion model may capture extraneous context, introducing noise into the subsequent segmentation stages.
Implications and Future Directions
Practically, DiffuSAM demonstrates that hybrid pipelines leveraging diffusion models and foundation segmentation architectures can achieve robust, data-efficient text-guided object grounding in remote sensing. Theoretically, this blueprint establishes a path for further integration of generative models for spatial reasoning, potentially increasing the autonomy of remote sensing systems in complex multi-object and unstructured environments.
Future avenues include refining diffusion model control to mitigate hallucinations, and fine-tuning generative models on remote sensing corpora to enhance their resilience to environmental obstructions such as haze. Additional improvements may focus on bounding box orientation and advanced spatial relationship parsing in prompts.
Conclusion
DiffuSAM introduces an adaptive, hybrid framework for zero-shot object grounding in remote sensing imagery, evidencing strong empirical gains over baseline models. By harnessing generative diffusion models in conjunction with segmentation architectures, the pipeline pushes forward both the practical efficacy and theoretical foundation for text-guided localization in Earth observation, with implications for scalable urban, environmental, and infrastructure analysis (2604.18201).
