UV-SAM: Adapting Segment Anything Model for Urban Village Identification (2401.08083v2)

Abstract: Urban villages, defined as informal residential areas in or around urban centers, are characterized by inadequate infrastructures and poor living conditions, closely related to the Sustainable Development Goals (SDGs) on poverty, adequate housing, and sustainable cities. Traditionally, governments heavily depend on field survey methods to monitor the urban villages, which however are time-consuming, labor-intensive, and possibly delayed. Thanks to widely available and timely updated satellite images, recent studies develop computer vision techniques to detect urban villages efficiently. However, existing studies either focus on simple urban village image classification or fail to provide accurate boundary information. To accurately identify urban village boundaries from satellite images, we harness the power of the vision foundation model and adapt the Segment Anything Model (SAM) to urban village segmentation, named UV-SAM. Specifically, UV-SAM first leverages a small-sized semantic segmentation model to produce mixed prompts for urban villages, including mask, bounding box, and image representations, which are then fed into SAM for fine-grained boundary identification. Extensive experimental results on two datasets in China demonstrate that UV-SAM outperforms existing baselines, and identification results over multiple years show that both the number and area of urban villages are decreasing over time, providing deeper insights into the development trends of urban villages and sheds light on the vision foundation models for sustainable cities. The dataset and codes of this study are available at https://github.com/tsinghua-fib-lab/UV-SAM.

• The paper introduces UV-SAM, a novel framework that adapts the Segment Anything Model for urban village boundary detection in satellite images.
• It employs a generalist-specialist approach by combining SAM with SegFormer to generate initial segmentation masks and precise boundary prompts.
• Experimental results on Beijing and Xi'an datasets show that UV-SAM significantly outperforms existing methods in IoU and F1-score for urban planning applications.

An Overview of UV-SAM: Segment Anything Model for Urban Village Identification

The paper "UV-SAM: Adapting Segment Anything Model for Urban Village Identification" presents an innovative approach to identifying urban villages using satellite imagery, leveraging advances in computer vision and adaptation of foundational models. This paper addresses the complexity of demarcating urban village boundaries—a task integral to urban planning and governance aligned with Sustainable Development Goals (SDG), specifically SDG 11, which promotes sustainable cities.

Central to this approach is the use of the Segment Anything Model (SAM), which has demonstrated significant potential in general object segmentation tasks. However, the adaptation of SAM to the specific context of urban village identification from satellite images requires domain-specific modifications. The authors introduce the UV-SAM framework, which incorporates a refined prompting strategy to transform the generic capabilities of SAM into a specialized tool for urban planning applications.

Methodological Framework

UV-SAM utilizes a generalist-specialist framework, where the SAM acts as a generalist module with extensive vision capabilities, and a lightweight semantic segmentation model (SegFormer) functions as a specialist for urban village identification. The process begins with SegFormer, which produces initial segmentation masks, bounding box prompts, and semantic representations from satellite images. These outputs serve as specialized prompts for SAM, guiding it to pay focused attention to the urban village features within the images.

The mixing of multiple prompt types effectively provides the necessary context for SAM to achieve precise boundary delineation. This is particularly relevant in urban village identification, where traditional boundaries are often obscure due to overlapping features with surrounding urban infrastructure.

Experimental Validation and Results

The authors validate UV-SAM on datasets from two major Chinese cities, Beijing and Xi'an. The results demonstrate that UV-SAM significantly outperforms existing state-of-the-art approaches, particularly in terms of Intersection over Union (IoU) and F1-score, highlighting its superior capability in accurately identifying urban village boundaries. The model shows notable robustness in extracting fine-grained boundary details that other models, including those solely based on SAM, struggle to delineate.

The experimental results also reveal temporal changes in urban village distributions, showcasing UV-SAM's potential to monitor urban development trends effectively. This model not only identifies the decreasing number and size of urban villages over time but also provides insights into spatial distribution patterns critical for urban policy making.

Implications and Future Research Directions

The implications of UV-SAM are notable for urban planning professionals focusing on informal settlement monitoring and management. By providing a more accurate and efficient method for identifying urban village boundaries, UV-SAM facilitates data-driven decisions in urban development and socio-economic planning.

For future research, this paper opens up pathways for applying vision foundation models like SAM across various domain-specific segmentation tasks beyond urban villages. Of particular interest is the potential adaptation of UV-SAM to other forms of satellite-based urban analysis, as well as its integration with socio-economic datasets to deepen our understanding of urban sprawl and informal settlement dynamics.

Moreover, the research underscores the significance of prompt engineering—a field that promises further exploration in enhancing the functionality of generic machine learning models for specific organizational applications. As the community continues to refine prompting strategies, the deployment of foundation models in sector-specific contexts is expected to become increasingly sophisticated and impactful.