Deep Level Set for Box-supervised Instance Segmentation in Aerial Images (2112.03451v1)

Abstract: Box-supervised instance segmentation has recently attracted lots of research efforts while little attention is received in aerial image domain. In contrast to the general object collections, aerial objects have large intra-class variances and inter-class similarity with complex background. Moreover, there are many tiny objects in the high-resolution satellite images. This makes the recent pairwise affinity modeling method inevitably to involve the noisy supervision with the inferior results. To tackle these problems, we propose a novel aerial instance segmentation approach, which drives the network to learn a series of level set functions for the aerial objects with only box annotations in an end-to-end fashion. Instead of learning the pairwise affinity, the level set method with the carefully designed energy functions treats the object segmentation as curve evolution, which is able to accurately recover the object's boundaries and prevent the interference from the indistinguishable background and similar objects. The experimental results demonstrate that the proposed approach outperforms the state-of-the-art box-supervised instance segmentation methods. The source code is available at https://github.com/LiWentomng/boxlevelset.

Deep Level Set for Box-supervised Instance Segmentation in Aerial Images

The paper presents a novel approach to box-supervised instance segmentation, specifically tailored for aerial images, leveraging a deep-level set method. The focus on aerial images introduces unique challenges due to large intra-class variances, inter-class similarities, complex backgrounds, and the prevalence of tiny objects, aspects which are less pronounced in general object collections.

Key Contributions and Approach

The authors propose a method that departs from typical pairwise affinity modeling, which can incorporate noisy supervision and provide inferior results in the context of aerial images. Instead, their approach innovatively applies level set methods to guide curve evolution for object segmentation with only box annotations. This strategy circumvents the limitations of relying on pixel affinity by directly recovering object boundaries.

The deep level set method is embedded into an end-to-end trainable network composed of two branches: the detection and segmentation branches. The detection branch utilizes oriented bounding boxes to cater to the spatial characteristics of aerial objects, particularly those with arbitrary orientations. Meanwhile, the segmentation branch employs a novel framework that integrates an energy function designed to facilitate the level set evolution of potential object masks within enlarged box regions, thus enabling precise demarcation of object boundaries. This is achieved without requiring pixel-wise mask annotations, a process considered time-consuming and labor-intensive.

Methodological Insights

1. Level Set Methodology: Central to the approach is the application of the level set method in an aerial context. Here, segmentation is framed as a boundary evolution problem, which manages to handle noisy backgrounds and similar inter-class object appearances effectively. The curve is iteratively refined using designed energy functions, thereby addressing aerial images' unique segmentation challenges.
2. Box and Background Constraints: For effective convergence and to safeguard against noise, constraints are introduced. These leverage bounding box projections and background areas to ensure learned segmentations are consistent with provided box annotations, enabling the model to distinguish between foreground objects and background noise efficiently.
3. Sample Assignment Framework: The proposed system employs a unified sample assignment framework that selects potential positive samples for joint training purposes. This framework consolidates classification, box regression, and mask-level prediction processes, improving efficiency and segmentation accuracy.

Experimental Evaluation

The paper reports comprehensive experimental results on two prominent aerial datasets: iSAID and Potsdam. The authors demonstrate the efficacy of their method by comparison with existing state-of-the-art fully supervised and other box-supervised segmentation methods. Notably, their approach achieved competitive performance with fully supervised methods in several categories and demonstrated superior performance among box-supervised methods, with AP improvements across various metrics.

Implications and Future Directions

From a practical standpoint, the proposed method offers a more annotation-efficient approach to instance segmentation in aerial imagery, which could be beneficial for urban management and environmental monitoring where timely and accurate object segmentation is crucial. Theoretically, the adaptation of level set methods within deep learning frameworks for weakly supervised learning signals potential flexibility and robustness in tackling complex segmentation tasks.

Future work could expand on testing the method’s generalizability to nonspecialized datasets, such as COCO, to evaluate its strengths and weaknesses beyond the domain of aerial imagery. Additionally, further exploration into optimizing the class-wise parameter calculation within the energy functions could refine segmentation performance across diverse datasets.

In conclusion, the paper presents a methodologically sound and practically applicable approach for instance segmentation in aerial images, fulfilling an essential niche within remote sensing applications. The integration of level set methodologies into deep learning, facilitated by bounding box annotations, represents a substantive contribution to the field of computer vision.