Prior Guided Feature Enrichment Network for Few-Shot Segmentation (2008.01449v1)

Abstract: State-of-the-art semantic segmentation methods require sufficient labeled data to achieve good results and hardly work on unseen classes without fine-tuning. Few-shot segmentation is thus proposed to tackle this problem by learning a model that quickly adapts to new classes with a few labeled support samples. Theses frameworks still face the challenge of generalization ability reduction on unseen classes due to inappropriate use of high-level semantic information of training classes and spatial inconsistency between query and support targets. To alleviate these issues, we propose the Prior Guided Feature Enrichment Network (PFENet). It consists of novel designs of (1) a training-free prior mask generation method that not only retains generalization power but also improves model performance and (2) Feature Enrichment Module (FEM) that overcomes spatial inconsistency by adaptively enriching query features with support features and prior masks. Extensive experiments on PASCAL-5$^i$ and COCO prove that the proposed prior generation method and FEM both improve the baseline method significantly. Our PFENet also outperforms state-of-the-art methods by a large margin without efficiency loss. It is surprising that our model even generalizes to cases without labeled support samples. Our code is available at https://github.com/Jia-Research-Lab/PFENet/.

• The paper introduces PFENet, a novel model leveraging training-free prior masks to improve few-shot segmentation accuracy.
• It features a multi-scale Feature Enrichment Module that mitigates spatial discrepancies between support and query images.
• Experimental results on PASCAL-5i and COCO demonstrate state-of-the-art improvements in mIoU and FB-IoU metrics.

Prior Guided Feature Enrichment Network for Few-Shot Segmentation

The paper "Prior Guided Feature Enrichment Network for Few-Shot Segmentation" addresses the challenges inherent in few-shot segmentation, where models must effectively segment novel classes with limited labeled data. This task is complicated by issues such as inappropriate use of semantic information and spatial discrepancies between the support and query images.

Key Contributions

The authors introduce the Prior Guided Feature Enrichment Network (PFENet), which comprises two innovative components:

1. Training-Free Prior Mask Generation: This method leverages high-level features from pre-trained models to create prior masks that enhance target identification in query images. By capturing pixel-wise correspondence between query and support images without further training, these masks help maintain generalization power across unseen classes.
2. Feature Enrichment Module (FEM): FEM addresses spatial inconsistencies by using multi-scale representations to enrich query features. It adapts feature resolutions and selectively integrates information across different scales, enhancing the model’s ability to understand and predict targets accurately.

Experimental Results

Extensive experiments conducted on PASCAL-5$^i$ and COCO datasets reveal the efficacy of PFENet. The network achieves significant improvements over existing methods, establishing new state-of-the-art results. Notably, PFENet demonstrates robust performance even in zero-shot scenarios, indicating its potential adaptability beyond the few-shot paradigm.

• PFENet yields substantial mIoU (mean Intersection over Union) and FB-IoU (Foreground-Background IOU) enhancements compared to baselines.
• It maintains efficiency, with minimal impact on parameter count and processing speed.

Implications and Future Work

PFENet's innovations in prior mask generation and feature enrichment provide substantial benefits to few-shot segmentation tasks. The use of fixed high-level features for prior masks ensures that the network remains unbiased towards training classes, thus enhancing adaptability to novel classes. The FEM's design tackles spatial inconsistency, a critical challenge in segmentation tasks with limited data.

The paper opens avenues for extending these methodologies to related domains such as few-shot object detection and instance segmentation. Exploring different architectures and integrating with transformers could further boost performance. Additionally, analyzing the impact of different feature scales and alternative backbone networks may yield further insights and improvements.

Conclusion

In summary, the Prior Guided Feature Enrichment Network (PFENet) effectively enhances few-shot segmentation by innovating in prior mask generation and feature enrichment. It provides a comprehensive framework that successfully addresses generalization challenges and spatial inconsistencies, laying a foundation for future advancements in adaptive segmentation techniques.