SPLF-SAM: Self-Prompting Segment Anything Model for Light Field Salient Object Detection

Abstract: Segment Anything Model (SAM) has demonstrated remarkable capabilities in solving light field salient object detection (LF SOD). However, most existing models tend to neglect the extraction of prompt information under this task. Meanwhile, traditional models ignore the analysis of frequency-domain information, which leads to small objects being overwhelmed by noise. In this paper, we put forward a novel model called self-prompting light field segment anything model (SPLF-SAM), equipped with unified multi-scale feature embedding block (UMFEB) and a multi-scale adaptive filtering adapter (MAFA). UMFEB is capable of identifying multiple objects of varying sizes, while MAFA, by learning frequency features, effectively prevents small objects from being overwhelmed by noise. Extensive experiments have demonstrated the superiority of our method over ten state-of-the-art (SOTA) LF SOD methods. Our code will be available at https://github.com/XucherCH/splfsam.

• The paper introduces a self-prompting architecture that iteratively refines segmentation by updating a prompt bank, enhancing contextual aggregation.
• It employs multi-scale adaptive filtering using Fourier transforms to preserve high-frequency details, significantly improving small object detection in noisy backgrounds.
• Unified multi-scale feature embedding integrates diverse features via attention mechanisms, yielding superior quantitative results on multiple LF SOD benchmarks.

SPLF-SAM: Self-Prompting Segment Anything Model for Light Field Salient Object Detection

Introduction

The paper introduces SPLF-SAM, a novel architecture for light field salient object detection (LF SOD) that leverages the Segment Anything Model (SAM) paradigm with a self-prompting mechanism, multi-scale adaptive filtering, and unified multi-scale feature embedding. The motivation stems from the limitations of prior LF SOD models, which often neglect prompt information and frequency-domain analysis, resulting in suboptimal detection of small or complex objects in noisy backgrounds. SPLF-SAM addresses these gaps by integrating a multi-scale adaptive filtering adapter (MAFA) and a unified multi-scale feature embedding block (UMFEB), aiming to enhance both the robustness and precision of salient object segmentation in light field imagery.

Figure 1: The overall flowchart of the self-prompting light field segment anything model (SPLF-SAM).

Architecture and Methodology

Self-Prompting and Feature Extraction

SPLF-SAM builds upon the SAM backbone, freezing all encoder weights except for the proposed adapters. Feature maps are extracted from the 2nd, 5th, 8th, and 11th encoder layers. The deepest feature undergoes a U-Net block and is stored in a prompt bank, which is iteratively updated as the decoder generates new prompt information. This self-prompting mechanism enables the model to refine its segmentation iteratively, leveraging both current and historical prompt information for improved context aggregation.

Multi-Scale Adaptive Filtering Adapter (MAFA)

The MAFA module is designed to address the challenge of small object detection in noisy environments. Unlike conventional adapter-based fine-tuning, which often introduces noise through downsampling–upsampling, MAFA applies multi-scale convolutions followed by patch-wise Fourier transforms. Learnable frequency-domain kernels filter each patch, and the inverse Fourier transform reconstructs the filtered features. This process enhances the model's ability to suppress noise and preserve small, high-frequency structures.

Figure 2: The illustration of multi-scale adaptive filtering adapter (MAFA).

Unified Multi-Scale Feature Embedding Block (UMFEB)

UMFEB is introduced to overcome the limited receptive field and insufficient feature fusion of standard concatenation-convolution approaches. It employs a multi-scale fusion block (MFB) that applies parallel depthwise convolutions with varying kernel sizes, concatenates the outputs, and fuses them via a $1 \times 1$ convolution and residual connection. Channel attention and a query-key-value attention mechanism further enhance feature integration, enabling effective separation of targets from background and noise.

Figure 3: The illustration of unified multi-scale feature embedding block (UMFEB).

Decoder and Prompt Bank

The decoder receives fused features and prompt information, generating both saliency maps and new prompts. The prompt bank is updated iteratively, ensuring that multi-level semantic and frequency information is fully exploited. This design is inspired by recent advances in prompt-based segmentation and is tailored for the unique requirements of LF SOD.

Experimental Results

Quantitative Evaluation

SPLF-SAM is evaluated on four LF SOD datasets: PKU-LF, DUT-LF, HFUT, and Lytro Illum. The model is compared against 12 state-of-the-art methods using S-measure ($S_\alpha$), F-measure ($F_\beta$), E-measure ($E_\phi$), and Mean Absolute Error ($M$). SPLF-SAM achieves the best or second-best performance across all metrics and datasets. Notably, it demonstrates substantial improvements in $M$, with reductions of 30.8%, 16.7%, 19.6%, and 26.7% over the second-best model on the four datasets, respectively. These results underscore the effectiveness of the MAFA and UMFEB modules in enhancing detection accuracy, particularly for small and complex objects.

Qualitative Analysis

Visual comparisons reveal that SPLF-SAM produces more accurate and precise saliency maps, especially in challenging scenarios involving multiple objects, complex structures, or cluttered backgrounds. The model consistently delineates object boundaries with higher fidelity than competing methods.

Figure 4: Visual comparisons of SPLF-SAM and SOTA methods, highlighting superior boundary delineation and multi-object detection.

Ablation Studies

Ablation experiments confirm the incremental benefits of each architectural component. The addition of MAFA (with Fourier filtering), the prompt module, and UMFEB each yield measurable improvements in $F_\beta$ and $M$. The full model achieves the highest performance, validating the synergistic effect of the proposed modules.

Implementation Considerations

• Training Regime: The model is trained with AdamW, batch size 8, initial learning rate $5 \times 10^{-4}$, and weight decay $1 \times 10^{-4}$ for 50 epochs on a single NVIDIA RTX 5070Ti GPU.
• Loss Function: Binary Cross Entropy with Logits Loss is used to supervise four saliency predictions, with the total loss being the sum over all predictions.
• Resource Requirements: The architecture is computationally efficient due to frozen encoder weights and adapter-based fine-tuning, making it suitable for deployment on modern GPUs.
• Scalability: The modular design of MAFA and UMFEB allows for straightforward integration with other backbone architectures and potential extension to other multi-modal or 3D segmentation tasks.

Implications and Future Directions

SPLF-SAM demonstrates that integrating frequency-domain filtering and multi-scale feature fusion with self-prompting mechanisms can significantly advance the state of LF SOD. The approach is particularly effective in scenarios with small, complex, or overlapping objects and in the presence of substantial background noise. Theoretically, the work highlights the importance of frequency-aware processing and prompt-based iterative refinement in segmentation models.

Future research may explore:

• Incorporation of additional light field cues (e.g., angular or depth information) for further performance gains.
• Extension to other domains requiring robust small object detection, such as medical imaging or remote sensing.
• Investigation of more efficient or lightweight adapter designs for real-time applications.

Conclusion

SPLF-SAM sets a new benchmark for light field salient object detection by combining self-prompting, frequency-domain adaptive filtering, and unified multi-scale feature embedding. The architecture achieves superior quantitative and qualitative results compared to existing methods, particularly excelling in challenging detection scenarios. The modularity and efficiency of the design make it a promising foundation for future research in light field and multi-modal segmentation.