Modality-Agnostic Prompt Learning for Multi-Modal Camouflaged Object Detection

Abstract: Camouflaged Object Detection (COD) aims to segment objects that blend seamlessly into complex backgrounds, with growing interest in exploiting additional visual modalities to enhance robustness through complementary information. However, most existing approaches generally rely on modality-specific architectures or customized fusion strategies, which limit scalability and cross-modal generalization. To address this, we propose a novel framework that generates modality-agnostic multi-modal prompts for the Segment Anything Model (SAM), enabling parameter-efficient adaptation to arbitrary auxiliary modalities and significantly improving overall performance on COD tasks. Specifically, we model multi-modal learning through interactions between a data-driven content domain and a knowledge-driven prompt domain, distilling task-relevant cues into unified prompts for SAM decoding. We further introduce a lightweight Mask Refine Module to calibrate coarse predictions by incorporating fine-grained prompt cues, leading to more accurate camouflaged object boundaries. Extensive experiments on RGB-Depth, RGB-Thermal, and RGB-Polarization benchmarks validate the effectiveness and generalization of our modality-agnostic framework.

• The paper proposes a modality-agnostic, dual-domain architecture that integrates RGB and auxiliary modalities for enhanced camouflaged object detection.
• The paper achieves state-of-the-art performance on multiple COD benchmarks, outperforming modality-specific approaches with high efficiency (<3M parameters).
• The paper validates its method through extensive ablations and qualitative analyses, demonstrating accurate boundary recovery and robust cross-modal segmentation.

Modality-Agnostic Prompt Learning for Multi-Modal Camouflaged Object Detection

Introduction

The paper "Modality-Agnostic Prompt Learning for Multi-Modal Camouflaged Object Detection" (2604.12380) addresses the challenge of Camouflaged Object Detection (COD) in complex scenes by leveraging multi-modal input and enhancing the Segment Anything Model (SAM) with modality-agnostic prompt learning. The methodology moves beyond previous works constrained to modality-specific designs, instead offering a scalable, efficient, and robust framework applicable to RGB combined with arbitrary auxiliary modalities such as depth, thermal, and polarization.

Methodology

Dual-Domain Architecture

The core architecture introduces two synergistic domains:

• Content Domain: Fuses RGB and auxiliary modality features using a frozen ViT-based SAM encoder (for RGB) and a frozen PVT backbone (for the auxiliary modality), followed by element-wise summation. The auxiliary features are hierarchically extracted and adaptively fused with a learnable weighted mechanism to encode comprehensive scene perception.
• Prompt Domain: Comprises learnable prompt prototypes, modulated adaptively by global context derived from the content domain. These prototypes encode generalized semantic priors for camouflaged object structure, enabling the system to abstract and unify guidance across varying modality combinations.

Interaction between domains is enforced via symmetric cross-attention within the Content-Prompt Interaction Module (CPIM), distilling task-relevant evidence and enforcing spatial and structural regularization.

Figure 1: The dual-stream framework enables joint modeling of RGB and arbitrary auxiliary modalities, with content and prompt domain synergy via the CPIM.

Modality-Agnostic Information Flow

Auxiliary learnable "Auxi-tokens" complement the original SAM prompt and output tokens, specifically enhancing modality-agnostic representation and ensuring seamless integration of non-RGB cues for mask prediction.

Mask Refine Module

A lightweight refinement head is introduced to recalibrate SAM's initial masks, particularly addressing ambiguous boundaries. The module integrates enhanced prompt features with the decoder’s transposed convolution and token-to-image attention representations for precise mask refinement.

Experimental Results

Quantitative Performance

The proposed approach was evaluated on multiple multi-modal COD benchmarks (RGB-D, RGB-P, RGB-T), consistently outperforming both RGB-only and modality-specific multi-modal baselines in all canonical metrics, including $S_\alpha$, $F_\beta^w$, $E_\phi$, and MAE ($M$).

• On COD10K (RGB-D): Achieves $S_\alpha=0.901$, $F_\beta^w=0.863$, $E_\phi=0.958$, $M=0.018$.
• On PCOD-1200 (RGB-P): Yields $S_\alpha=0.945$, $F_\beta^w=0.924$, $F_\beta^w$0, $F_\beta^w$1.
• On VIAC (RGB-T): Records $F_\beta^w$2, $F_\beta^w$3, $F_\beta^w$4, $F_\beta^w$5.

The approach consistently uses fewer trainable parameters (<3M) than all tested alternatives, underlining high efficiency.

Figure 2: The $F_\beta^w$6 scores and trainable parameters highlight superior performance and parameter efficiency of the proposed method over competing approaches.

Qualitative Results

Qualitative comparison on multiple datasets shows the method’s capability to recover fine object boundaries and adhere to target structures under scenarios with low visual contrast or heavy camouflage. Visualization on RGB-P and RGB-T further demonstrates that the framework robustly leverages material and thermal cues for effective segmentation.

Figure 3: Example segmentation results on RGB-D and RGB-P datasets reveal precise detection and sharper mask boundaries.

Figure 4: Robust target segmentation on VIAC (RGB-T) highlights the framework's effectiveness in leveraging thermal information.

Ablation and Analysis

Ablation confirms the necessity of each architectural component (weighted fusion, Auxi-tokens, CPIM with cross-attention, and Mask Refine module). Joint and cross-modality training experiments further demonstrate the modality-agnostic property: minor performance degradation occurs when testing on combinations unseen during training, but accuracy remains competitive.

Visual ablation studies and attention heatmaps pinpoint that each module incrementally improves focus and localization fidelity.

Figure 5: Ablation visualizations show the effect of key component removal, especially on boundary and structure.

Figure 6: Attention map visualization from the transposed convolution branch within the mask decoder, underscoring improved spatial focus.

Discussion

The proposed framework addresses critical limitations of prior art by refraining from any modality-specific re-design and instead unifies multi-modal fusion in a SAM-compatible, parameter-efficient prompt learning context. Comparative analysis underscores fragility of competing approaches (e.g., DSAM, SAM-DSA) in cross-modal generalization, whereas the proposed method is robustly transferrable. Theoretical implications include a blueprint for scaling segmentation models to arbitrary modality combinations, aligning with the broader trend in universal segmentation and foundation model adaptation.

Joint training with heterogeneous datasets indicates that imbalanced data distributions can modestly impact performance, a consideration for practical deployment.

Figure 7: Scenario-wise radar analysis reveals stability across challenging conditions such as small, irregular, or multiple objects.

Limitations

Failure cases primarily occur when auxiliary modalities are either uninformative or themselves ambiguous, paralleling the limitations of all current COD systems in extreme camouflage with minimal signal contrast.

Figure 8: Failure example where both RGB and auxiliary modalities fail to provide sufficient discriminatory information.

Broader Applications

Generalizability analysis extends the framework to salient object detection (SOD), where it delivers SOTA performance in RGB-T based tasks, indicating that the dual-domain, modality-agnostic prompting paradigm is readily transferrable beyond COD.

Figure 9: Qualitative SOD results with RGB-T input accentuate the cross-task versatility of the learned representation.

Conclusion

This work establishes a modality-agnostic, prompt-based adaptation of SAM for robust multi-modal camouflaged object detection. The dual-domain interaction paradigm, lightweight integration strategy, and minimal parameter count yield consistent SOTA performance with broad cross-modal generalization. The approach sets an agenda for future foundation model adaptation strategies encompassing arbitrary sensor fusion and transferable universal segmentation frameworks in both COD and SOD, and potentially other dense prediction tasks.