MirrorSAM2: RGB-D Video Mirror Segmentation

• MirrorSAM2 is a prompt-free RGB-D video mirror segmentation framework that adapts SAM2 by using depth cues to automatically generate high-quality mask prompts.
• It employs a four-module approach—Depth Warping, Multi-Scale Point Prompt Generation, Frequency Detail Attention Fusion, and Mirror Mask Decoder—to address ambiguities in RGB mirror representations.
• Experiments on VMD and DVMD datasets demonstrate improved IoU, F-measure, accuracy, and MAE over baseline methods, proving its efficacy in challenging mirror segmentation tasks.

Searching arXiv for the cited papers to ground the article in current literature. MirrorSAM2 is a prompt-free RGB-D video mirror segmentation framework built by adapting Segment Anything Model 2 (SAM2) to the mirror detection setting. It is presented as the first framework to adapt SAM2 for RGB-D video mirror segmentation and, to the authors’ knowledge, the first work to enable SAM2 for automatic video mirror segmentation. Its central premise is that RGB alone is ambiguous for mirrors, whereas depth reveals scene structure; accordingly, MirrorSAM2 uses depth to guide prompt generation and feature refinement, then performs mirror-aware mask decoding through four task-specific modules: a Depth Warping Module, a Depth-guided Multi-Scale Point Prompt Generator, a Frequency Detail Attention Fusion Module, and a Mirror Mask Decoder with a learnable mirror token (Xu et al., 21 Sep 2025).

1. Definition, scope, and problem setting

MirrorSAM2 targets video mirror segmentation under RGB-D input. The task is difficult because mirrors reflect surrounding objects, often exhibit weak or blurred boundaries, can be small, and may share textures and semantics with non-mirror regions such as paintings, windows, doors, or bright surfaces. In video, motion cues may also be unreliable, especially when the camera is moving or the reflection changes (Xu et al., 21 Sep 2025).

The framework is explicitly positioned against the limitations of generic SAM2 prompting on rare-object video segmentation. Prior evaluation of SAM2 on Video Mirror Detection (VMD) showed that SAM2 can perform strongly when the first frame is initialized with a ground-truth mask prompt, but that point prompts are unstable and much worse than mask prompts; the same study concludes that SAM2 is suitable for mirror and shadow video segmentation only when given a high-quality mask prompt on the first frame (Jie, 2024). MirrorSAM2 addresses this bottleneck by making prompt generation automatic rather than relying on human-provided prompts, and by using depth to reduce reflection ambiguity and texture confusion (Xu et al., 21 Sep 2025).

Within the broader SAM lineage, SAM is a promptable zero-shot image segmentation model, while SAM2 extends the paradigm to images and videos through temporal memory, memory attention, and a Hiera-based image encoder with multiscale features (Geetha et al., 2024). MirrorSAM2 does not replace SAM2’s video-capable core; instead, it adapts SAM2 to a specialized prompt-free RGB-D mirror segmentation setting (Xu et al., 21 Sep 2025).

2. Architectural organization and data flow

MirrorSAM2 extends SAM2 with four modules: Depth Warping (DW), Depth-guided Multi-Scale Point Prompt Generator (DMS-PPG), Frequency Detail Attention Fusion (FDAF), and Mirror Mask Decoder (MMD). The pipeline is summarized as

$$\text{RGB-D video frames} \rightarrow \text{backbone features} \rightarrow \text{DW} \rightarrow \text{DMS-PPG} \rightarrow \text{SAM2 prompt encoder/decoder} \rightarrow \text{FDAF} \rightarrow \text{MMD} \rightarrow \text{final mask}.$$

The model uses a lightweight temporal setup with two adjacent frames and one distant frame, instead of SAM2’s usual eight-frame input, to reduce cost while keeping short- and long-range temporal context (Xu et al., 21 Sep 2025).

This organization reflects a division of labor across modules. DW aligns RGB and depth features and produces structure-guided fused features. DMS-PPG converts depth-enhanced evidence into sparse prompts suitable for SAM2’s prompt encoder. FDAF strengthens fine structural details by operating in the frequency domain. MMD introduces an explicit mirror-aware decoding stage through a learnable mirror token and a context contrast pathway (Xu et al., 21 Sep 2025).

A key conceptual distinction from earlier SAM2 evaluations on mirror videos is that the “method” is not merely a prompted benchmark protocol. In the earlier prompted setting, SAM2 was initialized on the first frame with either sampled points or the ground-truth mask, and then propagated masks to subsequent frames (Jie, 2024). MirrorSAM2 instead internalizes prompt creation via DMS-PPG, thereby converting mirror segmentation into a prompt-driven problem without manual prompts at inference time (Xu et al., 21 Sep 2025).

3. Core modules

The Depth Warping Module is designed to align RGB and depth features using cross-modal correlation and depth-aware decoding. It operates on low-level and high-level image and depth features,

$$\mathbf{F}_{\text{img}} \in \mathbb{R}^{B \times C \times H \times W}, \qquad \mathbf{F}_{\text{depth}} \in \mathbb{R}^{B \times C' \times H \times W},$$

computes modality-specific correlation maps,

$$\mathbf{C}_{\text{img}} = \text{Corr}(\mathbf{F}_{\text{img}}, \hat{\mathbf{F}}_{\text{img}}), \qquad \mathbf{C}_{\text{depth}} = \text{Corr}(\mathbf{F}_{\text{depth}}, \hat{\mathbf{F}}_{\text{depth}}),$$

and averages them as

$$\mathbf{C} = \frac{1}{2}\left(\mathbf{C}_{\text{img}} + \mathbf{C}_{\text{depth}}\right).$$

After compression by a $1\times1$ conv and $3\times3$ conv block $\phi(\mathbf{C})$, the result is fused back into each modality,

$$\tilde{\mathbf{F}}_{\text{img}} = \text{Fuse}_{\text{img}}\left([\mathbf{F}_{\text{img}}, \phi(\mathbf{C})]\right), \qquad \tilde{\mathbf{F}}_{\text{depth}} = \text{Fuse}_{\text{depth}}\left([\mathbf{F}_{\text{depth}}, \phi(\mathbf{C})]\right),$$

and a PAC-guided decoder produces

$$\mathbf{F}_{\text{fused}} = \text{Decoder}(\tilde{\mathbf{F}}_{\text{img}}, \tilde{\mathbf{F}}_{\text{depth}}).$$

The outputs are refined depth features and structure-guided fused features, which become the basis for automatic prompt generation (Xu et al., 21 Sep 2025).

The Depth-guided Multi-Scale Point Prompt Generator creates foreground point prompts from depth-augmented multi-scale features. Let $R \in \mathbb{R}^{H \times W}$ be the response map. Prompt selection is defined by

$$\mathbf{F}_{\text{img}} \in \mathbb{R}^{B \times C \times H \times W}, \qquad \mathbf{F}_{\text{depth}} \in \mathbb{R}^{B \times C' \times H \times W},$$0

Here, $$\mathbf{F}_{\text{img}} \in \mathbb{R}^{B \times C \times H \times W}, \qquad \mathbf{F}_{\text{depth}} \in \mathbb{R}^{B \times C' \times H \times W},$$1 denotes the top-$$\mathbf{F}_{\text{img}} \in \mathbb{R}^{B \times C \times H \times W}, \qquad \mathbf{F}_{\text{depth}} \in \mathbb{R}^{B \times C' \times H \times W},$$2 highest-response points, $$\mathbf{F}_{\text{img}} \in \mathbb{R}^{B \times C \times H \times W}, \qquad \mathbf{F}_{\text{depth}} \in \mathbb{R}^{B \times C' \times H \times W},$$3 is a minimum distance threshold, NMS-like filtering ensures spatial diversity, and a fallback center point is used if no valid point remains. The coordinates are normalized to $$\mathbf{F}_{\text{img}} \in \mathbb{R}^{B \times C \times H \times W}, \qquad \mathbf{F}_{\text{depth}} \in \mathbb{R}^{B \times C' \times H \times W},$$4 and fed to the SAM2 prompt encoder as sparse tokens (Xu et al., 21 Sep 2025).

The Frequency Detail Attention Fusion module is introduced to recover fine details that generic SAM2 may omit, especially mirror boundaries or thin regions. It takes a low-level structure-guided feature

$$\mathbf{F}_{\text{img}} \in \mathbb{R}^{B \times C \times H \times W}, \qquad \mathbf{F}_{\text{depth}} \in \mathbb{R}^{B \times C' \times H \times W},$$5

and applies a 1D FFT along the channel dimension to extract magnitude and phase. The projected frequency features act as keys and values, while memory features

$$\mathbf{F}_{\text{img}} \in \mathbb{R}^{B \times C \times H \times W}, \qquad \mathbf{F}_{\text{depth}} \in \mathbb{R}^{B \times C' \times H \times W},$$6

serve as queries in

$$\mathbf{F}_{\text{img}} \in \mathbb{R}^{B \times C \times H \times W}, \qquad \mathbf{F}_{\text{depth}} \in \mathbb{R}^{B \times C' \times H \times W},$$7

with residual fusion

$$\mathbf{F}_{\text{img}} \in \mathbb{R}^{B \times C \times H \times W}, \qquad \mathbf{F}_{\text{depth}} \in \mathbb{R}^{B \times C' \times H \times W},$$8

The stated effect is improved recovery of sharp boundaries, fine textures, symmetric structure, and missing mirror regions (Xu et al., 21 Sep 2025).

The Mirror Mask Decoder is the final segmentation head. It adds a learnable mirror token $$\mathbf{F}_{\text{img}} \in \mathbb{R}^{B \times C \times H \times W}, \qquad \mathbf{F}_{\text{depth}} \in \mathbb{R}^{B \times C' \times H \times W},$$9 to the transformer input sequence,

$$\mathbf{C}_{\text{img}} = \text{Corr}(\mathbf{F}_{\text{img}}, \hat{\mathbf{F}}_{\text{img}}), \qquad \mathbf{C}_{\text{depth}} = \text{Corr}(\mathbf{F}_{\text{depth}}, \hat{\mathbf{F}}_{\text{depth}}),$$0

which is processed jointly with the spatial feature map $$\mathbf{C}_{\text{img}} = \text{Corr}(\mathbf{F}_{\text{img}}, \hat{\mathbf{F}}_{\text{img}}), \qquad \mathbf{C}_{\text{depth}} = \text{Corr}(\mathbf{F}_{\text{depth}}, \hat{\mathbf{F}}_{\text{depth}}),$$1,

$$\mathbf{C}_{\text{img}} = \text{Corr}(\mathbf{F}_{\text{img}}, \hat{\mathbf{F}}_{\text{img}}), \qquad \mathbf{C}_{\text{depth}} = \text{Corr}(\mathbf{F}_{\text{depth}}, \hat{\mathbf{F}}_{\text{depth}}),$$2

The mirror token output

$$\mathbf{C}_{\text{img}} = \text{Corr}(\mathbf{F}_{\text{img}}, \hat{\mathbf{F}}_{\text{img}}), \qquad \mathbf{C}_{\text{depth}} = \text{Corr}(\mathbf{F}_{\text{depth}}, \hat{\mathbf{F}}_{\text{depth}}),$$3

is mapped through an MLP to produce $$\mathbf{C}_{\text{img}} = \text{Corr}(\mathbf{F}_{\text{img}}, \hat{\mathbf{F}}_{\text{img}}), \qquad \mathbf{C}_{\text{depth}} = \text{Corr}(\mathbf{F}_{\text{depth}}, \hat{\mathbf{F}}_{\text{depth}}),$$4, and the final dense mask is

$$\mathbf{C}_{\text{img}} = \text{Corr}(\mathbf{F}_{\text{img}}, \hat{\mathbf{F}}_{\text{img}}), \qquad \mathbf{C}_{\text{depth}} = \text{Corr}(\mathbf{F}_{\text{depth}}, \hat{\mathbf{F}}_{\text{depth}}),$$5

The decoder also includes a Context Contrast Module adapted from CSFwinformer: $$\mathbf{C}_{\text{img}} = \text{Corr}(\mathbf{F}_{\text{img}}, \hat{\mathbf{F}}_{\text{img}}), \qquad \mathbf{C}_{\text{depth}} = \text{Corr}(\mathbf{F}_{\text{depth}}, \hat{\mathbf{F}}_{\text{depth}}),$$6 where $$\mathbf{C}_{\text{img}} = \text{Corr}(\mathbf{F}_{\text{img}}, \hat{\mathbf{F}}_{\text{img}}), \qquad \mathbf{C}_{\text{depth}} = \text{Corr}(\mathbf{F}_{\text{depth}}, \hat{\mathbf{F}}_{\text{depth}}),$$7 is the depth-warped feature map and $$\mathbf{C}_{\text{img}} = \text{Corr}(\mathbf{F}_{\text{img}}, \hat{\mathbf{F}}_{\text{img}}), \qquad \mathbf{C}_{\text{depth}} = \text{Corr}(\mathbf{F}_{\text{depth}}, \hat{\mathbf{F}}_{\text{depth}}),$$8 is the frequency-enhanced feature (Xu et al., 21 Sep 2025).

4. Optimization, datasets, and evaluation protocol

MirrorSAM2 is trained with a hybrid loss over multiple frames and both intermediate and final predictions: $$\mathbf{C}_{\text{img}} = \text{Corr}(\mathbf{F}_{\text{img}}, \hat{\mathbf{F}}_{\text{img}}), \qquad \mathbf{C}_{\text{depth}} = \text{Corr}(\mathbf{F}_{\text{depth}}, \hat{\mathbf{F}}_{\text{depth}}),$$9 where $$\mathbf{C} = \frac{1}{2}\left(\mathbf{C}_{\text{img}} + \mathbf{C}_{\text{depth}}\right).$$0 denotes the final mirror prediction, $$\mathbf{C} = \frac{1}{2}\left(\mathbf{C}_{\text{img}} + \mathbf{C}_{\text{depth}}\right).$$1 the intermediate prediction, and $$\mathbf{C} = \frac{1}{2}\left(\mathbf{C}_{\text{img}} + \mathbf{C}_{\text{depth}}\right).$$2 the ground-truth mask for frame $$\mathbf{C} = \frac{1}{2}\left(\mathbf{C}_{\text{img}} + \mathbf{C}_{\text{depth}}\right).$$3 (Xu et al., 21 Sep 2025).

The reported training setup is explicit: implementation in PyTorch, AdamW optimizer, learning rate $$\mathbf{C} = \frac{1}{2}\left(\mathbf{C}_{\text{img}} + \mathbf{C}_{\text{depth}}\right).$$4, weight decay $$\mathbf{C} = \frac{1}{2}\left(\mathbf{C}_{\text{img}} + \mathbf{C}_{\text{depth}}\right).$$5, 30 epochs, 4 NVIDIA A100 GPUs with 40 GB each, batch size 1, and input size $$\mathbf{C} = \frac{1}{2}\left(\mathbf{C}_{\text{img}} + \mathbf{C}_{\text{depth}}\right).$$6 (Xu et al., 21 Sep 2025).

Evaluation is conducted on VMD, an RGB video mirror detection dataset, and DVMD, an RGB-D video mirror detection dataset. The metrics are IoU, $$\mathbf{C} = \frac{1}{2}\left(\mathbf{C}_{\text{img}} + \mathbf{C}_{\text{depth}}\right).$$7-measure $$\mathbf{C} = \frac{1}{2}\left(\mathbf{C}_{\text{img}} + \mathbf{C}_{\text{depth}}\right).$$8, Accuracy, and MAE (Xu et al., 21 Sep 2025). The baseline set contains 15 methods, including mirror detection methods such as MirrorNet, PMD, SANet, HetNet, SATNet, and CSFwinformer; video mirror detection methods such as VMDNet, MG-VMD, and DVMDNet; related detection-task methods such as TVSD, SCOTCH, GDNet, VGDNet, and PDNet; and a SAM-family adaptation baseline, SAM2-adapter (Xu et al., 21 Sep 2025).

In the surrounding literature, an earlier SAM2 evaluation on video mirror detection used ground-truth point or mask prompts on the first frame and then propagated masks, reporting that mask prompts are vastly more reliable than point prompts (Jie, 2024). MirrorSAM2’s protocol differs in that prompt generation is automatic and depth-guided, which is precisely the mechanism used to extend SAM2 to the prompt-free setting (Xu et al., 21 Sep 2025).

5. Quantitative performance and ablation evidence

On VMD, MirrorSAM2 reports IoU $$\mathbf{C} = \frac{1}{2}\left(\mathbf{C}_{\text{img}} + \mathbf{C}_{\text{depth}}\right).$$9, $1\times1$0 $1\times1$1, Accuracy $1\times1$2, and MAE $1\times1$3. On DVMD, it reports IoU $1\times1$4, $1\times1$5 $1\times1$6, Accuracy $1\times1$7, and MAE $1\times1$8. The paper states that these results outperform all compared methods, including DVMDNet and SAM2-adapter (Xu et al., 21 Sep 2025).

The reported strengths are concentrated in cases involving small mirrors, weak boundaries, strong reflections, and confusing objects such as paintings, doors, or other reflective-looking surfaces. The authors state that MirrorSAM2 reduces both false positives from RGB ambiguity and false negatives or omissions due to weak edges or reflection clutter (Xu et al., 21 Sep 2025).

The ablation on DVMD isolates the cumulative contributions of the four modules relative to a SAM2 baseline $1\times1$9. The baseline reports IoU $3\times3$0, $3\times3$1 $3\times3$2, Accuracy $3\times3$3, and MAE $3\times3$4. Adding DW yields $3\times3$5, $3\times3$6, $3\times3$7, and $3\times3$8, respectively. Adding DW and DMS-PPG yields $3\times3$9, $\phi(\mathbf{C})$0, $\phi(\mathbf{C})$1, and $\phi(\mathbf{C})$2. Adding FDAF alone yields $\phi(\mathbf{C})$3, $\phi(\mathbf{C})$4, $\phi(\mathbf{C})$5, and $\phi(\mathbf{C})$6. Adding MMD alone yields $\phi(\mathbf{C})$7, $\phi(\mathbf{C})$8, $\phi(\mathbf{C})$9, and $$\tilde{\mathbf{F}}_{\text{img}} = \text{Fuse}_{\text{img}}\left([\mathbf{F}_{\text{img}}, \phi(\mathbf{C})]\right), \qquad \tilde{\mathbf{F}}_{\text{depth}} = \text{Fuse}_{\text{depth}}\left([\mathbf{F}_{\text{depth}}, \phi(\mathbf{C})]\right),$$0. The full MirrorSAM2 obtains the best values across all four metrics: IoU $$\tilde{\mathbf{F}}_{\text{img}} = \text{Fuse}_{\text{img}}\left([\mathbf{F}_{\text{img}}, \phi(\mathbf{C})]\right), \qquad \tilde{\mathbf{F}}_{\text{depth}} = \text{Fuse}_{\text{depth}}\left([\mathbf{F}_{\text{depth}}, \phi(\mathbf{C})]\right),$$1, $$\tilde{\mathbf{F}}_{\text{img}} = \text{Fuse}_{\text{img}}\left([\mathbf{F}_{\text{img}}, \phi(\mathbf{C})]\right), \qquad \tilde{\mathbf{F}}_{\text{depth}} = \text{Fuse}_{\text{depth}}\left([\mathbf{F}_{\text{depth}}, \phi(\mathbf{C})]\right),$$2 $$\tilde{\mathbf{F}}_{\text{img}} = \text{Fuse}_{\text{img}}\left([\mathbf{F}_{\text{img}}, \phi(\mathbf{C})]\right), \qquad \tilde{\mathbf{F}}_{\text{depth}} = \text{Fuse}_{\text{depth}}\left([\mathbf{F}_{\text{depth}}, \phi(\mathbf{C})]\right),$$3, Accuracy $$\tilde{\mathbf{F}}_{\text{img}} = \text{Fuse}_{\text{img}}\left([\mathbf{F}_{\text{img}}, \phi(\mathbf{C})]\right), \qquad \tilde{\mathbf{F}}_{\text{depth}} = \text{Fuse}_{\text{depth}}\left([\mathbf{F}_{\text{depth}}, \phi(\mathbf{C})]\right),$$4, and MAE $$\tilde{\mathbf{F}}_{\text{img}} = \text{Fuse}_{\text{img}}\left([\mathbf{F}_{\text{img}}, \phi(\mathbf{C})]\right), \qquad \tilde{\mathbf{F}}_{\text{depth}} = \text{Fuse}_{\text{depth}}\left([\mathbf{F}_{\text{depth}}, \phi(\mathbf{C})]\right),$$5 (Xu et al., 21 Sep 2025).

A plausible implication is that the four modules act as complementary interventions rather than redundant add-ons. This interpretation is consistent with the authors’ own summary: DW improves localization, DMS-PPG improves prompt quality and spatial coverage, FDAF improves boundary and detail recovery, and MMD improves completeness while suppressing false positives (Xu et al., 21 Sep 2025).

6. Position in the literature, interpretation, and limitations

MirrorSAM2 occupies a specific position in the emerging literature on adapting SAM-family models to non-generic dense prediction tasks. Earlier analysis of SAM2 on camouflaged object detection argued that SAM2 is clearly better than SAM when prompts are available, but that SAM remains stronger than SAM2 in auto mode for prompt-free object discovery; the interpretation offered there is that SAM2 has become more prompt-dependent (Tang et al., 2024). MirrorSAM2 can be read as a task-specific response to that broader issue: rather than assuming that generic SAM2 auto mode is sufficient, it introduces a depth-guided prompt generator and mirror-aware decoding to recover automatic segmentation behavior in a difficult domain (Xu et al., 21 Sep 2025).

The framework is also distinct from zero-shot prompted evaluation. In video mirror detection, SAM2 with a ground-truth first-frame mask can already be very strong, whereas point prompts are unreliable and may degrade severely as the number of points changes (Jie, 2024). MirrorSAM2 addresses the prompt-initialization problem by replacing manual first-frame prompting with DMS-PPG, which mines prompts from depth-salient regions and normalizes them for SAM2’s sparse prompt interface (Xu et al., 21 Sep 2025). This suggests that, for mirror video segmentation, prompt quality is not merely an auxiliary factor but a central systems bottleneck.

Subsequent SAM-family adaptation work on image mirror detection, such as SAM3-UNet, follows a different design trajectory: a frozen SAM3 image encoder, learnable adapters, and a lightweight U-Net-style decoder for downstream dense prediction, including mirror detection on MSD and PMD (Xiong et al., 1 Dec 2025). Relative to that line, MirrorSAM2 is specialized to RGB-D video and retains SAM2 as the temporal core while modifying prompting, fusion, and decoding (Xu et al., 21 Sep 2025).

The limitations stated for MirrorSAM2 are also task-specific. The paper notes that some extreme reflection or highly cluttered scenes remain challenging, with further discussion deferred to supplementary material (Xu et al., 21 Sep 2025). More generally, the method’s dependence on depth as a structural cue implies that its gains are tied to RGB-D availability rather than RGB-only video. That boundary condition is integral to the framework’s design rather than an incidental implementation choice.