MoonSeg3R: Monocular 3D Segmentation

• MoonSeg3R is an online, zero-shot monocular framework that generates per-instance 3D masks and partial scene geometry from uncalibrated RGB video without ground-truth depth.
• It leverages reconstructive (CUT3R) and vision foundation models (CropFormer/FastSAM) via query refinement and 3D memory to ensure temporal consistency and competitive segmentation performance.
• The architecture integrates spatial–semantic distillation and state-distribution tokens to robustly fuse segmentation proposals, enabling real-time 3D instance segmentation in challenging conditions.

MoonSeg3R is an online, zero-shot, monocular 3D instance segmentation framework that leverages reconstructive and visual foundation models to produce temporally consistent, per-instance 3D masks and partial scene geometry from a single uncalibrated RGB video stream, without access to any ground-truth depth, camera pose, or 3D masks. In contrast to previous approaches relying on posed RGB-D data, MoonSeg3R operates entirely from monocular input and integrates reconstructive priors from CUT3R, a reconstructive foundation model (RFM), with segmentation proposals from a vision foundation model (VFM) such as CropFormer or FastSAM. The system demonstrates competitive segmentation performance relative to state-of-the-art RGB-D pipelines while operating in a fully monocular, online fashion (Du et al., 17 Dec 2025).

1. Online Monocular 3D Instance Segmentation: Task and Motivation

MoonSeg3R addresses the problem of performing per-frame, real-time 3D instance segmentation from a continuous stream of uncalibrated monocular RGB frames $\{I_t\}_{t=1}^T$, producing both partial 3D reconstructions and per-instance masks with no ground-truth 3D information or explicit supervision. Unlike RGB-D-based models such as EmbodiedSAM and OnlineAnySeg which assume access to accurate depth and pose data per frame, MoonSeg3R infers geometry implicitly from the monocular stream, enabling deployment in environments lacking depth sensors.

Key challenges in this setting include handling partial and occluded observations (as objects may only be seen intermittently or from limited viewpoints), coping with noisy geometry reconstructions subject to drift, and ensuring temporal consistency so instance identities remain coherent across viewpoint changes despite the absence of explicit 3D supervision (Du et al., 17 Dec 2025).

2. System Architecture and Component Roles

The MoonSeg3R architecture couples CUT3R (RFM) and a 2D vision foundation model (CropFormer or FastSAM). For each input frame $I_t$:

• CUT3R produces explicit geometry (pointmap $X_t \in \mathbb{R}^{H \times W \times 3}$, camera pose $P_t$), implicit 3D features ($F_t^{3d} \in \mathbb{R}^{H \times W \times d_1}$), and a state-attention tensor ($A_t \in \mathbb{R}^{n_s \times (H\cdot W)}$) from persistent state tokens.
• VFM yields per-pixel 2D masks $\{M_t^i\}$ and 2D semantic features ($F_t^{2d} \in \mathbb{R}^{H \times W \times d_2}$).
• Query Refinement Module lifts each 2D mask into a 3D prototype $q_t^i$, applies transformer-based refinement and context injection via cross-attention with features $[F_t^{3d}, F_t^{2d}]$, and contextual queries from memory.
• 3D Query Index Memory (QIM) maintains refined queries and their historical spatial association for temporal consistency.
• Mask Fusion and Cross-Frame Association merges over-segmented regions (within frames) and matches masks temporally across frames using a state-distribution token derived from CUT3R's state-attention (Du et al., 17 Dec 2025).

3. Self-Supervised Query Refinement, Memory, and State-Distribution Mechanisms

a) Query Refinement

Masks from the VFM are lifted to initial prototypes via masked average pooling over concatenated geometric and semantic features: $$q_t^i = \eta \left( \frac{\sum_{u,v} F_t(u,v)\,M_t^i(u,v)}{\sum_{u,v} M_t^i(u,v)} \right)$$ with $F_t(u,v) = [F_t^{3d}(u,v), F_t^{2d}(u,v)]$ and $\eta(\cdot)$ as a learned projection.

These prototypes undergo transformer-based refinement, where a 3-layer decoder $\phi$ first attends to current frame features, then to contextual queries from QIM: $$q'_t := \phi(\mathrm{Attn}(q_t, F_t, M_t)), \quad q'_t := \phi(\mathrm{Attn}(q'_t, \mathcal{Q}_t^\mathrm{ctx}))$$ The refined prototype $q'_t$ is trained to reconstruct the 2D mask: $$\mathcal{L}_\mathrm{seg} = \mathrm{BCE}\left(\sigma(\psi(F_t) \odot q'_t), M_t \right)$$ where $\psi$ is an MLP, $\odot$ denotes channel-wise multiplication, and $\sigma$ is sigmoid activation.

b) Spatial–Semantic Distillation

To conserve both geometry and semantic signals, Gram matrices of the features are matched for both 2D and 3D channels: $$\mathcal{L}_\mathrm{dist} = \|G - G^{2d}\|_F^2 + \|G - G^{3d}\|_F^2$$ where $G$ is computed on concatenated features, $G^{2d}$ and $G^{3d}$ on respective channels.

c) 3D Query Index Memory (QIM) and Cross-Frame Consistency

QIM stores global query vectors and a spatial index. For contextual retrieval, keys are projected into the current frame, and corresponding queries are retrieved to support temporal fusion. Cross-frame supervision enforces query consistency via binary cross-entropy loss on warped and fused memory features.

d) State-Distribution Token

From CUT3R's state-attention $A_t$, an instance-specific state-distribution token $s_t^i$ is defined: $$s_t^i = \sum_{j=1}^{H \cdot W} (A_t \odot M_t^i)_{:,j}$$ This serves as a temporally stable identity descriptor, supporting intra-frame merge (cosine similarity $>0.8$) and cross-frame matching (combined cosine similarity and IoU score, threshold $1.8$).

4. Training, Inference, and Implementation Details

The total training objective is a weighted sum: $$\min_{\eta,\phi,\psi} \, \lambda_\mathrm{seg} \mathcal{L}_\mathrm{seg} + \lambda_\mathrm{dist} \mathcal{L}_\mathrm{dist} + \lambda_\mathrm{xseg} \mathcal{L}_\mathrm{xseg}$$ with $\lambda_\mathrm{seg}=1$, $\lambda_\mathrm{dist}=0.1$, $\lambda_\mathrm{xseg}=0.5$.

• Architectural summary: $\phi$ is a 3-layer transformer decoder; $\eta, \psi$ are 3-layer MLPs.
• Training: Sequences of $16$ frames ($512{\times}384$) from ScanNet (no 3D label supervision); FastSAM for training masks, CropFormer at inference. Foundation models fixed (CUT3R, DINOv3). AdamW optimizer, $lr=10^{-4}\rightarrow10^{-5}$, batch 4, 100 epochs on 4$\times$A6000 GPUs.
• Inference: Merge threshold 0.8 (intra-frame), match threshold 1.8 (cross-frame). Overall speed: 55ms (mask fusion) + 66ms (CUT3R) per frame on a single GPU (Du et al., 17 Dec 2025).

5. Quantitative and Qualitative Evaluation

Quantitative Performance: ScanNet200 and SceneNN

Model	AP	AP$_{50}$	AP$_{25}$
OnlineAnySeg-M (monocular)	13.4	26.8	43.2
MoonSeg3R	16.7	33.3	50.0

MoonSeg3R approaches RGB-D SOTA performance while using only monocular video. On SceneNN the model achieves $AP=14.3$, $AP_{50}=31.4$, $AP_{25}=48.4$.

Ablation Studies

Configuration	AP	AP$_{50}$
$F^{2d}+F^{3d}$ only	8.1	19.7
+ Query Refinement	12.5	27.7
+ SSD	13.5	29.3
+ QIM	15.9	32.8
+ State-Distribution Token	16.7	33.3

Each module contributes incremental improvements to both detection and mask quality.

Qualitative Analysis

MoonSeg3R produces more complete, temporally stable segmentation under challenging conditions, notably outperforming monocular baselines under occlusions.

6. Integration of Foundation Model Priors

CUT3R provides explicit geometry (for unprojection and memory), dense 3D features (used in all query refinement), and state attention for building temporally consistent mask descriptors. The learning objectives enforce that the segmentation queries distill both geometric and semantic content, while the pose estimates from CUT3R support spatial memory alignment. Mask fusion and cross-frame instance association depend critically on these reconstructive priors (Du et al., 17 Dec 2025).

7. Limitations, Strengths, and Future Directions

MoonSeg3R is the first method to achieve truly monocular, online, zero-shot 3D instance segmentation without any access to depth or pose ground truth. Its runtime is competitive with RGB-D alternatives (approximately 120 FPS end-to-end with CUT3R), and its modular framework permits substitution of different RFMs or VFMs for further generalization or performance gains.

The principal limitation is the inherited geometric drift from the RFM component over extended sequences, which degrades temporal consistency. Additionally, the evaluation suite is presently restricted to static scenes and does not address dynamic or deformable objects.

Proposed future directions include integrating explicit long-term memory or loop closure mechanisms to mitigate drift, extending to dynamic environments via motion segmentation, leveraging multi-view monocular or stereo inputs, and exploring tighter end-to-end joint training of RFM and segmentation modules to improve overall performance (Du et al., 17 Dec 2025).