MonoMRN: Monocular Semantic Scene Completion

• MonoMRN is a two-stage framework for monocular semantic scene completion that first performs a coarse 3D lifting then refines predictions with a Masked Recurrent Network.
• It employs a Masked Sparse GRU to focus computation on occupied voxels, reducing computational cost while iteratively correcting the coarse estimates.
• The framework integrates Distance Attention Projection to assign ray-wise feature weights, thereby mitigating projection errors and enhancing depth-informed 3D reconstructions.

MonoMRN is a two-stage framework for Monocular Semantic Scene Completion (MSSC) that infers voxel-wise occupancy and semantic categories from a single-view RGB image. It was introduced in “Monocular Semantic Scene Completion via Masked Recurrent Networks” as a decomposition of MSSC into a coarse MSSC stage followed by a Masked Recurrent Network (MRN) that iteratively refines the initial 3D estimate. The design centers on two technical components: a Masked Sparse GRU (MS-GRU) that concentrates computation on occupied voxels through dynamic masking and sparse convolutions, and a Distance Attention Projection (DAP) that assigns ray-wise feature weights according to distance from the observed surface in order to reduce projection errors (Wang et al., 23 Jul 2025).

1. Problem definition and formulation

MonoMRN addresses MSSC under a strictly monocular setting. The input is a single RGB image $I_{\mathrm{RGB}}$, and the output is a voxel grid with two predictions for each voxel: occupancy and semantic class. The semantic label set is defined as $C_i$ with $C_0$ denoting empty and $i \in \{0,1,\dots,N\}$ (Wang et al., 23 Jul 2025).

The framework is explicitly formulated as a two-stage network $\{F_1,F_2\}$ with learnable parameters $\theta_1$ and $\theta_2$. The first stage produces a coarse hidden feature and an initial occupancy mask,

$h_0, m_0 = F_1(I_{\mathrm{RGB}};\theta_1),$

and the second stage refines the estimate recurrently,

$\hat{Y} = F_2(h_0, m_0, x_0; \theta_2),$

where $x_0$ is the 3D feature obtained from DAP (Wang et al., 23 Jul 2025).

The motivation for this decomposition is that single-stage MSSC methods must simultaneously solve visible-region segmentation and occluded-region hallucination while also inheriting errors from monocular depth estimation. MonoMRN separates these functions: the coarse stage performs the initial 2D-to-3D lifting and first-pass prediction, while the recurrent stage corrects errors iteratively and focuses computation on likely occupied regions. This suggests a deliberate reduction of task entanglement rather than a purely architectural scaling of prior monocular scene-completion pipelines.

2. Two-stage architecture

The first stage, coarse MSSC, begins with 2D feature extraction from $C_i$0 using ResNet-50 pre-trained on ImageNet. An off-the-shelf monocular depth estimator, AdaBins, predicts a depth map. The 2D features are then lifted into 3D by surface projection using camera intrinsics and extrinsics, after which a 3D network composed of stacked AIC modules, an intermediate channel-wise attention module, and two deconvolutions produces coarse MSSC logits, the hidden feature $C_i$1, and the initial mask $C_i$2 (Wang et al., 23 Jul 2025).

The second stage, Masked Recurrent Network, takes $C_i$3 as input. A sequence of MS-GRU blocks updates the hidden state and emits MSSC logits at each recurrent step. In the main configuration, the number of recurrent iterations is $C_i$4, and tied weights across iterations are reported to perform best while reducing parameters (Wang et al., 23 Jul 2025).

The forward pass is described in the source as a fixed sequence:

1. $C_i$5 ResNet-50 $C_i$6 2D features.
2. $C_i$7 AdaBins $C_i$8 depth map $C_i$9.
3. 2D-to-3D lifting via surface projection using camera intrinsics/extrinsics.
4. 3D encoder with AIC blocks, channel attention, and deconvolutions $C_0$0 coarse semantic logits, $C_0$1, and $C_0$2.
5. DAP on RGB features and depth along line-of-sight rays $C_0$3 attention-weighted 3D features $C_0$4 AIC $C_0$5.
6. For $C_0$6, MS-GRU updates $C_0$7, the MSSC head predicts $C_0$8, and the Mask Updating Module produces $C_0$9.
7. The final output is $i \in \{0,1,\dots,N\}$0 (Wang et al., 23 Jul 2025).

MonoMRN is presented as a unified indoor/outdoor framework. The same overall pipeline is used for NYUv2 and SemanticKITTI, with dataset-specific voxelizations: $i \in \{0,1,\dots,N\}$1 for NYUv2, and $i \in \{0,1,\dots,N\}$2 intermediate features upsampled to $i \in \{0,1,\dots,N\}$3 for SemanticKITTI (Wang et al., 23 Jul 2025).

3. Masked Recurrent Network and occupancy-focused refinement

The core recurrent operator is the Masked Sparse GRU (MS-GRU). At recurrent step $i \in \{0,1,\dots,N\}$4, with previous hidden state $i \in \{0,1,\dots,N\}$5, previous binary mask $i \in \{0,1,\dots,N\}$6, and DAP feature $i \in \{0,1,\dots,N\}$7, the update is

$i \in \{0,1,\dots,N\}$8

Here, $i \in \{0,1,\dots,N\}$9 if voxel $\{F_1,F_2\}$0 is occupied and $\{F_1,F_2\}$1 otherwise; $\{F_1,F_2\}$2 is sigmoid; $\{F_1,F_2\}$3 is hyperbolic tangent; and SubConv and SConv denote submanifold sparse convolution and sparse convolution, respectively (Wang et al., 23 Jul 2025).

The masking is not auxiliary. It gates both the hidden state and the input feature before the gate computations and candidate-state update, thereby restricting recurrent computation to occupied or potentially occupied voxels. Submanifold sparse convolutions are used for the update and reset gates in order to preserve the active set, whereas sparse convolutions are used for the candidate update so that information can propagate into occupied neighborhoods (Wang et al., 23 Jul 2025).

The reported computational effect is substantial. Replacing a dense GRU with MS-GRU reduces MACs from 171.99 G to 52.44 G per iteration at the same 1.33 M parameter count. With tied weights, two iterations require approximately 104.88 G MACs, while tied weights also reduce parameters relative to untied weights (1.33 M versus 2.66 M) and slightly improve accuracy (Wang et al., 23 Jul 2025).

MonoMRN also includes a Mask Updating Module that refines the occupancy mask across iterations. The initial occupancy confidence is defined as

$\{F_1,F_2\}$4

and with threshold $\{F_1,F_2\}$5, the initial mask is set by thresholding $\{F_1,F_2\}$6 (Wang et al., 23 Jul 2025).

Subsequent mask updates are produced from MSSC-head features rather than directly from $\{F_1,F_2\}$7, which decouples mask prediction from hidden-state dynamics. The module uses 3×3 convolution, global average pooling with twofold downsampling, dropout with rate 0.1, another 3×3 convolution, and softmax to produce occupancy and emptiness probabilities. A top-$\{F_1,F_2\}$8 strategy with $\{F_1,F_2\}$9 adds the five voxels with highest occupancy probability and removes the five masked voxels with highest emptiness probability (Wang et al., 23 Jul 2025).

4. Distance Attention Projection and 2D-to-3D lifting

DAP is the mechanism by which MonoMRN modulates line-of-sight feature propagation. The geometric mapping between 2D and 3D is written as

$\theta_1$0

where $\theta_1$1 is the camera intrinsic matrix and $\theta_1$2 is the extrinsic matrix (Wang et al., 23 Jul 2025).

The paper contrasts two projection regimes. Surface projection places features only at the estimated surface and therefore leaves occluded voxels without features. Sight projection propagates features along rays and therefore reaches occluded regions, but introduces many incorrect assignments. DAP is introduced to preserve the reach of line-of-sight propagation while attenuating its projection errors (Wang et al., 23 Jul 2025).

The attention weight along a ray is defined by the distance $\theta_1$3 from the camera to a voxel and the estimated surface distance $\theta_1$4 on that ray:

$\theta_1$5

where $\theta_1$6 is set according to the RMS of predicted depth. The source gives AdaBins’ RMS on NYUv2 as the reference for choosing $\theta_1$7 (Wang et al., 23 Jul 2025).

The line-of-sight-projected 3D feature is multiplied element-wise by $\theta_1$8, and the result is passed through an AIC module to produce the recurrent input feature $\theta_1$9. In ablation, Distance Attention Projection outperforms both surface projection and sight projection alone on NYUv2: 53.16 / 30.73 for SC IoU / SSC mIoU, compared with 51.23 / 29.36 for surface projection and 52.56 / 30.12 for sight projection (Wang et al., 23 Jul 2025).

A plausible implication is that DAP is not merely a feature-weighting heuristic but a mechanism for reconciling depth-conditioned and depth-agnostic lifting in monocular 3D reasoning.

5. Training objectives and optimization

MonoMRN is trained with sequential supervision over both semantic predictions and occupancy masks. The sequential MSSC loss is

$\theta_2$0

where $\theta_2$1 is the coarse prediction, $\theta_2$2 are recurrent outputs, $\theta_2$3 is cross-entropy, and the decay factor is $\theta_2$4 (Wang et al., 23 Jul 2025).

The sequential mask loss is

$\theta_2$5

where $\theta_2$6 is the occupancy ground truth obtained by summing all non-empty classes, $\theta_2$7 is weighted binary cross-entropy, and the decay factor is $\theta_2$8 (Wang et al., 23 Jul 2025).

The full objective is

$\theta_2$9

where $h_0, m_0 = F_1(I_{\mathrm{RGB}};\theta_1),$0 is the Scene-Class Affinity Loss from MonoScene (Wang et al., 23 Jul 2025).

The reported training configuration uses PyTorch, SGD, and an initial learning rate of 0.1 with polynomial decay. Training lasts 200 epochs on NYUv2 and 30 epochs on SemanticKITTI. The main recurrent configuration uses 2 MS-GRU iterations with tied weights (Wang et al., 23 Jul 2025).

The source emphasizes that coarse predictions and all recurrent MSSC states are supervised, and recurrent masks are supervised against occupancy ground truth. No separate visible/occluded supervision is introduced; instead, robustness to noisy depth is attributed to the combination of recurrent refinement and DAP attenuation.

6. Evaluation, ablations, robustness, and limitations

MonoMRN is evaluated on NYUv2 and SemanticKITTI using SC IoU for scene completion and SSC mIoU for semantic scene completion. NYUv2 uses 1,449 RGB-D frames with 795 train and 654 test samples under the SSCNet protocol. SemanticKITTI uses sequences 00–10 for training, 08 for validation, and 11–21 for testing, with camera-2 RGB cropped to 1220 × 370 and evaluation on 256 × 256 × 32 voxel grids at 0.2 m resolution (Wang et al., 23 Jul 2025).

Dataset	Main result
NYUv2	SC IoU 53.16%, SSC mIoU 30.73%
SemanticKITTI hidden test	SC IoU 42.0%, SSC mIoU 13.8%

On NYUv2, the paper reports that MonoMRN improves over NDC-Scene by +8.99% SC IoU and +1.60% SSC mIoU, and over MonoScene by +10.65% SC IoU and +3.79% SSC mIoU. On SemanticKITTI, it improves over VoxFormer-S monocular by +3.3% SC IoU and over NDC-Scene by +1.1% SSC mIoU (Wang et al., 23 Jul 2025).

The ablation sequence on NYUv2 attributes the final performance to cumulative gains from the recurrent design. The coarse baseline achieves 48.23 / 27.47. Adding MS-GRU raises this to 50.61 / 29.16; adding DAP yields 51.86 / 30.11; and adding Mask Updating yields the final 53.16 / 30.73 (Wang et al., 23 Jul 2025).

Further ablations show that:

• MS-GRU outperforms dense GRU (53.16 / 30.73 versus 50.48 / 29.67).
• Tied weights outperform untied weights (53.16 / 30.73 versus 52.85 / 30.34).
• Increasing recurrent depth helps up to a point: 1× gives 53.01 / 30.11, 2× gives 53.16 / 30.73, 3× gives 53.51 / 30.86, and 4× reduces SSC mIoU to 30.17, indicating over-iteration (Wang et al., 23 Jul 2025).

The paper also reports robustness analysis under indoor corruptions (darkness, motion blur) and outdoor corruptions (brightness, fog), each in weak and strong settings. After MRN refinement, the model consistently shows higher robustness than the coarse baseline, with improvements attributed to iterative masked updates and DAP’s attenuation of noisy ray propagation (Wang et al., 23 Jul 2025).

Efficiency is reported explicitly. On NYUv2, throughput is 2.56 FPS, compared with 3.68 FPS for AICNet, 3.12 FPS for 3DSketch, and 1.96 FPS for MonoScene. This places MonoMRN between lightweight feed-forward baselines and slower monocular scene-completion systems, while retaining recurrent refinement and sparse 3D computation (Wang et al., 23 Jul 2025).

The limitations identified in the source remain centered on depth quality and small classes. The paper notes that indoor AdaBins RMS (0.364 m) is much larger than the 0.08 m voxel size used on NYUv2, which constrains surface-projection fidelity. It also identifies extreme occlusions, highly inaccurate depth predictions, and rare small classes such as bicycle and motorcycle as persistent difficulties (Wang et al., 23 Jul 2025).

MonoMRN therefore occupies a specific position within monocular 3D scene understanding: it is a monocular semantic scene completion system that combines coarse 3D lifting with occupancy-focused recurrent correction, sparse 3D recurrence, and distance-aware ray projection. Its empirical behavior suggests that its principal contribution lies in restructuring the completion problem around iterative refinement rather than treating visible-region parsing and occluded-region hallucination as a single monolithic prediction stage.