Masked Sparse GRU for 3D Scene Completion

• MS-GRU is a recurrent neural module that employs binary masking and sparse convolutions to restrict computation to occupied voxels.
• It integrates into the MonoMRN pipeline to achieve efficient 3D semantic scene completion, reducing MACs by approximately 3.3x while maintaining accuracy.
• The module features a dynamic mask updating strategy and distance attention projection to enhance voxel-wise occupancy and semantic labeling.

The Masked Sparse Gated Recurrent Unit (MS-GRU) is a recurrent neural module designed for efficient 3D reasoning in the context of Monocular Semantic Scene Completion (MSSC). MS-GRU extends the conventional GRU by introducing binary masking to restrict computation to occupied voxels and by leveraging sparse convolutions, reducing resource requirements while maintaining state-of-the-art performance in voxel-wise occupancy and semantic estimation from single RGB images. Integrated within the MonoMRN pipeline, MS-GRU supports efficient and robust scene understanding in both indoor and outdoor environments (Wang et al., 23 Jul 2025).

1. Architectural Foundations and GRU Recap

The standard Gated Recurrent Unit (GRU) operates on input $x_t \in \mathbb{R}^d$ and hidden state $h_{t-1} \in \mathbb{R}^k$ by computing update and reset gates as:

• $z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$
• $r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$
• $$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$
• $$h_t = (1-z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$

where $\sigma$ is the sigmoid function and $\odot$ denotes element-wise multiplication.

MS-GRU modifies this structure for sparse, masked operation suitable for volumetric scenes represented on a 3D voxel grid.

2. Masked Sparse GRU Mechanism

MS-GRU utilizes a binary mask $m_{t-1} \in \{0,1\}^{X \times Y \times Z}$ to restrict computation to predicted occupied voxels. The masking zeros out both the input tensor $x_t$ and hidden state $h_{t-1} \in \mathbb{R}^k$0 for empty locations before applying a mixture of two types of sparse convolutions:

• Submanifold sparse convolution (SubConv): Applied so that only active voxels are processed and the sparseness pattern is preserved.
• Neighborhood-expanding sparse convolution (SConv): Operates on a sparse grid but can activate neighboring voxels, still maintaining computational efficiency over standard dense convolutions.

The MS-GRU recurrence at time $h_{t-1} \in \mathbb{R}^k$1 is:

• $h_{t-1} \in \mathbb{R}^k$2
• $h_{t-1} \in \mathbb{R}^k$3
• $h_{t-1} \in \mathbb{R}^k$4
• $h_{t-1} \in \mathbb{R}^k$5

The channel-wise concatenation $h_{t-1} \in \mathbb{R}^k$6 is used within convolutional gates. All convolutions use shared weights per iteration.

3. Mask Initialization and Update Strategy

A core aspect of MS-GRU is the management of the occupancy mask sequence $h_{t-1} \in \mathbb{R}^k$7.

• Initialization: From the coarse-stage predicted emptiness probability $h_{t-1} \in \mathbb{R}^k$8, the initial mask is set as $h_{t-1} \in \mathbb{R}^k$9 if $z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$0, with $z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$1.
• Sequential Mask Update: During each recurrent step $z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$2, a “mask-head” module (comprised of 3x3 convolutions, global average pooling, dropout, and softmax) computes new voxel-wise occupancy scores $z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$3 and emptiness scores $z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$4. The top-K voxels (K=5) by $z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$5 are added to the mask (set to $z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$6), and the top-K by $z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$7 are removed (set to $z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$8), yielding

$z_t = \sigma(W_z x_t + U_z h_{t-1} + b_z)$9

This mechanism dynamically adjusts the spatial support of MS-GRU to refine predictions while constraining computation.

4. Sparse Operations and Computational Efficiency

MS-GRU’s computational advantage derives from restricting all recurrent operations to the nonzero mask region, with complexity further controlled by employing SubConv and SConv:

• The sparse convolutions process only nonzero voxels and their local neighborhoods.
• In empirical evaluation, a standard dense GRU over the full 3D grid incurs approximately 171.99 G multiply–accumulate operations (MACs), whereas MS-GRU reduces this to 52.44 G MACs, achieving a $r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$03.3x reduction in computational cost without parameter count increase (1.33 M parameters) (Wang et al., 23 Jul 2025).

A pseudo-code trace for a gate variable, e.g., $r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$1, highlights selection of indices where $r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$2, computation of convolutional outputs, and application of $r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$3 at those locations only.

5. Distance Attention Projection

To generate high-fidelity 3D context features from monocular input, the Distance Attention Projection (DAP) assigns attention weights along the viewing ray of each pixel based on its observed or estimated depth $r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$4:

• For voxel at distance $r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$5 on the same ray,
  • $r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$6 for $r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$7
  • $r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$8 for $r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$9
  • $$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$0 for $$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$1
  • where $$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$2 is the depth estimation RMS error (0.364 m indoors).
• Projected features $$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$3 are fused into a 3D tensor via an attention-integrated context (AIC) block, providing spatially-aware input $$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$4 for MS-GRU.

6. Integration within the Two-Stage Scene Completion Framework

MS-GRU is positioned within MonoMRN’s two-stage MSSC architecture:

• Stage 1: ResNet-50 backbone with AdaBins depth estimation and surface projection produces initial coarse features $$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$5 and occupancy mask $$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$6 via a 3D AIC network.
• Stage 2: For $$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$7 (T=2 in practice), inputs $$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$8 are obtained from the RGB/depth via DAP and AIC; MS-GRU updates the hidden state and mask $$\tilde{h}_t = \tanh(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h)$$9 accompanied by the mask-head logic. The final semantic prediction is obtained from $$h_t = (1-z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$0 via a softmax layer.

This design enables iterative refinement focused on dynamically-selected occupied voxels, with propagation of context informed by learned mask updates.

7. Quantitative Results and Implementation Details

MonoMRN and MS-GRU are validated on NYUv2 (60%%%%51$r_t = \sigma(W_r x_t + U_r h_{t-1} + b_r)$652%%%%60 voxel grid) and SemanticKITTI (128$$h_t = (1-z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$3128$$h_t = (1-z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$416, upsampled to 256$$h_t = (1-z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$5256$$h_t = (1-z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$632). Key measurements:

• NYUv2: Scene Completion IoU (SC-IoU) = 53.16% (an increase of 8.99% over prior best); Semantic Scene Completion mIoU = 30.73% (+1.60%). Framerate $$h_t = (1-z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$72.56 FPS.
• SemanticKITTI: SC-IoU = 42.0% (+3.35%), SSC-mIoU = 13.8% (+1.07%).
• Ablation on NYUv2: Baseline (coarse only): SC-IoU 48.23%, SSC-mIoU 27.47%; +MS-GRU: 50.61%, 29.16%; +DAP: 51.86%, 30.11%; +mask updating: 53.16%, 30.73%.
• Robustness: The Masked Recurrent Network (incl. MS-GRU) maintains superior SC-IoU under visual disturbances (dark, bright, motion blur, or fog).

Implementation parameters include hidden state matching coarse features (32–64 dimensions), SGD optimizer (initial lr=0.1, poly schedule, 200 epochs for NYUv2, 30 for KITTI), mask threshold $$h_t = (1-z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$8, top-K=5 for mask-head, and sequential loss decays of $$h_t = (1-z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$9 (MSSC) / $\sigma$0 (mask).

8. Context, Significance, and Applicability

MS-GRU directly addresses the inefficiency of dense 3D recurrent units by focusing learning and computation on meaningful regions within large voxel spaces, enabled by dynamic masking and sparse convolutions. Its role in MonoMRN exemplifies advanced strategies in monocular scene understanding, specifically for tasks requiring simultaneous semantic labeling and completion of partially-observed scenes (Wang et al., 23 Jul 2025).

This framework—comprising formal recurrent equations, a mask-driven sparse strategy, depth-aware attention projection, and an efficient recurrent training loop—provides a direct blueprint for reimplementing or extending MS-GRU in high-dimensional, sparsity-dominated 3D reasoning tasks.