---
title: 3D Spatial Grid Masking
url: https://www.emergentmind.com/topics/3d-spatial-grid-masking
type: topic
---

# 3D Spatial Grid Masking

3D Spatial Grid Masking denotes a family of masking mechanisms in which visibility, preservation, sparsity, or conditioning is controlled by explicit spatial structure rather than by purely random token omission or purely sequential attention. Recent uses span multi-view 3D style transfer, text-driven 3D editing, camera-only 3D occupancy prediction, rotation-invariant point cloud masked autoencoding, and 3D Gaussian Splatting compression. Depending on the representation, the mask may live on a non-overlapping image patch grid used to condition 3D stylization, on voxel lattices such as $16^3$ and $64^3$, on height-band partitions tied to a semantic occupancy grid, on rank-space grids over point-cloud patch centers, or on a binary hash grid coupled to learned offset masks [2510.10497] [2604.13688] [2409.07972] [2509.14975] [2403.14530].

## 1. Formal definitions and representational domains

The term is not tied to a single data structure. In "Jigsaw3D" [2510.10497], the mask is defined on a spatial patch grid over a reference image, but the masked reference is used as a 3D style-conditioning signal for a multi-view diffusion pipeline. Let a reference image be $I \in \mathbb{R}^{C \times H \times W}$, partitioned into non-overlapping $S \times S$ patches $P_{i,j}$. After a random permutation $\sigma$ and Bernoulli mask $M_{i,j}$ with mask ratio $p$, the jigsawed reference is
$$
I_{jigsaw} = \bigcup_{i,j} \left[ M_{i,j} \odot P_{\sigma(i,j)} + (1-M_{i,j}) \odot \mu \right].
$$
Training uses $S=64$ and $p \in [0,0.25]$; inference uses $S=128$, and by default only shuffling is applied.

In "Beyond Voxel 3D Editing" [2604.13688], the preservation mask is explicitly 3D and binary. The model follows TRELLIS’s structured latent design with a coarse SS grid of $16^3$ and a finer SLAT grid of $64^3$. For a canonical grid point $p \in \Omega$, after rigid alignment $T^*$ between the original and edited point clouds, the mask is
$$
M(p) = \left[ \|T^*p - q^*\|_2 < \tau \right],
$$
where $q^*$ is the nearest neighbor in the edited point cloud. The resulting tensor $M \in \{0,1\}^{H \times W \times D}$ is indexed either densely or sparsely.

In "Deep Height Decoupling" [2409.07972], masking is a height-aware gating mechanism for 2D-to-3D projection. HeightNet predicts a discretized height map with $B=16$ bins, and Mask Guided Height Sampling defines interval-specific masks
$$
M_k(u,v) = \mathbb{1}[H_{map}(u,v) \in I_k].
$$
The default partition is $I_1=[1,4]$, $I_2=[5,8]$, $I_3=[9,16]$, and masked image features are projected only into the corresponding 3D height subspaces.

In the dual-stream rotation-invariant point cloud MAE [2509.14975], 3D Spatial Grid Masking is not voxelization but rank-space grid construction over patch centers. After FPS selects $K$ centroids and KNN forms patch tokens, each centroid receives axis-wise ranks $pos_d[i]$, grid bits
$$
grid_d[i] = \left\lfloor \frac{pos_d[i]}{G_d} \right\rfloor \bmod 2,
$$
and a bit-packed type
$$
grid\_type[i] = grid_x[i] + 2grid_y[i] + 4grid_z[i] \in \{0,\dots,7\}.
$$
The default is $G_x=G_y=G_z=4$.

In HAC [2403.14530], the grid is a learned binary hash grid queried at anchor locations, while masking acts on offsets. If $o \in \mathbb{R}^{3K}$ is the per-anchor offset vector and $M \in \{0,1\}^K$ is a binary gate, masked offsets become $M \odot o$; if all gates are zero, the anchor is invalid and removed.

| Work | Mask support | Primary function |
|---|---|---|
| Jigsaw3D | Non-overlapping $S \times S$ patch grid | Suppress semantics, preserve local style statistics |
| BVE | Canonical voxel grids $16^3$ and $64^3$ | Preserve unedited regions during 3D editing |
| DHD | Height-band masks tied to occupancy grid projection | Reduce cross-height feature contamination |
| Dual-stream RI MAE | Rank-space grid over patch centers | Impose structured masking for geometric continuity |
| HAC | Binary hash grid plus offset mask | Context modeling and removal of invalid Gaussians |

## 2. Functional roles of masking

A central use of spatial grid masking is semantic suppression without destroying local statistics. Jigsaw3D states that patch-level shuffling breaks global shape and layout cues, which are essential for object semantics, while preserving local statistics such as color means and variances, stroke textures, and frequency content. The paper quantitatively shows that as the grid gets finer, CNN-based content classification accuracy drops sharply while Gram matrix similarity to the source increases, and its appendix proof shows that shuffling preserves first- and second-order statistics per channel [2510.10497]. In that setting, masking is not primarily for occlusion simulation; it is a disentanglement device.

In BVE, the role is local invariance. The edited-original paired training objective contains a standard edit term and a mask-weighted preservation term,
$$
\mathcal{L}_{edit} = \mathbb{E}_{t,x,\epsilon}\left[\|v_\theta - v_{edit}\|^2 + \|M \odot (v_\theta - v_{orig})\|^2\right].
$$
Masked regions are therefore driven toward the original asset’s flow target, while unmasked regions follow the edited target. The paper explicitly states that the second term increases the penalty in masked, preserved regions to match the original asset’s flow target, enforcing local invariance [2604.13688].

In DHD, masking is a height prior for projection. Forward projection methods mix features across height ranges, producing “confusing features” that harm voxel-wise semantic classification. MGHS filters per-pixel features before lifting, so only pixels whose predicted height falls into the appropriate interval contribute to the corresponding $z$ slices. Dataset analysis reported in the paper shows that lower height bands $[1\!-\!4]$ concentrate drivable surface, sidewalk, and terrain, while upper bands $[9\!-\!16]$ favor vegetation, manmade, and trailer; a weighted entropy metric indicates that $[1,4]$, $[5,8]$, $[9,16]$ yields the lowest entropy [2409.07972].

In the point-cloud MAE setting, the function is structured pretext generation rather than preservation or projection gating. The paper argues that random masking treats patches independently and ignores long-range spatial relationships, whereas 3D Spatial Grid Masking creates structured patterns through coordinate sorting so the encoder sees coherent spatial context during pretraining. The intended benefit is not exact invariance of the mask itself, but preservation of adjacency and regularity in the visible/hidden layout [2509.14975].

In HAC, masking is rate-distortion control. The paper observes an impulse at zero in empirical histograms of offsets and uses a learned adaptive offset masking module to remove redundant or non-contributing Gaussians. Here the mask reduces the number of serialized symbols while the binary hash grid provides spatial context for entropy coding [2403.14530].

## 3. Integration into learning, generation, and compression pipelines

The masked object or grid is typically not an isolated preprocessing artifact; it is integrated directly into the core inference or training objective. In Jigsaw3D, the jigsawed reference is encoded by a pretrained VAE encoder, passed through a frozen or reference pretrained diffusion U-Net at $t=0$, and reference features are extracted from self-attention hidden states. These features then enter the Style U-Net through reference-to-view cross-attention,
$$
\alpha = softmax\left(\frac{QK^T}{\sqrt{d}}\right), \qquad RefAttention(f_{in}, f_{ref}) = \alpha V.
$$
The Style U-Net combines standard self-attention per view, efficient row-wise multi-view attention, and reference attention after each residual layer, followed by text-conditioned cross-attention. Geometry enters through rendered position and normal maps from $K$ predefined viewpoints, passed through a T2I-Adapter-style condition encoder. The stylized renders are then baked back to the mesh by visibility-aware reprojection, seam-aware confidence-weighted blending, tangent-space normal baking, and UV-space inpainting [2510.10497].

In BVE, masking is present only in the backward signal. The paper states that the forward pass does not gate activations: the network predicts $v_\theta$ as usual with multimodal conditioning, and the preservation term is multiplied by $M$ only in the loss. Stage-specific masks $M_{16}$ and $M_{64}$ are used for SS and SLAT respectively, and there is no mask computation at inference; inference edits are driven by conditioning learned during training [2604.13688].

In DHD, MGHS sits inside the camera-to-occupancy view transform. HeightNet predicts per-pixel bins, interval masks are applied in image space to context features, masked features are projected into disjoint 3D subspaces, and the resulting height-refined volume $F_{hr}$ is fused with an unmasked depth-based stream $F_{db}$ by Synergistic Feature Aggregation. SFA first applies channel attention and then spatial attention:
$$
F_{agg} = A_2 \odot F_{db}^{cs} + (1-A_2)\odot F_{hr}^{cs}.
$$
The total loss combines occupancy, height, optional depth, and scaling losses with default weights $\lambda_{depth}=0.05$, $\lambda_{height}=0.1$, $\lambda_{occ}=10$, and $\lambda_{sem}=\lambda_{geo}=0.2$ [2409.07972].

In the dual-stream point-cloud MAE, grid masking is fused with Progressive Semantic Masking by curriculum. The paper defines
$$
\alpha(t) = (t/T)^\gamma, \qquad P_i^{(t)} = (1-\alpha^{(t)})P_i^{grid} + \alpha^{(t)}P_{i,sem}^{(t)},
$$
with $\gamma=2$ and a target mask ratio $m=0.75$. Early training is therefore dominated by grid masking, and later training shifts toward semantic masking [2509.14975].

In HAC, masking is introduced after a staged warmup. The system first trains Scaffold-GS anchors, then introduces quantization noise, and only after iteration 10000 enables the binary hash grid, context-conditioned probability modeling, and adaptive offset masking. The total loss combines rendering fidelity, attribute entropy, hash-grid entropy, and masking:
$$
Loss = L_{Scaffold} + \lambda_e \cdot \frac{1}{N(D_a + 6 + 3K)}(L_{entropy}+L_{hash}) + \lambda_m L_m.
$$
Thus, the grid and the mask are optimized jointly with compression targets rather than appended after training [2403.14530].

## 4. Reported empirical behavior

Across domains, spatial grid masking is reported to improve either fidelity, invariance, semantic localization, or compression efficiency.

Jigsaw3D evaluates on 20 Objaverse meshes unseen in training and 30 WikiArt styles plus 40 collected styles. Style fidelity is measured by Gram matrix similarity and AdaIN distance using VGG19 features, and style-content disentanglement is measured by CLIP score, with lower being better. The paper reports that Jigsaw3D achieves the best Gram and AdaIN on both datasets, competitive CLIP, and latency of approximately 40 seconds, whereas SDS optimization in StyleTex is approximately 15 minutes. Ablations report that moderate masking in the range $0$ to $0.25$ improves robustness and style capture, while too high a mask ratio such as $0.75$ harms geometric consistency and increases artifacts during baking [2510.10497].

BVE reports a direct ablation of masking. For Ours(full) versus w/o MASK, the paper gives CD $0.013$ versus $0.019$, SSIM $0.960$ versus $0.824$, LPIPS $0.039$ versus $0.048$, FID $28.9$ versus $49.2$, FVD $1139$ versus $1274$, and DINO-I $0.960$ versus $0.939$. It further states that Ours(full) achieves the best scores across CD, SSIM, LPIPS, FID, FVD, DINO-I, and CLIP-T, together with user study preferences of $91.0\%$ for Text Alignment and $88.5\%$ for Overall 3D Quality [2604.13688].

DHD reports state-of-the-art occupancy performance on Occ3D-nuScenes even with minimal input frames. DHD-S reaches mIoU $36.50$, outperforming FlashOcc at $31.95$ by $+4.55$; DHD-M reaches $41.49$, slightly above COTR(BEVDetOcc) at $41.39$ and OSP at $39.41$; DHD-L reaches $45.53$, above FlashOcc at $43.52$ and GEOcc at $44.67$. The ablation on DHD-S reports $33.72$ for the baseline without MGHS decoupling or SFA, $35.15$ for height decoupling only, $35.60$ after adding mask projection, and $35.90$ after adding SFA [2409.07972].

For rotation-invariant point-cloud MAE, the reported gains are for the full dual-stream method rather than the grid stream in isolation. The paper reports consistent improvements across RI-MAE, MaskLRF, and HFBRI-MAE on ModelNet40, ScanObjectNN, and OmniObject3D under aligned, $z$-rotated, and full $SO(3)$ random rotations. It also explicitly notes that it does not provide ablations isolating grid-only versus random masking [2509.14975].

HAC reports over $75\times$ size reduction versus vanilla 3DGS and over $11\times$ versus Scaffold-GS on average, while often maintaining similar or better fidelity. On Mip-NeRF360, for example, size drops from $744.71$ MB in 3DGS to $15.26$ MB in HAC lowrate while PSNR is approximately $27.53$; on Tanks and Temples, size drops from $430.95$ MB to $8.10$ MB [2403.14530].

## 5. Relation to adjacent masking paradigms

Not every spatial mask in 3D learning is a grid mask in the strict voxel sense. "Masking Matters" introduces 3D-SLIM, which replaces the causal decoder mask with a Geometry-adaptive Mask and an Instruction-aware Mask. The object-object mask is defined by density-adaptive nearest-neighbor connectivity from object centers, and the object-to-instruction block is explicitly opened. The paper is explicit that 3D-SLIM itself is purely geometry- and density-based, does not build a voxel or grid, and offers a voxelized neighborhood mask only as a contextual alternative rather than as the implemented method [2512.02487]. This makes 3D-SLIM relevant as a neighboring concept in 3D spatial masking, but not as an instance of grid masking in the narrow sense.

A related distinction appears in GeoMask3D. That method performs patch-level masking on FPS+KNN partitions, not on a regular grid, and uses a teacher-student mechanism to rank patches by learned geometric complexity rather than by fixed spatial bins. However, the paper explicitly describes how its teacher-student geometric complexity ranking can be adapted to a grid-based regime by replacing FPS+KNN with voxelization or a regular 3D spatial grid and computing GC per grid cell [2405.12419]. GeoMask3D therefore supplies a mask-selection principle that is orthogonal to grid topology: the grid may determine where masking can occur, while GC determines which cells should be masked.

A different boundary case is SVR-GS. Its implemented mask is per-pixel and per-ray, built from the effective contribution $w_k(r)=\alpha_k(r)T_k(r)$ of each Gaussian. The rendered spatial mask is
$$
F(x)=\frac{1}{\log(1+N(x))}\sum_i M_i[1-\alpha_i(x)T_i(x)].
$$
The main implementation is tile-based rasterization rather than a 3D grid. Nevertheless, the paper provides an explicit conversion from per-ray masks to a voxel grid through per-voxel accumulation $m_i$ and a grid-level regularizer
$$
R_{grid} = \frac{1}{|G|}\sum_i m_i^2.
$$
That formulation shows how spatial masking can be lifted from image-space rendering into occupancy-aware 3D pruning [2509.11116].

The comparison with MAE-style masking is also important. Jigsaw3D contrasts its joint patch shuffling and random masking with MAE-style masking, patch drop, and ViT grid masking, which it characterizes as pretext tasks for representation learning. Its novelty lies in using the jigsawed reference to destroy global semantics while preserving local style statistics and to provide style-aware conditioning for supervised multi-view diffusion [2510.10497].

## 6. Hyperparameters, sensitivities, and failure modes

Reported behavior is highly sensitive to mask granularity, thresholding, and the coupling between masking and representation.

In Jigsaw3D, the recommended settings are patch size approximately $64$ in training, approximately $128$ in inference, and mask ratio $p \le 0.25$ in training, with shuffle-only inference for full-reference stylization. The paper identifies three primary failure modes: over-masking, which yields insufficient style signal and geometric artifacts after baking; under-masking or overly large patches, which causes semantic leakage from the reference layout; and fine patterns such as text or symbols, which the SDXL backbone may reconstruct poorly [2510.10497].

In BVE, sensitivity centers on alignment and discretization. If RANSAC or ICP fails, the preservation mask can bleed into edited areas or under-select preserved ones. The paper notes that $16^3$ may be too coarse for fine parts and $64^3$ may be memory intensive; threshold $\tau$ that is too small yields under-selection, while too large causes mask bleeding. It recommends multi-scale masking, soft masks for robustness when alignment is imperfect, and setting $\tau$ proportional to voxel size [2604.13688].

In DHD, errors arise from ambiguous height predictions near class boundaries, calibration errors in intrinsics or extrinsics, and temporal blur for moving objects. Because MGHS filters features before projection, inaccurate height estimates can starve valid voxels or exclude valid deposits. The paper also notes diminishing returns when the number of height partitions grows beyond three, since per-band occupancy becomes dilute and computational overhead increases [2409.07972].

In the point-cloud grid-masking setting, the mask is not strictly rotation-invariant unless canonicalization is used. The paper emphasizes that without canonicalization, coordinate ranks change under rotation, and with PCA canonicalization, symmetric shapes can produce unstable principal axes and sign or permutation ambiguity. It recommends $G_x=G_y=G_z=4$, mask ratio approximately $0.75$, and curriculum exponent $\gamma=2$, while also stating that it does not prescribe a specific policy for the per-type masking probabilities $\{p_t\}$ [2509.14975].

In HAC, aggressive masking risks removing fine details in sparse regions. The paper identifies sensitivity to grid resolution and reports that HAC is approximately $0.9\times$ longer to train than Scaffold-GS. Its default masking strength is $\lambda_m=5e^{-4}$, and stronger sparsity requires careful tuning against fidelity [2403.14530].

A recurring misconception is that spatial grid masking is synonymous with random dropout on a voxel lattice. The surveyed works show a broader spectrum: masks may act as preservation weights in a loss, as source-side gates for 2D-to-3D lifting, as structured pretraining patterns over rank-space patch layouts, as semantic-suppression devices for style conditioning, or as binary gates coupled to entropy models. What unifies these cases is not a single implementation, but the use of explicit spatial structure to decide where information should be preserved, suppressed, or emphasized.

Source: https://www.emergentmind.com/topics/3d-spatial-grid-masking