3D Spatial Grid Masking

• 3D Spatial Grid Masking is a technique that employs structured spatial grids—such as voxel, patch, and rank-space grids—to control which regions are preserved or suppressed.
• The method is applied in multi-view 3D style transfer, text-driven editing, height-aware projections, and point cloud pretraining to enforce spatial continuity and invariance.
• Empirical results indicate that carefully tuning grid granularity and masking parameters improves fidelity, reduces artifacts, and maintains semantic integrity in 3D processing tasks.

3D Spatial Grid Masking denotes a family of masking mechanisms in which visibility, preservation, sparsity, or conditioning is controlled by explicit spatial structure rather than by purely random token omission or purely sequential attention. Recent uses span multi-view 3D style transfer, text-driven 3D editing, camera-only 3D occupancy prediction, rotation-invariant point cloud masked autoencoding, and 3D Gaussian Splatting compression. Depending on the representation, the mask may live on a non-overlapping image patch grid used to condition 3D stylization, on voxel lattices such as $16^3$ and $64^3$, on height-band partitions tied to a semantic occupancy grid, on rank-space grids over point-cloud patch centers, or on a binary hash grid coupled to learned offset masks (Ye et al., 12 Oct 2025, Xu et al., 15 Apr 2026, Wu et al., 2024, Yin et al., 18 Sep 2025, Chen et al., 2024).

1. Formal definitions and representational domains

The term is not tied to a single data structure. In "Jigsaw3D" (Ye et al., 12 Oct 2025), the mask is defined on a spatial patch grid over a reference image, but the masked reference is used as a 3D style-conditioning signal for a multi-view diffusion pipeline. Let a reference image be $I \in \mathbb{R}^{C \times H \times W}$, partitioned into non-overlapping $S \times S$ patches $P_{i,j}$. After a random permutation $\sigma$ and Bernoulli mask $M_{i,j}$ with mask ratio $p$, the jigsawed reference is

$$I_{jigsaw} = \bigcup_{i,j} \left[ M_{i,j} \odot P_{\sigma(i,j)} + (1-M_{i,j}) \odot \mu \right].$$

Training uses $S=64$ and $64^3$0; inference uses $64^3$1, and by default only shuffling is applied.

In "Beyond Voxel 3D Editing" (Xu et al., 15 Apr 2026), the preservation mask is explicitly 3D and binary. The model follows TRELLIS’s structured latent design with a coarse SS grid of $64^3$2 and a finer SLAT grid of $64^3$3. For a canonical grid point $64^3$4, after rigid alignment $64^3$5 between the original and edited point clouds, the mask is

$64^3$6

where $64^3$7 is the nearest neighbor in the edited point cloud. The resulting tensor $64^3$8 is indexed either densely or sparsely.

In "Deep Height Decoupling" (Wu et al., 2024), masking is a height-aware gating mechanism for 2D-to-3D projection. HeightNet predicts a discretized height map with $64^3$9 bins, and Mask Guided Height Sampling defines interval-specific masks

$I \in \mathbb{R}^{C \times H \times W}$0

The default partition is $I \in \mathbb{R}^{C \times H \times W}$1, $I \in \mathbb{R}^{C \times H \times W}$2, $I \in \mathbb{R}^{C \times H \times W}$3, and masked image features are projected only into the corresponding 3D height subspaces.

In the dual-stream rotation-invariant point cloud MAE (Yin et al., 18 Sep 2025), 3D Spatial Grid Masking is not voxelization but rank-space grid construction over patch centers. After FPS selects $I \in \mathbb{R}^{C \times H \times W}$4 centroids and KNN forms patch tokens, each centroid receives axis-wise ranks $I \in \mathbb{R}^{C \times H \times W}$5, grid bits

$I \in \mathbb{R}^{C \times H \times W}$6

and a bit-packed type

$I \in \mathbb{R}^{C \times H \times W}$7

The default is $I \in \mathbb{R}^{C \times H \times W}$8.

In HAC (Chen et al., 2024), the grid is a learned binary hash grid queried at anchor locations, while masking acts on offsets. If $I \in \mathbb{R}^{C \times H \times W}$9 is the per-anchor offset vector and $S \times S$0 is a binary gate, masked offsets become $S \times S$1; if all gates are zero, the anchor is invalid and removed.

Work	Mask support	Primary function
Jigsaw3D	Non-overlapping $S \times S$2 patch grid	Suppress semantics, preserve local style statistics
BVE	Canonical voxel grids $S \times S$3 and $S \times S$4	Preserve unedited regions during 3D editing
DHD	Height-band masks tied to occupancy grid projection	Reduce cross-height feature contamination
Dual-stream RI MAE	Rank-space grid over patch centers	Impose structured masking for geometric continuity
HAC	Binary hash grid plus offset mask	Context modeling and removal of invalid Gaussians

2. Functional roles of masking

A central use of spatial grid masking is semantic suppression without destroying local statistics. Jigsaw3D states that patch-level shuffling breaks global shape and layout cues, which are essential for object semantics, while preserving local statistics such as color means and variances, stroke textures, and frequency content. The paper quantitatively shows that as the grid gets finer, CNN-based content classification accuracy drops sharply while Gram matrix similarity to the source increases, and its appendix proof shows that shuffling preserves first- and second-order statistics per channel (Ye et al., 12 Oct 2025). In that setting, masking is not primarily for occlusion simulation; it is a disentanglement device.

In BVE, the role is local invariance. The edited-original paired training objective contains a standard edit term and a mask-weighted preservation term,

$S \times S$5

Masked regions are therefore driven toward the original asset’s flow target, while unmasked regions follow the edited target. The paper explicitly states that the second term increases the penalty in masked, preserved regions to match the original asset’s flow target, enforcing local invariance (Xu et al., 15 Apr 2026).

In DHD, masking is a height prior for projection. Forward projection methods mix features across height ranges, producing “confusing features” that harm voxel-wise semantic classification. MGHS filters per-pixel features before lifting, so only pixels whose predicted height falls into the appropriate interval contribute to the corresponding $S \times S$6 slices. Dataset analysis reported in the paper shows that lower height bands $S \times S$7 concentrate drivable surface, sidewalk, and terrain, while upper bands $S \times S$8 favor vegetation, manmade, and trailer; a weighted entropy metric indicates that $S \times S$9, $P_{i,j}$0, $P_{i,j}$1 yields the lowest entropy (Wu et al., 2024).

In the point-cloud MAE setting, the function is structured pretext generation rather than preservation or projection gating. The paper argues that random masking treats patches independently and ignores long-range spatial relationships, whereas 3D Spatial Grid Masking creates structured patterns through coordinate sorting so the encoder sees coherent spatial context during pretraining. The intended benefit is not exact invariance of the mask itself, but preservation of adjacency and regularity in the visible/hidden layout (Yin et al., 18 Sep 2025).

In HAC, masking is rate-distortion control. The paper observes an impulse at zero in empirical histograms of offsets and uses a learned adaptive offset masking module to remove redundant or non-contributing Gaussians. Here the mask reduces the number of serialized symbols while the binary hash grid provides spatial context for entropy coding (Chen et al., 2024).

3. Integration into learning, generation, and compression pipelines

The masked object or grid is typically not an isolated preprocessing artifact; it is integrated directly into the core inference or training objective. In Jigsaw3D, the jigsawed reference is encoded by a pretrained VAE encoder, passed through a frozen or reference pretrained diffusion U-Net at $P_{i,j}$2, and reference features are extracted from self-attention hidden states. These features then enter the Style U-Net through reference-to-view cross-attention,

$P_{i,j}$3

The Style U-Net combines standard self-attention per view, efficient row-wise multi-view attention, and reference attention after each residual layer, followed by text-conditioned cross-attention. Geometry enters through rendered position and normal maps from $P_{i,j}$4 predefined viewpoints, passed through a T2I-Adapter-style condition encoder. The stylized renders are then baked back to the mesh by visibility-aware reprojection, seam-aware confidence-weighted blending, tangent-space normal baking, and UV-space inpainting (Ye et al., 12 Oct 2025).

In BVE, masking is present only in the backward signal. The paper states that the forward pass does not gate activations: the network predicts $P_{i,j}$5 as usual with multimodal conditioning, and the preservation term is multiplied by $P_{i,j}$6 only in the loss. Stage-specific masks $P_{i,j}$7 and $P_{i,j}$8 are used for SS and SLAT respectively, and there is no mask computation at inference; inference edits are driven by conditioning learned during training (Xu et al., 15 Apr 2026).

In DHD, MGHS sits inside the camera-to-occupancy view transform. HeightNet predicts per-pixel bins, interval masks are applied in image space to context features, masked features are projected into disjoint 3D subspaces, and the resulting height-refined volume $P_{i,j}$9 is fused with an unmasked depth-based stream $\sigma$0 by Synergistic Feature Aggregation. SFA first applies channel attention and then spatial attention:

$\sigma$1

The total loss combines occupancy, height, optional depth, and scaling losses with default weights $\sigma$2, $\sigma$3, $\sigma$4, and $\sigma$5 (Wu et al., 2024).

In the dual-stream point-cloud MAE, grid masking is fused with Progressive Semantic Masking by curriculum. The paper defines

$\sigma$6

with $\sigma$7 and a target mask ratio $\sigma$8. Early training is therefore dominated by grid masking, and later training shifts toward semantic masking (Yin et al., 18 Sep 2025).

In HAC, masking is introduced after a staged warmup. The system first trains Scaffold-GS anchors, then introduces quantization noise, and only after iteration 10000 enables the binary hash grid, context-conditioned probability modeling, and adaptive offset masking. The total loss combines rendering fidelity, attribute entropy, hash-grid entropy, and masking:

$\sigma$9

Thus, the grid and the mask are optimized jointly with compression targets rather than appended after training (Chen et al., 2024).

4. Reported empirical behavior

Across domains, spatial grid masking is reported to improve either fidelity, invariance, semantic localization, or compression efficiency.

Jigsaw3D evaluates on 20 Objaverse meshes unseen in training and 30 WikiArt styles plus 40 collected styles. Style fidelity is measured by Gram matrix similarity and AdaIN distance using VGG19 features, and style-content disentanglement is measured by CLIP score, with lower being better. The paper reports that Jigsaw3D achieves the best Gram and AdaIN on both datasets, competitive CLIP, and latency of approximately 40 seconds, whereas SDS optimization in StyleTex is approximately 15 minutes. Ablations report that moderate masking in the range $M_{i,j}$0 to $M_{i,j}$1 improves robustness and style capture, while too high a mask ratio such as $M_{i,j}$2 harms geometric consistency and increases artifacts during baking (Ye et al., 12 Oct 2025).

BVE reports a direct ablation of masking. For Ours(full) versus w/o MASK, the paper gives CD $M_{i,j}$3 versus $M_{i,j}$4, SSIM $M_{i,j}$5 versus $M_{i,j}$6, LPIPS $M_{i,j}$7 versus $M_{i,j}$8, FID $M_{i,j}$9 versus $p$0, FVD $p$1 versus $p$2, and DINO-I $p$3 versus $p$4. It further states that Ours(full) achieves the best scores across CD, SSIM, LPIPS, FID, FVD, DINO-I, and CLIP-T, together with user study preferences of $p$5 for Text Alignment and $p$6 for Overall 3D Quality (Xu et al., 15 Apr 2026).

DHD reports state-of-the-art occupancy performance on Occ3D-nuScenes even with minimal input frames. DHD-S reaches mIoU $p$7, outperforming FlashOcc at $p$8 by $p$9; DHD-M reaches $$I_{jigsaw} = \bigcup_{i,j} \left[ M_{i,j} \odot P_{\sigma(i,j)} + (1-M_{i,j}) \odot \mu \right].$$0, slightly above COTR(BEVDetOcc) at $$I_{jigsaw} = \bigcup_{i,j} \left[ M_{i,j} \odot P_{\sigma(i,j)} + (1-M_{i,j}) \odot \mu \right].$$1 and OSP at $$I_{jigsaw} = \bigcup_{i,j} \left[ M_{i,j} \odot P_{\sigma(i,j)} + (1-M_{i,j}) \odot \mu \right].$$2; DHD-L reaches $$I_{jigsaw} = \bigcup_{i,j} \left[ M_{i,j} \odot P_{\sigma(i,j)} + (1-M_{i,j}) \odot \mu \right].$$3, above FlashOcc at $$I_{jigsaw} = \bigcup_{i,j} \left[ M_{i,j} \odot P_{\sigma(i,j)} + (1-M_{i,j}) \odot \mu \right].$$4 and GEOcc at $$I_{jigsaw} = \bigcup_{i,j} \left[ M_{i,j} \odot P_{\sigma(i,j)} + (1-M_{i,j}) \odot \mu \right].$$5. The ablation on DHD-S reports $$I_{jigsaw} = \bigcup_{i,j} \left[ M_{i,j} \odot P_{\sigma(i,j)} + (1-M_{i,j}) \odot \mu \right].$$6 for the baseline without MGHS decoupling or SFA, $$I_{jigsaw} = \bigcup_{i,j} \left[ M_{i,j} \odot P_{\sigma(i,j)} + (1-M_{i,j}) \odot \mu \right].$$7 for height decoupling only, $$I_{jigsaw} = \bigcup_{i,j} \left[ M_{i,j} \odot P_{\sigma(i,j)} + (1-M_{i,j}) \odot \mu \right].$$8 after adding mask projection, and $$I_{jigsaw} = \bigcup_{i,j} \left[ M_{i,j} \odot P_{\sigma(i,j)} + (1-M_{i,j}) \odot \mu \right].$$9 after adding SFA (Wu et al., 2024).

For rotation-invariant point-cloud MAE, the reported gains are for the full dual-stream method rather than the grid stream in isolation. The paper reports consistent improvements across RI-MAE, MaskLRF, and HFBRI-MAE on ModelNet40, ScanObjectNN, and OmniObject3D under aligned, $S=64$0-rotated, and full $S=64$1 random rotations. It also explicitly notes that it does not provide ablations isolating grid-only versus random masking (Yin et al., 18 Sep 2025).

HAC reports over $S=64$2 size reduction versus vanilla 3DGS and over $S=64$3 versus Scaffold-GS on average, while often maintaining similar or better fidelity. On Mip-NeRF360, for example, size drops from $S=64$4 MB in 3DGS to $S=64$5 MB in HAC lowrate while PSNR is approximately $S=64$6; on Tanks and Temples, size drops from $S=64$7 MB to $S=64$8 MB (Chen et al., 2024).

5. Relation to adjacent masking paradigms

Not every spatial mask in 3D learning is a grid mask in the strict voxel sense. "Masking Matters" introduces 3D-SLIM, which replaces the causal decoder mask with a Geometry-adaptive Mask and an Instruction-aware Mask. The object-object mask is defined by density-adaptive nearest-neighbor connectivity from object centers, and the object-to-instruction block is explicitly opened. The paper is explicit that 3D-SLIM itself is purely geometry- and density-based, does not build a voxel or grid, and offers a voxelized neighborhood mask only as a contextual alternative rather than as the implemented method (Jeon et al., 2 Dec 2025). This makes 3D-SLIM relevant as a neighboring concept in 3D spatial masking, but not as an instance of grid masking in the narrow sense.

A related distinction appears in GeoMask3D. That method performs patch-level masking on FPS+KNN partitions, not on a regular grid, and uses a teacher-student mechanism to rank patches by learned geometric complexity rather than by fixed spatial bins. However, the paper explicitly describes how its teacher-student geometric complexity ranking can be adapted to a grid-based regime by replacing FPS+KNN with voxelization or a regular 3D spatial grid and computing GC per grid cell (Bahri et al., 2024). GeoMask3D therefore supplies a mask-selection principle that is orthogonal to grid topology: the grid may determine where masking can occur, while GC determines which cells should be masked.

A different boundary case is SVR-GS. Its implemented mask is per-pixel and per-ray, built from the effective contribution $S=64$9 of each Gaussian. The rendered spatial mask is

$64^3$00

The main implementation is tile-based rasterization rather than a 3D grid. Nevertheless, the paper provides an explicit conversion from per-ray masks to a voxel grid through per-voxel accumulation $64^3$01 and a grid-level regularizer

$64^3$02

That formulation shows how spatial masking can be lifted from image-space rendering into occupancy-aware 3D pruning (Taghipour et al., 14 Sep 2025).

The comparison with MAE-style masking is also important. Jigsaw3D contrasts its joint patch shuffling and random masking with MAE-style masking, patch drop, and ViT grid masking, which it characterizes as pretext tasks for representation learning. Its novelty lies in using the jigsawed reference to destroy global semantics while preserving local style statistics and to provide style-aware conditioning for supervised multi-view diffusion (Ye et al., 12 Oct 2025).

6. Hyperparameters, sensitivities, and failure modes

Reported behavior is highly sensitive to mask granularity, thresholding, and the coupling between masking and representation.

In Jigsaw3D, the recommended settings are patch size approximately $64^3$03 in training, approximately $64^3$04 in inference, and mask ratio $64^3$05 in training, with shuffle-only inference for full-reference stylization. The paper identifies three primary failure modes: over-masking, which yields insufficient style signal and geometric artifacts after baking; under-masking or overly large patches, which causes semantic leakage from the reference layout; and fine patterns such as text or symbols, which the SDXL backbone may reconstruct poorly (Ye et al., 12 Oct 2025).

In BVE, sensitivity centers on alignment and discretization. If RANSAC or ICP fails, the preservation mask can bleed into edited areas or under-select preserved ones. The paper notes that $64^3$06 may be too coarse for fine parts and $64^3$07 may be memory intensive; threshold $64^3$08 that is too small yields under-selection, while too large causes mask bleeding. It recommends multi-scale masking, soft masks for robustness when alignment is imperfect, and setting $64^3$09 proportional to voxel size (Xu et al., 15 Apr 2026).

In DHD, errors arise from ambiguous height predictions near class boundaries, calibration errors in intrinsics or extrinsics, and temporal blur for moving objects. Because MGHS filters features before projection, inaccurate height estimates can starve valid voxels or exclude valid deposits. The paper also notes diminishing returns when the number of height partitions grows beyond three, since per-band occupancy becomes dilute and computational overhead increases (Wu et al., 2024).

In the point-cloud grid-masking setting, the mask is not strictly rotation-invariant unless canonicalization is used. The paper emphasizes that without canonicalization, coordinate ranks change under rotation, and with PCA canonicalization, symmetric shapes can produce unstable principal axes and sign or permutation ambiguity. It recommends $64^3$10, mask ratio approximately $64^3$11, and curriculum exponent $64^3$12, while also stating that it does not prescribe a specific policy for the per-type masking probabilities $64^3$13 (Yin et al., 18 Sep 2025).

In HAC, aggressive masking risks removing fine details in sparse regions. The paper identifies sensitivity to grid resolution and reports that HAC is approximately $64^3$14 longer to train than Scaffold-GS. Its default masking strength is $64^3$15, and stronger sparsity requires careful tuning against fidelity (Chen et al., 2024).

A recurring misconception is that spatial grid masking is synonymous with random dropout on a voxel lattice. The surveyed works show a broader spectrum: masks may act as preservation weights in a loss, as source-side gates for 2D-to-3D lifting, as structured pretraining patterns over rank-space patch layouts, as semantic-suppression devices for style conditioning, or as binary gates coupled to entropy models. What unifies these cases is not a single implementation, but the use of explicit spatial structure to decide where information should be preserved, suppressed, or emphasized.