Multi-Scale HexPlanes in 3D/4D Modeling

• Multi-Scale HexPlanes is a hierarchical extension that explicitly decomposes high-dimensional 3D or 4D spaces into six complementary 2D views for efficient feature extraction.
• In 3D semantic scene understanding, it leverages multi-resolution plane projections and cross-attention fusion with standard 2D backbones to balance global context with fine detail.
• For dynamic 4D modeling, its factorization of radiance fields into six learned planes dramatically reduces computation while maintaining high-quality rendering and rapid training.

Searching arXiv for the cited HexPlane papers and closely related work to ground the article. Multi-Scale HexPlanes denotes a hierarchical extension of HexPlane representations in which the same six-plane decomposition is made explicit across multiple spatial, and in dynamic settings temporal, resolutions. The term is not defined explicitly in the two principal HexPlane papers, but the underlying ingredients are present in both: a six-view 3D point-cloud representation for semantic scene understanding in "HexPlane Representation for 3D Semantic Scene Understanding" (Chen et al., 7 Mar 2025), and a six-plane 4D spacetime factorization for dynamic radiance fields in "HexPlane: A Fast Representation for Dynamic Scenes" (Cao et al., 2023). In both cases, the central idea is to replace expensive direct processing of high-dimensional 3D or 4D structure with a small set of 2D planes that can be queried efficiently and fused back into task-specific features.

1. Two technical meanings of HexPlane

A common source of confusion is that the term HexPlane refers to two related but technically distinct constructions.

In 3D semantic scene understanding, HexPlane is a multi-view representation of a point cloud. An input point cloud

$$\mathbf{P} \in \mathbb{R}^{N \times 3}, \quad \mathbf{P}=\{p_n=(p_n^x,p_n^y,p_n^z)\}_{n=1}^N$$

is projected into six 2D planes. The paper describes these as the $XY$, $\pm XZ$, $\pm YZ$, and cylindrical range view, with

$$V \in \mathbb{R}^{M \times H \times W \times D}, \quad M=6.$$

Equivalently, the representation can be written as

$$\mathcal{V}=\{V_i\}_{i=1}^{6}, \quad V_i \in \mathbb{R}^{H_i \times W_i \times C}.$$

The purpose of the six views is to “maximally retain the original spatial information” while allowing the use of highly optimized 2D operators.

In dynamic scene modeling, HexPlane is an explicit 4D feature representation over space and time. A 4D feature volume

$\mathbf{D} \in \mathbb{R}^{X Y Z T F}$

is factorized into six learned 2D planes: three spatial planes $XY$, $XZ$, $YZ$, and three spatio-temporal planes $XY$0, $XY$1, $XY$2, plus a learned matrix $XY$3. The core sampled form is

$XY$4

Here, six bilinear interpolations, three elementwise products, one concatenation, and one vector-matrix product replace direct access to a dense 4D grid.

Multi-Scale HexPlanes therefore has two legitimate readings. In the 3D setting, it means explicit multi-resolution plane projections and multi-scale point-plane fusion. In the 4D setting, it means a pyramid of HexPlane levels with different spatial and temporal resolutions. Both readings are grounded in the same structural motif: six complementary planes, efficient 2D sampling, and fusion back into a higher-dimensional representation (Chen et al., 7 Mar 2025, Cao et al., 2023).

2. HexPlane representation for 3D semantic scene understanding

The 3D formulation is organized around the View Projection Module (VPM), a 2D encoder, and the HexPlane Association Module (HAM). VPM projects the point cloud into six planes. The orthographic planes are obtained by dropping one coordinate and discretizing the remaining two into a grid. The cylindrical view uses a LiDAR-style projection in which the horizontal coordinate is derived from azimuth $XY$5, the vertical coordinate from elevation $XY$6, and $XY$7 is the radial depth from the sensor.

The choice of six planes is motivated by projection loss and occlusion. The $XY$8 plane provides a top view; the split $XY$9 and $\pm XZ$0 planes separate front/back and left/right structure; the cylindrical range view captures LiDAR geometry and nonuniform point density. The paper’s interpretation is that these views are complementary: surfaces ambiguous or occluded in one projection can remain accessible in another. This is why the representation is presented as a means to “maximally retain the original spatial information.”

After projection, each plane is processed by a 2D backbone. The reported implementation uses resolution $\pm XZ$1 for the $\pm XZ$2 plane and $\pm XZ$3 for the other five planes. The backbone is SegFormer, and the paper states that it fuses the multi-scale features output by SegFormer before later fusion stages. In parallel, a lightweight 12-layer 3D CNN produces point-wise 3D features $\pm XZ$4.

HAM then maps the 2D plane features back into point space through cross-attention. Queries come from $\pm XZ$5, while keys and values come from the six encoded view features $\pm XZ$6. The module also incorporates a positional embedding $\pm XZ$7 computed from the difference between the spatial locations of the 3D query points and those of points projected on views. When a point is exactly projected onto a view, the positional offset is zero; when projection is occluded, the offset is larger. This makes the attention mechanism geometrically informed rather than purely appearance-based.

The resulting fused point features are passed to task heads for segmentation or detection. The architectural significance of HexPlane in this setting is not only the view decomposition itself, but the transfer pathway it opens: standard 2D backbones, pre-trained 2D weights, and 2D training recipes can be imported into 3D scene understanding with only a relatively lightweight point-space fusion mechanism (Chen et al., 7 Mar 2025).

3. Multi-scale structure in HexNet3D

The 3D paper does not define “Multi-Scale HexPlanes” as a separate model, but it already contains an implicit multi-scale hierarchy. The crucial statement is that, for each 2D view $\pm XZ$8, SegFormer produces hierarchical features and that the model “fuse[s] the multi-scale features output by SegFormer.” In other words, each HexPlane is already encoded as a feature pyramid before the planes are associated back to points.

This built-in hierarchy matters because it partially resolves the usual trade-off between global context and fine detail. Coarser levels in a hierarchical transformer capture scene layout and long-range context; finer levels preserve local geometric detail. In HexNet3D, however, this hierarchy remains internal to the 2D backbone. HAM receives a per-view feature representation after multi-scale fusion, rather than reasoning explicitly over several scales of the same plane.

The proposed notion of Multi-Scale HexPlanes in the 3D setting makes this hidden hierarchy explicit. One formulation is to represent each plane $\pm XZ$9 at several resolutions,

$\pm YZ$0

and treat each $\pm YZ$1 pair as a distinct feature map. A generalized HAM can then attend across both planes and scales, allowing each point to choose not only which view is informative but also whether coarse or fine context is more relevant. The same synthesis also outlines hierarchical fusion schemes in which coarse HexPlanes provide global context, intermediate HexPlanes refine structure, and fine HexPlanes add detail.

Another proposed design is a HexPlane feature pyramid, or “Hex-FPN,” in which multi-scale outputs are fused across planes at each scale and then queried by points. The rationale is straightforward: a single resolution per view may force an undesirable compromise between dense high-resolution grids and lower-resolution views that lose thin structures and boundaries. An explicit multi-scale formulation separates these roles. Coarse planes can capture large indoor layouts or large outdoor context cheaply, while fine planes focus on local detail.

These extensions remain proposals rather than reported implementations. Their status is therefore inferential. What is established is that HexNet3D already uses multi-scale 2D features internally, and that the paper’s own description naturally supports a more explicit multi-resolution HexPlane hierarchy (Chen et al., 7 Mar 2025).

4. HexPlane as a 4D spacetime factorization

The dynamic-scene formulation begins from a different problem: efficient modeling and rendering of a time-varying radiance field. Instead of projecting an observed point cloud into six external views, the method stores six learned feature planes that parameterize a 4D volume. A query point $\pm YZ$2 is mapped to the six planes, bilinearly interpolated, fused pairwise by elementwise multiplication, concatenated across the three spatial-temporal pairs, and projected through a learned linear basis.

This representation is derived from a low-rank factorization of a 4D tensor. At the tensor level,

$\pm YZ$3

The sampled form yields memory complexity

$\pm YZ$4

rather than $\pm YZ$5 for a dense 4D grid.

The dynamic HexPlane model uses separate appearance and opacity HexPlanes. Appearance features are combined with view direction and decoded by a very small MLP to RGB; a HexPlane-SH variant replaces the MLP with spherical harmonic decoding. Rendering uses standard NeRF-style volumetric compositing along rays. The paper reports that this produces image quality matching prior work while greatly reducing training time, because most per-sample computation becomes plane lookup and low-dimensional linear algebra rather than repeated evaluation of a large MLP.

Several ablations are directly relevant to any multi-scale interpretation. The default fusion is “Multiply–Concat–Linear,” and the reported D-NeRF ablation gives Multiply–Concat the best performance among tested combinations, with PSNR $\pm YZ$6, SSIM $\pm YZ$7, and LPIPS $\pm YZ$8. The plane arrangement itself is also critical: spatial planes only and spatio-temporal planes only perform poorly, whereas the full symmetric six-plane design performs best. This indicates that any multi-scale extension should preserve the complementary spatial and spatio-temporal pairing rather than replacing it with an arbitrary hierarchy (Cao et al., 2023).

5. Explicit multi-scale designs for dynamic HexPlanes

The dynamic HexPlane paper already contains an implicit multi-scale training strategy through coarse-to-fine grid growth. For Plenoptic Video, the spatial resolution is increased from $\pm YZ$9 to $$V \in \mathbb{R}^{M \times H \times W \times D}, \quad M=6.$$0, $$V \in \mathbb{R}^{M \times H \times W \times D}, \quad M=6.$$1, and $$V \in \mathbb{R}^{M \times H \times W \times D}, \quad M=6.$$2 over iterations at $$V \in \mathbb{R}^{M \times H \times W \times D}, \quad M=6.$$3, $$V \in \mathbb{R}^{M \times H \times W \times D}, \quad M=6.$$4, and $$V \in \mathbb{R}^{M \times H \times W \times D}, \quad M=6.$$5. For D-NeRF, the schedule is $$V \in \mathbb{R}^{M \times H \times W \times D}, \quad M=6.$$6 at $$V \in \mathbb{R}^{M \times H \times W \times D}, \quad M=6.$$7, $$V \in \mathbb{R}^{M \times H \times W \times D}, \quad M=6.$$8, and $$V \in \mathbb{R}^{M \times H \times W \times D}, \quad M=6.$$9 iterations. This is not merely an optimization trick; it acts as a form of multi-scale regularization by first fitting low-frequency structure and then progressively introducing finer detail.

An explicit Multi-Scale HexPlane in the dynamic setting generalizes this schedule into a persistent pyramid. One may define levels $$\mathcal{V}=\{V_i\}_{i=1}^{6}, \quad V_i \in \mathbb{R}^{H_i \times W_i \times C}.$$0, from coarse to fine, each with spatial resolution $$\mathcal{V}=\{V_i\}_{i=1}^{6}, \quad V_i \in \mathbb{R}^{H_i \times W_i \times C}.$$1, temporal resolution $$\mathcal{V}=\{V_i\}_{i=1}^{6}, \quad V_i \in \mathbb{R}^{H_i \times W_i \times C}.$$2, and basis dimensions $$\mathcal{V}=\{V_i\}_{i=1}^{6}, \quad V_i \in \mathbb{R}^{H_i \times W_i \times C}.$$3. Each level has its own six-plane HexPlane structure, and the level-specific features can be combined additively, by concatenation followed by linear mixing, or by gated fusion. The synthesis proposes that coarse temporal grids capture slow or global motion, whereas fine temporal grids capture rapid or localized changes.

The same logic extends naturally to ray sampling. Coarse HexPlanes can be queried first to estimate density and identify informative ray intervals. Fine HexPlanes are then evaluated only where coarse predictions indicate occupancy, variation, or dynamic content. The paper’s emptiness voxel, which marks voxels as empty when their maximum opacity over time is below a threshold, fits directly into this view: a multi-scale emptiness structure would provide coarse pruning followed by fine refinement.

Scale-aware regularization is also suggested by the original loss design. The base model uses photometric supervision together with total variation loss on planes and a depth smoothness loss. A direct multi-scale extension would place stronger total variation on coarse planes and weaker total variation on fine planes, preserving global coherence while allowing high-frequency residuals. The paper’s broader observations about robustness to different coordinate systems, including NDC and spherical coordinates, further suggest that multi-scale HexPlanes need not be restricted to Cartesian bounded scenes (Cao et al., 2023).

6. Empirical performance, advantages, and limitations

In the 3D semantic setting, HexNet3D reports $$\mathcal{V}=\{V_i\}_{i=1}^{6}, \quad V_i \in \mathbb{R}^{H_i \times W_i \times C}.$$4 mIoU on ScanNet v2 validation for 3D segmentation, surpassing Point Transformer V2 at $$\mathcal{V}=\{V_i\}_{i=1}^{6}, \quad V_i \in \mathbb{R}^{H_i \times W_i \times C}.$$5 by $$\mathcal{V}=\{V_i\}_{i=1}^{6}, \quad V_i \in \mathbb{R}^{H_i \times W_i \times C}.$$6 mIoU. On ScanNet v2 detection, the HexPlane-based extension of FCAF3D reports $$\mathcal{V}=\{V_i\}_{i=1}^{6}, \quad V_i \in \mathbb{R}^{H_i \times W_i \times C}.$$7 mAP$$\mathcal{V}=\{V_i\}_{i=1}^{6}, \quad V_i \in \mathbb{R}^{H_i \times W_i \times C}.$$8 and $$\mathcal{V}=\{V_i\}_{i=1}^{6}, \quad V_i \in \mathbb{R}^{H_i \times W_i \times C}.$$9 mAP$\mathbf{D} \in \mathbb{R}^{X Y Z T F}$0, compared with $\mathbf{D} \in \mathbb{R}^{X Y Z T F}$1 and $\mathbf{D} \in \mathbb{R}^{X Y Z T F}$2 for FCAF3D. On SemanticKITTI segmentation, integration into CENet yields $\mathbf{D} \in \mathbb{R}^{X Y Z T F}$3 mIoU, compared with $\mathbf{D} \in \mathbb{R}^{X Y Z T F}$4 for CENet. These results are presented as evidence that the representation can be “seamlessly integrated into existing voxel-based, point-based, and range-based approaches” and can bring gains across several families of 3D pipelines (Chen et al., 7 Mar 2025).

In the dynamic-scene setting, the reported computational savings are substantial. On Plenoptic Video, training is about $\mathbf{D} \in \mathbb{R}^{X Y Z T F}$5 GPU hours on a single V100, compared with $\mathbf{D} \in \mathbb{R}^{X Y Z T F}$6 GPU hours for DyNeRF, a $\mathbf{D} \in \mathbb{R}^{X Y Z T F}$7 speedup; with fewer iterations, the paper reports about $\mathbf{D} \in \mathbb{R}^{X Y Z T F}$8 hours, or $\mathbf{D} \in \mathbb{R}^{X Y Z T F}$9 speedup, with comparable quality. On D-NeRF, training is about $XY$0 minutes versus about $XY$1 hours for D-NeRF MLP baselines. The reported model size for Plenoptic Video scenes is about $XY$2MB. These results support the broader claim that storing capacity in planes while keeping decoding small is an effective route to fast explicit dynamic representations (Cao et al., 2023).

Across both literatures, the main advantages are consistent. HexPlanes shift computation toward dense 2D operations that are highly optimized; they exploit complementary views or complementary plane pairs to mitigate information loss; and they separate representational capacity from the cost of heavy point-wise or sample-wise neural decoding. In the 3D semantic case, they also enable reuse of off-the-shelf 2D models, weights, and training recipes.

The principal limitations are equally consistent. A single resolution per plane can impose a trade-off between global context and fine detail. Explicit multi-scale extensions increase the number of grids and therefore the memory footprint and lookup cost. In the 3D semantic case, computation and resource consumption must be managed carefully if several plane resolutions are exposed simultaneously. In the dynamic case, more scales introduce additional plane storage and regularization choices. This suggests that Multi-Scale HexPlanes are best understood not as a single fixed architecture, but as a design family: one that makes the latent hierarchical structure of HexPlane representations explicit, so that coarse and fine information can be organized, queried, and fused in a controlled way.