VoxelHound: Panoramic Multimodal Occupancy

• VoxelHound is a panoramic multimodal semantic occupancy framework that integrates RGB, thermal, polarization, and LiDAR data to generate dense 3D voxel grids for legged robot navigation.
• It employs techniques like Vertical Jitter Compensation and Multimodal Information Prompt Fusion to stabilize image features and effectively fuse multimodal sensor data.
• Empirical results on the PanoMMOcc benchmark demonstrate significant mIoU improvements over camera-only and baseline methods, enhancing robust navigation in complex terrains.

VoxelHound is a panoramic multimodal semantic occupancy prediction framework for quadruped robots introduced together with the PanoMMOcc benchmark (Zhao et al., 13 Mar 2026). It estimates a dense 3D semantic occupancy grid around the robot from panoramic RGB, thermal, polarization, and LiDAR inputs, and is explicitly tailored to the low viewpoint, strong ego-motion, vertical jitter, and terrain variability associated with legged locomotion rather than wheeled autonomous driving. In its published form, the system predicts a $64 \times 64 \times 16$ voxel grid over $x,y \in [-12.8,12.8]$ m and $z \in [-2.4,4.0]$ m at $$0.4\text{ m} \times 0.4\text{ m} \times 0.4\text{ m}$$ resolution, with 12 semantic categories, and achieves state-of-the-art performance on PanoMMOcc, including a reported $+4.16\%$ mIoU gain over the best baseline under comparable $C+L$ conditions (Zhao et al., 13 Mar 2026).

1. Problem formulation and semantic occupancy objective

VoxelHound is formulated as a multimodal mapping function

$$\mathbf{O} = \Phi \left( \mathcal{I}^{pal}, \mathcal{I}^{th}, \mathcal{I}^{pol}, \mathcal{P} \right),$$

where $\mathcal{I}^{pal}$ is the panoramic RGB image from the PAL camera, $\mathcal{I}^{th}$ is the thermal image, $\mathcal{I}^{pol}$ is the polarization image, $x,y \in [-12.8,12.8]$0 is the LiDAR point cloud, and $x,y \in [-12.8,12.8]$1 is the predicted 3D semantic occupancy grid (Zhao et al., 13 Mar 2026). In this setting, “semantic occupancy” denotes a dense voxel field whose cells encode occupancy state together with one of 12 labels: car, motorcycle, bicycle, pedestrian, driveable surface, sidewalk, terrain, vegetation, building, barrier, manmade, and pillar (Zhao et al., 13 Mar 2026).

The framework is motivated by a robotic regime that differs materially from conventional automotive occupancy prediction. The quadruped viewpoint is low, self-occlusion by the body and legs is common, and body pitch and roll induce severe vertical perturbations. The paper emphasizes that detrended vertical acceleration $x,y \in [-12.8,12.8]$2 exhibits strong high-frequency oscillations, which degrade image stability and, by extension, spatial reasoning if untreated (Zhao et al., 13 Mar 2026). The environment distribution is also atypical for driving benchmarks: forest, rural, green-space, campus, residential, and urban scenes all appear, including weakly structured terrain and non-flat traversable surfaces.

A central distinction from standard bird’s-eye-view semantic segmentation is that VoxelHound predicts full 3D vertical structure rather than a ground-plane abstraction. This includes vegetation volume, terrain undulations, pillars, and overhanging structure, all of which matter for legged navigation. The emphasis on full volumetric semantics is therefore not incidental; it is part of the problem definition and one reason camera-only BEV methods transfer poorly to this setting (Zhao et al., 13 Mar 2026).

2. PanoMMOcc dataset, sensing stack, and calibration model

VoxelHound is developed and evaluated on PanoMMOcc, described as the first real-world panoramic multimodal semantic occupancy dataset for quadruped robots (Zhao et al., 13 Mar 2026). The platform is a Unitree Go2 quadruped with a $x,y \in [-12.8,12.8]$3 cm body, 7 kg payload, 15 cm obstacle-climbing capability, and $x,y \in [-12.8,12.8]$4 slope capability. Its sensor suite comprises a panoramic annular-lens camera with $x,y \in [-12.8,12.8]$5 field of view and $x,y \in [-12.8,12.8]$6 resolution, a MID360 3D LiDAR running at 10 Hz, an N-Driver P302B thermal camera at $x,y \in [-12.8,12.8]$7, and a LUCID TRI050S-QC polarization camera up to $x,y \in [-12.8,12.8]$8 resolution (Zhao et al., 13 Mar 2026).

The polarization modality is derived from four polarizer-angle measurements $x,y \in [-12.8,12.8]$9. The Stokes components used are

$z \in [-2.4,4.0]$0

with $z \in [-2.4,4.0]$1 unused, and the downstream channels are the Degree of Linear Polarization and Angle of Linear Polarization,

$z \in [-2.4,4.0]$2

These channels are treated as the polarization input to the perception stack (Zhao et al., 13 Mar 2026).

PanoMMOcc contains 54 sequences at 10 Hz, each 40 s long, totaling 21,600 frames. Of these, 42 sequences are annotated for occupancy, with 30 used for training and 12 for testing; after keyframe sampling with stride 5, the benchmark contains 16.8k labeled frames. Scene types are campus, urban streets, residential, green spaces, rural, and forest, with 31 daytime and 11 nighttime sequences, and the test set includes two sequences per scene type (Zhao et al., 13 Mar 2026). The occupancy grid follows a SurroundOcc/OpenOccupancy-style protocol, aggregating manually labeled LiDAR point clouds into voxels with occupancy and semantics decided from points, visibility, and cross-frame consistency (Zhao et al., 13 Mar 2026).

All sensors are registered into a common LiDAR-centric frame $z \in [-2.4,4.0]$3. Time synchronization uses a single host PC and GigE switch as time base, with clocks synchronized before each session and timestamps aligned in post-processing (Zhao et al., 13 Mar 2026). Thermal and polarization cameras use a pinhole model with radial and tangential distortion, while the PAL camera uses the OCam/Taylor omnidirectional model with

$z \in [-2.4,4.0]$4

LiDAR-camera extrinsics are estimated by minimizing reprojection error of a planar whiteboard observed in both modalities (Zhao et al., 13 Mar 2026). The calibration procedure is not a peripheral implementation detail; it is an enabling condition for the multimodal fusion strategy.

3. Architecture: camera branch, LiDAR branch, VJC, and MIPF

VoxelHound has three principal components: a camera branch, a LiDAR branch, and a fusion branch centered on Vertical Jitter Compensation (VJC) and Multimodal Information Prompt Fusion (MIPF) (Zhao et al., 13 Mar 2026). Each visual modality $z \in [-2.4,4.0]$5 is processed by a ResNet-18 backbone producing multiscale features

$z \in [-2.4,4.0]$6

followed by an FPN neck yielding

$z \in [-2.4,4.0]$7

These features are passed through VJC and then lifted from image space to BEV, producing modality-specific BEV tensors

$z \in [-2.4,4.0]$8

In parallel, the LiDAR branch voxelizes $z \in [-2.4,4.0]$9 within the predefined 3D range, retains up to 10 points per voxel with mean-pooled features, processes them with a sparse 3D CNN encoder like SECOND at overall stride 8, and splats the output to BEV,

$$0.4\text{ m} \times 0.4\text{ m} \times 0.4\text{ m}$$0

This establishes the geometry-dominant BEV representation on which multimodal fusion operates (Zhao et al., 13 Mar 2026).

VJC is inserted between the 2D encoder and the view transformation. For a feature map $$0.4\text{ m} \times 0.4\text{ m} \times 0.4\text{ m}$$1, it first computes a vertical summary

$$0.4\text{ m} \times 0.4\text{ m} \times 0.4\text{ m}$$2

then encodes $$0.4\text{ m} \times 0.4\text{ m} \times 0.4\text{ m}$$3 with two Conv1D+ReLU layers, predicts a scalar offset $$0.4\text{ m} \times 0.4\text{ m} \times 0.4\text{ m}$$4, normalizes it by $$0.4\text{ m} \times 0.4\text{ m} \times 0.4\text{ m}$$5, shifts the normalized sampling grid by $$0.4\text{ m} \times 0.4\text{ m} \times 0.4\text{ m}$$6, and resamples the original feature map using bilinear GridSample (Zhao et al., 13 Mar 2026). The intent is to stabilize vertically displaced scene structure before BEV lifting. In ablation, adding VJC improves mIoU from 22.74 to 22.92, with best performance at hidden channel $$0.4\text{ m} \times 0.4\text{ m} \times 0.4\text{ m}$$7 (Zhao et al., 13 Mar 2026).

MIPF addresses multimodal fusion under a geometry-dominant design. It projects LiDAR and image BEV features into a shared embedding,

$$0.4\text{ m} \times 0.4\text{ m} \times 0.4\text{ m}$$8

derives modality prompts $$0.4\text{ m} \times 0.4\text{ m} \times 0.4\text{ m}$$9 from global-average-pooled image BEV features, stacks them into

$+4.16\%$0

and then applies geometry-guided prompt attention using LiDAR BEV tokens as queries and prompts as keys and values: $+4.16\%$1 The resulting attention output modulates LiDAR features through a residual gate,

$+4.16\%$2

This preserves LiDAR geometry while injecting semantic bias from the three visual modalities (Zhao et al., 13 Mar 2026). In ablation, MIPF raises mIoU from 22.74 to 23.14, and the full VJC+MIPF configuration reaches 23.34, with best prompt dimension $+4.16\%$3 and 8 attention heads (Zhao et al., 13 Mar 2026).

After fusion, the BEV feature $+4.16\%$4 is processed by a SECOND-FPN-based BEV encoder, and the occupancy head reshapes channels into the vertical axis to produce per-voxel semantic logits. The entire pipeline therefore maps multimodal observations into volumetric semantics through 2D feature extraction, view transformation, geometry-preserving fusion, BEV reasoning, and channel-to-height decoding (Zhao et al., 13 Mar 2026).

4. Supervision, baselines, and empirical performance

VoxelHound is trained with a composite occupancy loss,

$+4.16\%$5

where $+4.16\%$6 is voxel-wise cross-entropy over the 12 semantic classes, $+4.16\%$7 is Lovász-Softmax, and $+4.16\%$8 and $+4.16\%$9 are MonoScene-style geometric and semantic affinity losses between neighboring voxels (Zhao et al., 13 Mar 2026). Evaluation uses class mean IoU,

$C+L$0

with $C+L$1, alongside per-class IoU, per-scene-category mIoU, and day/night breakdowns (Zhao et al., 13 Mar 2026).

The benchmark adapts several occupancy baselines to the PanoMMOcc setting: MonoScene as a camera-only method, EFFOcc in camera, LiDAR, and multimodal variants, and OpenOccupancy’s CONet in camera, LiDAR, and multimodal variants (Zhao et al., 13 Mar 2026). The main quantitative result is that the best baseline under comparable $C+L$2 conditions, EFFOcc-T, achieves 19.18% mIoU, whereas VoxelHound reaches 22.87% mIoU for $C+L$3 and 23.34% for $C+L$4 (Zhao et al., 13 Mar 2026). Against MonoScene at 8.94% mIoU, the gain is reported as $C+L$5 (Zhao et al., 13 Mar 2026).

The modality study is unusually revealing. Camera-only configurations are weak in this regime: VoxelHound with $C+L$6 reaches 5.79% mIoU, and $C+L$7 reaches 6.14%, whereas adding LiDAR yields 22.87% for $C+L$8 and 23.34% for $C+L$9 (Zhao et al., 13 Mar 2026). The day/night analysis follows the same pattern: $$\mathbf{O} = \Phi \left( \mathcal{I}^{pal}, \mathcal{I}^{th}, \mathcal{I}^{pol}, \mathcal{P} \right),$$0 yields 6.93 by day and 3.52 at night, $$\mathbf{O} = \Phi \left( \mathcal{I}^{pal}, \mathcal{I}^{th}, \mathcal{I}^{pol}, \mathcal{P} \right),$$1 yields 6.85 and 4.07, $$\mathbf{O} = \Phi \left( \mathcal{I}^{pal}, \mathcal{I}^{th}, \mathcal{I}^{pol}, \mathcal{P} \right),$$2 yields 22.56 and 19.17, and $$\mathbf{O} = \Phi \left( \mathcal{I}^{pal}, \mathcal{I}^{th}, \mathcal{I}^{pol}, \mathcal{P} \right),$$3 yields 23.34 and 18.68 (Zhao et al., 13 Mar 2026). The paper interprets this as evidence that LiDAR dominates geometry, while thermal and polarization provide modest but useful gains in challenging lighting and materials.

Per-class performance for the full multimodal model includes 26.69 for car, 21.77 for motorcycle, 49.53 for pedestrian, 34.97 for driveable surface, 34.41 for vegetation, 37.35 for building, and 18.76 for pillar (Zhao et al., 13 Mar 2026). Scene-type analysis further shows strong dependence on structural regularity: moving from camera-only to full multimodal input raises urban mIoU from 5.04 to 16.96, residential from 4.70 to 20.30, campus from 5.30 to 26.50, green spaces from 3.82 to 15.35, forest from 4.08 to 13.10, and rural from 3.21 to 15.48 (Zhao et al., 13 Mar 2026). A common misconception is that panoramic cameras alone should suffice because they provide full angular coverage; the reported numbers contradict that view in this dataset, where geometry from LiDAR remains the decisive signal.

5. Robotic significance and deployment implications

VoxelHound is designed for embodied perception rather than generic 3D scene understanding. The panoramic field of view is explicitly linked to quadruped behaviors such as turning in place, backing up, and stepping sideways, and the 3D semantic occupancy output is intended to support collision avoidance, driveable-surface preference, and traversability reasoning in cluttered environments (Zhao et al., 13 Mar 2026). The framework therefore occupies the middle layer between raw multimodal sensing and motion planning.

Its operational advantages are described in four terms. First, panoramic coverage avoids blind spots around the robot. Second, true 3D occupancy captures overhanging obstacles, vegetation, and complex terrain geometry more faithfully than 2D BEV maps. Third, multimodal sensing improves robustness in low light, overexposed daylight, high-glare conditions, and geometry-sparse textured regions. Fourth, VJC and MIPF are targeted responses to embodied sensing pathologies: VJC stabilizes vertically unstable visual features before view transformation, and MIPF preserves LiDAR geometry while injecting semantic information from RGB, thermal, and polarization inputs (Zhao et al., 13 Mar 2026).

In deployment terms, the framework assumes the full calibrated sensing stack and synchronization pipeline of PanoMMOcc: a $$\mathbf{O} = \Phi \left( \mathcal{I}^{pal}, \mathcal{I}^{th}, \mathcal{I}^{pol}, \mathcal{P} \right),$$4 PAL RGB image, thermal image, polarization image or DoLP/AoLP channels, and a LiDAR scan for every frame (Zhao et al., 13 Mar 2026). The output is a tensor

$$\mathbf{O} = \Phi \left( \mathcal{I}^{pal}, \mathcal{I}^{th}, \mathcal{I}^{pol}, \mathcal{P} \right),$$5

of semantic logits or labels in the robot-centered workspace. The paper explicitly states that this representation can be consumed directly by motion planners as a mid-level perception layer (Zhao et al., 13 Mar 2026).

At the same time, the empirical profile clarifies what VoxelHound is not. It is not a camera-first occupancy model in the style of automotive monocular BEV prediction; pure camera variants perform poorly on PanoMMOcc. It is also not merely a LiDAR occupancy baseline with auxiliary imagery; the gain from VJC and MIPF shows that the multimodal design is not reducible to late concatenation. The system is best understood as a LiDAR-anchored, panoramic, multimodal occupancy model specialized for legged mobility (Zhao et al., 13 Mar 2026).

6. Limitations, assumptions, and broader voxel-centric context

The paper notes four main limitations: dataset scale is smaller than large driving corpora; voxel resolution at 0.4 m is sufficient for navigation but too coarse for fine manipulation or precise footstep planning near small obstacles; very dark nighttime scenes reduce MID360 density at long range; and forests or rural scenes remain difficult because of irregular terrain, dense vegetation, and ambiguous boundaries (Zhao et al., 13 Mar 2026). It also assumes the specific Unitree Go2 sensor placement, high-quality calibration, and environments broadly similar to those collected in PanoMMOcc (Zhao et al., 13 Mar 2026). Future work is directed toward more scenes and weather conditions, higher-resolution occupancy, other downstream tasks such as navigation and detection, and richer temporal modeling (Zhao et al., 13 Mar 2026).

Placed in the wider literature, VoxelHound occupies one branch of a broader voxel-centric research program. Large-scale sparse voxel hierarchies have been used for generative modeling in XCube, which generates millions of voxels at effective resolution up to $$\mathbf{O} = \Phi \left( \mathcal{I}^{pal}, \mathcal{I}^{th}, \mathcal{I}^{pol}, \mathcal{P} \right),$$6 via hierarchical latent diffusion over sparse voxel hierarchies (Ren et al., 2023). Sequence-based voxel tokenization has been explored in SnakeVoxFormer, which converts $$\mathbf{O} = \Phi \left( \mathcal{I}^{pal}, \mathcal{I}^{th}, \mathcal{I}^{pol}, \mathcal{P} \right),$$7 occupancy grids into run-length-encoded token sequences for transformer-based single-image reconstruction (Lee et al., 2023). Task-agnostic large-scale mapping has been addressed by a hash table-based voxel mapping system that manages voxels through spatial and temporal priorities and supports map sharing with over 95% bandwidth reduction from raw sensor data (La et al., 2024). Open-vocabulary semantic mapping has been developed in OpenVox, which maintains probabilistic instance voxels and an instance embedding codebook for real-time retrieval and incremental reconstruction (Deng et al., 23 Feb 2025). Off-road voxel mapping systems such as G-VOM have emphasized GPU-accelerated local 3D grids for hard and soft obstacle detection, slope estimation, roughness estimation, and negative obstacle inference at 10 Hz (Overbye et al., 2021).

These systems address distinct tasks—generation, reconstruction, mapping, open-vocabulary semantics, and off-road traversability—rather than panoramic quadruped occupancy. A plausible implication is that VoxelHound belongs to a convergent design space in which voxel representations serve as explicit 3D intermediates for reasoning, control, and semantic inference. Within that space, its specific contribution is to combine panoramic sensing, multimodal BEV lifting, motion-aware feature stabilization, and geometry-preserving fusion for semantic occupancy around legged robots (Zhao et al., 13 Mar 2026).