Voxel-Based 3D Intersection-over-Union (IoU)

• Voxel-based 3D IoU is defined as the ratio of intersecting to united occupied voxels in a discretized 3D grid.
• It serves as a key metric for evaluating reconstruction fidelity and semantic occupancy by comparing thresholded probabilistic predictions with binary ground truth.
• The method highlights practical challenges related to grid resolution, fixed spatial alignment, and the handling of empty space in volumetric predictions.

Searching arXiv for the cited papers to ground the article in the current literature. Voxel-based 3D Intersection-over-Union (IoU) is a volumetric overlap measure defined on discrete 3D grids, where predictions and ground truth are represented as voxels and compared by the ratio of intersection to union. In volumetric reconstruction, it is typically computed from thresholded per-voxel occupancy predictions against binary ground-truth occupancy, and is also referred to as the Jaccard index (Wilson et al., 2021). In 3D semantic occupancy prediction, the same underlying ratio is used in two related forms: a geometric occupancy IoU for occupied-versus-free space, and a class-averaged semantic IoU over voxel labels (Wei et al., 22 Apr 2025). Across these settings, voxel-based 3D IoU functions as a central quantitative measure of dense 3D understanding because it evaluates agreement over the full voxel volume rather than over sparse points, boxes, or surfaces alone (Wilson et al., 2021).

1. Definition in voxel occupancy grids

In voxelized 3D reconstruction, the formulation is given explicitly as the overlap between the reconstructed voxel grid and the ground-truth voxel grid (Wilson et al., 2021). The paper states that methods are evaluated using voxel Intersection-over-Union between the 3D reconstruction and ground-truth voxels, represented as area of overlap divided by the area of union:

$$IoU = \frac{\sum_{i,j,k} [I(p_{(i,j,k)} > t)I(y_{(i,j,k)})]}{\sum_{i,j,k} [I(I(p_{(i,j,k)} > t) + I(y_{(i,j,k)}))]}.$$

Here, $y_{i,j,k} \in \{0,1\}$ denotes the ground-truth occupancy of voxel $(i,j,k)$, $p_{i,j,k}$ denotes the predicted Bernoulli output at that voxel, $I(\cdot)$ is the indicator function, and $t$ is a probability threshold (Wilson et al., 2021). The numerator counts voxels that are both predicted occupied after thresholding and ground-truth occupied; the denominator counts voxels where either prediction or ground truth is occupied (Wilson et al., 2021).

This formulation is defined over all voxels in the 3D grid, with no special handling of empty space beyond the fact that empty voxels contribute neither to the intersection nor to the union (Wilson et al., 2021). Predictions are probabilistic in $[0,1]$, but IoU is computed on a binarized grid via threshold $t$ (Wilson et al., 2021). The same paper reports visualization thresholds such as $0.3$ and $0.4$ for rendering, while leaving the evaluation formula threshold generic (Wilson et al., 2021).

A related but distinct voxel-space formulation appears in semantic occupancy prediction. There, scene completion IoU is defined as

$y_{i,j,k} \in \{0,1\}$0

where $y_{i,j,k} \in \{0,1\}$1 is the number of voxels correctly predicted as occupied, $y_{i,j,k} \in \{0,1\}$2 is the number predicted occupied but actually free, and $y_{i,j,k} \in \{0,1\}$3 is the number occupied in ground truth but predicted free (Wei et al., 22 Apr 2025). In that setting, all occupied semantic classes are grouped together and free space acts as the negative class (Wei et al., 22 Apr 2025). This is still voxel-based 3D IoU, but applied to binary occupancy extracted from multi-class voxel labels rather than from thresholded Bernoulli occupancies.

2. Representations, discretization, and metric computation

Voxel-based 3D IoU depends directly on how 3D space is discretized. In 3D-MOV, the decoder produces a dense $y_{i,j,k} \in \{0,1\}$4 voxel grid through a five deconvolutional layer 3D decoder, and the paper explicitly contrasts this with a baseline using a $y_{i,j,k} \in \{0,1\}$5 voxel grid (Wilson et al., 2021). Ground truth is binary occupancy, predictions are Bernoulli probabilities, and binary cross entropy is calculated between ground-truth and reconstructed 3D voxel grids during training (Wilson et al., 2021). The evaluation operation is correspondingly simple: compute the probability tensor, threshold it to obtain binary occupancy, and compute IoU against the ground-truth binary grid (Wilson et al., 2021).

The same work uses voxelized versions of ShapeNet’s ModelNet10 and ModelNet40 models, and also evaluates on Sound20K scenes generated synthetically from rigid body simulation with impact sound synthesis (Wilson et al., 2021). The paper does not detail the voxelization pipeline, but because the decoder architecture is fixed to $y_{i,j,k} \in \{0,1\}$6, both ground-truth and predicted shapes must be aligned to the same normalized coordinate frame (Wilson et al., 2021). This suggests that voxel-based 3D IoU in such reconstruction settings is inseparable from the normalization, centering, and fixed-grid assumptions that define the volume being compared.

In MS-Occ, the grid specification is substantially larger and tied to autonomous-driving scene understanding rather than single-object reconstruction. Each sample in nuScenes-OpenOccupancy is annotated using a voxel resolution of $y_{i,j,k} \in \{0,1\}$7, forming a dense voxel grid of size $y_{i,j,k} \in \{0,1\}$8 (Wei et al., 22 Apr 2025). IoU is computed within this fixed 3D voxel volume aligned to the ego-vehicle coordinate frame (Wei et al., 22 Apr 2025). Each voxel is assigned one of 17 classes, comprising 16 semantic categories and one class representing free space (Wei et al., 22 Apr 2025).

Prediction handling differs accordingly. In MS-Occ, the network outputs a 17-channel logit tensor over the voxel grid; evaluation applies softmax and then argmax along the class dimension to obtain hard voxel labels (Wei et al., 22 Apr 2025). For geometric IoU, occupied voxels are defined as all labels other than free space, while free space is treated as the negative class (Wei et al., 22 Apr 2025). For semantic evaluation, per-class voxel IoUs are computed separately and averaged across the 16 semantic classes to obtain mean IoU:

$y_{i,j,k} \in \{0,1\}$9

with $(i,j,k)$0 in practice (Wei et al., 22 Apr 2025).

3. Evaluation protocols and reported use cases

Voxel-based 3D IoU is used differently across reconstruction and occupancy-prediction benchmarks, but in both cases it serves as a test-time measure of volumetric agreement. In 3D-MOV, binary cross entropy is both a training loss and a reported quantity, while IoU is the reconstruction-accuracy metric used to compare audio, visual, and audio-visual methods (Wilson et al., 2021). The reported protocol is to compute IoU per test object and then average over objects, yielding a single IoU percentage for each method and condition (Wilson et al., 2021).

For single-object ShapeNet and ModelNet experiments, 3D-MOV reports IoU values for audio-only, visual-only, and audio-visual inputs under one-view and five-view conditions (Wilson et al., 2021). The reported reconstruction accuracies are 21.2% for 3D-MOV-A with one view, 22.7% for 3D-MOV-V with one view, 22.5% for 3D-MOV-V with five views, 18.0% for the adapted T-L Network baseline, 32.6% for 3D-MOV-AV with one view, and 31.0% for 3D-MOV-AV with five views (Wilson et al., 2021). For Sound20K, which contains multi-object, multi-frame video scenes, the reported IoUs are 37.2% for audio-only, 65.7% for visual-only, and 69.8% for audio-visual input over 10 views (Wilson et al., 2021).

In MS-Occ, voxel-based 3D IoU is part of a paired evaluation with semantic mIoU on nuScenes-OpenOccupancy (Wei et al., 22 Apr 2025). The main reported validation result is an IoU of 32.1% and an mIoU of 25.3% (Wei et al., 22 Apr 2025). The paper also reports per-class voxel IoUs for the 16 semantic categories, including 29.4 for barrier, 23.8 for bicycle, 24.2 for bus, 28.7 for car, 17.5 for construction vehicle, 27.5 for motorcycle, 29.0 for pedestrian, 21.1 for traffic cone, 16.9 for trailer, 25.3 for truck, 33.4 for driveable surface, 23.5 for other flat, 24.4 for sidewalk, 24.6 for terrain, 27.6 for manmade, and 28.4 for vegetation (Wei et al., 22 Apr 2025).

The benchmark role of voxel-based 3D IoU therefore spans at least two regimes. In single-object reconstruction it measures fidelity of reconstructed shape occupancy (Wilson et al., 2021). In semantic occupancy prediction it measures scene completion geometry over a fixed region of interest, while related class-wise IoUs quantify semantic correctness on the same voxel grid (Wei et al., 22 Apr 2025).

4. Relationship to losses, architectures, and fusion strategies

Voxel-based 3D IoU is often the primary evaluation metric without being optimized directly. In 3D-MOV, the network is a multimodal LSTM autoencoder with separate visual and audio encoders, temporal aggregation via LSTM, fusion by addition, concatenation, or factorized bilinear pooling, and a 3D decoder with five Conv3D transpose layers that output a $(i,j,k)$1 voxel grid of Bernoulli probabilities (Wilson et al., 2021). The 3D reconstruction stage is trained with binary cross entropy rather than with an IoU loss (Wilson et al., 2021). The paper states explicitly that there is no dice loss, IoU loss, or other direct overlap-based loss in the training objective (Wilson et al., 2021).

The architectural rationale is that better temporal and multimodal features should improve volumetric reconstruction quality and therefore improve IoU. The paper states that temporal encoding via LSTM aggregates information across multiple frames and helps with self-occlusion and multi-view consistency, while multimodal fusion can help recover concave interior or transparent regions where RGB views are insufficient (Wilson et al., 2021). It further states: “Our method is able to obtain better reconstruction results for concave internal structures and scenes with multiple objects by fusing temporal audio-visual inputs” (Wilson et al., 2021).

In MS-Occ, the relationship between architecture and voxel-based 3D IoU is more explicit because the task itself is voxel classification. The framework includes middle-stage fusion and late-stage fusion, with Gaussian-Geo and Semantic-Aware modules in the middle stage, and Adaptive Fusion and High Classification Confidence Voxel Fusion in the late stage (Wei et al., 22 Apr 2025). Training uses cross-entropy and Lovász-Softmax losses, along with affinity losses $(i,j,k)$2 and $(i,j,k)$3 “to optimize scene-level and class-level metrics,” following MonoScene (Wei et al., 22 Apr 2025).

The ablation results connect these modules directly to IoU and mIoU. Using only late-stage voxel fusion yields IoU 31.4 and mIoU 23.9; using only middle-stage fusion yields IoU 30.2 and mIoU 21.1; using both yields IoU 32.1 and mIoU 25.3 (Wei et al., 22 Apr 2025). Dropping Gaussian-Geo reduces performance to IoU 31.7 and mIoU 24.4, dropping Semantic-Aware to IoU 30.6 and mIoU 22.1, dropping Adaptive Fusion to IoU 31.9 and mIoU 24.9, and dropping HCCVF to IoU 31.6 and mIoU 24.3 (Wei et al., 22 Apr 2025). In a separate ablation, HCCVF improves over summation-based fusion from IoU 31.6 and mIoU 24.5 to IoU 32.1 and mIoU 25.3 (Wei et al., 22 Apr 2025).

These results illustrate two established patterns. First, voxel-based 3D IoU is routinely used as the target metric even when training relies on differentiable surrogates such as binary cross entropy or Lovász-Softmax (Wilson et al., 2021, Wei et al., 22 Apr 2025). Second, architectural changes that improve geometric alignment, semantic transfer, or multi-view consistency are evaluated through their effect on voxel overlap statistics (Wilson et al., 2021, Wei et al., 22 Apr 2025).

5. Limitations, sensitivities, and common sources of confusion

The principal limitations of voxel-based 3D IoU arise from resolution, fixed-grid structure, and the semantics of empty space. In 3D-MOV, the authors note that volumetric methods such as voxelized reconstructions are primarily limited by resolution, and they adopt a $(i,j,k)$4 grid partly to improve detail relative to a $(i,j,k)$5 baseline (Wilson et al., 2021). Because IoU is computed on that grid, thin structures, small parts, and fine concavities may disappear in voxelization and therefore not be reflected accurately in the score (Wilson et al., 2021). The same paper also states that the approach is currently implemented and evaluated with fixed-grid shapes, implying that IoU is measured within a fixed grid per object rather than under adaptive resolution or multi-scale representations (Wilson et al., 2021).

MS-Occ exhibits analogous issues at scene scale. The benchmark uses a fixed grid of size $(i,j,k)$6 at $(i,j,k)$7 resolution (Wei et al., 22 Apr 2025). The paper describes this as a compromise: finer resolution yields more precise surfaces but more voxels and more chances for false positives and false negatives, while coarser resolution blurs small objects and edges (Wei et al., 22 Apr 2025). It also emphasizes that small objects such as pedestrians, bicycles, motorcycles, and traffic cones occupy very few voxels, so minor misalignment or misprediction can severely affect per-class IoU (Wei et al., 22 Apr 2025).

Another recurring source of confusion is that “3D IoU” does not always denote the same object of comparison. In 3D-MOV and MS-Occ it refers to overlap in voxel occupancy grids (Wilson et al., 2021, Wei et al., 22 Apr 2025). By contrast, 3D IoU-Net and “From Voxel to Point” define 3D IoU for oriented 3D bounding boxes rather than for voxels (Li et al., 2020, Li et al., 2021). Those works use geometric volume overlap between boxes, not occupancy overlap in a discretized voxel field (Li et al., 2020, Li et al., 2021). “Bounding Box Disparity” goes further by deriving an analytic closed-form IoU for full 6-DoF oriented cuboids, and explicitly frames voxel-based 3D IoU as a discretized approximation of analytic volume overlap (Adam et al., 2022).

A different ambiguity appears in “A Dense Siamese U-Net trained with Edge Enhanced 3D IOU Loss for Image Co-segmentation,” where “3D IoU” refers not to spatial voxels but to a signed normalized distance map treated as a 3D surface over a 2D image grid (Liu et al., 2021). The paper itself states that the underlying grid is 2D and the third dimension is the distance-map value (Liu et al., 2021). This distinction matters because voxel-based 3D IoU is specifically tied to occupancy in a 3D grid rather than to distance-map overlap or box-volume overlap.

6. Relation to analytic box IoU and broader 3D vision practice

Within 3D vision, voxel-based 3D IoU coexists with exact geometric IoU for cuboids and with IoU-guided detection methods. “Bounding Box Disparity” formalizes 3D IoU for arbitrary oriented cuboids as

$(i,j,k)$8

and presents a closed-form method for computing the exact intersection volume for full 6-DoF boxes (Adam et al., 2022). That work explicitly states that voxel-based 3D IoU typically approximates the overlap by discretizing space into an axis-aligned voxel grid and counting occupied voxels, and identifies resolution trade-offs, orientation error, and partial occupancy as sources of approximation error (Adam et al., 2022). It also notes that the analytic method can serve as a ground truth against which voxel-based approximations are evaluated and calibrated (Adam et al., 2022).

In 3D object detection, IoU is often elevated from an evaluation criterion to a learned quantity. 3D IoU-Net predicts a scalar IoU per proposal and uses it as the detection confidence for NMS, but the method is point-based rather than voxel-based (Li et al., 2020). “From Voxel to Point” transfers related ideas into a voxel-based two-stage detector with a voxel-to-point decoder, where an IoU estimation branch predicts 3D IoU between proposals and ground truth and then re-aligns IoU predictions to refined boxes via a second inference pass (Li et al., 2021). These works show that IoU in 3D vision can function as a metric, a supervision signal, and a localization-confidence estimate, but their object of comparison remains oriented bounding boxes rather than voxel occupancies (Li et al., 2020, Li et al., 2021).

In reconstruction and occupancy benchmarks, however, voxel-based 3D IoU remains closely aligned with standard volumetric practice. 3D-MOV states that its IoU definition and usage follow standard practice in voxel-based 3D reconstruction benchmarks, including fixed voxel grids, binary occupancy, cross-entropy training, and IoU evaluation (Wilson et al., 2021). MS-Occ similarly adopts the exact OpenOccupancy metric definitions for scene completion IoU and semantic mIoU over the full voxel grid of the benchmark (Wei et al., 22 Apr 2025). Taken together, these works situate voxel-based 3D IoU as the canonical overlap measure for dense volumetric prediction, while also clarifying that it is only one member of a larger family of 3D IoU formulations used across reconstruction, occupancy, and detection (Wilson et al., 2021, Wei et al., 22 Apr 2025, Adam et al., 2022).