---
title: Voxel-Based 3D Intersection-over-Union (IoU)
url: https://www.emergentmind.com/topics/voxel-based-3d-intersection-over-union-iou
type: topic
---

# Voxel-Based 3D Intersection-over-Union (IoU)

Searching arXiv for the cited papers to ground the article in the current literature.
Voxel-based 3D Intersection-over-Union (IoU) is a volumetric overlap measure defined on discrete 3D grids, where predictions and ground truth are represented as voxels and compared by the ratio of intersection to union. In volumetric reconstruction, it is typically computed from thresholded per-voxel occupancy predictions against binary ground-truth occupancy, and is also referred to as the Jaccard index [2110.02404]. In 3D semantic occupancy prediction, the same underlying ratio is used in two related forms: a geometric occupancy IoU for occupied-versus-free space, and a class-averaged semantic IoU over voxel labels [2504.15888]. Across these settings, voxel-based 3D IoU functions as a central quantitative measure of dense 3D understanding because it evaluates agreement over the full voxel volume rather than over sparse points, boxes, or surfaces alone [2110.02404].

## 1. Definition in voxel occupancy grids

In voxelized 3D reconstruction, the formulation is given explicitly as the overlap between the reconstructed voxel grid and the ground-truth voxel grid [2110.02404]. The paper states that methods are evaluated using voxel Intersection-over-Union between the 3D reconstruction and ground-truth voxels, represented as area of overlap divided by the area of union:

$$
IoU = \frac{\sum_{i,j,k} [I(p_{(i,j,k)} > t)I(y_{(i,j,k)})]}{\sum_{i,j,k} [I(I(p_{(i,j,k)} > t) + I(y_{(i,j,k)}))]}.
$$

Here, $y_{i,j,k} \in \{0,1\}$ denotes the ground-truth occupancy of voxel $(i,j,k)$, $p_{i,j,k}$ denotes the predicted Bernoulli output at that voxel, $I(\cdot)$ is the indicator function, and $t$ is a probability threshold [2110.02404]. The numerator counts voxels that are both predicted occupied after thresholding and ground-truth occupied; the denominator counts voxels where either prediction or ground truth is occupied [2110.02404].

This formulation is defined over all voxels in the 3D grid, with no special handling of empty space beyond the fact that empty voxels contribute neither to the intersection nor to the union [2110.02404]. Predictions are probabilistic in $[0,1]$, but IoU is computed on a binarized grid via threshold $t$ [2110.02404]. The same paper reports visualization thresholds such as $0.3$ and $0.4$ for rendering, while leaving the evaluation formula threshold generic [2110.02404].

A related but distinct voxel-space formulation appears in semantic occupancy prediction. There, scene completion IoU is defined as

$$
IoU = \frac{TP}{TP + FP + FN},
$$

where $TP$ is the number of voxels correctly predicted as occupied, $FP$ is the number predicted occupied but actually free, and $FN$ is the number occupied in ground truth but predicted free [2504.15888]. In that setting, all occupied semantic classes are grouped together and free space acts as the negative class [2504.15888]. This is still voxel-based 3D IoU, but applied to binary occupancy extracted from multi-class voxel labels rather than from thresholded Bernoulli occupancies.

## 2. Representations, discretization, and metric computation

Voxel-based 3D IoU depends directly on how 3D space is discretized. In 3D-MOV, the decoder produces a dense $30^3$ voxel grid through a five deconvolutional layer 3D decoder, and the paper explicitly contrasts this with a baseline using a $20^3$ voxel grid [2110.02404]. Ground truth is binary occupancy, predictions are Bernoulli probabilities, and binary cross entropy is calculated between ground-truth and reconstructed 3D voxel grids during training [2110.02404]. The evaluation operation is correspondingly simple: compute the probability tensor, threshold it to obtain binary occupancy, and compute IoU against the ground-truth binary grid [2110.02404].

The same work uses voxelized versions of ShapeNet’s ModelNet10 and ModelNet40 models, and also evaluates on Sound20K scenes generated synthetically from rigid body simulation with impact sound synthesis [2110.02404]. The paper does not detail the voxelization pipeline, but because the decoder architecture is fixed to $30^3$, both ground-truth and predicted shapes must be aligned to the same normalized coordinate frame [2110.02404]. This suggests that voxel-based 3D IoU in such reconstruction settings is inseparable from the normalization, centering, and fixed-grid assumptions that define the volume being compared.

In MS-Occ, the grid specification is substantially larger and tied to autonomous-driving scene understanding rather than single-object reconstruction. Each sample in nuScenes-OpenOccupancy is annotated using a voxel resolution of $(0.2\,\text{m}, 0.2\,\text{m}, 0.2\,\text{m})$, forming a dense voxel grid of size $(512, 512, 40)$ [2504.15888]. IoU is computed within this fixed 3D voxel volume aligned to the ego-vehicle coordinate frame [2504.15888]. Each voxel is assigned one of 17 classes, comprising 16 semantic categories and one class representing free space [2504.15888].

Prediction handling differs accordingly. In MS-Occ, the network outputs a 17-channel logit tensor over the voxel grid; evaluation applies softmax and then argmax along the class dimension to obtain hard voxel labels [2504.15888]. For geometric IoU, occupied voxels are defined as all labels other than free space, while free space is treated as the negative class [2504.15888]. For semantic evaluation, per-class voxel IoUs are computed separately and averaged across the 16 semantic classes to obtain mean IoU:

$$
mIoU = \frac{1}{S} \sum_{i = 1}^{S} \frac{TP_i}{TP_i + FP_i + FN_i},
$$

with $S=16$ in practice [2504.15888].

## 3. Evaluation protocols and reported use cases

Voxel-based 3D IoU is used differently across reconstruction and occupancy-prediction benchmarks, but in both cases it serves as a test-time measure of volumetric agreement. In 3D-MOV, binary cross entropy is both a training loss and a reported quantity, while IoU is the reconstruction-accuracy metric used to compare audio, visual, and audio-visual methods [2110.02404]. The reported protocol is to compute IoU per test object and then average over objects, yielding a single IoU percentage for each method and condition [2110.02404].

For single-object ShapeNet and ModelNet experiments, 3D-MOV reports IoU values for audio-only, visual-only, and audio-visual inputs under one-view and five-view conditions [2110.02404]. The reported reconstruction accuracies are 21.2% for 3D-MOV-A with one view, 22.7% for 3D-MOV-V with one view, 22.5% for 3D-MOV-V with five views, 18.0% for the adapted T-L Network baseline, 32.6% for 3D-MOV-AV with one view, and 31.0% for 3D-MOV-AV with five views [2110.02404]. For Sound20K, which contains multi-object, multi-frame video scenes, the reported IoUs are 37.2% for audio-only, 65.7% for visual-only, and 69.8% for audio-visual input over 10 views [2110.02404].

In MS-Occ, voxel-based 3D IoU is part of a paired evaluation with semantic mIoU on nuScenes-OpenOccupancy [2504.15888]. The main reported validation result is an IoU of 32.1% and an mIoU of 25.3% [2504.15888]. The paper also reports per-class voxel IoUs for the 16 semantic categories, including 29.4 for barrier, 23.8 for bicycle, 24.2 for bus, 28.7 for car, 17.5 for construction vehicle, 27.5 for motorcycle, 29.0 for pedestrian, 21.1 for traffic cone, 16.9 for trailer, 25.3 for truck, 33.4 for driveable surface, 23.5 for other flat, 24.4 for sidewalk, 24.6 for terrain, 27.6 for manmade, and 28.4 for vegetation [2504.15888].

The benchmark role of voxel-based 3D IoU therefore spans at least two regimes. In single-object reconstruction it measures fidelity of reconstructed shape occupancy [2110.02404]. In semantic occupancy prediction it measures scene completion geometry over a fixed region of interest, while related class-wise IoUs quantify semantic correctness on the same voxel grid [2504.15888].

## 4. Relationship to losses, architectures, and fusion strategies

Voxel-based 3D IoU is often the primary evaluation metric without being optimized directly. In 3D-MOV, the network is a multimodal LSTM autoencoder with separate visual and audio encoders, temporal aggregation via LSTM, fusion by addition, concatenation, or factorized bilinear pooling, and a 3D decoder with five Conv3D transpose layers that output a $30^3$ voxel grid of Bernoulli probabilities [2110.02404]. The 3D reconstruction stage is trained with binary cross entropy rather than with an IoU loss [2110.02404]. The paper states explicitly that there is no dice loss, IoU loss, or other direct overlap-based loss in the training objective [2110.02404].

The architectural rationale is that better temporal and multimodal features should improve volumetric reconstruction quality and therefore improve IoU. The paper states that temporal encoding via LSTM aggregates information across multiple frames and helps with self-occlusion and multi-view consistency, while multimodal fusion can help recover concave interior or transparent regions where RGB views are insufficient [2110.02404]. It further states: “Our method is able to obtain better reconstruction results for concave internal structures and scenes with multiple objects by fusing temporal audio-visual inputs” [2110.02404].

In MS-Occ, the relationship between architecture and voxel-based 3D IoU is more explicit because the task itself is voxel classification. The framework includes middle-stage fusion and late-stage fusion, with Gaussian-Geo and Semantic-Aware modules in the middle stage, and Adaptive Fusion and High Classification Confidence Voxel Fusion in the late stage [2504.15888]. Training uses cross-entropy and Lovász-Softmax losses, along with affinity losses $\mathcal{L}_\text{scal}^\text{geo}$ and $\mathcal{L}_\text{scal}^\text{sem}$ “to optimize scene-level and class-level metrics,” following MonoScene [2504.15888].

The ablation results connect these modules directly to IoU and mIoU. Using only late-stage voxel fusion yields IoU 31.4 and mIoU 23.9; using only middle-stage fusion yields IoU 30.2 and mIoU 21.1; using both yields IoU 32.1 and mIoU 25.3 [2504.15888]. Dropping Gaussian-Geo reduces performance to IoU 31.7 and mIoU 24.4, dropping Semantic-Aware to IoU 30.6 and mIoU 22.1, dropping Adaptive Fusion to IoU 31.9 and mIoU 24.9, and dropping HCCVF to IoU 31.6 and mIoU 24.3 [2504.15888]. In a separate ablation, HCCVF improves over summation-based fusion from IoU 31.6 and mIoU 24.5 to IoU 32.1 and mIoU 25.3 [2504.15888].

These results illustrate two established patterns. First, voxel-based 3D IoU is routinely used as the target metric even when training relies on differentiable surrogates such as binary cross entropy or Lovász-Softmax [2110.02404] [2504.15888]. Second, architectural changes that improve geometric alignment, semantic transfer, or multi-view consistency are evaluated through their effect on voxel overlap statistics [2110.02404] [2504.15888].

## 5. Limitations, sensitivities, and common sources of confusion

The principal limitations of voxel-based 3D IoU arise from resolution, fixed-grid structure, and the semantics of empty space. In 3D-MOV, the authors note that volumetric methods such as voxelized reconstructions are primarily limited by resolution, and they adopt a $30^3$ grid partly to improve detail relative to a $20^3$ baseline [2110.02404]. Because IoU is computed on that grid, thin structures, small parts, and fine concavities may disappear in voxelization and therefore not be reflected accurately in the score [2110.02404]. The same paper also states that the approach is currently implemented and evaluated with fixed-grid shapes, implying that IoU is measured within a fixed grid per object rather than under adaptive resolution or multi-scale representations [2110.02404].

MS-Occ exhibits analogous issues at scene scale. The benchmark uses a fixed grid of size $(512,512,40)$ at $0.2\,\text{m}$ resolution [2504.15888]. The paper describes this as a compromise: finer resolution yields more precise surfaces but more voxels and more chances for false positives and false negatives, while coarser resolution blurs small objects and edges [2504.15888]. It also emphasizes that small objects such as pedestrians, bicycles, motorcycles, and traffic cones occupy very few voxels, so minor misalignment or misprediction can severely affect per-class IoU [2504.15888].

Another recurring source of confusion is that “3D IoU” does not always denote the same object of comparison. In 3D-MOV and MS-Occ it refers to overlap in voxel occupancy grids [2110.02404] [2504.15888]. By contrast, 3D IoU-Net and “From Voxel to Point” define 3D IoU for oriented 3D bounding boxes rather than for voxels [2004.04962] [2108.03648]. Those works use geometric volume overlap between boxes, not occupancy overlap in a discretized voxel field [2004.04962] [2108.03648]. “Bounding Box Disparity” goes further by deriving an analytic closed-form IoU for full 6-DoF oriented cuboids, and explicitly frames voxel-based 3D IoU as a discretized approximation of analytic volume overlap [2207.03720].

A different ambiguity appears in “A Dense Siamese U-Net trained with Edge Enhanced 3D IOU Loss for Image Co-segmentation,” where “3D IoU” refers not to spatial voxels but to a signed normalized distance map treated as a 3D surface over a 2D image grid [2108.07491]. The paper itself states that the underlying grid is 2D and the third dimension is the distance-map value [2108.07491]. This distinction matters because voxel-based 3D IoU is specifically tied to occupancy in a 3D grid rather than to distance-map overlap or box-volume overlap.

## 6. Relation to analytic box IoU and broader 3D vision practice

Within 3D vision, voxel-based 3D IoU coexists with exact geometric IoU for cuboids and with IoU-guided detection methods. “Bounding Box Disparity” formalizes 3D IoU for arbitrary oriented cuboids as

$$
\text{IoU}(T^1,T^2) = \frac{V_I}{V_U} = \frac{\text{Vol}(T^1 \cap T^2)}{\text{Vol}(T^1 \cup T^2)},
$$

and presents a closed-form method for computing the exact intersection volume for full 6-DoF boxes [2207.03720]. That work explicitly states that voxel-based 3D IoU typically approximates the overlap by discretizing space into an axis-aligned voxel grid and counting occupied voxels, and identifies resolution trade-offs, orientation error, and partial occupancy as sources of approximation error [2207.03720]. It also notes that the analytic method can serve as a ground truth against which voxel-based approximations are evaluated and calibrated [2207.03720].

In 3D object detection, IoU is often elevated from an evaluation criterion to a learned quantity. 3D IoU-Net predicts a scalar IoU per proposal and uses it as the detection confidence for NMS, but the method is point-based rather than voxel-based [2004.04962]. “From Voxel to Point” transfers related ideas into a voxel-based two-stage detector with a voxel-to-point decoder, where an IoU estimation branch predicts 3D IoU between proposals and ground truth and then re-aligns IoU predictions to refined boxes via a second inference pass [2108.03648]. These works show that IoU in 3D vision can function as a metric, a supervision signal, and a localization-confidence estimate, but their object of comparison remains oriented bounding boxes rather than voxel occupancies [2004.04962] [2108.03648].

In reconstruction and occupancy benchmarks, however, voxel-based 3D IoU remains closely aligned with standard volumetric practice. 3D-MOV states that its IoU definition and usage follow standard practice in voxel-based 3D reconstruction benchmarks, including fixed voxel grids, binary occupancy, cross-entropy training, and IoU evaluation [2110.02404]. MS-Occ similarly adopts the exact OpenOccupancy metric definitions for scene completion IoU and semantic mIoU over the full voxel grid of the benchmark [2504.15888]. Taken together, these works situate voxel-based 3D IoU as the canonical overlap measure for dense volumetric prediction, while also clarifying that it is only one member of a larger family of 3D IoU formulations used across reconstruction, occupancy, and detection [2110.02404] [2504.15888] [2207.03720].

Source: https://www.emergentmind.com/topics/voxel-based-3d-intersection-over-union-iou