---
title: 'VoxelNet: End-to-End LiDAR 3D Detection'
url: https://www.emergentmind.com/topics/voxelnet
type: topic
---

# VoxelNet: End-to-End LiDAR 3D Detection

VoxelNet is an end-to-end LiDAR-based 3D object detection architecture that replaces hand-crafted bird’s-eye-view feature engineering with learned volumetric representations derived directly from raw point clouds. In its original formulation, a point cloud is partitioned into equally spaced 3D voxels, each non-empty voxel is encoded by stacked Voxel Feature Encoding (VFE) layers, the resulting sparse voxel tensor is processed by 3D convolutional middle layers, and a region proposal network predicts oriented 3D bounding boxes in a single stage [1711.06396]. The model was introduced for autonomous driving settings in which sparsity, irregular point distribution, and the cost of manual feature design had limited earlier LiDAR pipelines [1711.06396].

## 1. Origin, problem setting, and design objective

VoxelNet was introduced to address three coupled difficulties in point-cloud detection: the extreme sparsity and non-uniform density of LiDAR scans, the irregular and unordered structure of point sets, and the information bottleneck induced by hand-crafted projections and summary statistics [1711.06396]. Earlier pipelines commonly projected points into bird’s-eye-view maps or handcrafted voxel features before applying 2D or 3D CNNs, but this discarded fine geometric structure before learning began [1711.06396]. VoxelNet’s central claim was that feature extraction and box prediction could be unified in a single trainable network operating directly on points grouped by voxels, eliminating manual feature engineering [1711.06396].

The original system targets 3D object detection and orientation estimation for autonomous driving, with evaluation on KITTI for cars, pedestrians, and cyclists [1711.06396]. For car detection, the LiDAR region of interest is \(Z \in [-3,1]\), \(Y \in [-40,40]\), and \(X \in [0,70.4]\) meters, voxelized with \(v_D=0.4\), \(v_H=0.2\), and \(v_W=0.2\) meters, yielding a grid of \(10 \times 400 \times 352\) cells [1711.06396]. The model processes only non-empty voxels, which is essential because more than 90% of the grid is empty in typical scans [1711.06396].

A recurring later observation is that “VoxelNet” became both a specific architecture and a template for voxel-based 3D detection. Subsequent work uses it as a baseline detector, a fusion backbone, a domain-adaptation host, or a lightweight BEV backbone, but these later systems generally preserve the original macro-structure of voxelization, voxel feature learning, spatial aggregation, and anchor-based detection unless explicitly redesigned [1911.10150][2011.07784][2503.07133].

## 2. Architectural composition

VoxelNet has three principal components: a feature learning network operating on voxelized points, convolutional middle layers that aggregate spatial context in 3D, and a region proposal network operating on a bird’s-eye-view feature map [1711.06396].

The input point cloud is \(\mathbf{M}=\{\mathbf{p}_i=[x_i,y_i,z_i,r_i]^T\}_{i=1}^N\), where \(r_i\) is reflectance. Each point is assigned to a voxel, and each voxel keeps at most \(T\) points; for cars, \(T=35\), while for pedestrians and cyclists \(T=45\) is used to better capture small-object shape [1711.06396]. For a non-empty voxel with \(t \le T\) points, VoxelNet computes the centroid
\[
v_x = \frac{1}{t}\sum_{i=1}^t x_i,\quad
v_y = \frac{1}{t}\sum_{i=1}^t y_i,\quad
v_z = \frac{1}{t}\sum_{i=1}^t z_i
\]
and augments each point as
\[
\hat{\mathbf{p}}_i = [x_i, y_i, z_i, r_i, x_i-v_x, y_i-v_y, z_i-v_z]^T \in \mathbb{R}^7.
\]
These augmented point features are the input to the VFE layers [1711.06396].

A VFE layer applies a point-wise fully connected network \(\phi\) to each point,
\[
\mathbf{f}_i = \phi(\hat{\mathbf{p}}_i) \in \mathbb{R}^m,
\]
computes a voxel-local aggregated feature by element-wise max pooling,
\[
\tilde{\mathbf{f}} = \max_{i=1,\ldots,t}\mathbf{f}_i,
\]
and concatenates point-wise and aggregated features,
\[
\mathbf{f}_i^{\text{out}} = [\mathbf{f}_i^T,\tilde{\mathbf{f}}^T]^T \in \mathbb{R}^{2m}.
\]
This permits inter-point interaction within each voxel while retaining point-level detail [1711.06396]. For cars, the original stack uses \(\text{VFE-1}(7,32)\) followed by \(\text{VFE-2}(32,128)\), then an FCN to 128 channels and voxel-wise max pooling, producing a sparse tensor of size \(128 \times 10 \times 400 \times 352\) [1711.06396].

The convolutional middle layers are 3D convolutions with BN and ReLU. For cars, the sequence is \(\text{Conv3D}(128,64,k=3,s=(2,1,1),p=(1,1,1))\), then \(\text{Conv3D}(64,64,k=3,s=(1,1,1),p=(0,1,1))\), then \(\text{Conv3D}(64,64,k=3,s=(2,1,1),p=(1,1,1))\), reducing the tensor to \(64 \times 2 \times 400 \times 352\), which is reshaped into a BEV feature map of \(128 \times 400 \times 352\) [1711.06396]. These layers aggregate local spatial context while collapsing the vertical axis into a representation suitable for planar detection.

The RPN is a multi-scale 2D convolutional detector operating on the BEV map. It predicts classification logits and seven regression parameters per anchor, using three convolutional blocks with downsampling and upsampling before concatenation into a high-resolution feature map [1711.06396]. The original paper parameterizes a 3D box by center \((x_c,y_c,z_c)\), size \((l,w,h)\), and yaw \(\theta\), with anchor-normalized regression targets
\[
\Delta x = \frac{x_c^g - x_c^a}{d^a},\quad
\Delta y = \frac{y_c^g - y_c^a}{d^a},\quad
\Delta z = \frac{z_c^g - z_c^a}{h^a},
\]
\[
\Delta l = \log\frac{l^g}{l^a},\quad
\Delta w = \log\frac{w^g}{w^a},\quad
\Delta h = \log\frac{h^g}{h^a},\quad
\Delta \theta = \theta^g - \theta^a,
\]
where \(d^a=\sqrt{(l^a)^2+(w^a)^2}\) [1711.06396].

## 3. Training protocol, anchors, and optimization behavior

VoxelNet is trained with a joint classification-and-regression objective. Let \(a_i^{\text{pos}}\) denote positive anchors and \(a_j^{\text{neg}}\) negative anchors. The original loss is
\[
L =
\alpha \frac{1}{N_{\text{pos}}}\sum_i L_{\text{cls}}(p_i^{\text{pos}},1)
+
\beta \frac{1}{N_{\text{neg}}}\sum_j L_{\text{cls}}(p_j^{\text{neg}},0)
+
\frac{1}{N_{\text{pos}}}\sum_i L_{\text{reg}}(\mathbf{u}_i,\mathbf{u}_i^*),
\]
with binary cross-entropy classification and Smooth L1 regression; for cars, \(\alpha=1.5\) and \(\beta=1\) [1711.06396]. Positive and negative assignment is IoU-based in BEV. For cars, anchors are positive if they have the highest IoU with a ground-truth box or IoU \(\ge 0.6\), negative if IoU \(< 0.45\), and ignored otherwise [1711.06396]. Car anchors use size \(l^a=3.9\) m, \(w^a=1.6\) m, \(h^a=1.56\) m, center height \(z_c^a=-1.0\) m, and two yaw orientations \(0^\circ\) and \(90^\circ\) [1711.06396].

The original training schedule on KITTI uses SGD with learning rate 0.01 for the first 150 epochs and 0.001 for the last 10 epochs, batch size 16, and on-the-fly augmentation including per-object perturbation, global scaling, and global rotation [1711.06396]. Per-object perturbation rotates each ground-truth box by \(\Delta\theta \sim \mathcal{U}[-\pi/10,\pi/10]\) and translates it by Gaussian offsets, with collision checks to revert invalid perturbations; global scaling uses \(s \sim \mathcal{U}[0.95,1.05]\), and global rotation uses \(\Delta\theta \sim \mathcal{U}[-\pi/4,\pi/4]\) [1711.06396]. These details became foundational for later voxel-based detectors.

Subsequent work showed that the original classification branch remains sensitive to foreground–background imbalance. In a focused study of one-stage 3D detectors, replacing VoxelNet’s BCE classification with focal loss improved KITTI validation AP for the car class under several \(\gamma\) values, with last-model gains up to \(+9.1\) AP on hard 3D detection at \(\gamma=0.5\) and best-model gains of \(+4.66\) AP on easy 3D detection at \(\gamma=0.2\) [1809.06065]. That study reports roughly 70k anchors per frame but fewer than 30 positives, making the imbalance approximately \(1:2300\), and concludes that in VoxelNet focal loss primarily helps hard positive examples rather than already well-managed negatives [1809.06065]. This suggests that the original anchor-dense formulation is effective but classification calibration is a persistent optimization issue.

## 4. Empirical performance and computational trade-offs

On the KITTI validation set, the original VoxelNet established strong LiDAR-only performance. For BEV detection, it achieved \(89.60/84.81/78.57\) AP for car, \(65.95/61.05/56.98\) for pedestrian, and \(74.41/52.18/50.49\) for cyclist across easy, moderate, and hard splits [1711.06396]. For full 3D detection on validation, it reached \(81.97/65.46/62.85\) AP for car, \(57.86/53.42/48.87\) for pedestrian, and \(67.17/47.65/45.11\) for cyclist [1711.06396]. On the KITTI test benchmark, it reported car 3D AP of \(77.47/65.11/57.73\), car BEV AP of \(89.35/79.26/77.39\), pedestrian 3D AP of \(39.48/33.69/31.51\), pedestrian BEV AP of \(46.13/40.74/38.11\), cyclist 3D AP of \(61.22/48.36/44.37\), and cyclist BEV AP of \(66.70/54.76/50.55\) [1711.06396].

These results were notable because VoxelNet used LiDAR only, yet exceeded several earlier multimodal systems on KITTI validation and test [1711.06396]. The gain was especially pronounced in full 3D detection rather than merely BEV detection, indicating that learned volumetric features improved vertical and geometric localization rather than only planar occupancy cues [1711.06396].

The principal drawback of the original design is computational cost. The original implementation reports approximately 5 ms for voxel input feature computation, 20 ms for feature learning, 170 ms for convolutional middle layers, and 30 ms for the RPN, for a total of about 225 ms per frame on TitanX GPU plus 1.7 GHz CPU, or roughly 4.4 FPS [1711.06396]. The 3D convolutional middle layers dominate latency, and later comparative work repeatedly identifies dense or quasi-dense 3D processing as the architectural bottleneck [1812.05784][2303.11301].

PointPillars made the contrast explicit by characterizing itself as “VoxelNet without voxels in z,” replacing 3D voxelization and 3D convolutions with learned pillar features and a purely 2D backbone. It reports 62 Hz on KITTI test, versus VoxelNet’s 4.4 Hz, while also improving moderate test-set mAP for car, pedestrian, and cyclist in both BEV and 3D detection [1812.05784]. This comparison established the classical trade-off: VoxelNet offers expressive local 3D aggregation but incurs substantial latency, whereas later descendants collapse or sparsify the representation to regain efficiency [1812.05784][2303.11301].

## 5. Fusion, adaptation, and architectural descendants

A large fraction of later 3D detection research treats VoxelNet as a reusable LiDAR backbone. In PointPainting, for example, the publicly released SECOND/VoxelNet implementation is used without changing voxelization, sparse 3D CNN structure, anchor settings, losses, training schedule, or optimizer; only the input point dimensionality is increased by appending semantic segmentation scores from an image network before voxelization [1911.10150]. On KITTI, this changes the original VoxelNet input from 7-D to 11-D and adapts the VFE channels to \((11,32)\) and \((64,128)\) while leaving the rest of the detector unchanged [1911.10150]. Under this sequential fusion scheme, VoxelNet moderate BEV mAP on KITTI validation increases from 71.83 to 73.55, and moderate 3D mAP increases from 67.12 to 68.01, with especially large gains for sparse classes such as cyclist-hard in both BEV and 3D AP [1911.10150]. This established that VoxelNet could benefit from image semantics without dedicated fusion layers.

MVX-Net extends the same intuition to earlier fusion. It builds both PointFusion and VoxelFusion on top of VoxelNet, injecting image features either before the VFE layers or after voxel encoding. On KITTI validation for the car class at IoU 0.7, a baseline LiDAR-only VoxelNet variant reports 79.5/65.7/64.6 AP in 3D, while MVX-Net PointFusion improves these to 85.5/73.3/67.4 and VoxelFusion to 82.3/72.2/66.8 [1904.01649]. At IoU 0.8 the gains are larger, which the paper interprets as improved localization rather than only detection confidence [1904.01649].

VoxelNet also became a host model for domain adaptation. DA-VoxelNet augments a standard VoxelNet detector with adversarial sample-level and anchor-level domain classifiers plus a consistency term, enabling synthetic-to-real adaptation from CARLA-derived LiDAR to KITTI without manual real-world labels [2011.07784]. In the unsupervised setting from their synthetic LIDAR dataset to KITTI, DA-VoxelNet reaches 76.66% BEV mAP and 56.64% 3D mAP on the moderate split with batch size 8, improving over plain VoxelNet trained on the same synthetic source [2011.07784]. This line of work demonstrates that the original voxel-and-anchor formulation remained adaptable to distribution shift once the backbone and detection heads were instrumented with adversarial alignment modules.

A different direction is lightweight redesign. “VoxelNet Light” preserves voxelization, voxel feature learning, 3D CNN backbone, BEV projection, and detection head, but replaces the 2D BEV backbone blocks with depthwise blocks using Group Normalization and H-swish [2503.07133]. In that formulation, LiDAR-only VoxelNet is reduced from 6.4M parameters and 157.64 GFlops to 2.0M parameters and 94.48 GFlops, with inference time decreasing from 105 ms to 99 ms and mAP dropping from 40.51 to 38.88 [2503.07133]. When fused with the lightweight camera backbone NextBEV, the resulting “VoxelNet Light + NextBEV” reaches 43.44 mAP and F1-score 0.4152 with 4.4M parameters and 125 ms inference, compared to the original LiDAR-only VoxelNet’s 40.51 mAP and 0.3107 F1-score [2503.07133].

The most radical descendant is VoxelNeXt, which retains voxelization and sparse voxel features but removes sparse-to-dense conversion, dense proxy heads, and NMS. It predicts objects directly from sparse voxel features, uses sparse height compression rather than dense BEV formation, and reports a better speed–accuracy trade-off than dense-head voxel detectors on nuScenes [2303.11301]. This suggests that the historical limitations of VoxelNet are attributable less to voxelization itself than to its dense head formulation and dense post-processing path.

## 6. Terminological extensions, ambiguities, and broader influence

The term “VoxelNet” later acquired multiple meanings. In cooperative perception, HEAD treats VoxelNet as one supported heterogeneous detector whose classification and regression head feature maps can be shared across vehicles for bandwidth-efficient fusion, without modifying the voxel encoder or backbone [2408.15428]. In patch-refinement systems, two separate VoxelNet-based networks are used as a coarse RPN and a fine local refinement network, allowing high-resolution local voxelization while preserving a voxel-encoded BEV detection paradigm [1910.04093]. In image-based 3D detection, ImGeoNet is explicitly contrasted with point-cloud voxelization by noting that image-lifted voxel volumes lack VoxelNet’s occupancy-induced geometry prior unless a learned surface-probability volume is introduced [2308.09098].

There is also direct terminological drift. In the Pixel-Voxel semantic mapping system, “VoxelNet” denotes a PointNet-style branch over unordered RGB-D point sets rather than the LiDAR voxel-and-3D-convolution detector introduced by Zhou and Tuzel [1710.00132]. In “Generative VoxelNet,” the phrase refers to a 3D convolutional voxel network used as an energy function for volumetric shape synthesis rather than detection [2012.13522]. These usages do not contradict the original VoxelNet, but they indicate that the label eventually denoted a family resemblance—3D voxel or point-set processing with volumetric inductive bias—rather than a single immutable architecture.

The clearest misconception is therefore that “VoxelNet” always names the original 2017 LiDAR detector. In strict usage, it does so: a network that voxelizes a point cloud, learns voxel features through VFE layers, aggregates them with 3D convolutions, and predicts 3D boxes via an anchor-based RPN [1711.06396]. In broader later usage, it can also refer to a voxel-based backbone or even a differently structured 3D branch whose role is to encode spatial geometry [1710.00132][2012.13522]. The literature surrounding PointPainting, MVX-Net, DA-VoxelNet, PointPillars, and VoxelNeXt shows that the original VoxelNet is best understood as the canonical first end-to-end learned voxel detector from raw LiDAR, and as the reference point from which sparse, pillar-based, multimodal, lightweight, and fully sparse successors were defined [1911.10150][1904.01649][1812.05784][2303.11301].

In that sense, VoxelNet’s enduring significance lies in formalizing a complete learned mapping from raw points to oriented 3D detections without hand-crafted LiDAR feature maps. Later work repeatedly preserves that premise even when altering almost every implementation detail around it [1711.06396][2303.11301].

Source: https://www.emergentmind.com/topics/voxelnet