VoxelNet: End-to-End LiDAR 3D Detection

• VoxelNet is an end-to-end LiDAR architecture that learns volumetric features from raw point clouds, bypassing handcrafted feature engineering.
• It employs Voxel Feature Encoding, 3D convolutions, and an anchor-based Region Proposal Network to predict oriented 3D bounding boxes with high accuracy.
• VoxelNet has become a foundational template for subsequent methods, influencing designs like PointPillars and adaptations for multimodal fusion and domain adaptation.

VoxelNet is an end-to-end LiDAR-based 3D object detection architecture that replaces hand-crafted bird’s-eye-view feature engineering with learned volumetric representations derived directly from raw point clouds. In its original formulation, a point cloud is partitioned into equally spaced 3D voxels, each non-empty voxel is encoded by stacked Voxel Feature Encoding (VFE) layers, the resulting sparse voxel tensor is processed by 3D convolutional middle layers, and a region proposal network predicts oriented 3D bounding boxes in a single stage (Zhou et al., 2017). The model was introduced for autonomous driving settings in which sparsity, irregular point distribution, and the cost of manual feature design had limited earlier LiDAR pipelines (Zhou et al., 2017).

1. Origin, problem setting, and design objective

VoxelNet was introduced to address three coupled difficulties in point-cloud detection: the extreme sparsity and non-uniform density of LiDAR scans, the irregular and unordered structure of point sets, and the information bottleneck induced by hand-crafted projections and summary statistics (Zhou et al., 2017). Earlier pipelines commonly projected points into bird’s-eye-view maps or handcrafted voxel features before applying 2D or 3D CNNs, but this discarded fine geometric structure before learning began (Zhou et al., 2017). VoxelNet’s central claim was that feature extraction and box prediction could be unified in a single trainable network operating directly on points grouped by voxels, eliminating manual feature engineering (Zhou et al., 2017).

The original system targets 3D object detection and orientation estimation for autonomous driving, with evaluation on KITTI for cars, pedestrians, and cyclists (Zhou et al., 2017). For car detection, the LiDAR region of interest is $Z \in [-3,1]$, $Y \in [-40,40]$, and $X \in [0,70.4]$ meters, voxelized with $v_D=0.4$, $v_H=0.2$, and $v_W=0.2$ meters, yielding a grid of $10 \times 400 \times 352$ cells (Zhou et al., 2017). The model processes only non-empty voxels, which is essential because more than 90% of the grid is empty in typical scans (Zhou et al., 2017).

A recurring later observation is that “VoxelNet” became both a specific architecture and a template for voxel-based 3D detection. Subsequent work uses it as a baseline detector, a fusion backbone, a domain-adaptation host, or a lightweight BEV backbone, but these later systems generally preserve the original macro-structure of voxelization, voxel feature learning, spatial aggregation, and anchor-based detection unless explicitly redesigned (Vora et al., 2019, Yang et al., 2020, Pederiva et al., 10 Mar 2025).

2. Architectural composition

VoxelNet has three principal components: a feature learning network operating on voxelized points, convolutional middle layers that aggregate spatial context in 3D, and a region proposal network operating on a bird’s-eye-view feature map (Zhou et al., 2017).

The input point cloud is $$\mathbf{M}=\{\mathbf{p}_i=[x_i,y_i,z_i,r_i]^T\}_{i=1}^N$$, where $r_i$ is reflectance. Each point is assigned to a voxel, and each voxel keeps at most $T$ points; for cars, $Y \in [-40,40]$0, while for pedestrians and cyclists $Y \in [-40,40]$1 is used to better capture small-object shape (Zhou et al., 2017). For a non-empty voxel with $Y \in [-40,40]$2 points, VoxelNet computes the centroid

$Y \in [-40,40]$3

and augments each point as

$Y \in [-40,40]$4

These augmented point features are the input to the VFE layers (Zhou et al., 2017).

A VFE layer applies a point-wise fully connected network $Y \in [-40,40]$5 to each point,

$Y \in [-40,40]$6

computes a voxel-local aggregated feature by element-wise max pooling,

$Y \in [-40,40]$7

and concatenates point-wise and aggregated features,

$Y \in [-40,40]$8

This permits inter-point interaction within each voxel while retaining point-level detail (Zhou et al., 2017). For cars, the original stack uses $Y \in [-40,40]$9 followed by $X \in [0,70.4]$0, then an FCN to 128 channels and voxel-wise max pooling, producing a sparse tensor of size $X \in [0,70.4]$1 (Zhou et al., 2017).

The convolutional middle layers are 3D convolutions with BN and ReLU. For cars, the sequence is $X \in [0,70.4]$2, then $X \in [0,70.4]$3, then $X \in [0,70.4]$4, reducing the tensor to $X \in [0,70.4]$5, which is reshaped into a BEV feature map of $X \in [0,70.4]$6 (Zhou et al., 2017). These layers aggregate local spatial context while collapsing the vertical axis into a representation suitable for planar detection.

The RPN is a multi-scale 2D convolutional detector operating on the BEV map. It predicts classification logits and seven regression parameters per anchor, using three convolutional blocks with downsampling and upsampling before concatenation into a high-resolution feature map (Zhou et al., 2017). The original paper parameterizes a 3D box by center $X \in [0,70.4]$7, size $X \in [0,70.4]$8, and yaw $X \in [0,70.4]$9, with anchor-normalized regression targets

$v_D=0.4$0

$v_D=0.4$1

where $v_D=0.4$2 (Zhou et al., 2017).

3. Training protocol, anchors, and optimization behavior

VoxelNet is trained with a joint classification-and-regression objective. Let $v_D=0.4$3 denote positive anchors and $v_D=0.4$4 negative anchors. The original loss is

$v_D=0.4$5

with binary cross-entropy classification and Smooth L1 regression; for cars, $v_D=0.4$6 and $v_D=0.4$7 (Zhou et al., 2017). Positive and negative assignment is IoU-based in BEV. For cars, anchors are positive if they have the highest IoU with a ground-truth box or IoU $v_D=0.4$8, negative if IoU $v_D=0.4$9, and ignored otherwise (Zhou et al., 2017). Car anchors use size $v_H=0.2$0 m, $v_H=0.2$1 m, $v_H=0.2$2 m, center height $v_H=0.2$3 m, and two yaw orientations $v_H=0.2$4 and $v_H=0.2$5 (Zhou et al., 2017).

The original training schedule on KITTI uses SGD with learning rate 0.01 for the first 150 epochs and 0.001 for the last 10 epochs, batch size 16, and on-the-fly augmentation including per-object perturbation, global scaling, and global rotation (Zhou et al., 2017). Per-object perturbation rotates each ground-truth box by $v_H=0.2$6 and translates it by Gaussian offsets, with collision checks to revert invalid perturbations; global scaling uses $v_H=0.2$7, and global rotation uses $v_H=0.2$8 (Zhou et al., 2017). These details became foundational for later voxel-based detectors.

Subsequent work showed that the original classification branch remains sensitive to foreground–background imbalance. In a focused study of one-stage 3D detectors, replacing VoxelNet’s BCE classification with focal loss improved KITTI validation AP for the car class under several $v_H=0.2$9 values, with last-model gains up to $v_W=0.2$0 AP on hard 3D detection at $v_W=0.2$1 and best-model gains of $v_W=0.2$2 AP on easy 3D detection at $v_W=0.2$3 (Yun et al., 2018). That study reports roughly 70k anchors per frame but fewer than 30 positives, making the imbalance approximately $v_W=0.2$4, and concludes that in VoxelNet focal loss primarily helps hard positive examples rather than already well-managed negatives (Yun et al., 2018). This suggests that the original anchor-dense formulation is effective but classification calibration is a persistent optimization issue.

4. Empirical performance and computational trade-offs

On the KITTI validation set, the original VoxelNet established strong LiDAR-only performance. For BEV detection, it achieved $v_W=0.2$5 AP for car, $v_W=0.2$6 for pedestrian, and $v_W=0.2$7 for cyclist across easy, moderate, and hard splits (Zhou et al., 2017). For full 3D detection on validation, it reached $v_W=0.2$8 AP for car, $v_W=0.2$9 for pedestrian, and $10 \times 400 \times 352$0 for cyclist (Zhou et al., 2017). On the KITTI test benchmark, it reported car 3D AP of $10 \times 400 \times 352$1, car BEV AP of $10 \times 400 \times 352$2, pedestrian 3D AP of $10 \times 400 \times 352$3, pedestrian BEV AP of $10 \times 400 \times 352$4, cyclist 3D AP of $10 \times 400 \times 352$5, and cyclist BEV AP of $10 \times 400 \times 352$6 (Zhou et al., 2017).

These results were notable because VoxelNet used LiDAR only, yet exceeded several earlier multimodal systems on KITTI validation and test (Zhou et al., 2017). The gain was especially pronounced in full 3D detection rather than merely BEV detection, indicating that learned volumetric features improved vertical and geometric localization rather than only planar occupancy cues (Zhou et al., 2017).

The principal drawback of the original design is computational cost. The original implementation reports approximately 5 ms for voxel input feature computation, 20 ms for feature learning, 170 ms for convolutional middle layers, and 30 ms for the RPN, for a total of about 225 ms per frame on TitanX GPU plus 1.7 GHz CPU, or roughly 4.4 FPS (Zhou et al., 2017). The 3D convolutional middle layers dominate latency, and later comparative work repeatedly identifies dense or quasi-dense 3D processing as the architectural bottleneck (Lang et al., 2018, Chen et al., 2023).

PointPillars made the contrast explicit by characterizing itself as “VoxelNet without voxels in z,” replacing 3D voxelization and 3D convolutions with learned pillar features and a purely 2D backbone. It reports 62 Hz on KITTI test, versus VoxelNet’s 4.4 Hz, while also improving moderate test-set mAP for car, pedestrian, and cyclist in both BEV and 3D detection (Lang et al., 2018). This comparison established the classical trade-off: VoxelNet offers expressive local 3D aggregation but incurs substantial latency, whereas later descendants collapse or sparsify the representation to regain efficiency (Lang et al., 2018, Chen et al., 2023).

5. Fusion, adaptation, and architectural descendants

A large fraction of later 3D detection research treats VoxelNet as a reusable LiDAR backbone. In PointPainting, for example, the publicly released SECOND/VoxelNet implementation is used without changing voxelization, sparse 3D CNN structure, anchor settings, losses, training schedule, or optimizer; only the input point dimensionality is increased by appending semantic segmentation scores from an image network before voxelization (Vora et al., 2019). On KITTI, this changes the original VoxelNet input from 7-D to 11-D and adapts the VFE channels to $10 \times 400 \times 352$7 and $10 \times 400 \times 352$8 while leaving the rest of the detector unchanged (Vora et al., 2019). Under this sequential fusion scheme, VoxelNet moderate BEV mAP on KITTI validation increases from 71.83 to 73.55, and moderate 3D mAP increases from 67.12 to 68.01, with especially large gains for sparse classes such as cyclist-hard in both BEV and 3D AP (Vora et al., 2019). This established that VoxelNet could benefit from image semantics without dedicated fusion layers.

MVX-Net extends the same intuition to earlier fusion. It builds both PointFusion and VoxelFusion on top of VoxelNet, injecting image features either before the VFE layers or after voxel encoding. On KITTI validation for the car class at IoU 0.7, a baseline LiDAR-only VoxelNet variant reports 79.5/65.7/64.6 AP in 3D, while MVX-Net PointFusion improves these to 85.5/73.3/67.4 and VoxelFusion to 82.3/72.2/66.8 (Sindagi et al., 2019). At IoU 0.8 the gains are larger, which the paper interprets as improved localization rather than only detection confidence (Sindagi et al., 2019).

VoxelNet also became a host model for domain adaptation. DA-VoxelNet augments a standard VoxelNet detector with adversarial sample-level and anchor-level domain classifiers plus a consistency term, enabling synthetic-to-real adaptation from CARLA-derived LiDAR to KITTI without manual real-world labels (Yang et al., 2020). In the unsupervised setting from their synthetic LIDAR dataset to KITTI, DA-VoxelNet reaches 76.66% BEV mAP and 56.64% 3D mAP on the moderate split with batch size 8, improving over plain VoxelNet trained on the same synthetic source (Yang et al., 2020). This line of work demonstrates that the original voxel-and-anchor formulation remained adaptable to distribution shift once the backbone and detection heads were instrumented with adversarial alignment modules.

A different direction is lightweight redesign. “VoxelNet Light” preserves voxelization, voxel feature learning, 3D CNN backbone, BEV projection, and detection head, but replaces the 2D BEV backbone blocks with depthwise blocks using Group Normalization and H-swish (Pederiva et al., 10 Mar 2025). In that formulation, LiDAR-only VoxelNet is reduced from 6.4M parameters and 157.64 GFlops to 2.0M parameters and 94.48 GFlops, with inference time decreasing from 105 ms to 99 ms and mAP dropping from 40.51 to 38.88 (Pederiva et al., 10 Mar 2025). When fused with the lightweight camera backbone NextBEV, the resulting “VoxelNet Light + NextBEV” reaches 43.44 mAP and F1-score 0.4152 with 4.4M parameters and 125 ms inference, compared to the original LiDAR-only VoxelNet’s 40.51 mAP and 0.3107 F1-score (Pederiva et al., 10 Mar 2025).

The most radical descendant is VoxelNeXt, which retains voxelization and sparse voxel features but removes sparse-to-dense conversion, dense proxy heads, and NMS. It predicts objects directly from sparse voxel features, uses sparse height compression rather than dense BEV formation, and reports a better speed–accuracy trade-off than dense-head voxel detectors on nuScenes (Chen et al., 2023). This suggests that the historical limitations of VoxelNet are attributable less to voxelization itself than to its dense head formulation and dense post-processing path.

6. Terminological extensions, ambiguities, and broader influence

The term “VoxelNet” later acquired multiple meanings. In cooperative perception, HEAD treats VoxelNet as one supported heterogeneous detector whose classification and regression head feature maps can be shared across vehicles for bandwidth-efficient fusion, without modifying the voxel encoder or backbone (Qu et al., 2024). In patch-refinement systems, two separate VoxelNet-based networks are used as a coarse RPN and a fine local refinement network, allowing high-resolution local voxelization while preserving a voxel-encoded BEV detection paradigm (Lehner et al., 2019). In image-based 3D detection, ImGeoNet is explicitly contrasted with point-cloud voxelization by noting that image-lifted voxel volumes lack VoxelNet’s occupancy-induced geometry prior unless a learned surface-probability volume is introduced (Tu et al., 2023).

There is also direct terminological drift. In the Pixel-Voxel semantic mapping system, “VoxelNet” denotes a PointNet-style branch over unordered RGB-D point sets rather than the LiDAR voxel-and-3D-convolution detector introduced by Zhou and Tuzel (Zhao et al., 2017). In “Generative VoxelNet,” the phrase refers to a 3D convolutional voxel network used as an energy function for volumetric shape synthesis rather than detection (Xie et al., 2020). These usages do not contradict the original VoxelNet, but they indicate that the label eventually denoted a family resemblance—3D voxel or point-set processing with volumetric inductive bias—rather than a single immutable architecture.

The clearest misconception is therefore that “VoxelNet” always names the original 2017 LiDAR detector. In strict usage, it does so: a network that voxelizes a point cloud, learns voxel features through VFE layers, aggregates them with 3D convolutions, and predicts 3D boxes via an anchor-based RPN (Zhou et al., 2017). In broader later usage, it can also refer to a voxel-based backbone or even a differently structured 3D branch whose role is to encode spatial geometry (Zhao et al., 2017, Xie et al., 2020). The literature surrounding PointPainting, MVX-Net, DA-VoxelNet, PointPillars, and VoxelNeXt shows that the original VoxelNet is best understood as the canonical first end-to-end learned voxel detector from raw LiDAR, and as the reference point from which sparse, pillar-based, multimodal, lightweight, and fully sparse successors were defined (Vora et al., 2019, Sindagi et al., 2019, Lang et al., 2018, Chen et al., 2023).

In that sense, VoxelNet’s enduring significance lies in formalizing a complete learned mapping from raw points to oriented 3D detections without hand-crafted LiDAR feature maps. Later work repeatedly preserves that premise even when altering almost every implementation detail around it (Zhou et al., 2017, Chen et al., 2023).