LidarNAS: Unified NAS for 3D LiDAR Backbones

• LidarNAS is a neural architecture search framework that factorizes 3D backbones into view transforms and neural layers.
• It unifies diverse representations—point, voxel, pillar, and perspective—to improve LiDAR detection on the Waymo Open Dataset.
• NAS evolution in LidarNAS shows that macro-level design changes in views and transforms drive significant AP gains while keeping latency in check.

Searching arXiv for the specified paper and closely related context papers. arXiv search query: (Liu et al., 2022) LidarNAS is a framework and NAS system for LiDAR / 3D point cloud backbones that does two things: it unifies the diverse family of 3D architectures into a single conceptual and modular representation, and it searches that unified space, including both macro-level view choices and micro-level layer details, to automatically discover high-performing LiDAR detection architectures on the Waymo Open Dataset (Liu et al., 2022). Its central claim is that 3D point-cloud backbones can be factorized into a sequence of view transforms and neural layers, so that point-based, voxel-based, BEV/pillar, range-view, and multi-view fusion designs can be expressed within one formalism and then explored by evolutionary neural architecture search.

1. Problem setting and design motivation

LidarNAS is motivated by the observation that existing neural architectures for 3D point clouds exhibit a large variety in their designs, including the views considered, the format of the neural features, and the neural operations used. A single 3D scene can be represented as unordered points $(x,y,z)$, discretized voxels in 3D, top-down pillars / BEV in 2D, or perspective / range views indexed by azimuth and elevation. Each view induces different neighborhoods and therefore different natural operations, such as point MLPs, 2D convolutions, 3D convolutions, or sparse convolutions (Liu et al., 2022).

The framework is also motivated by the sparsity and irregularity of LiDAR data. Voxels and pillars are mostly empty, and range images contain invalid or “infinite” depth pixels. This makes dense convolutions expensive or ineffective in some settings and motivates specialized sparse operators. The resulting ecosystem includes PointNet, PointNet++, voxel networks, PointPillars, LaserNet, RSN, MVF, SPVCNN, and related families, but without a single canonical family comparable to 2D CNNs for images.

LidarNAS formulates this diversity as a two-level hierarchy. The macro-level concerns the choice and sequencing of views and transforms, such as whether to use point, pillar, voxel, or perspective representations and where to fuse them. The micro-level concerns the neural layers within each view, including channels, depth, kernel shapes, and related hyperparameters. The lack of a unified conceptual lens makes it difficult to compare these design families fairly and difficult to search over both levels simultaneously. LidarNAS addresses both issues by making view choice, transform connectivity, and layer design part of one modular search space.

2. Unified representation: views, formats, transforms, and stages

The central idea in LidarNAS is to factorize a 3D backbone into a sequence of view transforms interspersed with neural layers. The framework defines four canonical views and, where applicable, dense and sparse feature formats.

View	Format(s)	Representation
Point view	Sparse	$\mathbf{F} \in \mathbb{R}^{N \times C}$, $\mathbf{P} \in \mathbb{R}^{N \times 3}$
Pillar view	Dense or sparse	Dense: $$\mathbf{F} \in \mathbb{R}^{B \times X \times Y \times C}$$; Sparse: $\mathbf{F} \in \mathbb{R}^{N \times C}$, $\mathbf{I} \in \mathbb{R}^{N \times 3}$
Voxel view	Sparse	$\mathbf{F} \in \mathbb{R}^{N \times C}$, $\mathbf{I} \in \mathbb{R}^{N \times 4}$
Perspective view	Dense or sparse	Dense: $$\mathbf{F} \in \mathbb{R}^{B \times H \times W \times C}$$, $$\mathbf{P} \in \mathbb{R}^{B \times H \times W \times 3}$$; Sparse: $\mathbf{F} \in \mathbb{R}^{N \times C}$0, $\mathbf{F} \in \mathbb{R}^{N \times C}$1

Each view defines how neighborhoods are measured: radius-based neighborhoods in $\mathbf{F} \in \mathbb{R}^{N \times C}$2 for points, grid neighborhoods in $\mathbf{F} \in \mathbb{R}^{N \times C}$3 for voxels, 2D neighborhoods in $\mathbf{F} \in \mathbb{R}^{N \times C}$4 for pillars, and 2D neighborhoods in pixel coordinates $\mathbf{F} \in \mathbb{R}^{N \times C}$5 for perspective view. This organization gives a common vocabulary for designs that had previously appeared disjoint.

A view transform is a mapping

$\mathbf{F} \in \mathbb{R}^{N \times C}$6

Examples given in the framework include identity, densification, voxelization, devoxelization, and projection. For instance, point-to-voxel transformation is written as

$\mathbf{F} \in \mathbb{R}^{N \times C}$7

while point-to-perspective projection maps 3D coordinates to angular or image coordinates. The paper states that there are up to $\mathbf{F} \in \mathbb{R}^{N \times C}$8 possible transforms once view-format states are enumerated, and that implementation-wise the framework covers 31 of these, excluding pillar$\mathbf{F} \in \mathbb{R}^{N \times C}$9voxel and some voxel$\mathbf{P} \in \mathbb{R}^{N \times 3}$0others (Liu et al., 2022).

A neural layer is a learned operator

$\mathbf{P} \in \mathbb{R}^{N \times 3}$1

with the constraint that input and output remain in the same view and format. For point view, layers are MLP stacks of Dense, normalization, and ReLU. For 2D dense views, the layer family is a U-Net with 2D residual blocks and up to 5 scales with channels $\mathbf{P} \in \mathbb{R}^{N \times 3}$2. For 2D sparse views, the framework uses U-Nets with $\mathbf{P} \in \mathbb{R}^{N \times 3}$3 sparse convolutions, up to 3 downsample and upsample scales, block counts down $\mathbf{P} \in \mathbb{R}^{N \times 3}$4 and up $\mathbf{P} \in \mathbb{R}^{N \times 3}$5, and a single channel size $\mathbf{P} \in \mathbb{R}^{N \times 3}$6 shared across scales. For 3D sparse voxel view, the U-Net uses sparse 3D convolutions with kernel either $\mathbf{P} \in \mathbb{R}^{N \times 3}$7 or $\mathbf{P} \in \mathbb{R}^{N \times 3}$8, and stride either $\mathbf{P} \in \mathbb{R}^{N \times 3}$9 or $$\mathbf{F} \in \mathbb{R}^{B \times X \times Y \times C}$$0.

The backbone is organized as a sequence of stages. A stage contains a set of view transforms from the previous stage into possible current-stage views and, for each present view and format, a neural layer. Architectures correspond to connected subsets of a stage trellis, so they may be purely sequential pipelines or multi-branch graphs with several views simultaneously present. This formalism is used to express PointNet and PointNet++, VoxelNet and SECOND-style voxel detectors, PointPillars, range-view detectors such as LaserNet, RSN, and multi-view fusion systems such as MVF. In the paper, RSN, PointPillars-like, LaserNet-like, and MVF++-like models are explicitly re-implemented within this framework.

3. Search space and evolutionary optimization

LidarNAS turns the unified formalism into a concrete NAS space over 3D LiDAR backbones. The search space includes four views—point, voxel, pillar, perspective—and six representational states: point (sparse), voxel (sparse), pillar (dense), pillar (sparse), perspective (dense), and perspective (sparse). Among the 36 possible view-to-view transforms, the implemented search space uses 31, excluding pillar$$\mathbf{F} \in \mathbb{R}^{B \times X \times Y \times C}$$1voxel and limiting voxel-to-other-view transforms to voxel$$\mathbf{F} \in \mathbb{R}^{B \times X \times Y \times C}$$2pillar. Available transforms include identity, sparse–dense conversions where applicable, point$$\mathbf{F} \in \mathbb{R}^{B \times X \times Y \times C}$$3pillar, point$$\mathbf{F} \in \mathbb{R}^{B \times X \times Y \times C}$$4voxel, point$$\mathbf{F} \in \mathbb{R}^{B \times X \times Y \times C}$$5perspective, perspective$$\mathbf{F} \in \mathbb{R}^{B \times X \times Y \times C}$$6pillar, perspective$$\mathbf{F} \in \mathbb{R}^{B \times X \times Y \times C}$$7voxel, voxel$$\mathbf{F} \in \mathbb{R}^{B \times X \times Y \times C}$$8pillar, and pillar–perspective variants (Liu et al., 2022).

The search space is stage-based with $$\mathbf{F} \in \mathbb{R}^{B \times X \times Y \times C}$$9. Each stage may contain multiple views, but the last stage has only one view, the detection view. There is also an option to allow foreground segmentation in the first stage once a perspective branch appears, in the style of RSN. When multiple transforms feed the same view in a given stage, their outputs are merged by concatenation in the paper.

Searchable layer parameters depend on the view and format. Point layers search over the number of MLP layers and hidden dimension $\mathbf{F} \in \mathbb{R}^{N \times C}$0. Dense 2D pillar and perspective layers search over the number of U-Net scales and base channels $\mathbf{F} \in \mathbb{R}^{N \times C}$1. Sparse 2D layers search over the number of downsampling and upsampling scales and channels $\mathbf{F} \in \mathbb{R}^{N \times C}$2. Sparse 3D voxel layers search over kernel shape and stride per scale, the number of scales, and channel count $\mathbf{F} \in \mathbb{R}^{N \times C}$3. The paper states that these layer specifics mirror RSN’s backbone design while allowing NAS to vary macro structure and scaling.

The NAS algorithm is regularized evolutionary NAS. Each individual encodes, for every stage, the views present, format choice, transform connections, pillar or voxel sizes, channels $\mathbf{F} \in \mathbb{R}^{N \times C}$4, and layer progression parameters such as numbers of scales or MLP layers. Mutation randomly selects a stage and one of six mutation types: add a view, remove a view, switch the view, adjust pillar or voxel size by $\mathbf{F} \in \mathbb{R}^{N \times C}$5 or $\mathbf{F} \in \mathbb{R}^{N \times C}$6, adjust channels by $\mathbf{F} \in \mathbb{R}^{N \times C}$7 or $\mathbf{F} \in \mathbb{R}^{N \times C}$8, or adjust layer progression. The paper explicitly interprets mutations $\mathbf{F} \in \mathbb{R}^{N \times C}$9–$\mathbf{I} \in \mathbb{R}^{N \times 3}$0 as changing the transform or macro part and mutations $\mathbf{I} \in \mathbb{R}^{N \times 3}$1–$\mathbf{I} \in \mathbb{R}^{N \times 3}$2 as changing the layer or micro part.

Selection uses a population of size $\mathbf{I} \in \mathbb{R}^{N \times 3}$3. At each step, a tournament of $\mathbf{I} \in \mathbb{R}^{N \times 3}$4 architectures is sampled, the best is chosen to produce a child via mutation, the child is trained and evaluated, the child is added to the population, and the oldest architecture is removed. During search, each candidate is trained for $\mathbf{I} \in \mathbb{R}^{N \times 3}$5k steps, approximately $\mathbf{I} \in \mathbb{R}^{N \times 3}$6 of full training, on V100 GPUs with batch size $\mathbf{I} \in \mathbb{R}^{N \times 3}$7 GPUs, cosine learning rate, single-frame input, and a common anchor-free detection head. The primary metric is Waymo validation L1 3D AP; the secondary metric is V100 GPU latency measured on a batch after 11k steps. The fitness function is

$\mathbf{I} \in \mathbb{R}^{N \times 3}$8

After search, selected architectures are retrained for $\mathbf{I} \in \mathbb{R}^{N \times 3}$9k steps with larger per-GPU batch size, from $\mathbf{F} \in \mathbb{R}^{N \times C}$0 to $\mathbf{F} \in \mathbb{R}^{N \times C}$1, using the same head. The reported total search cost is approximately $\mathbf{F} \in \mathbb{R}^{N \times C}$2 GPU-days, about $\mathbf{F} \in \mathbb{R}^{N \times C}$3 training a single RSN baseline.

4. Empirical setting and results on the Waymo Open Dataset

The empirical study is conducted on the Waymo Open Dataset, which contains 1150 LiDAR sequences split into 798 train, 202 validation, and 150 test, with 20 seconds per sequence at 10 FPS. Range images are provided, making perspective view natural in this setting. The task is single-frame 3D object detection for the Vehicle and Pedestrian classes, evaluated using official Waymo L1 3D AP and L1 BEV AP, where L1 denotes the easy difficulty level (Liu et al., 2022).

Before NAS, the framework re-implements RSN, PointPillars-like, LaserNet-like, and MVF++-like baselines. The stated purpose is to confirm the correctness of transforms and layer modules and to show that the modular implementation matches or closely approximates reported performance.

For vehicles (single-frame), the simplified table in the data reports the following L1 3D AP / BEV AP values: LaserNet $\mathbf{F} \in \mathbb{R}^{N \times C}$4, PointPillars $\mathbf{F} \in \mathbb{R}^{N \times C}$5, a pillar-based ECCV’20 model $\mathbf{F} \in \mathbb{R}^{N \times C}$6, MVF++ $\mathbf{F} \in \mathbb{R}^{N \times C}$7, RSN baseline $\mathbf{F} \in \mathbb{R}^{N \times C}$8, LidarNASNet-R $\mathbf{F} \in \mathbb{R}^{N \times C}$9, and LidarNASNet-P $\mathbf{I} \in \mathbb{R}^{N \times 4}$0. For pedestrians (single-frame), RSN reports $\mathbf{I} \in \mathbb{R}^{N \times 4}$1, while LidarNASNet-R reports $\mathbf{I} \in \mathbb{R}^{N \times 4}$2.

Latency values on V100 are also reported. For the vehicle model, RSN is $\mathbf{I} \in \mathbb{R}^{N \times 4}$3 ms and LidarNASNet-R is $\mathbf{I} \in \mathbb{R}^{N \times 4}$4 ms. For the pedestrian model, RSN is $\mathbf{I} \in \mathbb{R}^{N \times 4}$5 ms and LidarNASNet-R is $\mathbf{I} \in \mathbb{R}^{N \times 4}$6 ms. The paper further reports that RSN to LidarNASNet-R yields $\mathbf{I} \in \mathbb{R}^{N \times 4}$7 3D AP and $\mathbf{I} \in \mathbb{R}^{N \times 4}$8 BEV AP for vehicles, and $\mathbf{I} \in \mathbb{R}^{N \times 4}$9 3D AP and $$\mathbf{F} \in \mathbb{R}^{B \times H \times W \times C}$$0 BEV AP for pedestrians. For the PointPillars-like baseline, the corresponding vehicle 3D AP improves from $$\mathbf{F} \in \mathbb{R}^{B \times H \times W \times C}$$1 to $$\mathbf{F} \in \mathbb{R}^{B \times H \times W \times C}$$2, and BEV AP from $$\mathbf{F} \in \mathbb{R}^{B \times H \times W \times C}$$3 to $$\mathbf{F} \in \mathbb{R}^{B \times H \times W \times C}$$4.

These results are presented as evidence that the search space can produce state-of-the-art LiDAR backbones while preserving latency in roughly the same regime. A more specific comparison in the data states that manually scaling channels in RSN to match AP increases latency to $$\mathbf{F} \in \mathbb{R}^{B \times H \times W \times C}$$5 ms, whereas the searched design reaches higher AP with $$\mathbf{F} \in \mathbb{R}^{B \times H \times W \times C}$$6 ms latency. This suggests that the discovered architectural changes are not reducible to simple width scaling.

5. Discovered architectures and analytical findings

The search discovers two principal models: LidarNASNet-P, obtained by evolving a PointPillars-like starting point, and LidarNASNet-R, obtained by evolving RSN for both vehicle and pedestrian detection (Liu et al., 2022).

For LidarNASNet-P, the macro-level changes are explicit. The architecture adds a perspective branch before pillar encoding: instead of a direct point$$\mathbf{F} \in \mathbb{R}^{B \times H \times W \times C}$$7pillar$$\mathbf{F} \in \mathbb{R}^{B \times H \times W \times C}$$82D U-Net BEV pipeline, it first operates on the range image with a 2D U-Net, then transforms perspective features to pillar view and applies the pillar U-Net. The paper states that this enriches BEV features with high-resolution range-view context. It also increases pillar resolution from $$\mathbf{F} \in \mathbb{R}^{B \times H \times W \times C}$$9 m to $$\mathbf{P} \in \mathbb{R}^{B \times H \times W \times 3}$$0 m. The data attributes the gain from $$\mathbf{P} \in \mathbb{R}^{B \times H \times W \times 3}$$1 to $$\mathbf{P} \in \mathbb{R}^{B \times H \times W \times 3}$$2 3D AP to the added range-view U-Net and the further gain to $$\mathbf{P} \in \mathbb{R}^{B \times H \times W \times 3}$$3 to the finer pillar resolution.

For LidarNASNet-R, the discovered macro-level concept is stated to be the same for both vehicle and pedestrian. Relative to RSN’s original structure—stage 1 perspective sparse 2D U-Net for foreground segmentation and stages 2–3 voxel sparse 3D U-Net for detection—NAS adds a pillar branch in the first stage, parallel to perspective, and then fuses pillar and perspective views downstream. All four views are used somewhere: point, pillar, voxel, and perspective. The pillar branch is described as providing top-down context complementary to perspective. The channel allocation is also rebalanced: for vehicles, perspective channels are halved from $$\mathbf{P} \in \mathbb{R}^{B \times H \times W \times 3}$$4 to $$\mathbf{P} \in \mathbb{R}^{B \times H \times W \times 3}$$5 while the pillar branch uses $$\mathbf{P} \in \mathbb{R}^{B \times H \times W \times 3}$$6; for pedestrians, perspective channels are reduced from $$\mathbf{P} \in \mathbb{R}^{B \times H \times W \times 3}$$7 to $$\mathbf{P} \in \mathbb{R}^{B \times H \times W \times 3}$$8 to keep compute in check and allocate capacity to pillar and voxel.

Several analyses quantify why this macro-level search matters. Randomly sampled architectures have lower average AP and higher variance in AP and latency than NAS results, which the paper uses to argue that the space is nontrivial and that warm-starting from strong baselines helps. A direct comparison of mutations shows that AP standard deviation is approximately $$\mathbf{P} \in \mathbb{R}^{B \times H \times W \times 3}$$9 for layer-only mutations but approximately $\mathbf{F} \in \mathbb{R}^{N \times C}$00 for transform-related mutations. This is used to support the claim that macro-level changes in views and transforms cause larger performance shifts than micro-level channel or depth adjustments.

A linear regression from stage-wise view presence to AP yields a further pattern: later stages, especially Stage 3, have stronger influence on AP than earlier stages; in the last stage, voxel and pillar views have positive coefficients, while perspective view has a negative coefficient. The paper interprets this as indicating that detecting in a top-down view is generally more effective for 3D object detection, while perspective view is beneficial for feature extraction but less suitable as the final detection view unless specialized methods are used.

The latency analysis also addresses a common misconception that sparse formats are uniformly advantageous. For perspective view, the reported regression coefficients are $\mathbf{F} \in \mathbb{R}^{N \times C}$01 for empty, dense, and sparse branches, indicating that more sparse branches increase latency relative to dense. For pillar view, the reported coefficients are $\mathbf{F} \in \mathbb{R}^{N \times C}$02, indicating that more sparse branches decrease latency relative to dense. The paper interprets this as showing that sparse operations help when the underlying view is truly sparse, such as pillars, but not always in dense views such as range images.

6. Relation to prior work, limitations, and implications

LidarNAS is positioned against two kinds of prior work. First, NAS for 3D medical imaging, including V-NAS, SegNAS3D, and C2FNAS, treats volumes as dense 3D images with less sparsity and fixed grids, thereby simplifying the search space. Second, NAS for 3D scenes and point clouds, including SPV NAS and LC-NAS, searches efficient architectures under more constrained macro-structures. The specific distinction drawn in the paper is that earlier point-cloud NAS generally fixes the macro structure—such as which views are present and how point–voxel fusion is organized—and searches mainly micro-level hyperparameters and operations. LidarNAS differs by explicitly modeling the macro-level over views and transforms, including sequence versus parallelism, adding or removing views, and changing voxel or pillar sizes (Liu et al., 2022).

Relative to hand-crafted LiDAR detectors such as RSN, MVF, PointPillars, and PV-RCNN, the framework is presented as both a search method and a conceptual language. In that language, architectures are variations of “find neighborhoods, aggregate,” organized through views, transforms, layers, and stages. This suggests a unifying interpretation of sequential pipelines and multi-view fusion models as instances of the same underlying compositional scheme.

The paper also states several limitations. The search space has limited layer diversity: it includes point MLPs and 2D or 3D U-Nets with sparse or dense convolutions, but does not yet include Transformer-based layers, attention mechanisms, graph convolutions, or more exotic operators. The experiments are single-frame only, whereas multi-frame temporal fusion is important for autonomous driving and would introduce both memory pressure and additional choices about where temporal fusion should occur. The method also has a nontrivial search cost, reported as about $\mathbf{F} \in \mathbb{R}^{N \times C}$03 GPU-days, which is moderate in industrial settings but significant for many research labs. Finally, the demonstration is specific to 3D detection; segmentation, tracking, scene flow, and other tasks may require different heads or different optimal view combinations.

A plausible implication is that the most durable contribution of LidarNAS is not only the specific searched backbones, but the view-transform-layer factorization itself. Within the evidence given, the strongest recurring pattern is that range-view processing is useful early, top-down detection views are useful late, and macro-level view selection and connectivity have larger impact than incremental micro-level tuning. The paper’s reported discovery that NAS tends to find the same macro-level architecture concept for both vehicle and pedestrian classes suggests that this design pattern is robust across classes with different scales and spatial distributions.