VoxelFormer: Voxel-Centered Transformer Models

Updated 10 July 2026

VoxelFormer is a family of voxel-based transformer models used for diverse 3D vision tasks, defining voxel queries, BEV feature generation, and sparse tokenization.
It leverages sparse-to-dense reasoning and dual-view attention to effectively integrate image, LiDAR, and context features for improved 3D scene understanding.
Different variants address specific challenges, optimizing efficiency and accuracy across applications from semantic scene completion to multi-modal medical imaging and fMRI decoding.

VoxelFormer denotes an overlapping set of architectures built around voxelized or voxel-query representations, rather than a single canonical model. In 3D vision, the name most directly identifies a transformer-based multi-view camera detector that replaces BEVFormer’s spatial cross-attention with dual-view attention (Li et al., 2023), but closely related usage also includes sparse voxel transformers for camera-based semantic scene completion (Li et al., 2023), context- and geometry-aware successors in the same sparse-to-dense lineage (Yu et al., 2024), sparse voxel detectors for LiDAR–camera fusion (Son et al., 11 Mar 2025), and efficiency-oriented voxel-transformer backbones for LiDAR detection (He et al., 2024). The term is further complicated by name overlap with PointVoxelFormer, which is explicitly not a standard voxel transformer (Heinrich, 2024), by SnakeVoxFormer for image-to-voxel reconstruction (Lee et al., 2023), and by an unrelated fMRI visual decoder also titled VoxelFormer (Le et al., 10 Sep 2025).

1. Terminological scope and disambiguation

Current usage spans camera-based semantic scene completion, multi-view 3D object detection, sparse LiDAR backbones, hybrid point–voxel medical models, single-image voxel reconstruction, and multi-subject fMRI decoding (Li et al., 2023, Li et al., 2023, Yu et al., 2024, He et al., 2024, Son et al., 11 Mar 2025, Heinrich, 2024, Lee et al., 2023, Le et al., 10 Sep 2025).

Name	Primary task	Distinguishing mechanism
VoxFormer	Camera-based semantic scene completion	Sparse visible voxel queries followed by dense completion
VoxelFormer	Multi-view camera 3D object detection	Dual-view attention for BEV feature generation
CGFormer	Semantic scene completion	Context-aware queries and 3D deformable cross-attention
ScatterFormer	LiDAR 3D object detection	Scattered Linear Attention on variable-length voxel windows
SparseVoxFormer	Multi-modal 3D object detection	Sparse 3D voxel tokens instead of dense BEV
PointVoxelFormer	3D medical segmentation and registration	Alternating point-wise MLPs and voxel CNNs
SnakeVoxFormer	Single-image voxel reconstruction	Snake traversal, RLE, and dictionary-token decoding
VoxelFormer	Multi-subject fMRI visual decoding	ToMer token compression and Q-Former aggregation

A persistent source of confusion is that “VoxelFormer” does not imply a single architectural recipe. Some methods are query-based sparse-to-dense transformers over voxel grids, some are BEV-centric detectors that use an intermediate voxel lifting step, some are sparse voxel-token backbones, and one prominent name overlap—PointVoxelFormer—is explicitly positioned as an efficient alternative to both kNN-based graph or point transformers and dense voxel CNNs rather than as a conventional self-attention-heavy transformer (Heinrich, 2024).

2. Sparse-to-dense voxel transformers for semantic scene completion

In camera-based semantic scene completion, voxel transformers are used to infer dense 3D occupancy and semantics from RGB imagery, including occluded and unobserved regions. VoxFormer formulates SSC as learning a mapping

$\mathbf{Y}_t = \Theta(\mathbf{I}_t),$

where $\mathbf{Y}_t \in \{c_0, c_1, \ldots, c_M\}^{H \times W \times Z}$ . On SemanticKITTI, the target volume spans 51.2 m forward, 25.6 m left/right, and 6.4 m vertically, voxelized at 0.2 m into a $256 \times 256 \times 32$ grid with 20 classes total (Li et al., 2023).

The defining idea of VoxFormer is reconstruction-before-hallucination. Stage 1 is class-agnostic: estimated depth is back-projected, voxelized into $\mathbf{M}_{in}$ , corrected by an occupancy network $\Theta_{occ}$ , and used to select a sparse subset of voxel queries,

$\mathbf{Q}_p = \mathrm{Reshape}\big(\mathbf{Q}[\mathbf{M}_{out}]\big), \qquad \mathbf{Q}_p \in \mathbb{R}^{N_p \times d}.$

Stage 2 is class-specific: the proposed voxel queries attend to 2D image features through deformable cross-attention, and a masked-autoencoder-like deformable self-attention stage propagates information to non-proposed voxels. The implementation uses a ResNet-50 backbone, FPN features at $1/16$ resolution, feature dimension $d=128$ , 3 deformable cross-attention layers, 2 deformable self-attention layers, and 8 sampling points per head (Li et al., 2023).

This sparse-to-dense design is empirically motivated by the mismatch between 2D image evidence and the volumetric target. VoxFormer reports, on SemanticKITTI validation at full 51.2 m range, IoU/mIoU of 44.02/12.35 for VoxFormer-S and 44.15/13.35 for VoxFormer-T, compared with 36.80/11.30 for MonoScene; at 12.8 m, VoxFormer-T reaches 65.38 IoU and 21.55 mIoU. The paper also reports about 60M parameters, less than 16 GB GPU memory during training, and an ablation in which occupancy-based sparse query proposal outperforms both dense and random query selection (Li et al., 2023).

CGFormer inherits this family structure but argues that shared context-independent voxel queries and 2D-only deformable sampling are weak points. It replaces static queries with context-dependent voxel queries

$\mathbf{V}_{Q}=f(\mathbf{C},\mathbf{D}),$

where $\mathbf{C}$ is a context feature map and $\mathbf{Y}_t \in \{c_0, c_1, \ldots, c_M\}^{H \times W \times Z}$ 0 a depth probability volume, and it extends deformable cross-attention into 3D pixel space by constructing

$\mathbf{Y}_t \in \{c_0, c_1, \ldots, c_M\}^{H \times W \times Z}$ 1

Its 3D deformable cross-attention is written as

$\mathbf{Y}_t \in \{c_0, c_1, \ldots, c_M\}^{H \times W \times Z}$ 2

and the resulting voxel volume is refined by a combined voxel + TPV encoder. The paper reports state-of-the-art camera-based SSC performance, with the abstract giving 45.99 IoU / 16.87 mIoU on SemanticKITTI and 48.07 IoU / 20.05 mIoU on SSCBench-KITTI-360, while the main text notes a hidden-test result of 44.41 IoU / 16.63 mIoU and explicitly acknowledges the discrepancy between tables and appendix values (Yu et al., 2024).

3. VoxelFormer as BEV feature generation for multi-view 3D object detection

In its most literal usage, VoxelFormer is the detector introduced in “VoxelFormer: Bird’s-Eye-View Feature Generation based on Dual-view Attention for Multi-view 3D Object Detection” (Li et al., 2023). It addresses multi-view camera-based 3D detection through the lens of BEV generation, arguing that BEVFormer’s spatial cross-attention has two limitations: attention weights are generated only from BEV queries, which prevents direct lidar-point supervision of the attention-generation process, and image features are aggregated into BEV through deformable sampling that uses only a small subset of camera features (Li et al., 2023).

VoxelFormer’s solution is dual-view attention. Camera-view attention is provided by predicted depth maps, and BEV-side attention is provided by a 3D occupancy confidence

$\mathbf{Y}_t \in \{c_0, c_1, \ldots, c_M\}^{H \times W \times Z}$ 3

Pixels are lifted to candidate 3D points via

$\mathbf{Y}_t \in \{c_0, c_1, \ldots, c_M\}^{H \times W \times Z}$ 4

mapped into ego coordinates by

$\mathbf{Y}_t \in \{c_0, c_1, \ldots, c_M\}^{H \times W \times Z}$ 5

and accumulated into voxels. The depth-adjusted voxel feature is

$\mathbf{Y}_t \in \{c_0, c_1, \ldots, c_M\}^{H \times W \times Z}$ 6

and the updated BEV feature is

$\mathbf{Y}_t \in \{c_0, c_1, \ldots, c_M\}^{H \times W \times Z}$ 7

Conceptually, the module encodes all camera-view features into BEV rather than sparsely sampling a few (Li et al., 2023).

The detector keeps a BEVFormer-style encoder–decoder structure but uses an explicit 3D voxel lattice $\mathbf{Y}_t \in \{c_0, c_1, \ldots, c_M\}^{H \times W \times Z}$ 8, 3 encoders in the main setting, temporal fusion with 1 historical frame, 900 detection queries, and 6 decoders following Deformable DETR. The backbone examples are ResNet-101 or VoVNet with a SECOND FPN, and only one FPN level is used for BEV generation. The method is trained for 24 epochs on nuScenes without CBGS, with optimizer and hyperparameters following BEVFormer (Li et al., 2023).

The reported performance positions VoxelFormer as a BEVFormer-family detector with a different BEV construction mechanism. On the nuScenes test set, the VoVNet $\mathbf{Y}_t \in \{c_0, c_1, \ldots, c_M\}^{H \times W \times Z}$ 9 variant reports mAP $256 \times 256 \times 32$ 0 and NDS $256 \times 256 \times 32$ 1. In fair comparison with BEVFormer under identical 3-encoder settings and no historical frame, VoxelFormer improves NDS from 0.270 to 0.319, a gain of +4.9 NDS points; with one historical frame, NDS improves from 0.357 to 0.386. The paper also reports 8.197 FPS and 164.48k params for VoxelFormer versus 7.634 FPS and 180.93k params for BEVFormer under matched settings (Li et al., 2023).

A different VoxelFormer lineage uses sparse occupied voxels as transformer tokens rather than dense BEV grids or dense voxel queries. ScatterFormer is a voxel-based transformer backbone for LiDAR 3D object detection that targets the systems bottleneck created by windowed attention on highly variable numbers of occupied voxels per window. Instead of forcing each window into fixed-length groups through sorting and padding, it sorts voxels by window coordinate, flattens all voxels into a single matrix, and applies Scattered Linear Attention,

$256 \times 256 \times 32$ 2

with $256 \times 256 \times 32$ 3 in practice and XCiT-style cross-covariance attention as $256 \times 256 \times 32$ 4. Cross-window interaction is provided by an Inception-style depthwise-convolution module rather than shifted windows. The backbone has four encoding stages, Triton-based implementation, and a chunk-wise kernel inspired by FlashAttention; the abstract reports SLA latency below 1 ms on moderate GPUs, and the paper reports 28 FPS with a speed–accuracy tradeoff better than DSVT on Waymo validation (He et al., 2024).

The empirical role of ScatterFormer is as an efficiency-oriented alternative within voxel-transformer backbones rather than as a named VoxelFormer derivative. On Waymo validation it reports 79.3/73.6 all-category mAPH (L1/L2) in single-frame mode and 81.5/76.5 in four-frame mode; on the Waymo test set, ScatterFormer-4f reports 80.71/75.91 all-category mAPH (L1/L2). On nuScenes validation, ScatterFormer reports 70.5 NDS and 66.9 mAP, slightly below DSVT on NDS but above it on mAP. The abstract separately reports 73.8 mAP (L2) on Waymo, 72.4 NDS on nuScenes, and 23 FPS, which the paper places alongside the detailed validation tables (He et al., 2024).

SparseVoxFormer moves further away from BEV by treating sparse 3D voxels as the detector’s primary representation. Its central claim is that high-resolution voxel grids can remain computationally attractive because the number of occupied voxels can be smaller than the number of cells in a dense BEV map. For the main configuration, the final voxel feature resolution is $256 \times 256 \times 32$ 5, the dense BEV token count for $256 \times 256 \times 32$ 6 is 32,400, and the average sparse voxel token count is 18,000 on nuScenes, 9,500 on Waymo, and 3,000 on Argoverse2. Sparse voxel features are serialized as

$256 \times 256 \times 32$ 7

fused with image features collected by voxel-to-image projection

$256 \times 256 \times 32$ 8

and consumed by a DETR-like decoder with 900 queries, 6 transformer decoder layers, and 3D positional encoding $256 \times 256 \times 32$ 9 (Son et al., 11 Mar 2025).

This design is explicitly anti-BEV in its information geometry. SparseVoxFormer reports, on nuScenes validation, 70.8 mAP / 73.2 NDS for the base model and 72.2 / 74.4 for the full model; on the nuScenes test set it reports 72.9 / 75.3, with the best listed NDS among compared methods. The detector cost is 61.3 GFLOPs for SparseVoxFormer-base versus 163.6 GFLOPs for CMT, while total cost is 111.3 GFLOPs versus 318.7 GFLOPs, a 65% reduction. Token budgeting through an auxiliary foreground/background head shows that reducing from an average 18,000 sparse tokens to 10,000 preserves performance at 72.2 / 74.4, whereas 5,000 tokens reduces performance to 70.0 / 73.4 (Son et al., 11 Mar 2025).

PointVoxelFormer is directly relevant to the name but not to the canonical architecture family. The paper states that PointVoxelFormer is not a “VoxelFormer” in the usual sense of a voxel-based transformer architecture. It is a point–voxel hybrid network for 3D medical imaging that alternates point-wise MLPs on sparse point clouds with low-resolution voxel or grid processing through shallow 3D CNNs, exchanging information through differentiable rasterisation and interpolation. There are no transformer blocks in the actual proposed network in the sense of multi-head self-attention over tokens; the “Former” part is mostly motivated by a MetaFormer-like design philosophy. The model is evaluated on 3D ultrasound thyroid segmentation, abdominal organ labeling from CT, and pulmonary vessel tree registration, reporting 95.01% accuracy and $\mathbf{M}_{in}$ 0 Dice on thyroid segmentation, 93.84% accuracy and $\mathbf{M}_{in}$ 1 Dice on abdominal organ labeling, and 3.45 mm network-only TRE on PVT1010/DirLab after longer training (Heinrich, 2024).

SnakeVoxFormer is a separate transformer-based voxel reconstruction method in which the output voxel grid is converted into a sequence through traversal, run-length encoding, and dictionary encoding. The model reconstructs a $\mathbf{M}_{in}$ 2 binary voxel grid from a $\mathbf{M}_{in}$ 3 image using a frozen ViT-B/32 encoder and a transformer decoder. Its snake traversal, average RLE length of 458.97 bytes, and compression factor $\mathbf{M}_{in}$4 correspond to roughly 1.4% of the original voxel size on average. On ShapeNet, the paper reports mean IoU of 0.843, 0.874, 0.904, and 0.912 for 4, 8, 16, and 20 views with snake traversal, while spiral traversal reaches 0.933 at 20 views (Lee et al., 2023).

The name also appears outside 3D vision. VoxelFormer in fMRI decoding is a lightweight multi-subject visual decoder that uses a Token Merging Transformer (ToMer) followed by a query-driven Q-Former to map visual-cortex fMRI signals into a CLIP-aligned latent space. The input is tokenized voxel activity with coordinate-derived positional information; ToMer compresses tokens during training and inference, and the Q-Former converts variable-sized compressed neural tokens into a fixed-size representation. The paper reports a total of 39M parameters, mean top-1 retrieval of 74.3% forward and 73.1% backward across training subjects S2–S7 on a 300-image candidate pool, and a parameter count about 12× lower than MindEye2 and 24× lower than MindEye1 (Le et al., 10 Sep 2025).

6. Recurrent architectural themes, misconceptions, and limitations

A plausible synthesis is that “VoxelFormer” has become a shorthand for models that preserve a voxel or voxel-query space while using transformer-style token interaction to avoid the limitations of dense 3D convolution alone. In SSC, the dominant pattern is sparse-to-dense reasoning: propose or initialize a subset of visible occupied voxel queries, ground those queries in image evidence through cross-attention, and then propagate information into masked or invisible voxels through self-attention (Li et al., 2023, Yu et al., 2024). In multi-view camera detection, the emphasis shifts from volumetric completion to BEV feature construction, where voxel lifting is used as an intermediate geometric scaffold and dual-view attention combines camera-view depth weighting with BEV-derived occupancy weighting (Li et al., 2023). In sparse LiDAR and multi-modal detection, the common design pressure is computational: exploit the fact that occupied voxels are much fewer than dense grid cells, whether through scattered linear attention on variable-length windows or direct sparse voxel tokens with Top-K control (He et al., 2024, Son et al., 11 Mar 2025).

Several common misconceptions follow from the naming. First, not every “VoxelFormer” is a standard self-attention-heavy voxel transformer. PointVoxelFormer is an explicit counterexample: it is a hybrid point–voxel CNN/MLP architecture, not a canonical voxel transformer (Heinrich, 2024). Second, the term does not denote a single benchmark family. Semantic scene completion methods are evaluated by IoU and mIoU on SemanticKITTI and SSCBench-KITTI-360, detection methods by mAP, NDS, or mAPH on nuScenes and Waymo, medical variants by Dice, accuracy, and TRE, and the fMRI model by top-1 retrieval (Li et al., 2023, Yu et al., 2024, Li et al., 2023, He et al., 2024, Son et al., 11 Mar 2025, Heinrich, 2024, Le et al., 10 Sep 2025). Third, the role of voxels differs materially across papers: they may be the prediction target, the attention query space, an intermediate lifting lattice, or the primary sparse tokenization of the sensor input.

The limitations are likewise heterogeneous. VoxFormer’s long-range performance is bottlenecked by the quality of image-only depth estimation, and the authors explicitly suggest that decoupling short-range and long-range SSC could be promising (Li et al., 2023). CGFormer improves query initialization and geometric disambiguation but still depends on stereo-guided depth refinement and reports an internal discrepancy between validation and hidden-test numbers (Yu et al., 2024). VoxelFormer’s detector results are presented with explicit compute tradeoffs, and the authors note that they did not train with the heaviest settings used by the strongest baselines (Li et al., 2023). ScatterFormer’s efficiency depends on custom Triton kernels and linear-attention reformulation rather than exact softmax attention (He et al., 2024). SparseVoxFormer gains efficiency from LiDAR sparsity, but it only forms voxel tokens where LiDAR points exist (Son et al., 11 Mar 2025). The fMRI VoxelFormer is parameter-efficient but does not match the top retrieval accuracy of larger subject-specific or partially subject-specific baselines (Le et al., 10 Sep 2025).

Across these variants, the most stable encyclopedic description is therefore taxonomic rather than singular: VoxelFormer is a name shared by multiple voxel-centered transformer or transformer-adjacent systems, with its most established uses in 3D semantic scene completion and multi-view 3D object detection, and with meaning that is strongly conditioned by domain, tokenization strategy, and the exact role of voxels within the model.