Sparse 3D VQ-VAE Models

• Sparse 3D VQ-VAE is a family of models that compress 3D or spatiotemporal data into discrete latent tokens using sparsity and compact spatial parameterizations.
• The approach spans multiple regimes, from strict sparse architectures like L3DG to dense and discrete tokenizers handling video and medical volumes.
• Empirical studies show that optimized encoder–quantizer–decoder pipelines enhance reconstruction fidelity and compression efficiency in complex 3D tasks.

Sparse 3D VQ-VAE denotes a family of vector-quantized variational autoencoders that compress 3D or 3D-structured data into discrete latent tokens while exploiting sparsity, compact spatial parameterizations, or both. In the strictest usage, it refers to models such as L3DG, where a sparse convolutional VQ-VAE operates on voxel-aligned 3D Gaussian primitives and supplies a compact discrete latent space for downstream diffusion (Roessle et al., 2024). In broader usage, the label is also applied to dense spatio-temporal VQ-VAEs for video, triplane tokenizers, hierarchical volumetric codebooks for medical volumes, and 3D-aware VQ models whose efficiency comes from compact latent structure rather than explicit sparse occupancy (Yan et al., 2021); (Han et al., 14 Feb 2026); (Tudosiu et al., 2020); (Sargent et al., 2023). The adjacent literature further includes continuous-latent sparse autoencoders—most notably Hyper3D, Sparc3D, and FLUX3D—that are not VQ-VAEs in the strict sense but clarify what explicit 3D structure, sparsity, and modality consistency contribute to high-fidelity 3D compression (Guo et al., 13 Mar 2025); (Li et al., 20 May 2025); (Ji et al., 23 Jun 2026).

1. Terminological scope and model taxonomy

Taken together, the recent literature suggests that “Sparse 3D VQ-VAE” is not a single canonical architecture but a label spanning several distinct design regimes. One regime is a strict sparse 3D VQ-VAE, in which the encoder and decoder operate on sparse 3D coordinates or active voxels and vector quantization is applied to a sparse latent grid; L3DG is the clearest example (Roessle et al., 2024). A second regime consists of discrete but not explicitly sparse 3D tokenizers, such as VAR-3D’s multi-scale triplane VQ-VAE and VQ3D’s image-to-discrete-latent-to-NeRF pipeline (Han et al., 14 Feb 2026); (Sargent et al., 2023). A third regime contains dense 3D VQ-VAEs, exemplified by VideoGPT, whose representation is a standard dense spatio-temporal VQ-VAE built with 3D convolutions and axial attention rather than a sparse architecture (Yan et al., 2021). A fourth regime comprises continuous sparse or hybrid VAEs, including Hyper3D, Sparc3D, and FLUX3D, which are often conceptually adjacent to sparse latent 3D modeling but explicitly omit vector quantization (Guo et al., 13 Mar 2025); (Li et al., 20 May 2025); (Ji et al., 23 Jun 2026).

System family	Latent representation	Relation to “Sparse 3D VQ-VAE”
L3DG	Sparse 3D latent grid of Gaussian primitives with codebook quantization	Strict sparse 3D VQ-VAE
VideoGPT	Dense spatio-temporal grid of discrete video codes	3D VQ-VAE, but not sparse
VAR-3D	Multi-scale discrete triplane tokens	Discrete 3D tokenizer, not sparse voxel model
VQ3D	Discrete image tokens decoded by triplane-conditioned NeRF	3D-aware VQ model, not sparse voxel model
Hyper3D / Sparc3D / FLUX3D	Continuous sparse or hybrid structured latents	Adjacent, but not VQ-VAE

This taxonomic distinction matters because the word sparse is used differently across papers. In L3DG, sparsity means computation on active voxels using Minkowski Engine sparse convolutions and occupancy-aware decoding (Roessle et al., 2024). In the medical volumetric work, sparsity refers instead to extremely compact hierarchical discrete coding, where the latent representation is about 3.3% of the original image in number of variables and the bit-wise compression rate is about 0.825% of the original size (Tudosiu et al., 2020). In VideoGPT and VAR-3D, compactness comes from downsampling and tokenization rather than sparse occupancy masks (Yan et al., 2021); (Han et al., 14 Feb 2026).

2. Core encoder–quantizer–decoder mechanisms

The canonical sparse 3D VQ-VAE pipeline retains the standard VQ-VAE structure:

$$\mathbf z_e = E(\boldsymbol\theta), \qquad \mathbf z_q = \mathrm{quantize}(\mathbf z_e), \qquad \hat{\boldsymbol\theta} = D(\mathbf z_q),$$

where the encoder output is replaced by its nearest codebook entry and the decoder reconstructs the target representation. In L3DG, this pipeline is implemented on a sparse 3D grid of Gaussian parameters, and the codebook is updated with exponential moving average (EMA) rather than direct gradient descent (Roessle et al., 2024). The associated commitment term is

$$\mathcal L_{\mathrm{commit}} = \|\mathbf z_e - \mathbf e_{\bot}\|_2^2,$$

and the full compression-model objective is

$$\mathcal L_{\mathrm{comp}} = \lambda_{\mathrm{commit}}\mathcal L_{\mathrm{commit}} + \lambda_{\mathrm{RGB}}\mathcal L_{\mathrm{RGB}} + \lambda_{\mathrm{perc}}\mathcal L_{\mathrm{perc}} + \mathcal L_{\mathrm{occ}},$$

with perceptual supervision provided by VGG19 features and occupancy supervision provided by BCE loss (Roessle et al., 2024).

Architecturally, L3DG’s encoder begins with a sparse convolution block raising channels to 128, followed by two downsampling stages with residual blocks, then a bottleneck residual block at 512 channels, and finally a projection to 4 channels per latent voxel (Roessle et al., 2024). The decoder mirrors this hierarchy with two upsampling stages and generative sparse transpose convolution blocks. The decoder must be able to generate new coordinates during upsampling, because at test time it receives latent samples from diffusion rather than encoder feature maps with cached sparsity patterns (Roessle et al., 2024). After each upsampling step, a linear classifier predicts occupancy for each voxel, and the resulting BCE occupancy loss prunes free space and prevents voxel explosion (Roessle et al., 2024).

Other 3D VQ-VAE systems preserve the same high-level discrete bottleneck while changing the geometry of the latent. VideoGPT quantizes a dense spatio-temporal feature grid produced by 3D convolutions and axial self-attention; the main model uses one codebook, and ablations show that one codebook performs best (Yan et al., 2021). VAR-3D quantizes a triplane feature tensor at multiple resolutions using a shared codebook across scales, so token semantics remain aligned across coarse-to-fine serialization (Han et al., 14 Feb 2026). The neuromorphology-preserving MRI model inserts VQ blocks at three resolutions—$48^3$, $12^3$, and $3^3$—and emphasizes that higher-resolution codes are conditioned on the immediately lower-resolution ones so that the hierarchy encodes complementary information rather than redundantly learning the same content (Tudosiu et al., 2020).

3. Sparse 3D parameterizations and what is actually being quantized

A defining feature of a strict sparse 3D VQ-VAE is that quantization is applied not to an image-like feature map but to a structured 3D representation that remains meaningful after compression. L3DG first converts an unstructured Gaussian set into a grid-aligned sparse 3D representation by discretizing scene space into voxels of size $d$, with each occupied voxel containing at most one Gaussian primitive (Roessle et al., 2024). A primitive $i$ is parameterized as

$$\boldsymbol\theta_{\kappa_i} = (\boldsymbol\delta_i, \mathbf s_i, \mathbf r_i, \boldsymbol\gamma_i, \alpha_i),$$

with position reparameterized by

$$\boldsymbol\mu_i = \mathbf y_{\kappa_i} + \psi(\boldsymbol\delta_i), \qquad \psi(\boldsymbol\delta) = 1.5\,\tanh(\boldsymbol\delta)\, d.$$

This yields a sparse grid of Gaussian parameters $$\mathcal L_{\mathrm{commit}} = \|\mathbf z_e - \mathbf e_{\bot}\|_2^2,$$0, where only occupied voxels store a 3D Gaussian (Roessle et al., 2024). The latent is therefore a compressed 3D geometric field, and the decoder reconstructs Gaussian parameters rather than pixel values or a binary voxel occupancy field (Roessle et al., 2024).

Related work shows that sparsity can be instantiated differently even when vector quantization is absent. Sparc3D introduces SparCubes, a sparse deformable marching-cubes representation denoted $$\mathcal L_{\mathrm{commit}} = \|\mathbf z_e - \mathbf e_{\bot}\|_2^2,$$1, obtained by narrow-band voxel activation, UDF estimation,

$$\mathcal L_{\mathrm{commit}} = \|\mathbf z_e - \mathbf e_{\bot}\|_2^2,$$2

flood-fill sign labeling,

$$\mathcal L_{\mathrm{commit}} = \|\mathbf z_e - \mathbf e_{\bot}\|_2^2,$$3

and gradient-based deformation of the sparse cube grid (Li et al., 20 May 2025). SparConv-VAE then compresses the SparCubes parameters $$\mathcal L_{\mathrm{commit}} = \|\mathbf z_e - \mathbf e_{\bot}\|_2^2,$$4 into a continuous latent $$\mathcal L_{\mathrm{commit}} = \|\mathbf z_e - \mathbf e_{\bot}\|_2^2,$$5 with a KL regularizer rather than a codebook (Li et al., 20 May 2025). FLUX3D likewise uses a sparse set of active voxel positions $$\mathcal L_{\mathrm{commit}} = \|\mathbf z_e - \mathbf e_{\bot}\|_2^2,$$6 and continuous per-voxel features $$\mathcal L_{\mathrm{commit}} = \|\mathbf z_e - \mathbf e_{\bot}\|_2^2,$$7, where $$\mathcal L_{\mathrm{commit}} = \|\mathbf z_e - \mathbf e_{\bot}\|_2^2,$$8, $$\mathcal L_{\mathrm{commit}} = \|\mathbf z_e - \mathbf e_{\bot}\|_2^2,$$9, and $$\mathcal L_{\mathrm{comp}} = \lambda_{\mathrm{commit}}\mathcal L_{\mathrm{commit}} + \lambda_{\mathrm{RGB}}\mathcal L_{\mathrm{RGB}} + \lambda_{\mathrm{perc}}\mathcal L_{\mathrm{perc}} + \mathcal L_{\mathrm{occ}},$$0, but explicitly states that there is no discrete codebook and no explicit vector quantization step (Ji et al., 23 Jun 2026).

These distinctions clarify the role of quantization. In a strict sparse 3D VQ-VAE, the model first defines a sparse 3D support and then discretizes the corresponding latent features. In adjacent continuous models, the same sparse support may be retained while the discrete codebook is removed. This suggests that sparse geometry and vector quantization are separable design axes rather than a single indivisible recipe.

4. Dense 3D VQ-VAEs and discrete compact 3D tokenizers

Several influential systems are often discussed alongside sparse 3D VQ-VAEs because they share the discrete-token bottleneck even when they do not use sparse voxel computation. VideoGPT is a first-stage dense 3D VQ-VAE for video generation: the encoder is a stack of 3D convolutions followed by attention residual blocks with axial self-attention, the decoder is the reverse of the encoder with 3D transposed convolutions, and the learned discrete latents form a spatio-temporal grid such as $$\mathcal L_{\mathrm{comp}} = \lambda_{\mathrm{commit}}\mathcal L_{\mathrm{commit}} + \lambda_{\mathrm{RGB}}\mathcal L_{\mathrm{RGB}} + \lambda_{\mathrm{perc}}\mathcal L_{\mathrm{perc}} + \mathcal L_{\mathrm{occ}},$$1 for BAIR or $$\mathcal L_{\mathrm{comp}} = \lambda_{\mathrm{commit}}\mathcal L_{\mathrm{commit}} + \lambda_{\mathrm{RGB}}\mathcal L_{\mathrm{RGB}} + \lambda_{\mathrm{perc}}\mathcal L_{\mathrm{perc}} + \mathcal L_{\mathrm{occ}},$$2 for UCF-101 and TGIF (Yan et al., 2021). The paper explicitly states that the model is not a sparse 3D VQ-VAE in the architectural sense; sparsity enters only indirectly through downsampling and quantization (Yan et al., 2021).

VAR-3D moves the discrete bottleneck into a view-aware multi-view-to-triplane VQ-VAE. Its encoder consumes rendered multi-view RGB-D observations with Plücker-coordinate camera embeddings, applies per-view self-attention and cross-view interaction, fuses the last three downsampling stages through Multi-scaleFusion, and produces a triplane latent

$$\mathcal L_{\mathrm{comp}} = \lambda_{\mathrm{commit}}\mathcal L_{\mathrm{commit}} + \lambda_{\mathrm{RGB}}\mathcal L_{\mathrm{RGB}} + \lambda_{\mathrm{perc}}\mathcal L_{\mathrm{perc}} + \mathcal L_{\mathrm{occ}},$$3

This latent is interpolated to multiple scales and quantized with a shared codebook $$\mathcal L_{\mathrm{comp}} = \lambda_{\mathrm{commit}}\mathcal L_{\mathrm{commit}} + \lambda_{\mathrm{RGB}}\mathcal L_{\mathrm{RGB}} + \lambda_{\mathrm{perc}}\mathcal L_{\mathrm{perc}} + \mathcal L_{\mathrm{occ}},$$4; implementation uses ten scales, raster-scan serialization, and a coarse-to-fine token sequence in which indices corresponding to the same spatial location in the three planes are placed consecutively (Han et al., 14 Feb 2026). The representation is discrete and compact, but the paper does not describe sparse voxels, point clouds, or mesh token primitives as the main tokenizer representation (Han et al., 14 Feb 2026).

VQ3D is another discrete 3D-aware model, but its decoder is a conditional NeRF built from a contracted triplane representation, a proposal MLP, and a NeRF MLP (Sargent et al., 2023). Stage 1 is an image-to-discrete-latent-to-NeRF autoencoder, and Stage 2 is an autoregressive transformer prior over the discrete tokens; the codebook size is 8192 and the embedding dimension is 8 (Sargent et al., 2023). The latent is compact and discrete, yet the model has no explicit sparse voxel mechanism (Sargent et al., 2023).

The neuromorphological MRI model adapts VQ-VAE to full volumetric data by replacing all 2D blocks with 3D blocks, using FixUp residual blocks, transpose convolutions with kernel size 4, ICNR initialization, and a final subpixel convolution to reduce checkerboard artifacts (Tudosiu et al., 2020). Its hierarchical codes at $$\mathcal L_{\mathrm{comp}} = \lambda_{\mathrm{commit}}\mathcal L_{\mathrm{commit}} + \lambda_{\mathrm{RGB}}\mathcal L_{\mathrm{RGB}} + \lambda_{\mathrm{perc}}\mathcal L_{\mathrm{perc}} + \mathcal L_{\mathrm{occ}},$$5, $$\mathcal L_{\mathrm{comp}} = \lambda_{\mathrm{commit}}\mathcal L_{\mathrm{commit}} + \lambda_{\mathrm{RGB}}\mathcal L_{\mathrm{RGB}} + \lambda_{\mathrm{perc}}\mathcal L_{\mathrm{perc}} + \mathcal L_{\mathrm{occ}},$$6, and $$\mathcal L_{\mathrm{comp}} = \lambda_{\mathrm{commit}}\mathcal L_{\mathrm{commit}} + \lambda_{\mathrm{RGB}}\mathcal L_{\mathrm{RGB}} + \lambda_{\mathrm{perc}}\mathcal L_{\mathrm{perc}} + \mathcal L_{\mathrm{occ}},$$7 make the encoding extremely compact while preserving anatomical structure (Tudosiu et al., 2020). Here, “sparse” refers to low-rate discrete volumetric coding rather than sparse occupancy masks.

5. Empirical behavior, ablations, and reconstruction fidelity

Empirical evidence indicates that a true sparse 3D VQ-VAE can function as more than a memory-saving front end. In L3DG, the ablation w/o compression model, which diffuses directly on the low-resolution Gaussian parameter space, performs far worse than diffusion over the learned sparse VQ-VAE latents: on ABO Tables, FID jumps from 14.03 to 197.1 and KID from 3.15 to 166.8 (Roessle et al., 2024). The paper further attributes sharper structures such as thin chair and table legs to the full sparse VQ-VAE design, including its photometric, perceptual, and occupancy losses (Roessle et al., 2024). This establishes the discrete sparse bottleneck as a semantic representation, not merely a storage reduction.

Tokenizer-oriented discrete 3D models show a related pattern. VAR-3D reports VQ-VAE ablations in which the base tokenizer achieves PSNR 28.42, FID 34.72, KID 0.165, adding view-aware interaction yields PSNR 28.57, FID 33.35, KID 0.151, adding multi-scale fusion yields PSNR 28.68, FID 32.00, KID 0.150, and the full model reaches PSNR 28.97, FID 30.50, KID 0.140 (Han et al., 14 Feb 2026). A codebook-size ablation shows fidelity improving from PSNR 28.58, SSIM 0.932, LPIPS 0.069 at 4096 entries to PSNR 28.97, SSIM 0.938, LPIPS 0.063 at 16384 entries (Han et al., 14 Feb 2026). These results directly support the paper’s claim that reducing information loss before quantization improves discrete 3D token quality.

In medical volumetric encoding, the hierarchical 3D VQ-VAE substantially outperforms the $$\mathcal L_{\mathrm{comp}} = \lambda_{\mathrm{commit}}\mathcal L_{\mathrm{commit}} + \lambda_{\mathrm{RGB}}\mathcal L_{\mathrm{RGB}} + \lambda_{\mathrm{perc}}\mathcal L_{\mathrm{perc}} + \mathcal L_{\mathrm{occ}},$$8-WGAN baseline while maintaining morphology. Under healthy-control high-resolution training, VQ-VAE Baur reports MS-SSIM 0.998, log(MMD) 6.737, Dice WM 0.85, Dice GM 0.90, and Dice CSF 0.75, whereas VQ-VAE Adaptive reports MS-SSIM 0.991, log(MMD) 6.655, Dice WM 0.84, Dice GM 0.92, and Dice CSF 0.79; the corresponding $$\mathcal L_{\mathrm{comp}} = \lambda_{\mathrm{commit}}\mathcal L_{\mathrm{commit}} + \lambda_{\mathrm{RGB}}\mathcal L_{\mathrm{RGB}} + \lambda_{\mathrm{perc}}\mathcal L_{\mathrm{perc}} + \mathcal L_{\mathrm{occ}},$$9-WGAN baseline gives MS-SSIM 0.496, log(MMD) 15.676, Dice WM 0.77, Dice GM 0.86, and Dice CSF 0.68 (Tudosiu et al., 2020). The paper additionally reports that residual VBM analyses show much smaller residual $48^3$0-values for the VQ-VAE than for the baseline, indicating stronger morphology preservation (Tudosiu et al., 2020).

At larger scale, VQ3D demonstrates that a discrete 3D-aware tokenizer can support difficult image-domain generation tasks. On ImageNet, the paper reports FID 16.8, compared with 69.8 for the next best baseline method, StyleNeRF (Sargent et al., 2023). Although VQ3D is not a sparse voxel model, its results show that discrete latent tokenization and a 3D-aware decoder can remain effective even when the training signal comes only from 2D image collections (Sargent et al., 2023).

Continuous alternatives also sharpen the design trade-offs around sparse 3D latent modeling. Hyper3D reports that octree features outperform uniform surface sampling even with fewer points: 30,720 octree leaf-node points versus 81,920 uniformly sampled points improve F-score from 0.9931 to 0.9969, reduce Chamfer Distance from 9.5056 to 5.7283, improve normal consistency from 0.9529 to 0.9537, and raise Surface IoU from 0.5632 to 0.6502 (Guo et al., 13 Mar 2025). Because Hyper3D is explicitly not a VQ-VAE, this result isolates the contribution of explicit 3D structure and adaptive sparse input encoding apart from vector quantization.

6. Continuous-latent alternatives, misconceptions, and future trajectories

A recurring misconception is that any high-fidelity sparse 3D autoencoder used before diffusion is a VQ-VAE. The recent literature contradicts this directly. Hyper3D is trained as a standard VAE, not as a discrete VQ-VAE; it has no codebook, no nearest-neighbor assignment, and no discrete bottleneck (Guo et al., 13 Mar 2025). Sparc3D’s SparConv-VAE likewise uses a continuous latent with

$48^3$1

and the paper states that there is no codebook or discrete token assignment (Li et al., 20 May 2025). FLUX3D also states that it is not a classical VQ-VAE: there is no discrete codebook, no explicit vector quantization step, and the latents remain continuous (Ji et al., 23 Jun 2026).

These continuous systems nonetheless illuminate several pressures that also affect discrete sparse 3D VQ-VAEs. Hyper3D frames the central problem as a tension between compactness and geometric fidelity and argues that 1D vector sets and 2D triplanes are often too “flat” or too coarse to preserve sharp edges, thin structures, and surface micro-geometry (Guo et al., 13 Mar 2025). Sparc3D criticizes prior 3D VAEs for modality mismatch, arguing that encoder–decoder pairs that convert meshes into a different modality such as SDF samples, point normals, or global vector sets force the autoencoder to learn both compression and cross-modality translation (Li et al., 20 May 2025). FLUX3D identifies a representation bottleneck caused by discriminative 2D features and a cross-modal alignment bottleneck caused by standard diffusion transformers that do not respect sparse voxel topology (Ji et al., 23 Jun 2026).

Taken together, these works suggest that future sparse 3D VQ-VAE systems may combine discrete codebooks with three properties that have so far often been studied separately: modality-consistent sparse geometry, reconstructively rich latent features, and diffusion-aligned cross-modal conditioning. That implication is especially plausible because Hyper3D explicitly argues that its hybrid grid-plus-triplane design may inform future sparse or discrete 3D latent autoencoders (Guo et al., 13 Mar 2025), while Sparc3D and FLUX3D each show that sparse 3D generation quality depends not only on compression rate but also on how faithfully the latent space preserves geometry and appearance before the generative prior is trained (Li et al., 20 May 2025); (Ji et al., 23 Jun 2026).