Neural Graphics Texture Compression

• Neural graphics texture compression is a method that uses neural networks and MLP decoders to efficiently compress large, multi-channel and multi-mip textures used in real-time graphics.
• It exploits cross-channel and cross-mipmap redundancies to achieve high compression ratios, surpassing traditional block-based codecs across various bitrate ranges.
• The approach supports random-access decode on GPUs, ensuring efficient VRAM usage and scalable asset delivery for high-fidelity, physically-based rendering.

Neural graphics texture compression denotes a class of methods that leverage neural networks—typically in learned feature grid + MLP decoder architectures—to achieve high compression ratios for large, multi-channel, multi-mip texture sets commonly used in physically-based rendering and real-time graphics. Unlike classical block compression (e.g., BC1–BC7, ASTC), neural approaches can encode cross-channel and cross-mipmap redundancy, support random-access decompression compatible with GPU pipelines, and yield rate–distortion performance exceeding state-of-the-art hardware codecs across a wide bitrate range (Farhadzadeh et al., 2024, Vaidyanathan et al., 2023, Fujieda et al., 2024). This paradigm is foundational for scalable asset delivery, high-fidelity real-time rendering, and compact storage in modern computer graphics.

1. Motivations and Fundamental Requirements

The proliferation of photorealistic materials in real-time rendering pipelines has resulted in massively increased texture resolutions (often ≥$4K\times4K$), material channel diversity (albedo, normal, roughness, displacement, etc.), and deep mipmap hierarchies. Traditional image codecs and block-compressed formats are unsuited for this scale:

• Classical hardware block compression (BC1–BC7, ASTC) offers per-block random access and hardware-accelerated filtering, but is limited by fixed bitrates (e.g., $4-8$ bpp), cannot exploit redundancy across channels/mips, and becomes inefficient as scene assets scale into hundreds or thousands of textures (Fujieda et al., 2024, Laurent et al., 6 Jun 2025).
• Neural image compression (NIC) codecs (e.g., VAEs with entropy-coded global latents) fail to provide the necessary random-access properties and do not natively handle arbitrary channel counts or mipmapped sets (Farhadzadeh et al., 2024).
• Key requirements for neural texture compression:

1. On-demand, block- or texel-level random access: Each GPU thread must decode only the required texel, facilitating parallelism and efficient VRAM usage.
2. Multi-resolution/mipmap support: Ability to reconstruct any mip level by direct latent lookup/scanning; indexability is essential for GPU texture samplers.
3. Joint compression of multi-channel, multi-mip texture sets: Exploiting discovery of redundancy across all supplied channels and resolution pyramid levels.

2. Neural Texture Compression Architectures and Dataflows

Neural compression frameworks converge on asymmetric autoencoder designs: spatial feature encoders (often CNN-based) create compact, low-resolution latent grids, subsequently quantized and consumed by small MLP decoders at render time for random-access texel synthesis (Farhadzadeh et al., 2024, Vaidyanathan et al., 2023, Weinreich et al., 2023, Laurent et al., 6 Jun 2025). Representative architecture:

• Encoder ($E_\phi$): Deep CNN with residual blocks maps the input tensor $T\in[0,1]^{c\times H\times W}$ to latent grids $$z=E_\phi(T)\in[\text{-}0.5,0.5]^{c_z\times H/8\times W/8}$$.
• Grid construction: Linear projections and quantization yield feature banks $G_0$ (high-frequency bank) and $G_1$ (low-frequency bank), typically with $B_0=B_1=4$ bits per feature (Farhadzadeh et al., 2024), with the dimensioning controlled to achieve targeted compression ratios.
• Grid sampling: For arbitrary texel coordinate $(x,y)$ and mip level $m$, $4-8$0 performs nearest-neighbor gathering on $4-8$1; $4-8$2 performs bilinear interpolation over $4-8$3. Mip access is realized by increasing sampling stride as $4-8$4 increases.
• Decoder ($4-8$5): MLP stacks (e.g., 4 residual fully-connected blocks with skip connections) ingest the grid samples, normalized mip level, and sinusoidal positional encodings, outputting the reconstructed texel vector $4-8$6 at the requested mip and coordinate.
• Random access: Architectural design ensures all per-texel decode logic is local to a small latent grid (resident on GPU), supporting fully parallel, latency-tolerant access patterns necessary for real-time rendering (Farhadzadeh et al., 2024, Vaidyanathan et al., 2023, Weinreich et al., 2023).

3. Quantization, Compression Pipeline, and Rate-Distortion

Neural texture compression relies on fixed-rate scalar quantization of latent features for practical random access and minimal computational overhead:

• Quantization: Feature grids $4-8$7 are quantized per-channel to $4-8$8 bits. Quantization noise is simulated during training; the final model uses the straight-through estimator (STE) post-quantization for efficient backpropagation (Farhadzadeh et al., 2024, Mahmoud et al., 2023).
• Absence of sequential entropy coding: To guarantee local, independent decode (i.e., random access), context-based entropy coding is avoided; rate control is achieved by latent dimension and bit-depth tuning rather than context models (Farhadzadeh et al., 2024, Vaidyanathan et al., 2023).
• Rate–distortion objective: The total loss optimized is of the form $4-8$9, where $E_\phi$0 is typically MSE, SSIM, or a perceptual metric, and $E_\phi$1 is implicitly controlled via bit allocation.

Method	Decoding Granularity	Entropy Model	Compression Control
NeuralTxd (Farhadzadeh et al., 2024)	Per-texel	None	Bit-depth + dim
NTC (Vaidyanathan et al., 2023)	Per-texel	None	Bit-depth + dim
NTBC (Fujieda et al., 2024)	Per-block (BC)	None	Joint textures
CAwa-NeRF (Mahmoud et al., 2023)	Hash-grid	Explicit	Entropy penalty

The table summarizes tradeoffs among selected methods with respect to decode granularity, the use of entropy coding, and compression control strategies.

4. Random-Access Decoding, GPU Integration, and Filtering

Neural graphics texture compression pipelines are engineered for efficient GPU integration, mirroring the operational semantics of traditional block-compressed textures:

• Random-access decode: Each shader thread, texel lookup, or block-decode is independent, as latent feature access is direct (no entropy decode), and neural decoders are small-shallow MLPs amenable to streaming execution (Farhadzadeh et al., 2024, Weinreich et al., 2023, Laurent et al., 6 Jun 2025).
• Hardware filtering and mipmapping: When latent grids are stored in BC-format (e.g., BC6H or BC1), standard GPU samplers automatically provide continuous LOD transitions and filtering for neural features, with the decoder MLP trained to synthesize plausible outputs under interpolated latent input (Weinreich et al., 2023, Laurent et al., 6 Jun 2025).
• Tile-based hardware acceleration: With cooperative vector extensions (VK_NV_cooperative_vector), MLP matvecs can be distributed across threadgroups/tile units for significant runtime speedups (>20× compared to software FMA), enabling sub-millisecond full-G-buffer decode at 1080p and 28 MB VRAM per 4K 9-channel texture set (Laurent et al., 6 Jun 2025).

5. Empirical Results: Compression Quality, Throughput, and Comparisons

Experimental evaluations across multiple neural texture codecs establish substantial improvement over traditional and recent learned codecs, focusing on rate–distortion, VRAM footprint, and real-time performance:

• Compression ratios and visual quality: On standard multi-channel 4K materials, neural methods achieve 14×–108× compression relative to uncompressed textures, with PSNRs up to 50 dB at moderate rates. For example, the "CNTC" method provides ∼40.8% BD-rate reduction versus NTC, outperforming ASTC and NTC in PSNR/SSIM at all rates (0.2–1.0 bits per pixel per channel) (Farhadzadeh et al., 2024).
• Hardware-compatibility and decode speed: Neural block-format approaches (BC1/BC4/BC6H) achieve seamless drop-in of BC blocks with standard hardware decode at render time, and inference overhead per material is a one-time 25–50 ms at load (Fujieda et al., 2024, Weinreich et al., 2023, Laurent et al., 6 Jun 2025).
• Example rates: NTBC yields 44–72% storage reduction (6×4K textures from 48 MB to 13.4–26.7 MB) with only modest PSNR/SSIM reduction; BCf-1K model achieves PSNR 31.8 dB at 1.77 MB (4.5× smaller than raw FP16 features), matching or exceeding ASTC 12×12 at a similar size (Weinreich et al., 2023).
• Limiting factors: At extremely aggressive compression (e.g., BC1 with CR >100×), block artifacts and loss of high-frequency detail arise, particularly when latent spatial support is limited (Laurent et al., 6 Jun 2025).

6. Comparison with Alternative Architectures and Extensions

Several frameworks exist for neural graphics primitive compression, with varying emphasis on grid representation, entropy modeling, and rate–distortion optimization:

• Hash-grid–based representations: Compression-aware hash-table probing drastically reduces feature grid footprint (to 50–350 kB for PSNR ~30 dB), maintains random-access, and executes as fast as standard hash encoding (Instant NGP) (Takikawa et al., 2023).
• Two-stage scalable feature–texture models: Jointly optimize a compact base feature with high-level semantic accuracy (analysis-friendly) and an enhancement residual layer for photorealistic fidelity, employing hierarchical generative decoders for texture synthesis (Wang et al., 2020, Chang et al., 2020).
• Compression-aware latent entropy regularization: CAwa-NeRF applies rate-penalized objectives and post-training quantization+ZIP to standard hash-encoded feature grids, achieving 90–98% size reductions with <0.6 dB PSNR loss compared to uncompressed neural fields (Mahmoud et al., 2023).

7. Limitations, Open Challenges, and Future Directions

While neural graphics texture compression establishes new Pareto points in the storage-quality-performance landscape, several challenges remain:

• No per-block variable bitrate: Most methods use fixed-rate quantization for random access; further coding efficiency may be available via block- or context-aware entropy models that partially relax this constraint (Farhadzadeh et al., 2024, Vaidyanathan et al., 2023).
• Filtering and aliasing: Simple strided or block-wise interpolation may introduce aliasing at mismatched resolutions; data-driven or learned filtering/interpolation remain open research topics (Farhadzadeh et al., 2024, Weinreich et al., 2023).
• Block-artifact suppression at extreme compression: At high CR, low channel count or spatial support can yield visible blockiness; future extensions may utilize advanced BC formats (e.g., BC7 with wider partitioning) or increased latent map count (Laurent et al., 6 Jun 2025, Fujieda et al., 2024).
• Streaming, dynamic, and time-varying textures: Current frameworks target static PBR assets; dynamic or progressive/temporal support is a noted future direction (Farhadzadeh et al., 2024, Laurent et al., 6 Jun 2025).
• Cross-material generalization: Sharing decoder MLPs across multiple materials or learning global style/semantic mixing remains an area for further compression ratio scaling and renderer simplification (Laurent et al., 6 Jun 2025).

Neural graphics texture compression continues to be a rapidly advancing field, with hybrid block-neural methods, quantization-aware feature grid design, and hardware-software co-design at the forefront of research for next-generation real-time rendering systems (Farhadzadeh et al., 2024, Laurent et al., 6 Jun 2025, Weinreich et al., 2023).