Neural Texture Compression using Hypernetworks

Abstract: Recent work on neural texture compression has demonstrated that it is possible to learn small, per-material texture representations (composed of latent textures and a small Multi-Layer Perceptron decoder) that can be decoded in real-time during shading to reproduce the input to a physically based shading model. However, existing methods require performing gradient-descent optimization per material for a given MLP and latent configuration. In this work, we train a single hypernetwork that outputs both the latent features and the MLP's weights and biases. Though the solution space is high-dimensional, this approach produces results comparable in quality to the current reference neural texture compressors. We further extend this approach to infer multiple decoders at once or even produce decoders that learn super-resolution.

• The paper introduces a hypernetwork that jointly infers latent representations and MLP parameters to compress high-resolution PBR texture sets.
• It achieves rapid, single-pass compression and efficient batching, reducing asset conversion time from hours to minutes.
• Empirical results demonstrate maintained perceptual quality with robust MS-SSIM scores and detail preservation despite slight PSNR trade-offs.

Neural Texture Compression using Hypernetworks: Formal Summary and Analysis

Motivation and Context

The paper "Neural Texture Compression using Hypernetworks" (2606.26913) addresses the challenge of compressing high-resolution, multi-channel texture sets used in physically based rendering (PBR). This is a critical pain point in gaming and real-time graphics, where storage and bandwidth constraints prohibit storing large texture atlases. Recent advances in neural texture compression techniques leverage small neural networks (e.g., MLPs) per texture set, offering superior compression ratio and random access compared to conventional block-based schemes. However, prior methodologies depend on per-material gradient descent optimization, substantially increasing the computational overhead during asset preparation.

Hypernetwork-Based Solution

The central innovation is the introduction of a monolithic hypernetwork capable of inferencing both the latent compressed representation and the MLP weights/biases for any texture set. The hypernetwork ingests a full set of correlated textures (albedo, normal, ARM, etc.) and outputs compatible block-compressed latent textures ($l_0, \ldots, l_3$) plus the corresponding MLP parameters ($\mathbf{wb}$) needed for decompression.

Figure 1: Hypernetwork architecture for neural texture compression, showing inference of latents and MLP parameters for a target texture set.

This architecture enables rapid, single-pass compression for arbitrary texture sets, dramatically reducing the amortized optimization time compared to legacy methods. The authors utilize the BCF1 block-compressed feature variant and integrate a Diffusion Transformer backbone with cross-attention for tokenization and global context aggregation.

Architectural and Training Details

The hypernetwork comprises three stages: a convolution-plus-attention t-Encoder extracting global tokens, a DiT-style transformer block processing spatial patches via cross-attention with tokens, and BC1 conversion layers generating the latent textures and MLP parameters. Notably, quantized endpoints and blend weights are derived via convolutions and non-linear activation/quantization pipelines, reflecting the hardware constraints of graphics APIs.

Training is performed on a large-scale dataset (Polyhaven textures, Polyhaven assets, Sketchfab PBR assets) across variable resolutions ($32 \times 32$ to $512 \times 512$). Loss functions combine $l_1$ metrics and the DISTS perceptual loss. Random jitter/permutation techniques and color augmentations mitigate overfitting and ensure robustness.

Strong Numerical Results and Empirical Analysis

Qualitative and quantitative evaluation benchmarks include PSNR and MS-SSIM statistics on test and non-dataset texture sets. The hypernetwork method achieves visually indistinguishable decompression results while outperforming both U-Net and ConvNext-based architectures in explicit ablation studies.

Figure 2: Generalization of Hypernetwork outputs for in-dataset, out-of-dataset, and out-of-training-set texture sets.

Figure 3: Comparison of BCF1-compressed asset visuals, showing preservation of high-frequency details (wood grain, scratches) compared to reference.

On a random subset of Polyhaven textures, the hypernetwork provides similar MS-SSIM but slightly lower PSNR versus BCF1 optimized via gradient descent, consistent with the perceptual loss emphasis. Compression batch conversion is accelerated by orders of magnitude; scenes comprising $100$ separate materials are compressed in under 10 minutes, as opposed to half a day's computation with classical BCF1.

Figure 4: Efficient batching of neural texture compression, enabling conversion of entire asset scenes.

Extended Capabilities: Multi-Head, Super-Resolution, and Token Sharing

The hypernetwork can be repurposed to generate multiple MLP decoders from a single latent, offering variable quality/performance tradeoffs (e.g., $D_h = 16, 32, 64$). It also allows fine-tuning to share MLP parameters across texture sets, facilitating homogeneous compute pipelines during rendering.

Figure 5: Multi-head inference from shared latents, supporting multiple compression depths.

Additionally, the architecture supports super-resolution inference: by inputting downsampled textures and applying loss on full-resolution counterparts, the hypernetwork can enhance details beyond the native resolution, vital for SVBRDF upsampling without paired datasets.

Figure 6: Hypernetwork repurposed for super-resolution; input is downscaled texture, loss is applied on full-res target.

Figure 7: Qualitative material super-resolution results, showing enhancement and preservation of material detail.

Limitations and Theoretical Implications

The method may introduce global color shifts due to limited MLP depth, affecting color consistency—a limitation shared with prior neural compression approaches. The hypernetwork may also be sensitive to the choice of loss function and tokenization strategy. The extension to generic, format-agnostic compression (e.g., targeting both NTC and NBC) could benefit from meta-learning approaches or adaptive architectures, as discussed in the conclusion.

Future Directions and Impact

Practical impact is clear: asset pipeline acceleration, random access decompression, and robust quality across diverse materials. Theoretically, this demonstrates the viability of hypernetwork-based learned compression for structured data (textures, material sets), and opens avenues for joint learning across tasks such as compression and upscaling. Integrating meta-learning could further expand application domains and adaptability.

Conclusion

This work establishes hypernetwork-based neural texture compression as a scalable, efficient, and versatile solution for high-fidelity material storage in graphics. The approach matches perceptual quality standards of legacy optimizers in a fraction of the time, generalizes robustly, and supports novel tasks like super-resolution and multi-head inference. Limitations exist in color fidelity and potential format generalization, pointing to promising research directions in adaptive compression and broader neural asset management.