Neural Texture Compression via Hypernetworks

• The paper demonstrates a hypernetwork that amortizes per-material optimization, reducing bake time from minutes to seconds while retaining high visual quality.
• The approach combines compact latent textures with decoder weight prediction via a DiT, enabling flexible quality and performance trade-offs.
• Integration into GPU pipelines is seamless, allowing batch conversion and real-time decode using standard block-compressed formats.

Neural texture compression using hypernetworks is a recent paradigm in computer graphics for representing the high-dimensional, correlated 2D texture maps of physically based rendering (PBR) materials in a highly compact, learnable form that supports real-time decoding on GPUs. This approach replaces traditional bitmap compression methods with a learned combination of compact latent texture maps and neural decoders, and uniquely amortizes optimization cost by leveraging a single, task-specific hypernetwork to predict both the compressed texture representation and its decoder weights. The methodology achieves rapid bake times, flexible quality/performance trade-offs, and state-of-the-art visual quality with bitrates as low as 1–2 bits/texel, matching reference neural texture compressors while enabling batch conversion of large-scale asset libraries (Laurent, 25 Jun 2026).

1. Problem Formulation and Motivation

In modern rendering pipelines, each PBR material is described by a tuple of spatially aligned, correlated 2D texture maps, $T = \{T^0, T^1, \dots, T^m\}$, covering channels such as albedo, normal, roughness, and metallic. These texture maps are bandwidth- and memory-intensive, particularly in applications with thousands of materials, as in AAA game engines.

Recent neural texture compression (NTC) approaches represent each material using:

• A small set of latent “textures” $L = \{\ell_0, \ell_1, ..., \ell_N\}$ (block-compressed, e.g., BC1 format)
• A compact material-specific decoder $f(\cdot\ ;~\theta)$ (typically, a multi-layer perceptron) that reconstructs the original $m$-channel output from the $Nd$-dim input at each texel.

Previous state-of-the-art methods (e.g., BCF1 [Belcour & Benyoub 2025]) involve per-material, per-configuration gradient descent to optimize both latent maps and decoder weights. This produces competitive results at extreme compression but results in bake times on the order of tens of minutes per 4K texture set, scaling poorly to scenes with thousands of assets. Amortizing or eliminating the slow per-material optimization is thus a fundamental bottleneck for practical deployment (Laurent, 25 Jun 2026).

2. Hypernetwork-Based Compression Architecture

Motivated by the scalability limitations of per-material neural bake pipelines, the hypernetwork framework introduces a single, shared meta-model $H$, parameterized by $\varphi$, which receives a texture set $T$ and directly predicts both:

• Compressed latent textures $[\ell_0, \ell_1, ..., \ell_N]$
• The full set of decoder weights $\theta$ for the associated MLP.

Formally, the prediction step is:

$L = \{\ell_0, \ell_1, ..., \ell_N\}$0

with each $L = \{\ell_0, \ell_1, ..., \ell_N\}$1 (for BC1, $L = \{\ell_0, \ell_1, ..., \ell_N\}$2), and

$L = \{\ell_0, \ell_1, ..., \ell_N\}$3

for a small 2-layer MLP ($L = \{\ell_0, \ell_1, ..., \ell_N\}$4 standard).

The reconstructed texel at pixel $L = \{\ell_0, \ell_1, ..., \ell_N\}$5 is given by:

$L = \{\ell_0, \ell_1, ..., \ell_N\}$6

Hypernetwork $L = \{\ell_0, \ell_1, ..., \ell_N\}$7 is implemented as a DiT (Diffusion-Transformer) composed of:

1. A $L = \{\ell_0, \ell_1, ..., \ell_N\}$8-Encoder to pool global statistics.
2. Multiple cross-attention blocks for mixing texel-level (“image tokens”) and global features.
3. Linear heads projecting to decoder weights ($L = \{\ell_0, \ell_1, ..., \ell_N\}$9) and BC1-format latent maps ($f(\cdot\ ;~\theta)$0).

3. Training Objectives and Optimization

Training is end-to-end, minimizing a loss between $f(\cdot\ ;~\theta)$1 and the ground truth texture set $f(\cdot\ ;~\theta)$2. The loss function for each material is:

$f(\cdot\ ;~\theta)$3

where $f(\cdot\ ;~\theta)$4 denotes the DISTS perceptual loss computed over three randomly chosen texture channels.

Total training objective:

$f(\cdot\ ;~\theta)$5

Regularization such as weight decay ($f(\cdot\ ;~\theta)$6) on $f(\cdot\ ;~\theta)$7 is optionally used. The standard optimizer is Adam with a learning rate $f(\cdot\ ;~\theta)$8.

4. Extensions: Multi-Decoder and Super-Resolution Modes

The approach flexibly extends to meet diverse requirements:

4.1 Multi-Decoder Inference

For scenarios requiring multiple trade-offs (e.g., quality vs. speed), $f(\cdot\ ;~\theta)$9 is equipped with multiple output heads $m$0 for MLPs of different width. During training, reconstruction is computed for each decoder configuration, and the combined loss is summed:

$m$1

At inference, any $m$2 can be used without re-running $m$3.

4.2 Super-Resolution Decoders

For neural texture super-resolution, $m$4 is trained by inputting a low-res $m$5 and supervising with ground-truth full-resolution $m$6, enforcing:

$m$7

After training, the decoder MLP “hallucinates” plausible fine-scale details under relighting.

5. Empirical Evaluation and Quantitative Results

Datasets: ~2,200 unique PBR texture sets (4K×4K) from Polyhaven and Sketchfab.

Metrics: PSNR, MS-SSIM, FLIP (DSSIM), and visual inspection with Mitsuba.

Property	Per-Material BCF1	Hypernetwork (H)
Compression Ratio (bits/texel)	1–2	1–2 (for 4 latent maps + 32-channel MLP)
Median PSNR (dB)	~36	≈35
Median MS-SSIM	0.985	0.985 (≥ BCF1; due to perceptual loss)
Bake Time per 4Kx4K	~840 s (14 min)	~2.4 s (>300× speedup)

Visual quality under relighting is nearly identical. The reduction in bake time enables batch conversion of hundreds of assets in under 10 minutes, versus days for prior methods. Real-time shading proceeds by loading block-compressed latent maps and evaluating the MLP per shading point (32-thread GPU groups), matching the runtime of previous art (Laurent, 25 Jun 2026).

6. Integration into Physically Based Rendering Pipelines

The compressed representation is designed for seamless GPU integration:

• Latent textures $m$8 are stored as standard BC1-format block-compressed images, enabling random access, filtering, and mipmap support.
• At runtime, for each shading call, required $m$9 are fetched, concatenated to $Nd$0, and decoded by MLP $Nd$1 to yield the $Nd$2-channel output (e.g., diffuse RGB, normals, roughness).
• Standard GPU hardware suffices: the pipeline requires only block-compressed texture access and small matrix multiplies with standard nonlinearities (e.g., SILU).
• Single shared $Nd$3 decoding can optionally be used to avoid material-ID branches per tile.

This structure supports integration into any modern rendering system, leveraging the full performance of block texture formats and programmable shading hardware.

7. Connections and Context within Neural Compression

This hypernetwork-based approach reflects a general trend in neural compression: amortization of content-specific representation cost by means of a meta-network that learns to parameterize, predict, or generate representations for previously unseen instances with a forward pass. For textures, this accelerates asset authoring and enables large-scale batch conversion, while preserving real-time decode and fidelity. Comparable strategies have been applied to video INR compression (e.g., TeCoNeRV (Padmanabhan et al., 18 Feb 2026)), with similar architectural themes: separation of global “meta” encoding (hypernetwork), local compact IR (latent maps, INR), and decoders optimized for fast inference.

A plausible implication is that further work may incorporate richer meta-representations, shared decoders, or hybrid approaches with probabilistic entropy models, as seen in other INR compression literature, extending the efficiency and generality of neural block representations across texture, video, and scene domains.

---

References:

• "Neural Texture Compression using Hypernetworks" (Laurent, 25 Jun 2026)
• "TeCoNeRV: Leveraging Temporal Coherence for Compressible Neural Representations for Videos" (Padmanabhan et al., 18 Feb 2026)