Neural Shell Texture Synthesis

• Neural shell texture is a continuous, geometry-aware representation that maps 3D surface points to rich, view-dependent appearances using neural networks.
• It employs methods like MLPs, hash-grid encodings, and cellular automata to eliminate UV artifacts and support dynamic, high-frequency texture synthesis.
• The approach facilitates novel view rendering, texture baking, and real-time editing while offering efficient storage and high visual fidelity.

Neural shell texture refers to a class of learned, continuous texture representations defined on or around the surface (“shell”) of a 3D object, encoded and synthesized using neural networks. Unlike traditional UV-mapped images or per-vertex colorings, neural shell texture methods provide high-frequency, seamless, and geometrically-aware appearance models, supporting advanced applications such as view-dependent rendering, appearance transfer, real-time procedural modification, and efficient storage. Recent advances span diffusion-based multi-view projection (Georgiou et al., 19 Feb 2025), disentangled neural fields (Zhang et al., 27 Jul 2025, Huang et al., 2024), surface-based cellular automata (Pajouheshgar et al., 2023), and implicit neural representations (INRs) (Kwok et al., 2 Feb 2026), each contributing distinct inductive biases and practical tradeoffs for representing texture appearance on complex 3D geometry.

1. Conceptual Foundations and Motivation

Neural shell texture methods generalize classical texturing to learned, continuous, and surface-aware neural fields. In classical pipelines, texture is parameterized as a discrete image indexed by a fixed UV-mapping, often suffering from seams, stretching, or loss of geometric correlation. Neural shell textures—such as those encoded by MLPs, CNNs, or hash tables—directly map surface points (defined via world coordinates, barycentric mesh coordinates, or local surface parameterizations) to feature vectors representing the desired appearance. This decouples surface geometry from texture signal, enables seamless, consistent generation over curved and topologically complex surfaces, and supports view-dependent and multi-modal effects inherently (Georgiou et al., 19 Feb 2025, Zhang et al., 27 Jul 2025, Huang et al., 2024, Pajouheshgar et al., 2023, Kwok et al., 2 Feb 2026). Early motivations focused on eliminating UV artifacts and supporting texture synthesis that can adapt to the local geodesic structure, promote seamless blending, and learn from multi-modal high-dimensional data.

2. Neural Shell Texture Representations

Multiple formalizations coexist, distinguished by how the neural field is anchored to the geometry and by the structure of the neural parameterization:

• Global coordinate-MLP textures: A network $f_\theta(u, v)$ or $f_\theta(x, y, z)$ with $[u, v]$ or $[x, y, z]$ as input, typically a Fourier-encoded or SIREN MLP, returning RGB or feature codes. This supports seamless texture detail and native extension to curved surfaces, with the tradeoff of higher inference cost per-point (Kwok et al., 2 Feb 2026).
• Hash-grid shell features: Multi-resolution hash encodings map sampled shell coordinates (e.g., from surfels or base-mesh vertices) to feature vectors, as in NeST-Splatting (Zhang et al., 27 Jul 2025) or NeRF-Texture (Huang et al., 2024). Rendering uses features decoded by a compact MLP, often including viewing direction and signed distance, yielding both high parametric efficiency and extremely fine, continuous surface coverage.
• Mesh-based cellular automata (CA): Texture state vectors $x_i^t$ are iteratively updated on mesh vertices using learned neural message-passing, producing dynamic or static patterns and supporting zero-shot generalization to unseen meshes based on local neighborhood structure (Pajouheshgar et al., 2023).
• Cross-attention fusion in UV or geodesic space: Neural backprojection modules aggregate multi-view RGB/geometry evidence into surface texels, using deep attention architectures and local surface-aware encoding to enforce continuity and robustness to view inconsistency (Georgiou et al., 19 Feb 2025).

These strategies may reside on discrete mesh vertices/texels, continuous position in $\mathbb{R}^3$ or UV parameter space, or hybrid schemes (patch-based, tangent-frame quilters). Effective shell textures employ position/geometry encoding (e.g., geodesic distance, relative normal orientation, chart-barycentric coordinates) to promote local continuity and resolve spatial ambiguities, especially near seams, folds, or high-curvature regions (Georgiou et al., 19 Feb 2025, Zhang et al., 27 Jul 2025, Huang et al., 2024).

3. Architectures and Training Methodologies

Distinct architectures have been proposed for shell texture encoding:

• Attention-based Backprojection (Im2SurfTex): Multi-view diffused RGB projections are gathered at each UV texel; three stacked cross-attention blocks blend appearance and geometric encodings derived from local pixel neighborhoods, using geodesic and pointwise offset features as attention keys/queries. The network is trained with $L_1$ UV-atlas supervision, leveraging large mesh datasets (e.g., Objaverse) and noise-augmented partial renders for robustness (Georgiou et al., 19 Feb 2025).
• Hash-Decoded Shell Features (NeST-Splatting, NeRF-Texture): Features are mapped from 3D shell positions via a multi-level hash-grid and decoded with a lightweight MLP. Geometry primitives (Gaussians, mesh vertices) act as pure samplers, fully decoupling geometry from appearance. Training employs multi-loss objectives (photometric, geometric, and regularization terms), with parameter efficiency (2–28MB models, 20–80K surfels) and support for mesh extraction (Zhang et al., 27 Jul 2025, Huang et al., 2024).
• Mesh Neural Cellular Automata: Per-vertex update rules are realized as residual MLPs consuming neighbor-aggregated features built from spherical harmonics message-passing. The system supports multi-modal supervision (images, text, motion fields) and generalizes across meshes without re-training (Pajouheshgar et al., 2023).
• Implicit Neural Representations (INR): MLPs (plain ReLU, SIREN, Fourier-encoded) are trained to regress UV or barycentric coordinates to RGB, possibly extended with extra arguments (level-of-detail, normal, curvature). Losses include $L_2$, MAE, PSNR, SSIM, and perceptual metrics; efficient variants enable real-time applications with small memory footprints (Kwok et al., 2 Feb 2026).

Training protocols typically optimize only the neural parameterization (MLP weights, hash-embeddings, CA rules) over loss functions measuring per-pixel/voxel error, statistical similarity (FID, KID), or high-level alignment (CLIP-score, clustering constraints), sometimes employing augmentation for robustness to input noise and sampling biases.

4. Geometric and Positional Encoding Techniques

A critical aspect for shell texture coherence is geometry-aware positional encoding:

• Geodesic Distance Encoding: Geodesic distances $\delta_{p, u}$ between pairs of pixels or texels on the surface are computed through mesh graph traversal and fed into the attention blocks or MLPs, aligning neural attention with true local patches irrespective of mesh topology (Georgiou et al., 19 Feb 2025).
• Chart and Tangent Frame Parameterizations: Local tangent frames and projective barycentric coordinates are used to align feature patches, enable patch-matching schemes, and handle anisotropy or deformation (Huang et al., 2024).
• Fourier/Frequency Features: High-frequency information is encoded via Fourier-based input mappings, improving the expressiveness of INR architectures (Kwok et al., 2 Feb 2026).
• Spherical Harmonic Filters: Directionally-aware message-passing on the vertex graph leverages SH bases for isotropic neighborhood aggregation (MeshNCA) (Pajouheshgar et al., 2023).

This geometric anchoring enables texture patterns to propagate seamlessly across highly-curved regions, avoid seam artifacts, and respect local surface structure.

5. Evaluation Metrics and Empirical Comparisons

Quantitative and qualitative evaluation spans standard computer vision and graphics benchmarks:

• Im2SurfTex demonstrates FID drop from 29.13→27.34 (Paint3D backbone) and 28.68→26.68 (MatAtlas), with KID reductions, confirming the efficacy of neural cross-attention over heuristic backprojection (Georgiou et al., 19 Feb 2025).
• NeST-Splatting achieves PSNR/SSIM/LPIPS on NeRFSynthetic datasets matching or exceeding prior methods (33.50/0.967/0.032 with only 73K surfels), showing robust texture detail with a 3× reduction in the number of geometric primitives (Zhang et al., 27 Jul 2025).
• NeRF-Texture secures PSNR 32–43 dB, SSIM > 0.98, and SIFID ≈ 0.6 compared to ≈ 1.0 for classic 2D synthesis, robustly capturing view-dependent and meso-structure effects (Huang et al., 2024).
• MeshNCA exhibits strong generalization, outperforming direct UV and per-vertex coloring, with features such as zero-shot application to unseen meshes, grafting-based blending, and real-time interactive modification (Pajouheshgar et al., 2023).
• INR-based textures support fast training (0.5–2 it/s, 50–200 s per network), compact representation (e.g., 0.7 MB for PSNR > 30 dB), and controllable quality-performance tradeoffs (Kwok et al., 2 Feb 2026).

Ablation studies reinforce the importance of geometric encoding, local feature aggregation, and network depth. Qualitative gains include the elimination of seam artifacts, preservation of high-frequency and view-dependent appearance, and robust degradation with reduced model complexity.

6. Applications and Practical Considerations

Neural shell textures enable:

• Novel view synthesis and re-rendering: Supporting sharp, photorealistic appearance under arbitrary viewpoints with high parameter efficiency (Zhang et al., 27 Jul 2025, Huang et al., 2024).
• Interactive authoring and real-time shading: MeshNCA demonstrates real-time (20–50 fps) texture editing, including brush-based regeneration, grafting, and procedural control, entirely on-GPU (Pajouheshgar et al., 2023).
• Mesh extraction and texture baking: Learned shell fields can be sampled during surface marching (Marching Triangles/Marching Cubes) and “baked” into UV atlases or mesh vertex colors, facilitating export to graphics pipelines (Zhang et al., 27 Jul 2025, Huang et al., 2024).
• Procedural and dynamic texture generation: CA-based methods enable motion-controlled, density/orientation-adjustable, and grafted pattern synthesis (Pajouheshgar et al., 2023).
• Data-driven and generative modeling: INR-space diffusion and patch-based generation allow for stochastic, data-driven creation of new surface textures (Kwok et al., 2 Feb 2026, Huang et al., 2024).

Rendering performance is contingent on network size, feature decoding complexity, and sampling strategy. Efficient hash grids and compact MLPs render high-res textures with modest memory and compute, while CA and INR variants leverage parallelism for large-scale deployment. Limitations include handling of extreme topological complexity, strict semantic preservation, and potential artifacts in highly regular or topologically narrow regions (Huang et al., 2024, Kwok et al., 2 Feb 2026).

7. Technical Comparison Across Paradigms

Approach	Geometry Coupling	Texture Expressivity	Interactive Controls
Im2SurfTex (Georgiou et al., 19 Feb 2025)	UV/surface-aware	High, cross-attention	Limited
NeST-Splatting (Zhang et al., 27 Jul 2025)	Gaussian surfels + neural field	High, hash-grid/MLP	Mesh extraction supported
NeRF-Texture (Huang et al., 2024)	Mesh vertices/+latent	High, view-dependent	Patch-matching synthesis
MeshNCA (Pajouheshgar et al., 2023)	Mesh-local CA	Moderate-High, dynamic	Real-time, grafting, painting
INR (Kwok et al., 2 Feb 2026)	UV/3D coordinates	Moderate-High, continuous	Mipmap, generative (weight-space diffusion)

These approaches share a unifying trend: the continuous, geometry-aware, neural parameterization of surface appearance, supporting seamless texture mapping for arbitrary 3D shells with efficiency and flexibility previously unattainable in classical computer graphics pipelines.