Neural Image-Space Tessellation (NIST)

• Neural Image-Space Tessellation is a method that reformulates traditional geometric tessellation as a post-process, using discrepancies between geometric and shading normals to refine silhouettes.
• It operates by applying multi-scale neural deformation and feature warping on low-poly renders, tying computational cost to image resolution rather than mesh complexity.
• The approach leverages G-buffers and an encoder-decoder architecture to produce smooth, perceptually enhanced silhouettes without altering the underlying geometry.

Neural Image-Space Tessellation (NIST) is a real-time neural screen-space technique for silhouette smoothing that produces the visual effect of tessellated geometry while rendering only the original low-polygon meshes. Introduced in "Neural Image Space Tessellation" (Du et al., 27 Feb 2026), it reframes tessellation from an object-space geometry operation into a post-processing stage that consumes a low-poly rendered image together with G-buffers, notably depth, geometric normals, and shading normals. Its central premise is that the discrepancy between geometric normals and shading normals provides a minimal, view-dependent cue for where silhouette refinement is perceptually needed, allowing multi-scale image-space deformation and appearance warping to approximate the effect of geometric tessellation at a cost determined by image resolution rather than scene geometric complexity.

1. Problem Setting and Screen-Space Reformulation

In real-time graphics, tessellation in the broad practical sense refers to refining coarse meshes into higher-resolution surfaces and then displacing or smoothing them in order to smooth silhouettes, better approximate intended smooth shapes from coarse control meshes, and support displacement mapping. Classical solutions such as Catmull-Clark subdivision, Loop subdivision, PN-Triangles, Phong tessellation, and hardware tessellation stages operate in object space or in the GPU tessellation stage. Their cost scales with mesh complexity and visible surface area, and in large real-time scenes this increases vertex and triangle bandwidth, makes visibility and rasterization more expensive, and reduces frame rate, particularly under close-ups (Du et al., 27 Feb 2026).

NIST reformulates this workload as an image-space problem. Rather than refining geometry, it renders low-poly meshes conventionally to obtain a base image and G-buffers, then post-processes the resulting frame in screen space to smooth silhouettes and contours while consistently reassigning appearance. The method is motivated by the observation that the perceptual benefit of tessellation is view-dependent and manifests primarily at visible silhouettes in the final 2D image. By moving the operation from the geometry pipeline to a neural post-processing stage, NIST aligns computational cost with image resolution rather than with the number of triangles or patches in the scene.

This reformulation is also the basis for the paper’s claim that NIST is the first work to cast tessellation as a post-processing operation rather than a pre-rendering geometry operation. The distinction is not merely implementation-level: it changes the scaling law of the technique, from geometry-dependent to resolution-dependent, while preserving the target visual effect of silhouette smoothing.

2. Geometric Cue and Relation to Phong Tessellation

The key signal exploited by NIST is the discrepancy between geometric normals and shading normals. Geometric normals are the per-face normals of the coarse mesh and therefore represent the actual flat geometry. Shading normals are interpolated vertex normals used for smooth shading and therefore represent the smooth surface that the image already suggests perceptually. Where the two agree, the coarse mesh is already consistent with the perceived surface; where they differ, the silhouette implied by smooth shading diverges from the true coarse geometry, especially near contour regions (Du et al., 27 Feb 2026).

This cue is explicitly connected to Phong tessellation. In Phong tessellation, a point $p$ inside a triangle is projected onto the normal planes defined by the triangle’s vertices and vertex normals,

$$\pi_*(p) = p - \big((p - v_*)^T n_*\big) n_*, \quad * \in \{i,j,k\},$$

and the projected points are interpolated barycentrically,

$p^* = u\,\pi_i(p) + v\,\pi_j(p) + w\,\pi_k(p).$

The final position is obtained by blending the original point and the deformed point. This procedure still requires actual geometric subdivision and per-vertex displacement, but it already relies implicitly on the same criterion: when shading normals and geometric normals differ, the displaced surface departs from the original triangle plane; when they agree, deformation is minimal.

NIST inherits this conceptual criterion while discarding geometric refinement. Instead of moving 3D vertices, it moves 2D feature positions in image space, guided by normals and depth, so that contours bend toward shapes consistent with the implied smooth surface. The method therefore resembles Phong tessellation in the origin of its deformation cue, but differs fundamentally in representation and execution: it never constructs a tessellated mesh, never changes object-space geometry, and operates entirely after rendering.

3. Architecture and Multi-Scale Neural Tessellation

NIST takes as input the low-poly rendered color image $I^{input}$ and three camera-space G-buffers: depth $D$, geometric normals $N_g$, and shading normals $N_s$. No high-resolution geometry or tessellation output is required at run time. These inputs are stacked to form the network input, and the normals are transformed into camera space for better view-dependent generalization (Du et al., 27 Feb 2026).

The architecture is an encoder-decoder with a multi-scale neural tessellation pipeline. At each scale it contains two principal components. The first is an Implicit Deformation Module, which learns a latent deformation state $z_d^{(t)}$ that encodes contour-aware 2D deformation using guidance features derived from normals and depth rather than color. The second is a Feature Warping Module, which predicts an explicit backward warping field from the latent deformation state and uses it to warp both color features and guidance features.

Guidance encoding first transforms $(N_g, N_s, D)$ into a guidance feature volume $z_g$. This representation captures disagreement between geometric and shading normals together with depth discontinuities, both of which are important for silhouettes and visibility reasoning. The deformation module then updates the latent state at each scale through an attention-based mechanism:

$$\pi_*(p) = p - \big((p - v_*)^T n_*\big) n_*, \quad * \in \{i,j,k\},$$0

with $$\pi_*(p) = p - \big((p - v_*)^T n_*\big) n_*, \quad * \in \{i,j,k\},$$1 at the first scale and $$\pi_*(p) = p - \big((p - v_*)^T n_*\big) n_*, \quad * \in \{i,j,k\},$$2 at later scales. The update is

$$\pi_*(p) = p - \big((p - v_*)^T n_*\big) n_*, \quad * \in \{i,j,k\},$$3

followed by

$$\pi_*(p) = p - \big((p - v_*)^T n_*\big) n_*, \quad * \in \{i,j,k\},$$4

The Deform block is a residual convolutional block with a larger $$\pi_*(p) = p - \big((p - v_*)^T n_*\big) n_*, \quad * \in \{i,j,k\},$$5 kernel to expand receptive field and better capture structured contour changes.

The architecture does not predict an explicit displacement map as its primary representation of deformation. Instead, deformation is encoded implicitly in latent feature space. The explicit 2D motion appears in the Feature Warping Module:

$$\pi_*(p) = p - \big((p - v_*)^T n_*\big) n_*, \quad * \in \{i,j,k\},$$6

where $$\pi_*(p) = p - \big((p - v_*)^T n_*\big) n_*, \quad * \in \{i,j,k\},$$7 is the backward displacement for pixel $$\pi_*(p) = p - \big((p - v_*)^T n_*\big) n_*, \quad * \in \{i,j,k\},$$8. Warping is performed with a differentiable sampling operator, typically bilinear grid sampling. The network also maintains a cumulative warp,

$$\pi_*(p) = p - \big((p - v_*)^T n_*\big) n_*, \quad * \in \{i,j,k\},$$9

so that high-resolution encoder features can be mapped consistently back to the original low-poly image space. Guidance features are warped to remain aligned with already applied deformations, and color features from previous and current scales are warped and concatenated to form the refined feature representation passed forward.

A notable implementation characteristic is that the network performs most deformation and warping at reduced resolution, specifically 360p, while only early color feature extraction remains at full resolution. This design is directly tied to the method’s mild runtime scaling with output resolution.

4. Supervision, Training Configuration, and Engine Integration

NIST is supervised with high-quality reference images rendered with geometric tessellation, specifically PN-Triangles in UE4.27. Training pairs consist of a no-tessellation input render and a geometrically tessellated label render. The paper uses four scenes: SoulCave, Cowboy, and Bronze for training and testing splits, and Junkyard as a test-only scene for generalization (Du et al., 27 Feb 2026).

The learning objective combines three losses. The residual-relative loss,

$p^* = u\,\pi_i(p) + v\,\pi_j(p) + w\,\pi_k(p).$0

with $p^* = u\,\pi_i(p) + v\,\pi_j(p) + w\,\pi_k(p).$1, reweights errors by how much tessellation changes a pixel in the reference. This emphasizes deformed or silhouette-sensitive regions rather than unchanged regions. The shading-augmented loss focuses on the $p^* = u\,\pi_i(p) + v\,\pi_j(p) + w\,\pi_k(p).$2 worst pixels by absolute error,

$p^* = u\,\pi_i(p) + v\,\pi_j(p) + w\,\pi_k(p).$3

thereby concentrating supervision near the network’s current failure modes. A perceptual LPIPS loss is added to preserve high-frequency texture and reduce over-smoothing. The total objective is

$p^* = u\,\pi_i(p) + v\,\pi_j(p) + w\,\pi_k(p).$4

with fixed weights $p^* = u\,\pi_i(p) + v\,\pi_j(p) + w\,\pi_k(p).$5, $p^* = u\,\pi_i(p) + v\,\pi_j(p) + w\,\pi_k(p).$6, and $p^* = u\,\pi_i(p) + v\,\pi_j(p) + w\,\pi_k(p).$7.

The implementation uses PyTorch and Double Conv blocks, defined as two $p^* = u\,\pi_i(p) + v\,\pi_j(p) + w\,\pi_k(p).$8 convolutions followed by LeakyReLU, except for the Deform Block, whose first convolution uses a $p^* = u\,\pi_i(p) + v\,\pi_j(p) + w\,\pi_k(p).$9 kernel. Training uses Adam with learning rate $I^{input}$0, weight decay $I^{input}$1, batch size $I^{input}$2, and a single NVIDIA A100 GPU, with training time of approximately 24 hours per model.

At inference, the reported desktop platform is an i9-14900K with an RTX 5090. Mean latency figures are as follows:

Resolution	GPU Compute (ms)	Total (ms)
360p	4.520	4.699
720p	5.000	5.673
1080p	6.186	7.680

The GPU compute portion scales only mildly with resolution because the heavy deformation and warping stages are performed at 360p. Integration into a rendering engine is correspondingly simple: render the frame normally without tessellation, output $I^{input}$3, $I^{input}$4, $I^{input}$5, and $I^{input}$6, invoke NIST as a post-process compute pass, and use the resulting image as the final frame or as input to subsequent post-processing. The paper states that no changes to meshes, LOD systems, or material and shader workflows are necessary beyond outputting both types of normals.

5. Visual Behavior, Comparative Evaluation, and Ablations

The qualitative evaluation emphasizes four representative scenes. In SoulCave, NIST removes faceting and recovers smooth outlines on large curved structures. In Cowboy, it provides subtle smoothing on organic character surfaces while preserving details such as folds and highlights. In Bronze, it produces smooth edges and coherent shading transitions on a metallic statue with complex silhouettes. In Junkyard, a mechanically cluttered scene with many straight edges and rigid structures, it refines silhouettes while preserving straight beams and edges where normals align; the paper notes that PN-Triangles may bend such structures (Du et al., 27 Feb 2026).

The comparative behavior against geometric tessellation is therefore specific. Traditional tessellation produces geometrically defined surfaces and can recover sub-pixel geometric detail if tessellation density is sufficient, but its cost grows with tessellated primitives. NIST instead targets perceptual silhouette consistency rather than exact geometric reconstruction. The paper explicitly states that it is not pixel-exact and does not attempt a 1:1 geometry match. Its strong result is that smooth, visually coherent silhouettes can be obtained in many scenes without touching geometry, and that straight edges are preserved where geometric normals and shading normals are already consistent.

Quantitative evaluation is described more in terms of perceptual error and ablation than conventional image fidelity metrics. FLIP error maps are used in the ablation studies, where the full model shows lower errors than ablated variants. No explicit PSNR or SSIM tables are provided, which is consistent with the method’s emphasis on perceptual silhouette quality rather than exact pixel fidelity.

Three ablations clarify the role of the model components. Removing the implicit deformation module and replacing it with a simple $I^{input}$7 convolution causes the network to fail to learn structured image-space deformation, leaving silhouettes close to the low-poly input. Removing the feature warping module leads to visible seams and discontinuities because deformation in latent space is no longer accompanied by consistent appearance reassignment. Removing LPIPS yields structurally plausible deformations but overly smooth outputs, with loss of fine texture detail and shading nuance. The full model gives the best perceptual match to PN-Triangles in both visual inspection and FLIP error.

6. Limitations, Practical Scope, and Future Work

NIST inherits the strengths and the constraints of a screen-space method. Because it only sees what is currently visible on the screen, partially visible triangles may have incomplete geometric context, which can produce local deformation artifacts near their boundaries. Completely occluded geometry cannot be deformed at all. Thin structures such as wires or fences, and silhouettes defined by nearly sub-pixel triangles, are also difficult because the available image-space cues may be ambiguous (Du et al., 27 Feb 2026).

The current training regime is per-scene. The paper characterizes cross-scene generalization as acceptable but not scene-agnostic when training on a mixture of scenes, and states that this is not yet robust enough for plug-and-play deployment across arbitrary content. Shading normals in the present system are only interpolated vertex normals; normal maps are not integrated, so the method targets large-scale silhouette smoothing rather than microscopic detail enhancement. Temporal modeling is also absent: NIST operates per frame, without recurrent structure, temporal attention, or motion-vector conditioning. Temporal coherence is described as acceptable in practice, but explicit temporal constraints and sequence training are identified as future directions.

These limitations delimit the domain in which NIST is intended to operate. It is well suited to games and VR/AR, large-scale scenes, mobile or console GPUs with constrained geometry-processing bandwidth, and asset authoring workflows that favor lower-poly meshes with well-authored normals. The practical implication is that frame-time budgeting can become more stable because the method’s cost is geometry-independent; the paper explicitly contrasts visible-face counts ranging from hundreds of thousands to tens of millions with the constant-cost behavior of the neural post-process. At the same time, NIST cannot affect collision, invisible geometry, or geometry-dependent passes such as shadow maps that rely on independent geometric representations.

The future-work directions suggested in the paper are consistent with these boundaries. They include model compression through distillation, quantization, and architecture pruning; improved generalization through larger and more diverse datasets; explicit temporal modeling with recurrent or transformer-based modules and temporal losses; incorporation of normal maps and displacement-like effects into the guidance; and joint training with denoising, upsampling, frame extrapolation, or other screen-space effects to form a unified neural post-processing stack. Collectively, these directions indicate that NIST is less a replacement for geometry as such than a new rendering-stage abstraction: tessellation-like silhouette refinement performed entirely in image space, conditioned by geometric cues but no longer coupled to geometric complexity.