Sketch Gaussians in Structure-Aware Rendering

• Sketch Gaussians are structured Gaussian primitives that encode geometrically salient content such as edges, contours, and boundaries using explicit parametric sketches or boundary clustering.
• They are generated either from parametric 3D sketches (as in SketchSplat) or identified within 3D Gaussian Splatting models (as in Sketch and Patch and Sketch&Patch++), driving efficient scene and image representations.
• This structure-aware design pattern enables differentiable optimization, improved compression, and progressive streaming by exploiting spatial, directional, and color coherence among Gaussian clusters.

Searching arXiv for the cited papers and closely related work on Sketch Gaussians. Sketch Gaussians are Gaussian primitives used to encode structurally salient content—most commonly edges, contours, and boundary-defining features—under an explicit geometric or parametric prior rather than as fully unconstrained splats. In recent work, the term has been used in at least three technically distinct but related senses: as Gaussian samples derived from parametric 3D sketches for differentiable multi-view edge reconstruction in SketchSplat (Ying et al., 18 Mar 2025); as the boundary-focused subset of a 3D Gaussian Splatting model in hybrid compression schemes such as Sketch and Patch (Shi et al., 22 Jan 2025) and Sketch&Patch++ (Shi et al., 8 Jan 2026); and, more loosely, as Gaussian primitives arranged to represent sketch-like or edge-dominant image content in 2D Gaussian representations such as Image-GS (Zhang et al., 2024) and EigenGS (Tai et al., 10 Mar 2025). Across these formulations, the common principle is that Gaussian primitives are constrained to reflect structural coherence—curve topology, line support, or dense boundary-aligned clustering—rather than serving only as generic radiance carriers.

1. Terminological scope and core formulations

In SketchSplat, Sketch Gaussians are 3D Gaussian primitives whose parameters are derived from parametric 3D sketches rather than optimized independently (Ying et al., 18 Mar 2025). The underlying representation models 3D edges as sketches composed of either straight lines,

$l \in \mathbb{R}^{2\times 3},$

or cubic Bézier curves,

$c \in \mathbb{R}^{4\times 3}.$

A sketch $S_i$ is defined as

$S_i = \{\, \text{geometry},\ o_i,\ s_i \,\},$

where the geometry is either a line $l_i$ or a Bézier curve $c_i$, $o_i \in \mathbb{R}$ is opacity, and $s_i \in \mathbb{R}^3$ is a local scale controlling thickness and anisotropic spread. Gaussian primitives are then sampled along these sketches and rendered by 3D Gaussian splatting, so that Gaussian means lie on the sketch and covariances are induced by local tangent direction and scale.

In Sketch and Patch, by contrast, Sketch Gaussians are not generated from explicit sketches but are identified as a subset of an already trained 3DGS model (Shi et al., 22 Jan 2025). They are the Gaussians that lie close to reconstructed 3D line segments and exhibit coherent opacity, color, scale, and rotation behavior along those lines. Their defining characteristic is therefore not sampling from a parametric curve, but being boundary-aligned Gaussians that can be compactly encoded by per-line polynomial models.

Sketch&Patch++ generalizes this idea to arbitrary scenes without external line primitives (Shi et al., 8 Jan 2026). There, Sketch Gaussians are the subset of 3DGS splats that form dense, elongated, attribute-coherent clusters along edges and boundaries. These clusters are detected directly from the 3DGS representation by multi-criteria DBSCAN followed by polynomial-regression-based refinement, then encoded parametrically.

A plausible implication is that “Sketch Gaussians” names a family of structure-aware Gaussian representations rather than a single standardized object. The shared theme is the use of Gaussian primitives as a structural scaffold, especially for high-frequency or boundary-defining content.

2. SketchSplat: Gaussian primitives derived from parametric sketches

SketchSplat defines perhaps the most literal formulation of Sketch Gaussians: Gaussians sampled from explicit 3D line and curve primitives under differentiable multi-view supervision (Ying et al., 18 Mar 2025). The 3D Gaussian density follows the usual 3DGS form,

$$G(x) = e^{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)}, \qquad \mu \in \mathbb{R}^3,\ \Sigma \in \mathbb{R}^{3\times 3}.$$

However, the method does not optimize free Gaussians. Instead, all Gaussian attributes are derived from sketches. The mean $\mu$ lies on the sketch; the covariance $c \in \mathbb{R}^{4\times 3}.$0 is constructed from the sketch’s local scale $c \in \mathbb{R}^{4\times 3}.$1 and tangent direction; and the Gaussian opacity equals the sketch opacity $c \in \mathbb{R}^{4\times 3}.$2.

At each optimization step, SketchSplat uniformly samples points along each sketch with a fixed geometric step of 5 mm in all experiments (Ying et al., 18 Mar 2025). For a parametric curve $c \in \mathbb{R}^{4\times 3}.$3, sampled positions satisfy

$c \in \mathbb{R}^{4\times 3}.$4

and each point uses the local normalized tangent

$c \in \mathbb{R}^{4\times 3}.$5

to define Gaussian orientation. The resulting set of Gaussians forms a sampled Gaussian field subordinate to the sketch geometry.

Rendering uses 3D Gaussian splatting with calibrated pinhole cameras. For a ray $c \in \mathbb{R}^{4\times 3}.$6, intensity is accumulated as

$c \in \mathbb{R}^{4\times 3}.$7

where $c \in \mathbb{R}^{4\times 3}.$8 is the line integral of the 3D Gaussian along the ray. For edge reconstruction, color is effectively specialized to single-channel edge intensity (Ying et al., 18 Mar 2025). The rendered edge map $c \in \mathbb{R}^{4\times 3}.$9 is supervised by an image-space $S_i$0 loss,

$S_i$1

This construction makes the entire bridge from 3D sketches to 2D edge images differentiable. Because Gaussian mean, covariance, and opacity are all differentiable functions of sketch parameters, image-space error can be back-propagated directly to line endpoints, Bézier control points, local scale, and opacity. SketchSplat presents this as the key distinction from methods that first reconstruct 3D edge point sets and only later fit curves (Ying et al., 18 Mar 2025). In those pipelines, final parametric edges are downstream of reconstruction; here, parametric sketches are primary and Gaussians are secondary.

3. Differentiable optimization, supervision, and topology in SketchSplat

The differentiable nature of Sketch Gaussians in SketchSplat is central to its reconstruction behavior (Ying et al., 18 Mar 2025). For a sketch parameter $S_i$2, the gradient takes the chain-rule form

$S_i$3

with Gaussian parameters depending on the sketch through curve evaluation, tangent-dependent covariance construction, and shared opacity. The paper states that derivatives of the splatting equations with respect to $S_i$4, $S_i$5, and $S_i$6 are the same as in 3DGS and are implemented in automatic differentiation (Ying et al., 18 Mar 2025).

SketchSplat couples this differentiable rendering with adaptive topological operations. Sketches share a global optimizable point set $S_i$7, enabling endpoint merging when two endpoints are closer than

$S_i$8

Overlapping sketch merging uses sampled-point overlap with proximity threshold

$S_i$9

and overlap threshold

$S_i = \{\, \text{geometry},\ o_i,\ s_i \,\},$0

Co-linear line merging additionally requires angle difference

$S_i = \{\, \text{geometry},\ o_i,\ s_i \,\},$1

and orthogonal offset

$S_i = \{\, \text{geometry},\ o_i,\ s_i \,\},$2

After optimization, visibility filtering discards sketches if more than

$S_i = \{\, \text{geometry},\ o_i,\ s_i \,\},$3

of sampled points are invisible, where a point is considered invisible if it fails to land on an edge in more than 90% of views (Ying et al., 18 Mar 2025).

These operations act on sketches, but because Gaussians are sampled from sketches, they also change the Gaussian field. This yields a more compact and topologically cleaner structure. On the ABC-NEF benchmark with 82 CAD models and the paper’s 2DGS-SN edge detector, SketchSplat reports

$S_i = \{\, \text{geometry},\ o_i,\ s_i \,\},$4

and $S_i = \{\, \text{geometry},\ o_i,\ s_i \,\},$5-num $S_i = \{\, \text{geometry},\ o_i,\ s_i \,\},$6 edges, compared with EdgeGS at

$S_i = \{\, \text{geometry},\ o_i,\ s_i \,\},$7

and $S_i = \{\, \text{geometry},\ o_i,\ s_i \,\},$8-num $S_i = \{\, \text{geometry},\ o_i,\ s_i \,\},$9 edges under the same 2DGS-SN supervision (Ying et al., 18 Mar 2025). An ablation without merging operations yields nearly identical $l_i$0 but $l_i$1-num $l_i$2, indicating that the differentiable Gaussian-sketch coupling drives accuracy and completeness while topology control drives compactness.

The same paper also identifies the quality of 2D edge supervision as critical. Its 2DGS-SN detector combines foreground mask $l_i$3, depth map $l_i$4, and normal map $l_i$5 via

$l_i$6

with Sobel gradient magnitudes $l_i$7 and $l_i$8 (Ying et al., 18 Mar 2025). On ABC-NEF, replacing DexiNed with 2DGS-SN raises EdgeGS from 45.2% to 80.3% F5 and SketchSplat from 50.3% to 91.3% F5, showing that the Sketch Gaussian optimization is highly sensitive to geometric edge-map fidelity.

4. Sketch Gaussians in hybrid 3DGS compression: Sketch and Patch

In Sketch and Patch, Sketch Gaussians are a subset of 3D Gaussian splats that define scene boundaries in man-made scenes (Shi et al., 22 Jan 2025). The paper begins from a standard 3DGS parameterization with center $l_i$9, covariance $c_i$0, opacity $c_i$1, and spherical harmonics coefficients $c_i$2, trained with

$c_i$3

The method observes that adaptive densification produces dense Gaussian clusters along high-frequency regions such as edges and contours, while smoother surfaces are represented by more weakly structured Gaussians.

To detect Sketch Gaussians, the method reconstructs 3D line segments $c_i$4 using Line3D++, parameterized by

$c_i$5

A Gaussian $c_i$6 with center $c_i$7 is a candidate for line $c_i$8 if its distance to the segment,

$c_i$9

is within a search radius $o_i \in \mathbb{R}$0 (Shi et al., 22 Jan 2025). These candidates are then filtered by attribute coherence. For each attribute—opacity, color, scale, and rotation—the method fits a polynomial as a function of line coordinate $o_i \in \mathbb{R}$1, using RANSAC with residual threshold

$o_i \in \mathbb{R}$2

where MAD is the median absolute deviation of residuals. Only Gaussians that are inliers for all attributes are retained as Sketch Gaussians.

The resulting compression model stores, per line, polynomial coefficients for opacity,

$o_i \in \mathbb{R}$3

color,

$o_i \in \mathbb{R}$4

scale,

$o_i \in \mathbb{R}$5

and rotation,

$o_i \in \mathbb{R}$6

with polynomial degree selected by grid search in $o_i \in \mathbb{R}$7 (Shi et al., 22 Jan 2025). The original per-Gaussian attributes are then discarded, and decoded Sketch Gaussians are fixed during subsequent retraining of the remaining Patch Gaussians.

This decomposition yields a hybrid representation: a parametric sketch layer for boundary Gaussians and a pruned, retrained, vector-quantized patch layer for smooth regions. On four scenes at matched SSIM, Sketch+Patch reduces model size to roughly 2–5% of vanilla 3DGS: Playroom from 631.44 MB to 19.07 MB, Drjohnson from 844.48 MB to 41.46 MB, Room from 395.16 MB to 21.05 MB, and Truck from 630.23 MB to 36.59 MB (Shi et al., 22 Jan 2025). Across these scenes, the method reports up to 32.62% improvement in PSNR, 19.12% in SSIM, and 45.41% in LPIPS at equivalent model sizes, with matched-quality sizes of 2.35%, 5.41%, 4.83%, and 3.46% of the original model for Playroom, Drjohnson, Room, and Truck respectively (Shi et al., 22 Jan 2025).

The paper frames these results as evidence that edge-aligned Gaussians are substantially more valuable per byte than uniformly stored generic Gaussians. This suggests that Sketch Gaussians act as a structural scaffold whose preservation allows far more aggressive reduction of the remaining volumetric content.

5. Sketch&Patch++: structure-aware categorization without external lines

Sketch&Patch++ extends the Sketch/Patch distinction to arbitrary 3D scenes by removing the dependence on external 3D line reconstruction (Shi et al., 8 Jan 2026). Instead of line proximity, Sketch Gaussians are detected directly from the geometry and attributes of the 3DGS solution itself.

The method defines three pairwise distances between Gaussians $o_i \in \mathbb{R}$8 and $o_i \in \mathbb{R}$9: spatial distance,

$s_i \in \mathbb{R}^3$0

directional distance,

$s_i \in \mathbb{R}^3$1

where $s_i \in \mathbb{R}^3$2 is the principal direction derived from covariance; and color distance,

$s_i \in \mathbb{R}^3$3

A Gaussian is a neighbor only if all three satisfy thresholds $s_i \in \mathbb{R}^3$4, after which DBSCAN identifies dense, coherent clusters (Shi et al., 8 Jan 2026).

Candidate clusters are then tested for parametric compressibility by polynomial regression of scale, rotation, opacity, and color against normalized 3D position $s_i \in \mathbb{R}^3$5. Using polynomial basis

$s_i \in \mathbb{R}^3$6

with degree $s_i \in \mathbb{R}^3$7, the method computes per-attribute mean squared error and a combined score

$s_i \in \mathbb{R}^3$8

Clusters with

$s_i \in \mathbb{R}^3$9

are accepted as Sketch Gaussian clusters; others are recursively split by K-means over fused residual and spatial features until they either satisfy the threshold or are rejected as Patch Gaussians (Shi et al., 8 Jan 2026).

This procedure produces a broader notion of Sketch Gaussians than the line-based original. On the Room scene, the paper reports SketchGS: 1.16M Gaussians and PatchGS: 0.43M Gaussians; SketchGS clusters are about $$G(x) = e^{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)}, \qquad \mu \in \mathbb{R}^3,\ \Sigma \in \mathbb{R}^{3\times 3}.$$0 denser, have average elongation 2.82 versus 1.99, and occupy approximately $$G(x) = e^{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)}, \qquad \mu \in \mathbb{R}^3,\ \Sigma \in \mathbb{R}^{3\times 3}.$$1 smaller spatial volume, 1.30 versus 35.11 unit$$G(x) = e^{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)}, \qquad \mu \in \mathbb{R}^3,\ \Sigma \in \mathbb{R}^{3\times 3}.$$2 (Shi et al., 8 Jan 2026). Across datasets, up to 88.7% of Gaussians in indoor scenes and 65.9% in outdoor scenes are classified as SketchGS (Shi et al., 8 Jan 2026).

Compression is then applied separately to SketchGS and PatchGS. SketchGS centers are stored explicitly and attributes are reconstructed from polynomial coefficients; PatchGS are pruned, retrained, and quantized. On DeepBlending, the method reports compression from 703.77 MB to 4.03 MB, equivalent to $$G(x) = e^{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)}, \qquad \mu \in \mathbb{R}^3,\ \Sigma \in \mathbb{R}^{3\times 3}.$$3 compression and 0.5% of original size, with SketchGS compressed $$G(x) = e^{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)}, \qquad \mu \in \mathbb{R}^3,\ \Sigma \in \mathbb{R}^{3\times 3}.$$4 and PatchGS $$G(x) = e^{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)}, \qquad \mu \in \mathbb{R}^3,\ \Sigma \in \mathbb{R}^{3\times 3}.$$5 (Shi et al., 8 Jan 2026). At around 38 MB on Mip-NeRF360, Sketch&Patch++ improves over uniform prune-and-retrain by +1.74 dB PSNR, +6.7% SSIM, and 41.4% improvement in LPIPS (Shi et al., 8 Jan 2026).

A distinctive consequence of this decomposition is layered progressive streaming. SketchGS define layer $$G(x) = e^{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)}, \qquad \mu \in \mathbb{R}^3,\ \Sigma \in \mathbb{R}^{3\times 3}.$$6, a compact structural base; PatchGS form enhancement layers $$G(x) = e^{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)}, \qquad \mu \in \mathbb{R}^3,\ \Sigma \in \mathbb{R}^{3\times 3}.$$7. In the Playroom example, total compressed size is 17.45 MB, with SketchGS alone occupying 1.48 MB, or 8.5% of the total (Shi et al., 8 Jan 2026). Rendering only $$G(x) = e^{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)}, \qquad \mu \in \mathbb{R}^3,\ \Sigma \in \mathbb{R}^{3\times 3}.$$8 yields what the paper describes as a crisp structural sketch of edges, contours, and silhouettes.

6. Relation to 2D Gaussian image representations and volumetric Gaussian models

The term “Sketch Gaussians” also appears in looser or analogy-based senses outside edge-parametric and structure-aware 3DGS compression. Image-GS represents images as content-adaptive 2D anisotropic Gaussians,

$$G(x) = e^{-\frac{1}{2}(x-\mu)^T \Sigma^{-1}(x-\mu)}, \qquad \mu \in \mathbb{R}^3,\ \Sigma \in \mathbb{R}^{3\times 3}.$$9

with

$\mu$0

and per-Gaussian parameter vector

$\mu$1

(Zhang et al., 2024). Although the paper does not formalize a separate sketch layer, it explicitly notes that its sparse anisotropic Gaussians are conceptually close to the representation one would want for sketches, line drawings, or stylized content. It renders by order-invariant top-$\mu$2 normalized blending,

$\mu$3

and reports 0.3K MACs to decode a pixel, $\mu$4 speedup when increasing BSP blocks from 16 to 192, and 0.70 ms for rendering 10k pixels in pure PyTorch on RTX 3080 (Zhang et al., 2024). A plausible implication is that 2D Gaussian image representations provide a practical substrate for sketch-like Gaussian encoding even when explicit sketch topology is absent.

EigenGS pushes this further by constructing 2D Gaussian image representations from PCA eigenspaces (Tai et al., 10 Mar 2025). Each image is approximated by a shared Gaussian geometry and image-specific weights derived analytically from PCA coefficients,

$\mu$5

yielding an immediate Gaussian image decomposition

$\mu$6

Its frequency-aware training splits Gaussians into low- and high-frequency subsets, producing a bimodal size distribution and reducing the “penny-round-tile” artifact (Tai et al., 10 Mar 2025). On FFHQ with 20k Gaussians, GaussianImage reports PSNR $\mu$7 dB at iteration 100, 29.4 dB at 1000, and 40.1 dB at 10,000, while EigenGS reports 28.0 dB at iteration 0, 34.4 dB at 100, 37.5 dB at 1000, and 41.8 dB at 10,000; more than 80% of images exceed 35 dB already at 1000 iterations for EigenGS, versus 0% for GaussianImage (Tai et al., 10 Mar 2025). These results show a different but related interpretation of sketch-like Gaussian structure: not explicit boundaries, but sparse multi-scale image primitives that can serve as an immediate structural approximation.

At the opposite end of the representation spectrum, 3D Gaussian modeling for OpenVDB-based scientific visualization uses Gaussian primitives to encode volumetric density rather than edges or radiance (Sharma et al., 14 Sep 2025). The per-Gaussian density takes the form

$\mu$8

with axis-aligned covariance

$\mu$9

and opacity coefficient

$c \in \mathbb{R}^{4\times 3}.$00

for voxel group $c \in \mathbb{R}^{4\times 3}.$01 (Sharma et al., 14 Sep 2025). The paper explicitly relates its volume Gaussians to “Sketch Gaussians–style 3D Gaussian modeling,” but the semantics are different: these Gaussians represent extinction fields, not structural edges. Still, its analytic line integration,

$c \in \mathbb{R}^{4\times 3}.$02

demonstrates a fully volumetric alternative to rasterization-style Gaussian compositing. The paper reports, for example, PSNR/FPS/#Gaussians of 24.02 dB / 125.4 / 1.5M for explosion at dense 2$c \in \mathbb{R}^{4\times 3}.$03, 28.22 dB / 226.6 / 1.3M at dense 4$c \in \mathbb{R}^{4\times 3}.$04, and 21.87 dB / 248 / 1.2M at dense 8$c \in \mathbb{R}^{4\times 3}.$05, as well as reductions to as little as 0.08% of voxel count under the coarsest LOD (Sharma et al., 14 Sep 2025). This suggests that Gaussian structural priors can extend beyond boundary geometry into scientific volumetrics, though under different physical assumptions.

7. Conceptual significance, limitations, and open directions

Across these works, Sketch Gaussians provide a unifying answer to a recurring inefficiency in Gaussian representations: high-frequency structural content tends to attract a disproportionate number of primitives, yet that content is also the most geometrically regular. SketchSplat addresses this by making Gaussians subordinate to explicit 3D sketches and using differentiable rasterization as an optimization bridge from images to parametric edges (Ying et al., 18 Mar 2025). Sketch and Patch, and Sketch&Patch++, address it by separating a structural subset of a 3DGS model and encoding that subset parametrically (Shi et al., 22 Jan 2025, Shi et al., 8 Jan 2026). Image-GS and EigenGS show that sparse anisotropic Gaussians can also serve as explicit sketch-like image primitives in 2D (Zhang et al., 2024, Tai et al., 10 Mar 2025).

Several limitations recur. SketchSplat notes that fixed merging thresholds can erroneously merge nearby but distinct thin edges, that supervision errors can push hidden edges to incorrect locations, that 2DGS-SN depends on accurate depth and normal maps, and that the formulation is specialized to CAD-like edge-dominant objects rather than full radiance fields (Ying et al., 18 Mar 2025). Sketch and Patch is limited by dependence on external 3D line segments and is best suited to man-made scenes with straight or gently curved edges (Shi et al., 22 Jan 2025). Sketch&Patch++ removes that dependence but still faces challenges on highly textured natural regions, ambiguous structures, and dynamic scenes, and it does not incorporate semantic importance into its clustering (Shi et al., 8 Jan 2026). Image-GS notes optimization difficulties in spatial allocation and the absence of explicit stroke topology (Zhang et al., 2024), while EigenGS is limited by the quality of its PCA basis and performs best on aligned datasets (Tai et al., 10 Mar 2025).

The most plausible research trajectory is a convergence of these strands. One direction, explicitly suggested in SketchSplat, is combining edge-based sketch representations with full color 3DGS or NeRF-style radiance fields, using edges as structural priors (Ying et al., 18 Mar 2025). Another, suggested in Sketch&Patch++, is semantic-aware SketchGS that prioritize perceptually salient objects or regions of interest during structural categorization and streaming (Shi et al., 8 Jan 2026). A third is richer geometric parameterization: higher-order splines, improved visibility reasoning, or dynamic time-varying sketch layers (Ying et al., 18 Mar 2025, Shi et al., 8 Jan 2026). More broadly, the body of work suggests that the most effective Gaussian representations are increasingly hybrid: unconstrained splats where necessary, but explicit structural models where the scene exhibits strong geometric regularity.

In that sense, Sketch Gaussians are best understood not as a single primitive class, but as a structure-aware design pattern for Gaussian representations. Whether realized as Gaussian samples of parametric edge sketches, as boundary-coherent subsets of a 3DGS model, or as multi-scale 2D Gaussian surrogates for line-dominant image content, they encode the same methodological claim: the structural skeleton of a scene or image can be modeled more compactly, and often more faithfully, when Gaussian primitives are constrained by geometry rather than left entirely free.