SEPatch3D: Structure-Aware 3D Compression

• The paper introduces a dual-set Gaussian decomposition, splitting SketchGS for high-frequency structure and PatchGS for smooth volumetric fill.
• It uses multi-criteria density clustering and adaptive polynomial regression for precise Gaussian categorization and efficient compression.
• The hybrid encoding and layered streaming pipeline achieves up to 175× compression while maintaining high rendering quality in diverse 3D scenes.

SEPatch3D, referred to in the source paper as Sketch&Patch++ (SP++), is a structure-aware compression and representation framework for 3D Gaussian Splatting (3DGS) that separates the Gaussian set into two semantically distinct subsets: Sketch Gaussians (SketchGS), which concentrate on high-frequency, boundary-defining structure, and Patch Gaussians (PatchGS), which cover smooth, low-frequency volumetric regions. The framework uses this separation to build a hybrid encoding pipeline with layered progressive streaming, aiming to reduce the storage and transmission cost of vanilla 3DGS while preserving rendering quality across arbitrary 3D scenes (Shi et al., 8 Jan 2026).

1. Structural decomposition of the Gaussian field

The central premise of SEPatch3D is that optimized 3DGS models are highly non-uniform after densification. Because 3DGS allocates Gaussians through gradient-driven densification, high-frequency regions such as sharp edges, contours, and detailed boundaries accumulate dense Gaussian populations, while smoother surfaces remain more sparsely represented. SEPatch3D formalizes this asymmetry by partitioning the Gaussian set into SketchGS and PatchGS (Shi et al., 8 Jan 2026).

Component	Structural role	Reported characteristics
SketchGS	High-frequency, boundary-defining structure	dense, elongated, compact, structurally coherent
PatchGS	Smooth, low-frequency volumetric coverage	scattered, lower density, less elongated

In the Room scene, the paper reports that SketchGS are about 100× denser, 1.5× more elongated, and 27× more compact than PatchGS. This distinction is used operationally rather than descriptively: SketchGS are treated as a structural “outline” or “skeleton,” whereas PatchGS supply the “fill” that completes volumetric appearance (Shi et al., 8 Jan 2026).

The paper further links this decomposition to the behavior of the underlying 3DGS optimization. Different densification thresholds produce markedly different Gaussian counts, with $\tau=0.01$ yielding 10K Gaussians, $\tau=0.002$ yielding 27K Gaussians, and $\tau=0.0002$ yielding 589K Gaussians. This is presented as evidence that stricter densification disproportionately populates high-frequency regions, thereby making structural partitioning a natural basis for compression and progressive transmission (Shi et al., 8 Jan 2026).

2. Direct categorization on optimized 3DGS

SEPatch3D differs from the earlier Sketch&Patch method by operating directly on the converged 3DGS representation rather than relying on external 3D line primitives. The categorization stage uses only intrinsic Gaussian properties—position, covariance-derived orientation, and color—and consists of two steps: multi-criteria density-based clustering and adaptive quality-driven refinement (Shi et al., 8 Jan 2026).

The initial clustering extends DBSCAN by defining a neighborhood only when spatial, directional, and color consistency are all satisfied simultaneously:

$$\mathcal{G}_j \in \mathcal{N}(\mathcal{G}_i) \Leftrightarrow \left(d_{\text{spatial}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_s\right) \land \left(d_{\text{direction}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_d\right) \land \left(d_{\text{color}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_c\right).$$

The distances are defined as

$$d_{\text{spatial}}(\mathcal{G}_i,\mathcal{G}_j)=\|\boldsymbol{\mu}_i-\boldsymbol{\mu}_j\|_2,$$

$$d_{\text{direction}}(\mathcal{G}_i,\mathcal{G}_j) = 1- \left( \frac{\mathbf{n}_i\cdot \mathbf{n}_j}{\|\mathbf{n}_i\|\,\|\mathbf{n}_j\|} +1 \right)/2,$$

and

$$d_{\text{color}}(\mathcal{G}_i,\mathcal{G}_j)=\|\mathbf{c}_i-\mathbf{c}_j\|_2.$$

Here, $\mathbf{n}_i$ is the normalized principal direction derived from the Gaussian covariance. The stated purpose of this formulation is to capture structured coherence even for curved or irregular features, not only straight edges (Shi et al., 8 Jan 2026).

To scale to millions of Gaussians, the method uses a k-d tree for spatial candidate retrieval and then applies the directional and color tests only to those candidates. The paper reports a complexity reduction from roughly $\mathcal{O}(n^2)$ to about

$\mathcal{O}(n\log n + nk),$

where $\tau=0.002$0 is the average number of spatial neighbors. As in standard DBSCAN, Gaussians are labeled as core points, border points, or noise points, and clusters are formed as density-connected components by BFS over core points and their neighbors (Shi et al., 8 Jan 2026).

3. Adaptive refinement and the definition of SketchGS

The paper does not treat every coherent cluster as a valid SketchGS cluster. Instead, each cluster is tested for parametric regularity through polynomial regression over Gaussian attributes. For a cluster $\tau=0.002$1, the method fits polynomial models from normalized 3D positions $\tau=0.002$2 to four Gaussian attributes: scaling, rotation (quaternion), opacity, and color (Shi et al., 8 Jan 2026).

The attribute model is written as

$\tau=0.002$3

with polynomial features

$\tau=0.002$4

The polynomial degree $\tau=0.002$5 is selected by grid search over $\tau=0.002$6, and the number of coefficients for degree $\tau=0.002$7 in 3D is

$\tau=0.002$8

with at most 286 coefficients per attribute model (Shi et al., 8 Jan 2026).

Cluster quality is evaluated by per-attribute MSE and a combined score,

$\tau=0.002$9

with equal weights $\tau=0.0002$0 for scaling, rotation, opacity, and color. If $\tau=0.0002$1, the cluster is accepted as SketchGS; otherwise it is recursively split (Shi et al., 8 Jan 2026).

The splitting stage uses residual features

$\tau=0.0002$2

which are fused with spatial information as

$\tau=0.0002$3

K-means then partitions the cluster into

$\tau=0.0002$4

sub-clusters. Recursion terminates when clusters satisfy the quality threshold, when cluster size falls below $\tau=0.0002$5, or when the maximum number of iterations $\tau=0.0002$6 is reached. A final IQR-based outlier removal step is applied on scaling values, and outliers are reclassified as PatchGS rather than discarded (Shi et al., 8 Jan 2026).

This refinement stage is the formal criterion by which “structure-aware” classification is determined. A common simplification is to describe the method as density clustering alone; the paper’s actual formulation requires both density-connected grouping and parametric reconstructibility of Gaussian attributes (Shi et al., 8 Jan 2026).

4. Encoding pipeline and progressive layered streaming

After categorization, SEPatch3D applies different compression strategies to SketchGS and PatchGS. The distinction is motivated by the claim that SketchGS are sufficiently coherent for parametric encoding, whereas PatchGS are better handled by pruning, retraining, and quantization (Shi et al., 8 Jan 2026).

The underlying 3DGS representation associates each Gaussian with a center $\tau=0.0002$7, covariance $\tau=0.0002$8, opacity $\tau=0.0002$9, and SH color coefficients $$\mathcal{G}_j \in \mathcal{N}(\mathcal{G}_i) \Leftrightarrow \left(d_{\text{spatial}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_s\right) \land \left(d_{\text{direction}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_d\right) \land \left(d_{\text{color}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_c\right).$$0, with density

$$\mathcal{G}_j \in \mathcal{N}(\mathcal{G}_i) \Leftrightarrow \left(d_{\text{spatial}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_s\right) \land \left(d_{\text{direction}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_d\right) \land \left(d_{\text{color}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_c\right).$$1

and the original 3DGS training objective

$$\mathcal{G}_j \in \mathcal{N}(\mathcal{G}_i) \Leftrightarrow \left(d_{\text{spatial}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_s\right) \land \left(d_{\text{direction}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_d\right) \land \left(d_{\text{color}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_c\right).$$2

For SketchGS, the paper reports the following pipeline: polynomial regression for attributes, Draco for position compression, conversion of polynomial coefficients to half precision, and final gzip packing. The reported compression is about 18×–20× for polynomial-regression encoding alone and 200×–242× for full SketchGS compression with polynomial regression, Draco, and quantization (Shi et al., 8 Jan 2026).

For PatchGS, the method applies pruning and retraining, then vector quantization with 256-entry codebooks. Positions are stored with 16-bit half-floats rather than codebooks. The paper reports that PatchGS compression reaches roughly 77×–92× after quantization, with final gzip packing contributing an additional reduction of about 5% (Shi et al., 8 Jan 2026).

This separation also defines a layered streaming model. Layer $$\mathcal{G}_j \in \mathcal{N}(\mathcal{G}_i) \Leftrightarrow \left(d_{\text{spatial}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_s\right) \land \left(d_{\text{direction}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_d\right) \land \left(d_{\text{color}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_c\right).$$3 contains SketchGS only and is described as transmitting the structural skeleton, while later layers $$\mathcal{G}_j \in \mathcal{N}(\mathcal{G}_i) \Leftrightarrow \left(d_{\text{spatial}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_s\right) \land \left(d_{\text{direction}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_d\right) \land \left(d_{\text{color}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_c\right).$$4 progressively add PatchGS to refine volumetric detail. In the Playroom example, the paper uses four patch layers beyond the sketch base layer: $$\mathcal{G}_j \in \mathcal{N}(\mathcal{G}_i) \Leftrightarrow \left(d_{\text{spatial}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_s\right) \land \left(d_{\text{direction}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_d\right) \land \left(d_{\text{color}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_c\right).$$5 is SketchGS only; $$\mathcal{G}_j \in \mathcal{N}(\mathcal{G}_i) \Leftrightarrow \left(d_{\text{spatial}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_s\right) \land \left(d_{\text{direction}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_d\right) \land \left(d_{\text{color}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_c\right).$$6, $$\mathcal{G}_j \in \mathcal{N}(\mathcal{G}_i) \Leftrightarrow \left(d_{\text{spatial}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_s\right) \land \left(d_{\text{direction}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_d\right) \land \left(d_{\text{color}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_c\right).$$7, $$\mathcal{G}_j \in \mathcal{N}(\mathcal{G}_i) \Leftrightarrow \left(d_{\text{spatial}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_s\right) \land \left(d_{\text{direction}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_d\right) \land \left(d_{\text{color}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_c\right).$$8, and $$\mathcal{G}_j \in \mathcal{N}(\mathcal{G}_i) \Leftrightarrow \left(d_{\text{spatial}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_s\right) \land \left(d_{\text{direction}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_d\right) \land \left(d_{\text{color}}(\mathcal{G}_i,\mathcal{G}_j)\le \epsilon_c\right).$$9 contain the top 25%, 50%, 75%, and 100% of PatchGS respectively (Shi et al., 8 Jan 2026).

The paper characterizes this arrangement as a codec-like 3DGS representation. That description is significant because the method is framed not only as a compression technique but also as a representation designed for efficient delivery and progressive decoding (Shi et al., 8 Jan 2026).

5. Empirical evaluation

SEPatch3D is evaluated on 7 scenes drawn from 3 datasets: Deep Blending (Playroom, Drjohnson), Mip-NeRF360 (Room, Kitchen, Garden, Train), and Tanks and Temples (Truck). These scenes are intended to span indoor man-made environments, outdoor unbounded scenes, and natural or organic structures. The reported metrics are PSNR, SSIM, and LPIPS, using the same train/test split as the original 3DGS paper (Shi et al., 8 Jan 2026).

The principal baselines are Vanilla 3DGS, the earlier line-based Sketch&Patch (SP), Prune+Retrain, and a Sketch variant using only SketchGS encoding without Patch retraining. The paper also compares with ScaffoldGS, LightGaussian, EAGLES, CompactGS, ReducedGS, ContextGS, MesonGS, GaussSpa, HAC, and HAC++ (Shi et al., 8 Jan 2026).

The reported quantitative results emphasize quality retention at matched size. Compared to uniform pruning baselines at equivalent model sizes, the method achieves up to 1.74 dB PSNR improvement, 6.7% SSIM improvement, and 41.4% LPIPS improvement. At around 38 MB, SP++ surpasses Prune+Retrain by exactly those margins, which the paper attributes to preserving SketchGS while pruning only PatchGS (Shi et al., 8 Jan 2026).

The storage reductions reported for the full representation are:

• DeepBlending: 703.77 MB → 4.03 MB, about 174.6×
• Tanks and Temples: 436.50 MB → 3.76 MB, about 116.1×
• Mip-NeRF360: 738.74 MB → 5.90 MB, about 125.2×

The paper also states that indoor scenes can maintain visual quality with only 0.5% of the original model size, corresponding to up to 175× compression ratio (Shi et al., 8 Jan 2026).

Relative to the earlier line-based Sketch&Patch method, SP++ is reported to be smaller at equal quality by 2.7× on Mip-NeRF360, 3.3× on DeepBlending, and 3.9× on Tanks and Temples. Runtime on Mip-NeRF360 is also reported: vanilla 3DGS requires 1633.66 s, HAC++ requires 3038–3278 s, and SP++ requires 1558.48 s in its low-rate setting and 1659.76 s in its high-rate setting. On that basis, the paper describes SP++ as roughly as fast as vanilla 3DGS and about 1.9× faster than HAC++ (Shi et al., 8 Jan 2026).

6. Relation to prior sketch/patch methods and practical scope

A defining difference between SEPatch3D and the earlier Sketch&Patch method is the removal of dependence on Line3D++. The previous approach reconstructed 3D line segments from 2D line detections, which the paper states was effective for man-made scenes with clear straight edges but limited by external line extraction, error propagation from 2D detection and triangulation, and poor generalization to curved boundaries, irregular structures, organic objects, and natural scenes. SEPatch3D replaces that pipeline with direct analysis of the optimized Gaussian distribution itself (Shi et al., 8 Jan 2026).

This design supports an important clarification. SEPatch3D is not presented as a generic Gaussian pruning scheme. Its primary mechanism is a semantic and structural decomposition of the Gaussian field into subsets that are then compressed differently. Likewise, it is not restricted to architectural scenes, because the clustering criteria are based on intrinsic Gaussian position, orientation, and color rather than explicit straight-line primitives (Shi et al., 8 Jan 2026).

The paper’s practical implications are correspondingly broad. By compressing scene models to a few megabytes, the framework is positioned as useful for storage, adaptive streaming, rendering, and deployment on bandwidth-constrained networks and resource-limited mobile/AR/VR devices. The sketch-first transmission order is intended to allow a recognizable scene layout and object boundaries to appear before all detail data arrive, while later PatchGS layers refine appearance as bandwidth permits (Shi et al., 8 Jan 2026).

In that sense, SEPatch3D occupies a specific place in the 3DGS literature: it recasts the Gaussian set as a layered, structure-aware representation in which high-frequency boundary information becomes the base layer and low-frequency volumetric content becomes the enhancement layer. The paper’s results suggest that this reorganization, rather than uniform rate reduction alone, is the principal source of its compression-quality tradeoff (Shi et al., 8 Jan 2026).