Mesh-Guided 2D Gaussian Splatting

• Mesh-guided 2D Gaussian splatting is a technique that attaches 2D Gaussian primitives to mesh faces, ensuring precise surface alignment and detailed geometric reconstruction.
• It employs mesh-based barycentric parameterizations and localized covariance constructions to balance reconstruction flexibility with surface coherence.
• The method enables real-time interactive editing, advanced rendering through polygonal approximations, and robust optimization using tailored loss functions and gradient isolation.

Mesh-guided 2D Gaussian splatting denotes a class of Gaussian-splatting formulations in which flattened, surface-oriented Gaussian primitives are parameterized, initialized, constrained, or edited through an explicit mesh. In "FMGS-Avatar" (Fan et al., 18 Sep 2025), the formulation is used for 3D monocular avatar reconstruction by attaching 2D Gaussian primitives directly to template mesh faces with constrained position, rotation, and movement, thereby targeting superior surface alignment and geometric detail preservation. Related formulations parameterize Gaussians by mesh-face vertices, anchor splats to deformable template meshes, initialize 2DGS from semantic building meshes, or approximate flat Gaussians by polygonal meshes for ray tracing and editing (Waczyńska et al., 2024).

1. Conceptual scope and relationship to Gaussian splatting

Classical Gaussian splatting represents scenes with trainable Gaussian components carrying color and opacity. Mesh-guided variants modify this premise by introducing a surface scaffold. The mesh may act as a hard attachment domain, as in FMGS-Avatar where each 2D Gaussian is anchored in canonical space on a template mesh face, or as a face-based parameterization of the splat itself, as in GaMeS where Gaussian components are parameterized by the vertices of the mesh face (Fan et al., 18 Sep 2025).

Within this family, the role of the mesh differs substantially across methods. In GaMeS, the mesh provides barycentric coordinates for the Gaussian mean and a face-aligned covariance construction, so that mesh edits immediately update the position, orientation, and scale of the splats (Waczyńska et al., 2024). In DeMapGS, splats are attached to a fixed-topology triangle mesh with barycentric weights, a normal-offset, and a local rotation quaternion, enabling simultaneous mesh deformation and surface-attached 2D Gaussian optimization (Zhou et al., 11 Dec 2025). In GS4Buildings, the mesh is not merely a deformation scaffold but also a semantic prior source: LoD2 building geometry is rasterized into depth and normal buffers that supervise 2DGS optimization (Zhang et al., 10 Aug 2025).

A recurrent point of confusion is whether mesh guidance simply replaces Gaussian splats with ordinary triangle rendering. The published formulations do not do so uniformly. FMGS-Avatar, GaMeS, GS4Buildings, and DeMapGS all retain Gaussian primitives and differentiable splatting objectives, while REdiSplats explicitly converts each flat Gaussian into an $n$-sided polygon lying in its local plane so that ray tracing, physical simulation, and DCC-tool integration become possible (Byrski et al., 15 Mar 2025). This suggests that mesh guidance is best understood as a structural prior over Gaussian geometry rather than a single rendering recipe.

2. Primitive parameterization and surface attachment

In FMGS-Avatar, each primitive $k$ stores a 3D center $\mu_k\in\mathbb{R}^3$, a $2\times2$ covariance through an axis-aligned scale $s_k=(s_u,s_v)\in\mathbb{R}^2$ in its local $(u,v)$ plane, a $3\times3$ rotation $R_k=[t_u,t_v,t_w]$ whose columns are two orthonormal tangents and a normal $t_w=t_u\times t_v$, an opacity $\alpha_k\in\mathbb{R}$, and a view-dependent color $k$0 via a small SH MLP. All of $k$1 live in canonical space anchored on a template mesh; $k$2 is initialized from the barycentric center of triangle $k$3, $k$4 is oriented so that $k$5 is the mesh-face normal, and $k$6 are initialized as small constants (Fan et al., 18 Sep 2025).

The mesh attachment strategy is deliberately restrictive. FMGS-Avatar upsamples the SMPL template from $k$7k to $k$8k vertices, collects faces $k$9, fixes $\mu_k\in\mathbb{R}^3$0 for face $\mu_k\in\mathbb{R}^3$1, defines a local tangent plane through an orthonormal basis $\mu_k\in\mathbb{R}^3$2 with $\mu_k\in\mathbb{R}^3$3, constrains subsequent position moves to the normal direction, constrains rotation updates to $\mu_k\in\mathbb{R}^3$4 so it never “tilts” out of the tangent plane, and keeps one primitive per face to avoid proliferation (Fan et al., 18 Sep 2025).

The pose-dependent residual parameterization in FMGS-Avatar augments this anchored representation without discarding surface coherence. A $\mu_k\in\mathbb{R}^3$5-d geometry feature $\mu_k\in\mathbb{R}^3$6 is sampled from a hash grid at $\mu_k\in\mathbb{R}^3$7, concatenated with a $\mu_k\in\mathbb{R}^3$8-d pose code $\mu_k\in\mathbb{R}^3$9, and passed through an MLP $2\times2$0 to produce $2\times2$1, $2\times2$2, $2\times2$3, and $2\times2$4. The updates are

$2\times2$5

The residuals therefore act only within a mesh-defined local frame (Fan et al., 18 Sep 2025).

GaMeS uses a different but closely related attachment formalism. For a face $2\times2$6 and splat index $2\times2$7, the mean is written as

$2\times2$8

which guarantees that the center stays on the face. The covariance is face-aligned through a normal $2\times2$9, a local frame $s_k=(s_u,s_v)\in\mathbb{R}^2$0, and a scale matrix $s_k=(s_u,s_v)\in\mathbb{R}^2$1, with a learned scalar $s_k=(s_u,s_v)\in\mathbb{R}^2$2 controlling per-splat inflation or shrinkage:

$s_k=(s_u,s_v)\in\mathbb{R}^2$3

The barycentric alphas are re-parameterized via softmax to enforce positivity and sum-to-one (Waczyńska et al., 2024).

DeMapGS extends the attachment idea to a deformable surface. Each splat carries barycentric weights $s_k=(s_u,s_v)\in\mathbb{R}^2$4, a normal-offset $s_k=(s_u,s_v)\in\mathbb{R}^2$5, a local rotation quaternion $s_k=(s_u,s_v)\in\mathbb{R}^2$6, scale $s_k=(s_u,s_v)\in\mathbb{R}^2$7, opacity $s_k=(s_u,s_v)\in\mathbb{R}^2$8, and color $s_k=(s_u,s_v)\in\mathbb{R}^2$9. Its world-space center is

$(u,v)$0

so the splat remains glued to the deforming face while still admitting explicit displacement along the face normal (Zhou et al., 11 Dec 2025).

3. Rendering and compositing formulations

Mesh-guided 2D Gaussian splatting generally preserves front-to-back alpha compositing. In FMGS-Avatar, each Gaussian casts a small 2D disk of opacity onto pixels. For a pixel $(u,v)$1, if $(u,v)$2 are the splats sorted by depth and $(u,v)$3 are their induced opacity and color contribution, the final color is

$(u,v)$4

In practice, $(u,v)$5 is projected via skinning and camera to the image, the local Gaussian is evaluated as $(u,v)$6 in the primitive’s local plane, scaled by $(u,v)$7 to yield $(u,v)$8, and composited with the MLP-predicted color (Fan et al., 18 Sep 2025).

GaMeS writes the same principle through the standard projected covariance of a 3D Gaussian. With mean $(u,v)$9, covariance $3\times3$0, and projection Jacobian $3\times3$1, the image-space Gaussian is

$3\times3$2

Rendering splats $3\times3$3 at pixel locations with weight $3\times3$4 retains the usual differentiable-rasterization view of Gaussian splatting (Waczyńska et al., 2024).

REdiSplats makes the rendering substrate more explicit by approximating each flat Gaussian with an $3\times3$5-sided polygon, $3\times3$6 by default, lying in the Gaussian’s local OY–OZ plane. The local vertices are defined by

$3\times3$7

then transformed to world coordinates through $3\times3$8 and triangulated as a fan. This permits either OptiX ray tracing, where effective opacity is evaluated along the camera ray at the polygon hit point, or rasterization in Nvdiffrast or Blender with standard blending (Byrski et al., 15 Mar 2025).

DeMapGS also writes the 2DGS kernel in projected screen coordinates. If $3\times3$9 and $R_k=[t_u,t_v,t_w]$0, then the image-plane covariance is $R_k=[t_u,t_v,t_w]$1 and the accumulated alpha at screen-space sample $R_k=[t_u,t_v,t_w]$2 is

$R_k=[t_u,t_v,t_w]$3

Colors and normals are composited via front-to-back alpha blending (Zhou et al., 11 Dec 2025).

4. Loss design, optimization structure, and coordinated gradients

FMGS-Avatar couples photometric, geometric, semantic, skinning, and isometry losses. The photometric term is

$R_k=[t_u,t_v,t_w]$4

the silhouette term is $R_k=[t_u,t_v,t_w]$5, and the depth term is an ordinal-ranking loss

$R_k=[t_u,t_v,t_w]$6

with random pixel pairs. Normal supervision combines self-consistency, prior alignment, and TV smoothness:

$R_k=[t_u,t_v,t_w]$7

Semantic supervision uses cross-entropy $R_k=[t_u,t_v,t_w]$8 plus an intra-class feature regularizer $R_k=[t_u,t_v,t_w]$9, and the total loss is

$t_w=t_u\times t_v$0

A central technical issue is conflicting optimization objectives across modalities. FMGS-Avatar addresses this through selective gradient isolation: during backpropagation of $t_w=t_u\times t_v$1, gradients with respect to rotation updates $t_w=t_u\times t_v$2 and scale $t_w=t_u\times t_v$3 are zeroed so only $t_w=t_u\times t_v$4 via $t_w=t_u\times t_v$5 is adjusted; during $t_w=t_u\times t_v$6, gradients with respect to $t_w=t_u\times t_v$7 are zeroed so only orientation $t_w=t_u\times t_v$8 is adjusted; during $t_w=t_u\times t_v$9, gradients on both $\alpha_k\in\mathbb{R}$0 and $\alpha_k\in\mathbb{R}$1 are blocked so updates focus on the semantic MLP and semantic feature volume (Fan et al., 18 Sep 2025).

GS4Buildings also exemplifies mesh-guided supervision design, but at scene scale. It extends the standard 2DGS loss

$\alpha_k\in\mathbb{R}$2

with mesh depth and normal prior terms $\alpha_k\in\mathbb{R}$3 and $\alpha_k\in\mathbb{R}$4 obtained from LoD2 rasterization. During the first training phase, approximately $\alpha_k\in\mathbb{R}$5k iterations, $\alpha_k\in\mathbb{R}$6 to force Gaussians to snap onto the coarse planar mesh and fill missing regions; during the remaining $\alpha_k\in\mathbb{R}$7k iterations the mesh-prior weights decay toward zero while $\alpha_k\in\mathbb{R}$8 increase to refine fine-scale smoothness (Zhang et al., 10 Aug 2025).

2D-SuGaR uses another prior-guided optimization pattern. It minimizes

$\alpha_k\in\mathbb{R}$9

where $k$00 combines an $k$01 term and a D-SSIM term, $k$02 is the original depth-normal consistency loss, $k$03 is a monocular-normal prior loss, and $k$04 is a depth-distortion penalty. After TSDF fusion and marching cubes, a mesh-Gaussian joint refinement minimizes

$k$05

with Laplacian smoothing and mesh-normal consistency (R. et al., 1 May 2026).

DeMapGS emphasizes mesh regularity and optimization stability. Its objective combines photometric, SSIM, bi-Laplacian regularization, normal consistency, and depth-distortion terms, while a gradient diffusion operator

$k$06

smooths raw vertex gradients across the mesh. The method alternates 2DGS, 3DGS, and final 2DGS stages so that visible-surface alignment, concavity handling, and final surface consistency are optimized in separate phases (Zhou et al., 11 Dec 2025).

5. Initialization, priors, and canonical-space structure

Initialization quality is a decisive variable in mesh-guided 2DGS. FMGS-Avatar initializes all canonical-space primitive states from a template mesh and then augments the limited monocular visual cues using foundation models trained on large-scale datasets, such as Sapiens. The abstract states that the distilled prior knowledge within a shared canonical space naturally enables spatially and temporally consistent rendering under novel views and poses (Fan et al., 18 Sep 2025).

GS4Buildings demonstrates a stronger prior regime in which Gaussians are initialized directly from LoD2 semantic 3D building models rather than SfM/MVS. Surface points are sampled on mesh faces with probability proportional to face area, normals are inherited from the face, tangent frames are constructed per sample, and visibility filtering keeps points visible in at least $k$07 views. The same LoD2 mesh is ray-cast into per-view depth $k$08, normal $k$09, and mask $k$10 buffers with no additional smoothing or regularization. On the TUM2TWIN urban scenes, this prior-guided pipeline improves reconstruction completeness by $k$11 over vanilla 2DGS and reduces M3C2 error by $k$12 (Zhang et al., 10 Aug 2025).

2D-SuGaR addresses weak initialization differently: it seeds 2D surfel Gaussians from monocular depth and normal estimates, resolves monocular scale by aligning medians to SfM sparse depths, prunes disconnected “islands” of Gaussians with DBSCAN at iteration $k$13, and then extracts a mesh by TSDF fusion and marching cubes for joint refinement. On $k$14 DTU scenes, the full pipeline achieves an average Chamfer Distance of $k$15 units, compared to $k$16 for vanilla 2DGS and $k$17 for NeRF-based methods (R. et al., 1 May 2026).

GaMeS shows that mesh guidance need not require a mesh known a priori. If no mesh is available, a vanilla Gaussian splatting model can first be trained, then each learned Gaussian can be converted into a pseudo-triangle using the Gaussian mean and its two principal in-plane axes, after which the system is re-parameterized back into barycentric form and optimized as a mesh-guided model (Waczyńska et al., 2024). A plausible implication is that “mesh-guided” in this literature encompasses both explicit mesh-conditioned reconstruction and mesh recovery from a pre-existing Gaussian field.

6. Editing, efficiency, and limitations

The literature consistently links mesh guidance to editability and structured downstream graphics operations. GaMeS reports that once Gaussians are tied to mesh faces, any mesh edit such as vertex motion or face subdivision immediately updates the affected splats, and the splatting kernel can be replayed at $k$18–$k$19 Hz for real-time interactive editing (Waczyńska et al., 2024). REdiSplats pushes this logic further: because the Gaussians are instantiated as actual triangle meshes, they can participate in standard ray tracing with secondary rays for hard shadows or mirror reflections, can receive material properties such as index of refraction and roughness, and can be rendered using Blender or Nvdiffrast. Reported performance includes interactive $k$20–$k$21 fps at $k$22 on an RTX 3080 with $k$23 K Gaussians for the OptiX variant and $k$24–$k$25 fps for the rasterized variant (Byrski et al., 15 Mar 2025).

DeMapGS connects mesh-guided splats to asset extraction. After optimization, it samples UV texels per face and blends displacement, normal, and color from attached splats into full-resolution UV textures, yielding diffuse, normal, and displacement maps. The method reports PSNR approximately $k$26–$k$27 dB and OpenGL rendering at approximately $k$28 FPS for mesh plus UV maps, described as a $k$29 speed-up over SuGaR at approximately $k$30 FPS, while also enabling texture editing, geometry editing, and cross-object parameter transfer through a shared template (Zhou et al., 11 Dec 2025).

FMGS-Avatar emphasizes efficiency in a different operating regime: PyTorch with an Instant-NGP style hash grid for three separate $k$31-d volumes, geometry, appearance, and semantics; geometry, appearance, semantics, and skinning MLPs with specified widths; a batch of $k$32 K rays; Adam with learning rate $k$33 for hash volumes and $k$34 for MLPs; and a training schedule of approximately $k$35 minutes on one A100 for approximately $k$36k–$k$37k steps with no weight decay (Fan et al., 18 Sep 2025). The reported outcome is fast convergence, high-fidelity geometric detail, and robust multi-modal priors distilled from 2D foundation models while preserving surface coherence via the mesh-guided 2D Gaussian surfel representation.

A second recurrent misconception is that mesh guidance necessarily reduces reconstruction flexibility. The published designs instead distribute flexibility differently. FMGS-Avatar allows normal-direction displacement, log-scale adjustments, and axis-angle rotation residuals on top of per-face anchoring; DeMapGS adds per-splat barycentric motion, displacement, and quaternion rotation on a deformable template; GS4Buildings uses priors strongly in early training and decays them later; and 2D-SuGaR introduces mesh refinement only after prior-guided 2DGS optimization (Fan et al., 18 Sep 2025). This suggests that the central trade-off is not between “mesh” and “flexibility,” but between unconstrained Gaussian freedom and explicitly surface-biased regularization.