---
title: Mesh-Guided 2D Gaussian Splatting
url: https://www.emergentmind.com/topics/mesh-guided-2d-gaussian-splatting
type: topic
---

# Mesh-Guided 2D Gaussian Splatting

Mesh-guided 2D Gaussian splatting denotes a class of Gaussian-splatting formulations in which flattened, surface-oriented Gaussian primitives are parameterized, initialized, constrained, or edited through an explicit mesh. In "FMGS-Avatar" [2509.14739], the formulation is used for 3D monocular avatar reconstruction by attaching 2D Gaussian primitives directly to template mesh faces with constrained position, rotation, and movement, thereby targeting superior surface alignment and geometric detail preservation. Related formulations parameterize Gaussians by mesh-face vertices, anchor splats to deformable template meshes, initialize 2DGS from semantic building meshes, or approximate flat Gaussians by polygonal meshes for ray tracing and editing [2402.01459].

## 1. Conceptual scope and relationship to Gaussian splatting

Classical Gaussian splatting represents scenes with trainable Gaussian components carrying color and opacity. Mesh-guided variants modify this premise by introducing a surface scaffold. The mesh may act as a hard attachment domain, as in FMGS-Avatar where each 2D Gaussian is anchored in canonical space on a template mesh face, or as a face-based parameterization of the splat itself, as in GaMeS where Gaussian components are parameterized by the vertices of the mesh face [2509.14739].

Within this family, the role of the mesh differs substantially across methods. In GaMeS, the mesh provides barycentric coordinates for the Gaussian mean and a face-aligned covariance construction, so that mesh edits immediately update the position, orientation, and scale of the splats [2402.01459]. In DeMapGS, splats are attached to a fixed-topology triangle mesh with barycentric weights, a normal-offset, and a local rotation quaternion, enabling simultaneous mesh deformation and surface-attached 2D Gaussian optimization [2512.10572]. In GS4Buildings, the mesh is not merely a deformation scaffold but also a semantic prior source: LoD2 building geometry is rasterized into depth and normal buffers that supervise 2DGS optimization [2508.07355].

A recurrent point of confusion is whether mesh guidance simply replaces Gaussian splats with ordinary triangle rendering. The published formulations do not do so uniformly. FMGS-Avatar, GaMeS, GS4Buildings, and DeMapGS all retain Gaussian primitives and differentiable splatting objectives, while REdiSplats explicitly converts each flat Gaussian into an $n$-sided polygon lying in its local plane so that ray tracing, physical simulation, and DCC-tool integration become possible [2503.12284]. This suggests that mesh guidance is best understood as a structural prior over Gaussian geometry rather than a single rendering recipe.

## 2. Primitive parameterization and surface attachment

In FMGS-Avatar, each primitive $k$ stores a 3D center $\mu_k\in\mathbb{R}^3$, a $2\times2$ covariance through an axis-aligned scale $s_k=(s_u,s_v)\in\mathbb{R}^2$ in its local $(u,v)$ plane, a $3\times3$ rotation $R_k=[t_u,t_v,t_w]$ whose columns are two orthonormal tangents and a normal $t_w=t_u\times t_v$, an opacity $\alpha_k\in\mathbb{R}$, and a view-dependent color $c_k\in\mathbb{R}^3$ via a small SH MLP. All of $(\mu_k,s_k,R_k,\alpha_k)$ live in canonical space anchored on a template mesh; $\mu_k$ is initialized from the barycentric center of triangle $k$, $R_k$ is oriented so that $t_w$ is the mesh-face normal, and $s_k,\alpha_k$ are initialized as small constants [2509.14739].

The mesh attachment strategy is deliberately restrictive. FMGS-Avatar upsamples the SMPL template from $\approx 6.9$k to $30$k vertices, collects faces $\{f_k\}$, fixes $\mu_k=(v_1+v_2+v_3)/3$ for face $f_k$, defines a local tangent plane through an orthonormal basis $(t_u,t_v)$ with $t_w=t_u\times t_v$, constrains subsequent position moves to the normal direction, constrains rotation updates to $R_k'=R_k\cdot\exp([\delta r]_x)$ so it never “tilts” out of the tangent plane, and keeps one primitive per face to avoid proliferation [2509.14739].

The pose-dependent residual parameterization in FMGS-Avatar augments this anchored representation without discarding surface coherence. A $32$-d geometry feature $f_g$ is sampled from a hash grid at $\mu_k$, concatenated with a $16$-d pose code $Z_p$, and passed through an MLP $F_{\theta_g}$ to produce $\delta d\in\mathbb{R}$, $\delta s\in\mathbb{R}^2$, $\delta r\in\mathbb{R}^3$, and $z_g\in\mathbb{R}^{16}$. The updates are
$$
\mu_k'=\mu_k+n_k\cdot\delta d,\qquad
s_k'=s_k\cdot\exp(\delta s),\qquad
R_k'=R_k\cdot\exp([\delta r]_x).
$$
The residuals therefore act only within a mesh-defined local frame [2509.14739].

GaMeS uses a different but closely related attachment formalism. For a face $V=\{v_1,v_2,v_3\}$ and splat index $i$, the mean is written as
$$
\mu_V^i=\alpha_1^i v_1+\alpha_2^i v_2+\alpha_3^i v_3,
\qquad
\alpha_1^i+\alpha_2^i+\alpha_3^i=1,\quad \alpha_n^i\ge 0,
$$
which guarantees that the center stays on the face. The covariance is face-aligned through a normal $n$, a local frame $R_V$, and a scale matrix $S_V=\operatorname{diag}(\epsilon,s_2,s_3)$, with a learned scalar $\rho_i$ controlling per-splat inflation or shrinkage:
$$
\Sigma_V^i = R_V^\top (\rho_i S_V)^2 R_V.
$$
The barycentric alphas are re-parameterized via softmax to enforce positivity and sum-to-one [2402.01459].

DeMapGS extends the attachment idea to a deformable surface. Each splat carries barycentric weights $\beta_k$, a normal-offset $d_k$, a local rotation quaternion $\bar q_k$, scale $\mathbf s_k\in\mathbb{R}^2$, opacity $o_k$, and color $\mathbf c_k\in\mathbb{R}^3$. Its world-space center is
$$
\mathbf p_k=\sum_{m=1}^3 \beta_{k,m}\,\mathbf v'_{f_{\langle k\rangle},m}+d_k\,\mathbf n_{f_{\langle k\rangle}},
$$
so the splat remains glued to the deforming face while still admitting explicit displacement along the face normal [2512.10572].

## 3. Rendering and compositing formulations

Mesh-guided 2D Gaussian splatting generally preserves front-to-back alpha compositing. In FMGS-Avatar, each Gaussian casts a small 2D disk of opacity onto pixels. For a pixel $x$, if $\{k_1,\ldots,k_n\}$ are the splats sorted by depth and $\alpha_{k_i}^{2D}(x), c_{k_i}(x)$ are their induced opacity and color contribution, the final color is
$$
c(x)=\sum_{i=1}^n c_{k_i}(x)\cdot\alpha_{k_i}^{2D}(x)\cdot\prod_{j=1}^{i-1}(1-\alpha_{k_j}^{2D}(x)).
$$
In practice, $\mu_k'$ is projected via skinning and camera to the image, the local Gaussian is evaluated as $G(u,v)=\exp(-(u^2+v^2)/2)$ in the primitive’s local plane, scaled by $\alpha_k$ to yield $\alpha^{2D}$, and composited with the MLP-predicted color [2509.14739].

GaMeS writes the same principle through the standard projected covariance of a 3D Gaussian. With mean $\mu_i$, covariance $\Sigma_i$, and projection Jacobian $J_i=\partial\Pi/\partial x|_{x=\mu_i}$, the image-space Gaussian is
$$
G_i^{2D}(u)=\det(2\pi\Sigma_i^{2D})^{-1/2}\cdot
\exp\!\left[-\frac12 (u-u_i)^\top (\Sigma_i^{2D})^{-1}(u-u_i)\right],
\qquad
\Sigma_i^{2D}=J_i\Sigma_i J_i^\top.
$$
Rendering splats $G_i^{2D}$ at pixel locations with weight $\alpha_i\cdot G_i^{2D}(u)$ retains the usual differentiable-rasterization view of Gaussian splatting [2402.01459].

REdiSplats makes the rendering substrate more explicit by approximating each flat Gaussian with an $n$-sided polygon, $n=8$ by default, lying in the Gaussian’s local OY–OZ plane. The local vertices are defined by
$$
Q = F_{\chi^2(3)}^{-1}(\alpha_{\text{conf}}),\qquad
v_i^{\text{loc}}=
[0,\sqrt Q\cos(2\pi i/n),\sqrt Q\sin(2\pi i/n)]^\top,
$$
then transformed to world coordinates through $P_i=R_i S_i v_i^{\text{loc}}+m_i$ and triangulated as a fan. This permits either OptiX ray tracing, where effective opacity is evaluated along the camera ray at the polygon hit point, or rasterization in Nvdiffrast or Blender with standard blending [2503.12284].

DeMapGS also writes the 2DGS kernel in projected screen coordinates. If $\Sigma_k=R(q_k)\operatorname{diag}(s_{k,1}^2,s_{k,2}^2,\epsilon)R(q_k)^\top$ and $J(\mathbf p)=\frac1Z\begin{bmatrix}1&0&-u\\0&1&-v\end{bmatrix}$, then the image-plane covariance is $\Lambda_k=J(\mathbf p_k)\Sigma_k J(\mathbf p_k)^\top$ and the accumulated alpha at screen-space sample $\mathbf x$ is
$$
\alpha_k(\mathbf x)=
\exp\!\left(-\frac12(\boldsymbol\mu_k-\mathbf x)^\top\Lambda_k^{-1}(\boldsymbol\mu_k-\mathbf x)\right)o_k.
$$
Colors and normals are composited via front-to-back alpha blending [2512.10572].

## 4. Loss design, optimization structure, and coordinated gradients

FMGS-Avatar couples photometric, geometric, semantic, skinning, and isometry losses. The photometric term is
$$
L_c=(1-\lambda_{\text{ssim}})\,L_1(I,\bar I)+\lambda_{\text{ssim}}\,L_{\text{SSIM}}(I,\bar I),\qquad \lambda_{\text{ssim}}=0.2,
$$
the silhouette term is $L_m=\operatorname{BCE}(M,\bar M)$, and the depth term is an ordinal-ranking loss
$$
L_d=\left\|\tanh[\alpha\cdot(D(x_1)-D(x_2))]-I_{\text{ord}}(\bar D(x_1),\bar D(x_2))\right\|_1,
\qquad \alpha=10,
$$
with random pixel pairs. Normal supervision combines self-consistency, prior alignment, and TV smoothness:
$$
L_{\text{norm}}=L_{sn}+L_n+L_{tv}.
$$
Semantic supervision uses cross-entropy $L_s=\operatorname{CE}(S,\bar S)$ plus an intra-class feature regularizer $L_{\text{reg}}$, and the total loss is
$$
L=L_c+\lambda_m L_m+\lambda_d L_d+\lambda_n L_{\text{norm}}+\lambda_s L_s+\lambda_{\text{reg}}L_{\text{reg}}+\lambda_{\text{skin}}L_{\text{skin}}+\lambda_{\text{iso}}L_{\text{iso}}.
$$
A central technical issue is conflicting optimization objectives across modalities. FMGS-Avatar addresses this through selective gradient isolation: during backpropagation of $L_d$, gradients with respect to rotation updates $\delta r$ and scale $\delta s$ are zeroed so only $\mu$ via $\delta d$ is adjusted; during $L_{\text{norm}}$, gradients with respect to $\delta d$ are zeroed so only orientation $\delta r$ is adjusted; during $L_s$, gradients on both $\delta d$ and $\delta r$ are blocked so updates focus on the semantic MLP and semantic feature volume [2509.14739].

GS4Buildings also exemplifies mesh-guided supervision design, but at scene scale. It extends the standard 2DGS loss
$$
\mathcal L_{2DGS}=\mathcal L_c+\lambda_d\mathcal L_d+\lambda_n\mathcal L_n
$$
with mesh depth and normal prior terms $\mathcal L_{d_b}$ and $\mathcal L_{n_b}$ obtained from LoD2 rasterization. During the first training phase, approximately $10$k iterations, $\lambda_{d_b},\lambda_{n_b}\gg \lambda_d,\lambda_n$ to force Gaussians to snap onto the coarse planar mesh and fill missing regions; during the remaining $20$k iterations the mesh-prior weights decay toward zero while $\lambda_d,\lambda_n$ increase to refine fine-scale smoothness [2508.07355].

2D-SuGaR uses another prior-guided optimization pattern. It minimizes
$$
\mathcal L=\mathcal L_p+\lambda_c\mathcal L_c+\lambda_n\mathcal L_n+\lambda_d\mathcal L_d,
$$
where $\mathcal L_p$ combines an $\ell_1$ term and a D-SSIM term, $\mathcal L_c$ is the original depth-normal consistency loss, $\mathcal L_n=1-\langle n_{\text{rendered}},\tilde n_{\text{mono}}\rangle$ is a monocular-normal prior loss, and $\mathcal L_d$ is a depth-distortion penalty. After TSDF fusion and marching cubes, a mesh-Gaussian joint refinement minimizes
$$
\mathcal L_r=\mathcal L_p+\gamma\mathcal L_{\text{Lap}}+\delta\mathcal L_m,
$$
with Laplacian smoothing and mesh-normal consistency [2605.00569].

DeMapGS emphasizes mesh regularity and optimization stability. Its objective combines photometric, SSIM, bi-Laplacian regularization, normal consistency, and depth-distortion terms, while a gradient diffusion operator
$$
\Delta\mathbf v_i^*=-\eta\bigl[(\mathbf I+\lambda_\ell \mathbf L)^{-2}\bigr]_{ii'}
\Bigl[\frac{\partial\mathcal L}{\partial \mathbf V}\Bigr]_{i'}
$$
smooths raw vertex gradients across the mesh. The method alternates 2DGS, 3DGS, and final 2DGS stages so that visible-surface alignment, concavity handling, and final surface consistency are optimized in separate phases [2512.10572].

## 5. Initialization, priors, and canonical-space structure

Initialization quality is a decisive variable in mesh-guided 2DGS. FMGS-Avatar initializes all canonical-space primitive states from a template mesh and then augments the limited monocular visual cues using foundation models trained on large-scale datasets, such as Sapiens. The abstract states that the distilled prior knowledge within a shared canonical space naturally enables spatially and temporally consistent rendering under novel views and poses [2509.14739].

GS4Buildings demonstrates a stronger prior regime in which Gaussians are initialized directly from LoD2 semantic 3D building models rather than SfM/MVS. Surface points are sampled on mesh faces with probability proportional to face area, normals are inherited from the face, tangent frames are constructed per sample, and visibility filtering keeps points visible in at least $k\approx 3$ views. The same LoD2 mesh is ray-cast into per-view depth $D_j$, normal $N_j$, and mask $M_j$ buffers with no additional smoothing or regularization. On the TUM2TWIN urban scenes, this prior-guided pipeline improves reconstruction completeness by $20.5\%$ over vanilla 2DGS and reduces M3C2 error by $32.8\%$ [2508.07355].

2D-SuGaR addresses weak initialization differently: it seeds 2D surfel Gaussians from monocular depth and normal estimates, resolves monocular scale by aligning medians to SfM sparse depths, prunes disconnected “islands” of Gaussians with DBSCAN at iteration $7\,000$, and then extracts a mesh by TSDF fusion and marching cubes for joint refinement. On $15$ DTU scenes, the full pipeline achieves an average Chamfer Distance of $0.67\times10^{-3}$ units, compared to $0.80$ for vanilla 2DGS and $0.77$ for NeRF-based methods [2605.00569].

GaMeS shows that mesh guidance need not require a mesh known a priori. If no mesh is available, a vanilla Gaussian splatting model can first be trained, then each learned Gaussian can be converted into a pseudo-triangle using the Gaussian mean and its two principal in-plane axes, after which the system is re-parameterized back into barycentric form and optimized as a mesh-guided model [2402.01459]. A plausible implication is that “mesh-guided” in this literature encompasses both explicit mesh-conditioned reconstruction and mesh recovery from a pre-existing Gaussian field.

## 6. Editing, efficiency, and limitations

The literature consistently links mesh guidance to editability and structured downstream graphics operations. GaMeS reports that once Gaussians are tied to mesh faces, any mesh edit such as vertex motion or face subdivision immediately updates the affected splats, and the splatting kernel can be replayed at $30$–$60$ Hz for real-time interactive editing [2402.01459]. REdiSplats pushes this logic further: because the Gaussians are instantiated as actual triangle meshes, they can participate in standard ray tracing with secondary rays for hard shadows or mirror reflections, can receive material properties such as index of refraction and roughness, and can be rendered using Blender or Nvdiffrast. Reported performance includes interactive $10$–$20$ fps at $800\times800$ on an RTX 3080 with $30$ K Gaussians for the OptiX variant and $30$–$60$ fps for the rasterized variant [2503.12284].

DeMapGS connects mesh-guided splats to asset extraction. After optimization, it samples UV texels per face and blends displacement, normal, and color from attached splats into full-resolution UV textures, yielding diffuse, normal, and displacement maps. The method reports PSNR approximately $33$–$35$ dB and OpenGL rendering at approximately $40$ FPS for mesh plus UV maps, described as a $4\times$ speed-up over SuGaR at approximately $10$ FPS, while also enabling texture editing, geometry editing, and cross-object parameter transfer through a shared template [2512.10572].

FMGS-Avatar emphasizes efficiency in a different operating regime: PyTorch with an Instant-NGP style hash grid for three separate $32$-d volumes, geometry, appearance, and semantics; geometry, appearance, semantics, and skinning MLPs with specified widths; a batch of $8$ K rays; Adam with learning rate $10^{-2}$ for hash volumes and $10^{-3}$ for MLPs; and a training schedule of approximately $10$ minutes on one A100 for approximately $200$k–$300$k steps with no weight decay [2509.14739]. The reported outcome is fast convergence, high-fidelity geometric detail, and robust multi-modal priors distilled from 2D foundation models while preserving surface coherence via the mesh-guided 2D Gaussian surfel representation.

A second recurrent misconception is that mesh guidance necessarily reduces reconstruction flexibility. The published designs instead distribute flexibility differently. FMGS-Avatar allows normal-direction displacement, log-scale adjustments, and axis-angle rotation residuals on top of per-face anchoring; DeMapGS adds per-splat barycentric motion, displacement, and quaternion rotation on a deformable template; GS4Buildings uses priors strongly in early training and decays them later; and 2D-SuGaR introduces mesh refinement only after prior-guided 2DGS optimization [2509.14739]. This suggests that the central trade-off is not between “mesh” and “flexibility,” but between unconstrained Gaussian freedom and explicitly surface-biased regularization.

Source: https://www.emergentmind.com/topics/mesh-guided-2d-gaussian-splatting