Mesh-Embedded Gaussian Splatting

• Mesh-Embedded Gaussian Splatting is a hybrid 3D representation that couples Gaussian primitives with mesh topology to improve photoreal rendering, explicit structure, and dynamic editability.
• It leverages mesh-based techniques—such as face-conditioned parameterization and barycentric embeddings—to control Gaussian pose, scale, and deformation for both static and dynamic scenes.
• Multiple design variants enable tailored solutions for avatars, scene reconstruction, and high-fidelity geometry extraction, balancing detail capture with robust topology management.

Mesh-Embedded Gaussian Splatting denotes a family of hybrid 3D representations that couple Gaussian primitives with mesh structure rather than treating Gaussians as entirely free-floating volumetric elements. Across the literature, the mesh may serve as a face-wise parameterization for Gaussian means and covariances, a deformation proxy that transports Gaussian pose and scale, a textured background layer that offloads planar appearance, or a differentiably extracted surface that remains in the optimization loop. The common motivation is that vanilla 3D Gaussian Splatting (3DGS) is fast and photorealistic but lacks explicit topology, direct editability, and consistently reliable surface geometry, whereas meshes provide explicit structure, animation compatibility, and downstream utility for simulation or editing but are less effective at capturing fuzzy geometry, thin structures, or highly view-dependent detail (Waczyńska et al., 2024).

1. Conceptual scope and research trajectory

The modern literature does not define a single canonical formulation of mesh-embedded Gaussian splatting. Instead, it spans several closely related designs that all use mesh structure to regularize, parameterize, or control Gaussian primitives. Early work emphasized direct face attachment and editability, especially in GaMeS, which parameterizes each Gaussian component by the vertices of a mesh face and updates its mean, scale, and rotation automatically under mesh deformation (Waczyńska et al., 2024). In parallel, avatar methods such as SplattingAvatar used trainable mesh embeddings to disentangle low-frequency motion from high-frequency appearance, while GauMesh combined a UV-mapped mesh and 3DGS in one depth-sorted differentiable renderer for dynamic head avatars (Shao et al., 2024, Cai et al., 2024).

A second line of work treats the mesh less as a per-Gaussian parameterization and more as a complementary representation. Hybrid Mesh-Gaussian Representation for indoor scenes uses a textured mesh for texture-rich flat areas such as walls or ceilings and retains Gaussians in geometrically complex regions (Huang et al., 8 Jun 2025). MeshGS similarly distinguishes tightly-bound and loosely-bound splats according to distance from a mesh surface, using the former as surface-conforming primitives and the latter as a corrective layer for mesh artifacts and missing geometry (Choi et al., 2024). Direct Learning of Mesh and Appearance via 3DGS goes further by learning a mesh and its attached Gaussian appearance end-to-end from photometric supervision, rather than extracting geometry only after Gaussian optimization (Lin et al., 2024).

A third direction makes the mesh itself a differentiable participant in optimization. MILo activates mesh extraction during training, extracting a mesh at every iteration from Gaussian-derived pivots and learned scalar values so that mesh-based consistency losses directly constrain the Gaussian field (Guédon et al., 30 Jun 2025). UniMGS then addresses a separate systems problem: joint rasterization and deformation of triangles and Gaussians in a single pass with explicit handling of occlusion, transparency, and proxy-driven Gaussian deformation (Xiao et al., 27 Jan 2026).

Work	Mesh–Gaussian relation	Primary emphasis
GaMeS (Waczyńska et al., 2024)	Each Gaussian is parameterized by a mesh face	Editability and animation
SplattingAvatar (Shao et al., 2024)	Trainable barycentric embedding on a triangle mesh	Real-time human avatars
GauMesh (Cai et al., 2024)	UV mesh and 3DGS rendered in a unified depth-sorted pipeline	Dynamic head avatars
Direct Learning (Lin et al., 2024)	Gaussians bound to mesh faces extracted from a learnable SDF grid	End-to-end geometry and appearance learning
MeshGS (Choi et al., 2024)	Distance-based mesh alignment with tightly/loosely bound splats	High-quality scene rendering with fewer splats
Hybrid Mesh-Gaussian (Huang et al., 8 Jun 2025)	Textured mesh for flat texture-rich regions, Gaussians for complex geometry	Efficient indoor scene reconstruction
MILo (Guédon et al., 30 Jun 2025)	Mesh extracted differentiably from Gaussians at every iteration	Detailed and efficient surface reconstruction

2. Representation designs

The most explicit mesh-embedded formulation is face-conditioned parameterization. In GaMeS, for a triangle face with vertices $V=\{v_1,v_2,v_3\}$, the Gaussian mean is a convex combination of the three vertices,

$$\mu_V(\alpha_1,\alpha_2,\alpha_3)=\alpha_1 v_1+\alpha_2 v_2+\alpha_3 v_3,\qquad \alpha_1+\alpha_2+\alpha_3=1.$$

The covariance is derived from a face-based orthonormal frame and a diagonal scale matrix, yielding $\Sigma_V=R_V^TS_VS_VR_V$. One axis is set to a small constant $\varepsilon$, making the Gaussian nearly flat in the face-normal direction. This construction makes mesh deformation propagate automatically to Gaussian position, rotation, and scale (Waczyńska et al., 2024).

SplattingAvatar adopts a more flexible embedding. Each Gaussian is attached to one triangle of the canonical mesh via

$E=\{k,u,v,d\},$

where $k$ is the triangle index, $(u,v)$ are barycentric coordinates, and $d$ is a displacement along the interpolated normal. With triangle vertices $\{V_1,V_2,V_3\}$ and vertex normals $\{\mathbf n_1,\mathbf n_2,\mathbf n_3\}$,

$$\mu_V(\alpha_1,\alpha_2,\alpha_3)=\alpha_1 v_1+\alpha_2 v_2+\alpha_3 v_3,\qquad \alpha_1+\alpha_2+\alpha_3=1.$$0

and the Gaussian mean is $$\mu_V(\alpha_1,\alpha_2,\alpha_3)=\alpha_1 v_1+\alpha_2 v_2+\alpha_3 v_3,\qquad \alpha_1+\alpha_2+\alpha_3=1.$$1. This creates what the paper describes as a first-order continuous neighborhood around the mesh surface, permitting off-surface detail such as hair or glasses while keeping the representation explicitly mesh-controlled (Shao et al., 2024).

Several methods relax strict surface attachment to improve robustness. Mani-GS optimizes Gaussian position, rotation, and scale in triangle-local coordinates and then maps them to world coordinates with the local triangle frame. Under deformation, the triangle-local parameters stay fixed while world-space parameters are recomputed from the new triangle geometry, with explicit triangle-shape-aware adaptation of position and scale (Gao et al., 2024). MaGS similarly keeps Gaussians on or near the mesh surface but allows relative displacement between mesh and Gaussian through a Relative Deformation Field, so the Gaussians can “hover” with respect to the deforming mesh instead of remaining rigidly anchored (Ma et al., 2024).

A surface-centric variant appears in FMGS-Avatar, which replaces free-form 3D ellipsoids with mesh-guided 2D Gaussian surfels. One 2D Gaussian is attached to each upsampled mesh face; its center is the face barycenter, its orientation is the local tangent frame, and its residual motion is constrained to the surface normal direction,

$$\mu_V(\alpha_1,\alpha_2,\alpha_3)=\alpha_1 v_1+\alpha_2 v_2+\alpha_3 v_3,\qquad \alpha_1+\alpha_2+\alpha_3=1.$$2

This removes the volumetric freedom of 3DGS and makes the representation surface-aligned by construction (Fan et al., 18 Sep 2025).

3. Optimization and rendering couplings

A central difficulty is that mesh and Gaussian components have different rendering semantics. One class of solutions binds Gaussians to meshes but still uses a standard Gaussian rasterizer for photometric supervision. Direct Learning of Mesh and Appearance via 3DGS starts from a learnable SDF grid, extracts a mesh with FlexiCubes, binds a fixed number of Gaussians to each face by barycentric coordinates, and renders them with the standard 3DGS projection and alpha-compositing pipeline. The image loss

$$\mu_V(\alpha_1,\alpha_2,\alpha_3)=\alpha_1 v_1+\alpha_2 v_2+\alpha_3 v_3,\qquad \alpha_1+\alpha_2+\alpha_3=1.$$3

updates both geometry and appearance because the Gaussian centers depend on the extracted mesh and the mesh depends on the SDF grid (Lin et al., 2024).

Hybrid renderers instead keep a distinct mesh branch and a distinct Gaussian branch. GauMesh rasterizes the UV-mapped mesh with Nvdiffrast to obtain per-pixel depth, opacity, and color, merges the mesh result into the Gaussian depth list, and then performs front-to-back $$\mu_V(\alpha_1,\alpha_2,\alpha_3)=\alpha_1 v_1+\alpha_2 v_2+\alpha_3 v_3,\qquad \alpha_1+\alpha_2+\alpha_3=1.$$4-blending in the merged order. The paper emphasizes a stable hybrid depth sorting strategy to prevent Gaussian splats from crossing mesh facets visually (Cai et al., 2024). Hybrid Mesh-Gaussian Representation for indoor scenes also performs joint optimization of 3DGS and mesh, but its mesh branch acts as a textured background with a fixed depth value for flat, texture-dense areas. Its optimization introduces a warm-up stage and a transmittance-aware texture loss so that background mesh supervision does not absorb foreground appearance and create ghost artifacts (Huang et al., 8 Jun 2025).

A different strategy makes surface extraction part of the training loop. MILo samples Gaussian-derived “pivots” for Delaunay triangulation, assigns learned scalar values to those pivots, applies differentiable Marching Tetrahedra, and then compares mesh-rendered depth and normals to Gaussian-rendered depth and normals. Its total loss combines volumetric rendering terms, mesh consistency terms, and regularizers: $$\mu_V(\alpha_1,\alpha_2,\alpha_3)=\alpha_1 v_1+\alpha_2 v_2+\alpha_3 v_3,\qquad \alpha_1+\alpha_2+\alpha_3=1.$$5 The paper describes this as bidirectional consistency: Gaussians determine the mesh, and mesh-based supervision sends gradients back to Gaussian parameters and scalar values (Guédon et al., 30 Jun 2025).

UniMGS targets the remaining mismatch at the renderer level. Rather than rendering mesh and 3DGS in separate passes, it adapts the 3DGS rasterizer so that triangle fragments and Gaussian fragments are composited together in a single depth-ordered pass with anti-aliased $$\mu_V(\alpha_1,\alpha_2,\alpha_3)=\alpha_1 v_1+\alpha_2 v_2+\alpha_3 v_3,\qquad \alpha_1+\alpha_2+\alpha_3=1.$$6-blending. The paper argues that separate-pass pipelines fail in nested or transparent configurations because depth-map compositing cannot recover correct visibility relations in interleaved geometry (Xiao et al., 27 Jan 2026).

4. Mesh-driven deformation, manipulation, and avatar modeling

Human avatars provided some of the earliest and clearest demonstrations of why mesh embedding matters. SplattingAvatar uses the mesh as the driver of pose, rotation, and scale, avoiding an MLP-based linear blend skinning field. Triangle rotations are estimated from canonical and posed tangent-bitangent-normal frames, converted to quaternions, and interpolated barycentrically for each Gaussian. A lifted optimization procedure lets a Gaussian “walk” across neighboring triangles when a barycentric update leaves the current face, so embeddings are trainable rather than fixed. The paper reports over 300 FPS on an NVIDIA RTX 3090 and a stable 30 FPS on an iPhone 13, together with state-of-the-art results on both head and full-body benchmarks (Shao et al., 2024).

GauMesh addresses a related problem in dynamic head avatars but uses a different split of responsibilities: a UV-mapped mesh stores high-resolution texture and opacity, while time-deformed Gaussians capture complex geometry and fuzzy structures. After multiview tracking on Multiface, the method initializes 100,000 Gaussian points in canonical space according to tracked meshes and deforms them over time with a grid-based feature volume and a shallow MLP. Quantitatively, the paper reports PSNR 34.23, SSIM 0.8872, and LPIPS 0.1320, compared with 4DGaussians at PSNR 33.76, SSIM 0.8789, and LPIPS 0.1636 (Cai et al., 2024).

RMAvatar extends the mesh-driven avatar paradigm to monocular clothed-human reconstruction. Gaussians are embedded into triangular faces and moved by the mesh for coarse articulated motion, but a pose-related Gaussian rectification module predicts $$\mu_V(\alpha_1,\alpha_2,\alpha_3)=\alpha_1 v_1+\alpha_2 v_2+\alpha_3 v_3,\qquad \alpha_1+\alpha_2+\alpha_3=1.$$7 from encoded Gaussian position and pose to model fine non-rigid deformation: $$\mu_V(\alpha_1,\alpha_2,\alpha_3)=\alpha_1 v_1+\alpha_2 v_2+\alpha_3 v_3,\qquad \alpha_1+\alpha_2+\alpha_3=1.$$8 On PeopleSnapshot, the paper reports 34.12 PSNR, 0.985 SSIM, and 0.013 LPIPS on male-3-casual, and on ZJU-MoCap it reports 32.68 PSNR, 0.982 SSIM, and 0.015 LPIPS on subject 377 (Peng et al., 13 Jan 2025).

For editable scene manipulation, Mani-GS binds Gaussians to triangular meshes in local triangle coordinates and supports large deformation, local editing, and soft-body simulation through standard mesh tools. The paper reports average results on NeRF-Synthetic of PSNR 33.65, SSIM 0.966, and LPIPS 0.030, versus SuGaR at PSNR 30.61, SSIM 0.9531, and LPIPS 0.054 (Gao et al., 2024). MaGS tackles dynamic monocular reconstruction and simulation by combining ARAP mesh deformation with a Relative Deformation Field that updates Gaussian placement on a facet and adds an offset, allowing mesh-guided but non-rigid Gaussian motion; on D-NeRF it reports average PSNR 42.74, SSIM 0.9975, and LPIPS 0.0115 (Ma et al., 2024).

Editable Gaussian fields can also be driven indirectly through a mesh prior. Neural Surface Priors for Editable Gaussian Splatting learns a PermutoSDF surface, conditions Gaussian opacity on the SDF value at the Gaussian center, encodes each Gaussian as a triangle in a triangle-soup proxy, and transfers edits applied to the extracted mesh through local face-basis transforms back into Gaussian position, orientation, and scale. The method reports DTU Chamfer-$$\mu_V(\alpha_1,\alpha_2,\alpha_3)=\alpha_1 v_1+\alpha_2 v_2+\alpha_3 v_3,\qquad \alpha_1+\alpha_2+\alpha_3=1.$$9 averages of 0.61 with mask and 0.68 without mask, improving on 3DGSR at 0.81 (Szymkowiak et al., 2024).

5. Reconstruction quality, compactness, and efficiency

For static and indoor scenes, hybrid mesh–Gaussian designs are often motivated less by raw photometric gains than by a quality–efficiency trade-off. Hybrid Mesh-Gaussian Representation for Efficient Indoor Scene Reconstruction prunes large Gaussians, extracts and simplifies a mesh with QSlim, removes geometrically unreliable regions using a StableNormal-based normal-variation metric, an adjacent-angle rule with $\Sigma_V=R_V^TS_VS_VR_V$0, a triangle-area criterion, hole filling, and LS3 subdivision/smoothing, and then jointly optimizes mesh texture and 3DGS. On ScanNet++, the method reduces Gaussian primitives from 0.911M to 0.742M and raises FPS from 211 to 231 relative to 3DGS; on Deep Blending it reduces Gaussian primitives from 2.634M to 1.698M and raises FPS from 346 to 498. On Replica, it reports 0.973 SSIM / 36.32 PSNR / 0.034 LPIPS with 0.295M Gaussians and 446 FPS, compared with 3DGS at 1.667M Gaussians and 122 FPS (Huang et al., 8 Jun 2025).

MeshGS pursues a related objective in large unbounded scenes but keeps two Gaussian classes. Tightly-bound splats are flattened and aligned to the mesh surface through normal consistency, scale regularization, and projection loss, while loosely-bound splats are optimized mainly by image supervision and serve as a corrective layer for holes, missing thin structures, and background artifacts. On outdoor Mip-NeRF 360, the paper reports mean PSNR 25.7 dB, compared with SuGaR at 24.4 dB and GaMeS at 24.8 dB; it also reports a 30% reduction in Gaussian splats relative to original 3DGS, from a mean of 3263k to 2360k (Choi et al., 2024).

Direct Learning of Mesh and Appearance via 3DGS emphasizes the reciprocal benefit of explicit geometry and Gaussian rendering. After 3000 iterations of vanilla 3DGS and Alpha Shapes initialization, it performs a 10k-iteration joint learning stage and an optional 7k-iteration refinement stage. The paper states training time under 2 hours on a single GPU and reports, on NeRF-Synthetic, values such as 35.56 PSNR on Chair, 33.51 PSNR on Ficus, and 36.41 PSNR on Hotdog, while using 100k faces versus NeRF2Mesh at 192k faces in one cited comparison (Lin et al., 2024).

MILo is unusual in prioritizing explicit mesh quality and compactness during Gaussian training rather than only image metrics. The paper states that prior methods often output meshes with 14–16 million vertices or more, whereas MILo base outputs about 4.36M vertices on Tanks and Temples and does so while remaining competitive on DTU and strong on explicit-representation F1 metrics (Guédon et al., 30 Jun 2025). This suggests that mesh-in-the-loop optimization is especially relevant when the downstream consumer is the mesh itself rather than only the Gaussian renderer.

6. Limitations, distinctions, and current directions

A recurring limitation is dependence on mesh quality or proxy quality. MeshGS assumes a reasonably useful initial mesh and notes weaknesses with overly large triangles and the difficulty of assessing how well splats are bound to the surface (Choi et al., 2024). MaGS explicitly reports sensitivity to the quality of the initial mesh and weaker performance when viewpoints are concentrated or the extracted mesh is incomplete (Ma et al., 2024). UniMGS argues that many prior proxy-mesh deformation methods are sensitive to bad proxy topology, which motivates its Gaussian-centric binding and BBX-based multi-face association (Xiao et al., 27 Jan 2026).

Another distinction is that “mesh-embedded” does not always mean “surface-only.” GaMeS and some surfel-based formulations deliberately keep Gaussians nearly flat and face-aligned (Waczyńska et al., 2024), but Mani-GS and MaGS preserve off-surface degrees of freedom to remain robust to mesh inaccuracies or non-rigid residual motion (Gao et al., 2024, Ma et al., 2024). Hybrid Mesh-Gaussian indoor reconstruction and MeshGS go further by explicitly reserving some scene regions for Gaussians because the mesh is expected to be unreliable there (Huang et al., 8 Jun 2025, Choi et al., 2024). A plausible implication is that the field is converging on division-of-labor designs rather than a universal commitment to strict mesh adherence.

Editing pipelines also have structural constraints. Neural Surface Priors assumes edits that preserve face correspondence and notes that appearance may become physically inconsistent after geometry changes, for example when a shadow remains attached to a moved object (Szymkowiak et al., 2024). Ray-tracing-based integrations broaden lighting and tool compatibility but introduce specialized-system requirements: RaySplats requires a renderer supporting both ray tracing and 3D Gaussian Splatting, while REdiSplats approximates each flat Gaussian by an octagonal mesh proxy so that scenes can be rendered in Blender or Nvdiffrast and manipulated through mesh vertices (Byrski et al., 31 Jan 2025, Byrski et al., 15 Mar 2025).

Recent work also points toward broader system integration. FMGS-Avatar combines mesh-guided 2D Gaussian surfels with foundation-model priors and a coordinated training strategy with selective gradient isolation, reporting on ZJU-MoCap PSNR 30.89, SSIM 0.972, LPIPS 28.59, and 55 FPS (Fan et al., 18 Sep 2025). Proxy-GS uses a lightweight proxy mesh not to embed Gaussians directly but to provide occlusion-aware culling and proxy-guided densification, reporting more than $\Sigma_V=R_V^TS_VS_VR_V$1 speedup over Octree-GS in heavily occluded scenes and a 1000×1000 depth map in under 1 ms (Gao et al., 29 Sep 2025). A separate 2025 framework proposal combines 3DGS, SAM, GS2Mesh, LLM-based material assignment, and XPBD for non-rigid editing, but the paper explicitly states that the reported results are expected results rather than direct experimental benchmarks (B, 9 Jul 2025).

Taken together, the literature indicates that mesh-embedded Gaussian splatting is less a single algorithm than a research program for reconciling two explicit graphics primitives with complementary strengths. Meshes contribute topology, deformation structure, and downstream interoperability; Gaussians contribute differentiable rendering, photometric fidelity, and tolerance to geometric irregularity. The central technical question is therefore not whether one should replace the other, but how tightly they should be coupled for a given task: face-wise parameterization for editability, hybrid decomposition for efficiency, or mesh-in-the-loop extraction for surface fidelity.