FMGS-Avatar: Unity-Ready Gaussian Avatars

• FMGS-Avatar is a comprehensive pipeline that converts a single in-the-wild video into an animatable, photorealistic avatar using 3D Gaussian Splatting and SMPL-X rigging.
• The system integrates advanced preprocessing, a canonical Gaussian representation, and a custom Unity editor to achieve real-time VR/AR-ready performance with >60 FPS.
• It overcomes limitations of manual and NeRF-based methods by introducing novel 2D kinematic-chain compensation and efficient training for seamless avatar animation.

Searching arXiv for the specified FMGS-Avatar paper and closely related Gaussian-avatar work to ground the article in current literature. FMGS-Avatar denotes a monocular-video avatar reconstruction pipeline that converts a single “in-the-wild” video of a person rotating in front of a roughly static camera into an animatable, photorealistic human avatar inside Unity. In the formulation described in "GSAC: Leveraging Gaussian Splatting for Photorealistic Avatar Creation with Unity Integration," the system combines customized preprocessing, SMPL-X co-registration, 3D Gaussian Splatting in a canonical T-pose, facial-expression embedding, and a Unity-integrated Gaussian Splatting Avatar Editor, with the stated aim of delivering real-time VR/AR-ready avatars from ordinary smartphone capture rather than from manual authoring or slow NeRF-based optimization (Zhang et al., 17 Apr 2025).

1. System definition and scope

FMGS-Avatar is organized as an end-to-end pipeline with four major stages: monocular video preprocessing, 3D Gaussian Splatting training, avatar rigging and facial-expression embedding, and Unity integration through a custom editor. The input assumption is specific: a single monocular video, for example a phone rotation clip, showing a person rotating in front of a roughly static camera. The output is equally specific: a fully rigged avatar model whose Gaussian representation is directly usable inside Unity for animation and preview (Zhang et al., 17 Apr 2025).

The system is positioned against two limitations identified in prior avatar workflows. Manual creation methods such as MetaHuman are described as requiring extensive time and expertise, while automatic approaches such as NeRF-based pipelines are described as often lacking efficiency, detailed facial expression fidelity, and sufficient rendering speed for real-time applications. FMGS-Avatar is therefore defined not merely as a rendering method, but as a production pipeline linking capture, reconstruction, rigging, and engine deployment (Zhang et al., 17 Apr 2025).

Stage	Function
Monocular Video Preprocessing	Frame extraction, keypoints, face/body fitting, segmentation, co-registration
3D Gaussian Splatting Training	Canonical T-pose Gaussian optimization and rendering loss minimization
Avatar Rigging and Facial-Expression Embedding	SMPL-X binding, blendshape transfer, hand and face animation
Unity Integration and Gaussian Avatar Editor	GPU rendering, compute-shader deformation, preview, LOD, pose control

A common misconception is to treat Gaussian-avatar systems as isolated radiance-field optimizers. FMGS-Avatar is broader: its defining characteristic is the coupling of Gaussian rendering with body-model alignment, facial reconstruction, and an interactive Unity toolchain. This suggests that its novelty lies as much in systems integration as in primitive choice.

2. Monocular preprocessing and co-registration

The preprocessing stage standardizes “in-the-wild” monocular videos before Gaussian training. The pipeline applies camera calibration or initialization with a centered pinhole model, image undistortion if needed, cropping to $1080\times1080$, background subtraction via Sapiens depth segmentation, 2D keypoint detection with MMPose for body, hands, and face, detailed facial geometry from DECA, and whole-body pose from Hand4Whole using SMPL-X (Zhang et al., 17 Apr 2025).

A distinctive component is missing-hand compensation. If a hand disappears with low confidence, FMGS-Avatar estimates its 2D trajectory by computing segment angular velocities over the gap and reprojecting joint positions frame by frame. The stated purpose is to stabilize SMPL-X fitting when hand detections are incomplete. This is described as a “2D kinematic-chain angular-velocity heuristic,” and the contribution list identifies it as a novel 2D kinematic-velocity strategy for missing-joint compensation (Zhang et al., 17 Apr 2025).

The co-registration stage optimizes SMPL-X parameters by minimizing

$$L_{\mathrm{SMPLX}} = L_{\mathrm{kpt}} + 0.1\,L_{\mathrm{init}} + L_{\mathrm{face}} + L_{\mathrm{reg}},$$

with

$$L_{\mathrm{face}} = 10\,L_{\mathrm{vertex}} + 10{,}000\,L_{\mathrm{lap}} + L_{\mathrm{edge}},$$

and

$$L_{\mathrm{reg}} = 0.01\,L_{\mathrm{shape}} + 100\,L_{\mathrm{jo}} + L_{\mathrm{sym}}.$$

Here $L_{\mathrm{kpt}}$ is the $L_1$ reprojection error of 2D keypoints, $L_{\mathrm{init}}$ is the $L_1$ distance to the initial SMPL-X guess, $L_{\mathrm{face}}$ combines vertex, Laplacian, and edge regularization, and $L_{\mathrm{reg}}$ combines shape, joint-offset, and symmetry regularization. Per-frame camera intrinsics are initialized with nominal focal length and zero pose (Zhang et al., 17 Apr 2025).

The practical significance of this stage is explicit in the reported design: the preprocessing pipeline is intended to standardize custom monocular data so that subsequent 3DGS training remains stable even for ordinary captures rather than controlled multiview studios.

3. Canonical Gaussian representation and rendering objective

FMGS-Avatar represents the subject in a canonical T-pose by a set of $$L_{\mathrm{SMPLX}} = L_{\mathrm{kpt}} + 0.1\,L_{\mathrm{init}} + L_{\mathrm{face}} + L_{\mathrm{reg}},$$0 Gaussians $$L_{\mathrm{SMPLX}} = L_{\mathrm{kpt}} + 0.1\,L_{\mathrm{init}} + L_{\mathrm{face}} + L_{\mathrm{reg}},$$1 anchored to SMPL-X mesh polygons. Each Gaussian $$L_{\mathrm{SMPLX}} = L_{\mathrm{kpt}} + 0.1\,L_{\mathrm{init}} + L_{\mathrm{face}} + L_{\mathrm{reg}},$$2 is parameterized by a mean $$L_{\mathrm{SMPLX}} = L_{\mathrm{kpt}} + 0.1\,L_{\mathrm{init}} + L_{\mathrm{face}} + L_{\mathrm{reg}},$$3, covariance $$L_{\mathrm{SMPLX}} = L_{\mathrm{kpt}} + 0.1\,L_{\mathrm{init}} + L_{\mathrm{face}} + L_{\mathrm{reg}},$$4, color $$L_{\mathrm{SMPLX}} = L_{\mathrm{kpt}} + 0.1\,L_{\mathrm{init}} + L_{\mathrm{face}} + L_{\mathrm{reg}},$$5, opacity $$L_{\mathrm{SMPLX}} = L_{\mathrm{kpt}} + 0.1\,L_{\mathrm{init}} + L_{\mathrm{face}} + L_{\mathrm{reg}},$$6, rotation $$L_{\mathrm{SMPLX}} = L_{\mathrm{kpt}} + 0.1\,L_{\mathrm{init}} + L_{\mathrm{face}} + L_{\mathrm{reg}},$$7 as a quaternion, and scale $$L_{\mathrm{SMPLX}} = L_{\mathrm{kpt}} + 0.1\,L_{\mathrm{init}} + L_{\mathrm{face}} + L_{\mathrm{reg}},$$8 as a scalar. The paper also gives the compact scene model

$$L_{\mathrm{SMPLX}} = L_{\mathrm{kpt}} + 0.1\,L_{\mathrm{init}} + L_{\mathrm{face}} + L_{\mathrm{reg}},$$9

Because the Gaussians are polygon-anchored rather than fully free-floating, deformation is driven by the underlying SMPL-X motion. For each training frame, each Gaussian is updated relative to its polygon via

$$L_{\mathrm{face}} = 10\,L_{\mathrm{vertex}} + 10{,}000\,L_{\mathrm{lap}} + L_{\mathrm{edge}},$$0

where $$L_{\mathrm{face}} = 10\,L_{\mathrm{vertex}} + 10{,}000\,L_{\mathrm{lap}} + L_{\mathrm{edge}},$$1 are the polygon’s rotation, translation, and scale from the SMPL-X pose (Zhang et al., 17 Apr 2025).

Rendering is defined by ray-casting Gaussians into image space. If $$L_{\mathrm{face}} = 10\,L_{\mathrm{vertex}} + 10{,}000\,L_{\mathrm{lap}} + L_{\mathrm{edge}},$$2 denotes the 3D point on Gaussian $$L_{\mathrm{face}} = 10\,L_{\mathrm{vertex}} + 10{,}000\,L_{\mathrm{lap}} + L_{\mathrm{edge}},$$3 projected along ray $$L_{\mathrm{face}} = 10\,L_{\mathrm{vertex}} + 10{,}000\,L_{\mathrm{lap}} + L_{\mathrm{edge}},$$4, then the pixel radiance is

$$L_{\mathrm{face}} = 10\,L_{\mathrm{vertex}} + 10{,}000\,L_{\mathrm{lap}} + L_{\mathrm{edge}},$$5

with Gaussian occupancy weight

$$L_{\mathrm{face}} = 10\,L_{\mathrm{vertex}} + 10{,}000\,L_{\mathrm{lap}} + L_{\mathrm{edge}},$$6

The training objective for the Gaussian renderer is

$$L_{\mathrm{face}} = 10\,L_{\mathrm{vertex}} + 10{,}000\,L_{\mathrm{lap}} + L_{\mathrm{edge}},$$7

and the full training objective is

$$L_{\mathrm{face}} = 10\,L_{\mathrm{vertex}} + 10{,}000\,L_{\mathrm{lap}} + L_{\mathrm{edge}},$$8

The terms are specified as per-pixel $$L_{\mathrm{face}} = 10\,L_{\mathrm{vertex}} + 10{,}000\,L_{\mathrm{lap}} + L_{\mathrm{edge}},$$9 photometric loss, LPIPS for learned perceptual similarity, SSIM for structural similarity, Sobel-filtered $$L_{\mathrm{reg}} = 0.01\,L_{\mathrm{shape}} + 100\,L_{\mathrm{jo}} + L_{\mathrm{sym}}.$$0 for edge sharpness, and a KNN smoothness term penalizing deviations among nearest-neighbor Gaussians (Zhang et al., 17 Apr 2025).

Optimization is reported to run for approximately $$L_{\mathrm{reg}} = 0.01\,L_{\mathrm{shape}} + 100\,L_{\mathrm{jo}} + L_{\mathrm{sym}}.$$1 iterations, about $$L_{\mathrm{reg}} = 0.01\,L_{\mathrm{shape}} + 100\,L_{\mathrm{jo}} + L_{\mathrm{sym}}.$$2 minutes on an RTX 4090, and to produce approximately $$L_{\mathrm{reg}} = 0.01\,L_{\mathrm{shape}} + 100\,L_{\mathrm{jo}} + L_{\mathrm{sym}}.$$3K Gaussians. This is central to the system’s identity: the representation is explicit enough for rapid optimization and explicit enough for direct engine-side manipulation.

4. Rigging, facial-expression embedding, and animation semantics

After training, the final Gaussian set is bound to a rigged SMPL-X skeleton mesh with both blendshape-based facial controls and joint-based articulation. FMGS-Avatar therefore does not treat the learned Gaussian cloud as a static radiance model; it embeds the Gaussian representation into an animation stack already familiar in character pipelines (Zhang et al., 17 Apr 2025).

Dynamic facial expression reconstruction is performed by estimating expression and identity coefficients from DECA or FLAME for each frame and embedding those coefficients into SMPL-X blendshapes. In turn, Gaussians near the face update their color $$L_{\mathrm{reg}} = 0.01\,L_{\mathrm{shape}} + 100\,L_{\mathrm{jo}} + L_{\mathrm{sym}}.$$4 and opacity $$L_{\mathrm{reg}} = 0.01\,L_{\mathrm{shape}} + 100\,L_{\mathrm{jo}} + L_{\mathrm{sym}}.$$5 as functions of the blendshape parameters. The same binding principle extends to non-facial motion: hand and finger articulations come directly from SMPL-X joint angles, and the affected Gaussians deform through the same per-polygon $$L_{\mathrm{reg}} = 0.01\,L_{\mathrm{shape}} + 100\,L_{\mathrm{jo}} + L_{\mathrm{sym}}.$$6 update used during training (Zhang et al., 17 Apr 2025).

This design has an important interpretive consequence. In FMGS-Avatar, facial expression is not only geometric. The data explicitly states that blendshape coefficients modulate Gaussian colors and opacities in facial regions. A plausible implication is that expression realism is handled as a coupled geometry-appearance problem rather than as mesh displacement alone.

Another misconception is that Gaussian avatars are necessarily difficult to rig because the primitives are not vertices. FMGS-Avatar addresses this by using the underlying SMPL-X mesh as the deformation scaffold, so that Gaussian transforms inherit the mesh’s articulated motion while preserving Gaussian rendering semantics.

5. Unity editor architecture and runtime deployment

The Unity component is implemented as a custom package named the “Gaussian Splatting Avatar Editor.” It imports the trained Gaussian buffer and SMPL-X mesh, renders Gaussians as 2D quads on the GPU, composites them via alpha blending with exponential falloff for natural transparency, and updates Gaussian transforms in a compute shader whenever the SMPL-X skeleton animates through either default Unity animation or a custom pose controller (Zhang et al., 17 Apr 2025).

The editor architecture is described in shader-level terms. A compute shader reads SMPL-X skeleton and polygon transforms each frame and updates Gaussian buffers on the GPU. A fragment shader blends the splatted quads with alpha falloff. The UI exposes real-time pose and blendshape parameter adjustment through sliders and curves, LOD controls that vary the number of Gaussians drawn, target frame-rate and quality presets, and export or import of custom pose clips or live MoCap streams (Zhang et al., 17 Apr 2025).

The stated runtime target is real-time preview with 60+ FPS on a desktop GPU, and the experimental section reports sustained $$L_{\mathrm{reg}} = 0.01\,L_{\mathrm{shape}} + 100\,L_{\mathrm{jo}} + L_{\mathrm{sym}}.$$7 FPS during Unity animation. This integration is not incidental. The pipeline is defined so that the learned representation remains directly operable within a game engine rather than requiring offline conversion to a different asset type (Zhang et al., 17 Apr 2025).

From an application perspective, the Unity layer is what makes the system relevant to VR/AR production. The abstract explicitly situates photorealistic avatars in training simulations, telemedicine, and virtual collaboration, and the editor is the mechanism that turns the reconstruction output into an application-facing asset.

6. Empirical performance, limitations, and relation to adjacent Gaussian-avatar research

Quantitative evaluation on PeopleSnapshot and a custom volunteer dataset reports average training time of $$L_{\mathrm{reg}} = 0.01\,L_{\mathrm{shape}} + 100\,L_{\mathrm{jo}} + L_{\mathrm{sym}}.$$8–$$L_{\mathrm{reg}} = 0.01\,L_{\mathrm{shape}} + 100\,L_{\mathrm{jo}} + L_{\mathrm{sym}}.$$9 s with hand correction versus $L_{\mathrm{kpt}}$0 s for HAHA, Gaussian count of $L_{\mathrm{kpt}}$1K–$L_{\mathrm{kpt}}$2K, image quality around PSNR $L_{\mathrm{kpt}}$3–$L_{\mathrm{kpt}}$4 dB, SSIM $L_{\mathrm{kpt}}$5–$L_{\mathrm{kpt}}$6, and LPIPS $L_{\mathrm{kpt}}$7–$L_{\mathrm{kpt}}$8, with runtime on a desktop GPU of $L_{\mathrm{kpt}}$9–$L_1$0 FPS at $L_1$1p and sustained $L_1$2 FPS during Unity animation. The end-to-end pipeline is described as operating in under $L_1$3 minutes, and the contribution list characterizes it as the first open-source, end-to-end pipeline from single monocular video to a Unity-ready Gaussian avatar (Zhang et al., 17 Apr 2025).

The limitations are equally explicit. Unseen regions such as underarms and the occluded back are under-reconstructed; clothing is tied to the body mesh, with no independent cloth dynamics or outfit swapping; and mask accuracy can introduce minor artifacts, especially in hair. Proposed future directions include generative completion of occluded areas, modular clothing layers, cloth-aware Gaussian dynamics, and real-time MoCap integration for fully interactive VR/AR (Zhang et al., 17 Apr 2025).

Within the broader arXiv literature, related methods make clear that Gaussian-avatar research is not architecturally uniform. MonoGaussianAvatar reconstructs a monocular head avatar from explicit 3D Gaussian points plus a continuous Gaussian deformation field, with training reported at about $L_1$4 h on a 3090 and rendering at $L_1$5 ms per image (Chen et al., 2023). MixedGaussianAvatar instead attaches 2D Gaussians to the FLAME mesh and adds compensating 3D Gaussians in high-error regions, reporting on NeRSemble $L_1$6, PSNR $L_1$7 dB, SSIM $L_1$8, and LPIPS $L_1$9 (Chen et al., 2024). A later mesh-guided method uses one 2D Gaussian “surfel” per upsampled SMPL face and distills Sapiens depth, normal, and semantic priors under selective gradient isolation, with reported convergence in about $L_{\mathrm{init}}$0 minutes and $L_{\mathrm{init}}$1 FPS inference (Fan et al., 18 Sep 2025). FastAvatar replaces per-subject optimization with a feed-forward Large Gaussian Reconstruction Transformer that reconstructs a 3DGS model within seconds, reporting about $L_{\mathrm{init}}$2 s modeling and $L_{\mathrm{init}}$3 FPS in the single-image case (Wu et al., 27 Aug 2025).

These neighboring systems clarify FMGS-Avatar’s research niche. The Unity-oriented formulation in (Zhang et al., 17 Apr 2025) is distinguished less by pursuing the most specialized head-only reconstruction regime and more by coupling monocular full-body preprocessing, SMPL-X-grounded Gaussian deformation, facial-expression embedding, and engine-native deployment. This suggests that its principal significance lies in operationalizing Gaussian avatars as editable, animatable assets for real-time XR workflows rather than treating reconstruction quality in isolation.