EGSTalker: Audio-Driven 3D Gaussian Splatting

• EGSTalker is a real-time, audio-driven talking head framework that uses 3D Gaussian Splatting for identity-specific synthesis.
• It employs a two-stage pipeline: the first stage initializes a static Gaussian head from 3–5 minutes of video, and the second stage drives dynamic, audio-conditioned deformations.
• The system achieves competitive performance with efficient spatial–audio fusion, rendering at 68.5 FPS with high-quality lip synchronization and realistic facial details.

Searching arXiv for EGSTalker, GSTalker, and related Gaussian-splatting works to ground the article in current papers. arxiv_search(query="EGSTalker GSTalker Gaussian Splatting talking head", max_results=10) arxiv_search(query="GS-EVT event camera Gaussian Splatting tracking", max_results=10) Searching the arXiv entries directly for the named systems and close variants. EGSTalker is a real-time, audio-driven talking head generation framework built on 3D Gaussian Splatting (3DGS). In the 2025 formulation, it is designed for photorealistic, identity-specific synthesis from only 3–5 minutes of video, using a two-stage pipeline consisting of static Gaussian initialization and audio-driven deformation (Zhu et al., 3 Oct 2025). The name also has a documented ambiguity: in some discussions it refers to the earlier "GSTalker" talking-face model (Chen et al., 2024), while in a separate event-vision context an “EGSTalker” can denote a system conceptually equivalent to GS-EVT, an event-camera tracker based on Gaussian Splatting (Liu et al., 2024). In contemporary usage, however, EGSTalker most directly denotes the talking-head system introduced in "EGSTalker: Real-Time Audio-Driven Talking Head Generation with Efficient Gaussian Deformation" (Zhu et al., 3 Oct 2025).

1. Terminology and disambiguation

The label “EGSTalker” is not historically unique. The 2024 paper "GSTalker: Real-time Audio-Driven Talking Face Generation via Deformable Gaussian Splatting" states that “EGSTalker” refers to the model presented in the paper as GSTalker, and that there is no separate EGSTalker in that work (Chen et al., 2024). A later paper formalized the name in its title as "EGSTalker: Real-Time Audio-Driven Talking Head Generation with Efficient Gaussian Deformation" (Zhu et al., 3 Oct 2025).

A separate ambiguity appears in event-based visual localization. In that setting, “EGSTalker” can denote an “Event + Gaussian Splatting Tracker,” a concept described as essentially the same class of method as GS-EVT: an event-driven tracker that estimates the 6-DOF pose of an event camera by aligning integrated event measurements to photorealistic views rendered from a pre-built 3D Gaussian Splatting map (Liu et al., 2024).

Name	Domain	Relation to “EGSTalker”
"EGSTalker" (Zhu et al., 3 Oct 2025)	Audio-driven talking head generation	Direct use of the name
"GSTalker" (Chen et al., 2024)	Audio-driven talking face generation	Earlier model that some references call EGSTalker
"GS-EVT" (Liu et al., 2024)	Event-camera tracking	Separate use of an “Event + Gaussian Splatting Tracker” concept

For the talking-head literature, EGSTalker denotes a 3DGS-based, identity-specific synthesis system in which a compact Gaussian scene is first learned in a canonical state and then deformed per frame under audio conditioning (Zhu et al., 3 Oct 2025).

2. Problem setting and system formulation

EGSTalker addresses real-time, photorealistic talking-head synthesis driven by audio, from only 3–5 minutes of video of a specific identity (Zhu et al., 3 Oct 2025). The core challenges are stated as efficiency, cross-modal fusion, and structured dynamics. NeRF-based methods are characterized as achieving quality but being slow to train and render, while naïve 3DGS fusion with audio can be fast but may lose fidelity or synchronization. The model therefore combines an explicit Gaussian representation with an audio-aware deformation mechanism.

The end-to-end flow is organized into two stages. In preprocessing, the video is face-centric cropped and resized to $512\times512$ (with the Obama clip at $450\times450$); landmarks are used to crop lip regions for a lip reconstruction loss; eye features $f_e$ and pose embeddings $f_p$ are extracted and concatenated with audio tokens; and an audio sequence $f_a$ is extracted, although the paper does not specify the extractor type (Zhu et al., 3 Oct 2025). Stage 1 builds a multi-resolution hash triplane encoder, samples per-3D-point features $f_v(x)$, and uses a Kolmogorov–Arnold Network (KAN) to map those features to Gaussian parameters for a compact static head representation $G_{\text{static}}$. Stage 2 uses Efficient Spatial–Audio Attention (ESAA) to fuse spatial features with audio plus eye and pose tokens, then applies a KAN to predict per-Gaussian offsets of position, scale, rotation, spherical harmonics (SH), and opacity for each frame (Zhu et al., 3 Oct 2025).

This formulation places EGSTalker in the class of deformable 3DGS talking-head systems, but with three specific technical commitments: a multi-resolution hash triplane as spatial encoder, KANs instead of conventional MLPs for both static and dynamic mappings, and an agent-token attention design that reduces the cost of global audio–spatial interaction (Zhu et al., 3 Oct 2025).

3. Static Gaussian initialization and 3DGS representation

Stage 1 uses three mutually orthogonal 2D planes $(xy, yz, zx)$, each implemented as a multi-resolution hashed 2D grid in a HexPlane-style encoder (Zhu et al., 3 Oct 2025). For a 3D point $x=(x,y,z)\in\mathbb{R}^3$, the model bilinearly samples plane features at each resolution level and fuses the three plane descriptors by elementwise Hadamard product: $f_v(x)=f_{xy}(x)\odot f_{yz}(x)\odot f_{zx}(x).$ The paper highlights the Hadamard product as the plane fusion mechanism (Zhu et al., 3 Oct 2025).

A KAN then maps $450\times450$0 to Gaussian attributes: $450\times450$1 Each Gaussian $450\times450$2 carries $450\times450$3, $450\times450$4, $450\times450$5, $450\times450$6, and $450\times450$7. The covariance is formed as

$450\times450$8

where $450\times450$9 is the rotation matrix from $f_e$0 (Zhu et al., 3 Oct 2025).

Rendering follows standard 3DGS/EWA splatting. With pinhole projection $f_e$1, the projected mean and covariance are

$f_e$2

with $f_e$3 the first-order Jacobian of the projection. The projected Gaussian contributes by an elliptical weight $f_e$4, and front-to-back compositing accumulates

$f_e$5

where $f_e$6 is the radiance from SH evaluated at the view direction $f_e$7 (Zhu et al., 3 Oct 2025).

The static stage is optimized with a composite 3DGS-style objective,

$f_e$8

with $f_e$9, D-SSIM, and LPIPS explicitly listed, while the weights are not disclosed (Zhu et al., 3 Oct 2025). The paper states that initialization yields denser Gaussians in expression-critical regions such as the lips and eyes, improving dynamic fidelity, although exact Gaussian counts and pruning strategy are not specified (Zhu et al., 3 Oct 2025).

4. Audio-driven deformation, ESAA, and KAN dynamics

The dynamic stage centers on Efficient Spatial–Audio Attention. Its inputs are spatial tokens $f_p$0, audio tokens $f_p$1, optional eye features $f_p$2, pose embeddings $f_p$3, and a global zero-vector $f_p$4, all projected into a common latent space (Zhu et al., 3 Oct 2025). EGSTalker also introduces periodic positional encoding (PPE),

$f_p$5

to encode audio-driven periodicity (Zhu et al., 3 Oct 2025).

ESAA uses a two-step agent-token cross-attention. Agent tokens are generated as

$f_p$6

with $f_p$7 tokens, and the paper reports that empirically $f_p$8 is a good speed/quality trade-off (Zhu et al., 3 Oct 2025). The first step, Agent Aggregation, computes

$f_p$9

where $f_a$0. The second step, Agent Broadcast, returns audio information to the spatial tokens: $f_a$1 This reduces complexity from $f_a$2 to $f_a$3 while preserving global audio–spatial interactions (Zhu et al., 3 Oct 2025).

A second KAN maps the audio-aware features to per-Gaussian offsets: $f_a$4 The deformations are applied as

$f_a$5

with updated covariance

$f_a$6

Rendering then uses the same 3DGS/EWA pipeline to produce the frame $f_a$7, summarized as

$f_a$8

The deformation-stage objective augments the static loss with a lip reconstruction loss: $f_a$9 and the paper explicitly states that there is no explicit use of sync-network embeddings such as SyncNet in the loss; LSE-C and LSE-D are used only for evaluation (Zhu et al., 3 Oct 2025).

5. Training procedure, inference path, and quantitative performance

The paper reports training on a single NVIDIA Tesla T4 (16 GB), with typical training time of 3.7 hours for 3–5 minutes of video (Zhu et al., 3 Oct 2025). Inference is real-time at 68.5 FPS, corresponding to approximately 14.6 ms per frame in the reported self-driven setting (Zhu et al., 3 Oct 2025). Unlike the earlier GSTalker paper, which specifies optimizer settings and training iterations (Chen et al., 2024), the EGSTalker paper does not specify optimizer type, learning rate schedules, batch sizes, or number of iterations (Zhu et al., 3 Oct 2025).

The evaluation uses four high-resolution single-identity video clips, approximately 6500 frames each at 25 FPS, with center-cropped and resized frames and landmark detection for the lip crop during training (Zhu et al., 3 Oct 2025). Baselines include Wav2Lip, IP-LAP, AD-NeRF, RAD-NeRF, ER-NeRF, TalkingGaussian, and GaussianTalker (Zhu et al., 3 Oct 2025).

Setting	Metric bundle	EGSTalker result
Self-driven	PSNR / SSIM / LPIPS / FID	36.070 / 0.992 / 0.0223 / 2.424
Self-driven	LMD / LSE-D / LSE-C	2.536 / 8.237 / 6.966
Efficiency	Training / FPS	3.7 h / 68.51
Cross-driven, Testset A	LMD / LSE-C / LSE-D	7.459 / 6.945 / 8.470

In the self-driven setting, GaussianTalker is reported as achieving similar fidelity, with PSNR 36.034, SSIM 0.992, LPIPS 0.0224, and FID 2.431, but at a slower 59.24 FPS, while TalkingGaussian is reported as faster at 70.42 FPS but lower in PSNR at 35.21 (Zhu et al., 3 Oct 2025). The paper states that EGSTalker performs competitively in the cross-driven setting, matching or slightly outperforming state-of-the-art methods on several metrics (Zhu et al., 3 Oct 2025).

Ablations attribute measurable effects to each major component. Removing Stage 1 reduces PSNR from 36.07 to 35.865 and changes LPIPS from 0.0223 to 0.0248, which the paper uses to support the importance of a high-quality static Gaussian layout near the lips and eyes (Zhu et al., 3 Oct 2025). Replacing KAN with an MLP reduces reconstruction quality, and ESAA plus PPE improves lip synchronization. For agent-token ratio, “No ESAA” yields LMD $f_v(x)$0 at 57.4 FPS, while ESAA at 0.5% yields LMD $f_v(x)$1 at 68.5 FPS; 0.16% maximizes FPS at 74.1 with slightly degraded LMD (Zhu et al., 3 Oct 2025).

6. Relation to GSTalker and the deformable 3DGS talking-face line

EGSTalker belongs to a rapid sequence of 3DGS-based talking-face systems. The closely related GSTalker model of 2024 also uses audio-driven Gaussian deformation, person-specific training from 3–5 minutes of video, and a two-stage structure with static Gaussian initialization followed by conditional deformation (Chen et al., 2024). GSTalker differs in several explicit engineering choices: it uses approximately $f_v(x)$2 Gaussians, a multi-resolution hashed tri-plane with $f_v(x)$3 and feature dimension $f_v(x)$4, a tiny deformation MLP rather than KAN, and fixed SH color and opacity during speech, updating only $f_v(x)$5 (Chen et al., 2024).

GSTalker also includes a pose-conditioned torso deformation field and renders the torso under a fixed front-facing view $f_v(x)$6 for stabilization (Chen et al., 2024). Its reported efficiency is about 40 minutes of training on a single RTX 3090 and about 125 FPS at inference, with self-driven metrics PSNR 34.65, LPIPS 0.0151, LMD 2.695, Sync 5.775, and AUE 0.862 on the AD-NeRF benchmark (Chen et al., 2024). In that sense, the later EGSTalker can be situated as a 3DGS talking-head variant that replaces the earlier MLP-centered dynamic mapping with KAN-based mappings and replaces standard full attention with ESAA’s agent-token mechanism (Zhu et al., 3 Oct 2025).

This suggests a technical progression inside the deformable-3DGS family: explicit Gaussian scene structure is retained, while the conditioning pathway becomes more specialized for audio–spatial fusion and low-latency inference.

7. Limitations, scope, and unrelated homonyms

The EGSTalker paper does not enumerate failure modes, and it states that typical hard cases such as fast head motion, occlusions, extreme poses, and noisy audio are not systematically evaluated (Zhu et al., 3 Oct 2025). It also does not specify the audio feature type, triplane resolutions and feature sizes, SH degree, KAN or ESAA hyperparameters, optimizer, learning rates, or explicit temporal regularizers (Zhu et al., 3 Oct 2025). Cross-driven evaluation indicates robustness to unseen audio on a per-identity model, but multi-speaker or cross-identity training is not explored (Zhu et al., 3 Oct 2025).

The earlier GSTalker paper makes additional limitations explicit: the model is person-specific, requires a few minutes of video per identity, may degrade under large out-of-distribution head poses or rapid head motions, and still finds inner mouth and teeth challenging under extreme phonemes or tongue details (Chen et al., 2024). Those caveats remain relevant for the broader class of person-specific Gaussian-splatting talking-face systems.

Finally, EGSTalker should not be conflated with the event-vision usage attached to GS-EVT. In that separate literature, an “EGSTalker” is described as an event-camera tracker that aligns integrated event images to differential log-intensity renderings from a Gaussian Splatting map using a reference pose plus first-order dynamics, explicit Jacobians, and a staggered coarse-to-fine optimization (Liu et al., 2024). The shared term reflects common dependence on Gaussian Splatting rather than a shared application domain.

In the talking-head domain, EGSTalker is therefore best understood as a person-specific, real-time 3DGS talking-head generator whose defining features are static Gaussian initialization, KAN-based Gaussian parameterization, and Efficient Spatial–Audio Attention for sub-quadratic cross-modal fusion (Zhu et al., 3 Oct 2025).