StyGazeTalk: Audio-Driven Gaze-Head Animation

• The paper presents a novel audio-driven framework that jointly models 3D gaze and head motion with a contrastive style encoder to capture personalized motion traits.
• It employs a sliding-window LSTM-based architecture that fuses audio context, past motion, and continuous style embeddings to predict realistic temporal dynamics.
• Empirical evaluations using the high-precision HAGE dataset and custom metrics demonstrate improved naturalness, style transfer, and head-gaze coordination over prior isolated approaches.

Searching arXiv for StyGazeTalk and closely related talking-head style-control papers. StyGazeTalk is an audio-driven facial animation framework for generating synchronized 3D gaze and head motion with style control. It is formulated around the claim that gaze and head movement are not independent output channels but temporally coupled behaviors whose coordination is essential for realism, attention signaling, and expressive behavior. The method combines a joint gaze-head temporal generation model, a contrastive style encoder for personalized motion traits, and a new multimodal dataset, HAGE. In the paper’s framing, StyGazeTalk addresses limitations of prior talking-head systems that focus primarily on lip motion or head pose, rely on noisy vision-based annotations, or encode style through discrete speaker identities rather than transferable continuous traits (Shi et al., 21 Sep 2025).

1. Conceptual scope and problem setting

StyGazeTalk is situated within speech-driven facial animation, but its technical target is narrower and more specific than full talking-face synthesis. The model predicts motion in a space where each frame is a 7-dimensional vector containing 3D head rotation—pitch, yaw, roll—and binocular gaze—left and right pitch and yaw. The stated problem is to generate realistic speech-driven motion that jointly models gaze and head dynamics, respects temporal structure, and preserves speaker-specific motion style (Shi et al., 21 Sep 2025).

The method is motivated by the observation that most existing audio-driven talking-head systems treat facial components in isolation. In the paper’s summary, such systems typically model lip motion or lower-face expression, head pose, or only one of these under weak supervision. StyGazeTalk instead emphasizes that natural human communication contains structured gaze behavior, including fixations, saccades, and head-gaze compensation, where head movement counterbalances eye motion to keep attention stable. This suggests that gaze is not an auxiliary signal but a central part of communicative motion organization.

A common misconception is to treat StyGazeTalk as a conventional talking-head model with an added gaze regressor. The paper argues against that interpretation. Its central novelty is joint generation: gaze and head motion are learned together because their coordination, rather than either stream alone, is the relevant target behavior. Another misconception is that style control here means a one-hot speaker label. The method explicitly replaces one-hot speaker embeddings with a continuous style representation extracted from motion windows (Shi et al., 21 Sep 2025).

2. Sequence formulation and generation pipeline

StyGazeTalk is implemented as a sliding-window sequence generation model. At each time step, it consumes audio context, past gaze-head motion, and a style embedding extracted from recent motion, then predicts the next segment of gaze-head motion. Using the notation in the paper, audio features are denoted by $$\mathbf{A}_{1:T} = (\mathbf{a}_1,\ldots,\mathbf{a}_T)$$ and motion by $$\mathbf{X}_{1:T} = (\mathbf{x}_1,\ldots,\mathbf{x}_T)$$, with $\mathbf{x}_t \in \mathbb{R}^7$ (Shi et al., 21 Sep 2025).

For each window starting at time $t$, the model uses an audio window $\mathbf{A}_{t:t+M-1}$, a past-motion window $\mathbf{X}_{t-1:t+M-1}$, and a style code $\mathbf{s}_{t-1}$, and predicts the next motion segment $\hat{\mathbf{X}}_{t:t+N-1}$. The audio frontend is Wav2Vec 2.0, which produces contextualized features later projected into a latent space. The past motion window is projected into the same latent dimension, and the style code is extracted by a style encoder from the preceding motion segment. These representations are concatenated into a fused latent sequence $\mathbf{Z}_t \in \mathbb{R}^{M \times (2d + d_s)}$, which is then passed to the motion generator (Shi et al., 21 Sep 2025).

A notable architectural decision is the use of a multi-layer LSTM rather than a Transformer. The paper attributes this choice to three conditions: the dataset is not huge, sequence length is moderate, and the task benefits from a strong temporal inductive bias. The generator is written as

$$\hat{\mathbf{X}}_{t:t+N-1} = \text{MG}(\mathbf{Z}_t; \mathbf{h}_0, \mathbf{c}_0),$$

where $$\mathbf{X}_{1:T} = (\mathbf{x}_1,\ldots,\mathbf{x}_T)$$0 are learnable initial hidden and cell states. These initial states act as a trainable prior intended to improve temporal continuity across windows, long-range motion trends, and head-gaze coordination (Shi et al., 21 Sep 2025).

The paper distinguishes between shared patterns and sample-specific patterns. Shared patterns are described as universal behaviors such as fixation/saccade structure, head-gaze coupling, and conversational motion regularities. Sample-specific patterns include motion amplitude, velocity profile, and rhythmic tendencies for a particular speaker or session. In the proposed division of labor, the LSTM with learnable initial states models the former, while the style encoder models the latter.

3. Style representation and training objectives

A major component of StyGazeTalk is its contrastive style encoder, which extracts a continuous style representation from recent gaze-head motion. Given a motion window $$\mathbf{X}_{1:T} = (\mathbf{x}_1,\ldots,\mathbf{x}_T)$$1, the encoder first applies a linear projection and a temporal encoding module, producing

$$\mathbf{X}_{1:T} = (\mathbf{x}_1,\ldots,\mathbf{x}_T)$$2

The encoded sequence is processed by a Transformer encoder, and average pooling yields a fixed-length vector

$$\mathbf{X}_{1:T} = (\mathbf{x}_1,\ldots,\mathbf{x}_T)$$3

The paper emphasizes that this embedding is intended to describe how a window moves rather than what content it expresses (Shi et al., 21 Sep 2025).

The encoder is trained with NT-Xent contrastive loss. Positive pairs are adjacent windows from the same speaker or session, while negative pairs are drawn from different speakers or temporally distant windows. Cosine similarity is defined as

$$\mathbf{X}_{1:T} = (\mathbf{x}_1,\ldots,\mathbf{x}_T)$$4

and the contrastive objective pulls nearby windows of the same speaker together while separating different styles. The paper presents this as the mechanism that supports generalization, style transfer, and encoding of subtle motion traits such as gaze intensity, temporal rhythm, and head movement amplitude (Shi et al., 21 Sep 2025).

Motion prediction is trained with a combined generation objective consisting of a frame-wise mean squared error term and a velocity-consistency term:

$$\mathbf{X}_{1:T} = (\mathbf{x}_1,\ldots,\mathbf{x}_T)$$5

$$\mathbf{X}_{1:T} = (\mathbf{x}_1,\ldots,\mathbf{x}_T)$$6

and

$$\mathbf{X}_{1:T} = (\mathbf{x}_1,\ldots,\mathbf{x}_T)$$7

This design reflects the paper’s concern that coordinate regression alone is insufficient for realistic dynamics. A plausible implication is that the velocity term is intended to discourage over-smoothed trajectories, a concern reinforced by the reported ablation results (Shi et al., 21 Sep 2025).

4. HAGE dataset and data quality assumptions

StyGazeTalk introduces HAGE, expanded as Head Pose–Audio–Gaze–Expression. The dataset is described as the first high-precision face-to-face talking dataset with professional eye-tracker-based gaze capture. Its stated purpose is to provide reliable supervision for fine-grained gaze-head coordination, which the paper argues is difficult to learn from vision-estimated signals because gaze dynamics are subtle and easily corrupted by low resolution, occlusion, drift, or estimation noise (Shi et al., 21 Sep 2025).

HAGE contains approximately 2.5 hours of data from 8 subjects, with unscripted Chinese and English conversations, 16 kHz audio, 1080×1080 video, and head pose and gaze aligned to 25 FPS. Data were collected in a controlled indoor environment using a Pupil Labs eye tracker, microphone, camera, and a custom synchronization program built with the Pupil Labs API and OpenCV. Postprocessing included manual extraction of valid segments, alignment of gaze and head streams to 25 FPS, conversion to angular form, and removal of extreme-angle frames at $$\mathbf{X}_{1:T} = (\mathbf{x}_1,\ldots,\mathbf{x}_T)$$8 (Shi et al., 21 Sep 2025).

The dataset’s significance in the paper is not merely scale but measurement quality. HAGE is explicitly contrasted with datasets based on vision-estimated head pose or gaze. The paper claims that direct sensor-based gaze measurements provide lower noise and drift, less entanglement from occlusions or low resolution, and better ground truth for learning fine-grained gaze-head coordination. This claim is consistent with one of the ablations, MPGW, which replaces precise gaze information with vision-estimated gaze input and is reported to degrade performance substantially (Shi et al., 21 Sep 2025).

At the same time, HAGE is not presented as exhaustive. The dataset is relatively small—about 2.5 hours and 8 subjects—and the paper identifies cross-domain generalization as an open problem. This constrains the claims that can be made about universal deployment beyond the specific recording conditions used in the study.

5. Evaluation methodology and empirical results

The evaluation uses both conventional motion-error measures and gaze-specific pattern metrics. Standard metrics include MAE (Mean Absolute Error), Vel (velocity error), MEE (motion energy error), CE (cosine error for style similarity), and BAS (Beat Alignment Score). The paper argues that these are insufficient for assessing whether gaze behavior appears human-like, so it introduces additional pattern-aware measures: Fixation Ratio, Compensation Score, and SimScore (Shi et al., 21 Sep 2025).

Fixation detection is based on an I-DT criterion with

$$\mathbf{X}_{1:T} = (\mathbf{x}_1,\ldots,\mathbf{x}_T)$$9

where a window is treated as a fixation if $\mathbf{x}_t \in \mathbb{R}^7$0 and lasts at least 3 frames. The fixation ratio is then

$\mathbf{x}_t \in \mathbb{R}^7$1

The Compensation Score is defined from head velocity, eye velocity, and their combination, and is intended to quantify eye-head coordination. SimScore compares predicted and ground-truth fixation and compensation statistics through a composite similarity measure. These metrics formalize the paper’s claim that realistic gaze should be assessed as a dynamical pattern, not only as coordinate error (Shi et al., 21 Sep 2025).

The reported ablations compare Base, SE-32, SE-64, SE-64-VEL, TFM-SE-64, and MPGW. The main quantitative trends are directional rather than exhaustive. SE-64 is described as giving the best overall balance on the full task and the lowest MAE among the ablations, while SE-64-VEL yields the best motion-beat alignment and gaze-pattern realism. The Transformer-based generator, TFM-SE-64, performs worse, especially for gaze, and MPGW shows that noisy vision-based gaze supervision significantly harms results (Shi et al., 21 Sep 2025).

Several concrete numbers anchor these trends. The paper reports a ground-truth fixation ratio of 61.13%. The Base model yields 74.17%, which the paper characterizes as too static; SE-64 yields 69.22%; SE-64-VEL yields 67.33%; and TFM-SE-64 yields 84.57%, described as overly smoothed and unrealistic. SimScore improves from 0.8220 in Base to 0.9114 in SE-64-VEL. For style similarity, cosine error changes from 0.139 in Base to 0.138 in SE-32, 0.136 in SE-64, and 0.876 in MPGW, supporting the claim that precise supervision and continuous style encoding preserve speaker-specific traits (Shi et al., 21 Sep 2025).

The perceptual study involved 88 participants, with 78 valid after consistency filtering, each evaluating 12 videos along Naturalness, Realism, and Style Similarity. SE-64 (ours) received the highest preference in all three dimensions: 48.3% for Naturalness, 48.5% for Realism, and 47.9% for Style Similarity. The qualitative results reported in the paper align with these numbers: joint gaze-head motion appears more natural than head-only baselines, the style encoder produces more vivid and individualized behavior, velocity loss improves dynamics, Transformer-based generation is over-smoothed, and vision-based gaze initialization leads to unstable motion (Shi et al., 21 Sep 2025).

6. Relation to adjacent work, limitations, and interpretive context

StyGazeTalk belongs to a broader family of controllable talking-head systems, but its output space and control mechanism differ from nearby work. CapTalk is a text-guided stylized talking-head / 3D head animation system that takes style text + emotion text + audio and generates FLAME-based 3D head motions with separate control over style and emotion. Its emphasis is text-guided control and dynamic emotion updates across windows, rather than joint gaze-head generation from recent motion style (Chu et al., 28 May 2026). StyleTalk++ is a one-shot, style-controllable talking-head framework that transfers facial expression style and head-pose style from reference videos to a target identity through a 3DMM-based formulation. It uses style reference clips and style-aware decoders, whereas StyGazeTalk uses a continuous style encoder learned from gaze-head sequences and focuses on speech-driven head-gaze dynamics (Wang et al., 2024).

These comparisons help clarify what StyGazeTalk is and is not. It is not primarily a text-controlled system, unlike CapTalk, and it is not primarily a one-shot reference-video style-transfer framework, unlike StyleTalk++. It is also not a full facial synthesis pipeline in the narrow sense. The paper explicitly notes that the current model focuses on dynamic gaze/head motion, while full facial synthesis still requires external components such as EMOCA for shape, jaw, and expression visualization (Shi et al., 21 Sep 2025).

The limitations stated or implied in the paper are consequential. The dataset is relatively small, which constrains claims about broad generalization. Training is conducted in a single high-quality but limited domain, leaving cross-domain adaptation open. The model’s focus on dynamic gaze-head motion means that a full talking-face system must add other components. The authors identify future directions including hybrid training, semantic integration, and cross-domain adaptation. Another important finding is methodological rather than merely empirical: the Transformer ablation performs worse than the LSTM design, suggesting that higher-capacity sequence models are not automatically superior under small-data conditions and that task-specific temporal bias remains important (Shi et al., 21 Sep 2025).

In that sense, StyGazeTalk occupies a distinct position in the talking-head literature. It reframes gaze as a first-class structured motion variable, treats head and eye behavior as a coupled temporal process, and evaluates output with metrics intended to capture not only accuracy but also pattern realism. This suggests a broader shift from facial animation systems centered on lip synchronization alone toward models that represent speaking behavior as a coordinated multimodal dynamical system (Shi et al., 21 Sep 2025).