---
title: 'StyGazeTalk: Audio-Driven Gaze-Head Animation'
url: https://www.emergentmind.com/topics/stygazetalk
type: topic
---

# StyGazeTalk: Audio-Driven Gaze-Head Animation

Searching arXiv for StyGazeTalk and closely related talking-head style-control papers.
StyGazeTalk is an audio-driven facial animation framework for generating synchronized **3D gaze and head motion** with **style control**. It is formulated around the claim that gaze and head movement are not independent output channels but **temporally coupled** behaviors whose coordination is essential for realism, attention signaling, and expressive behavior. The method combines a **joint gaze-head temporal generation model**, a **contrastive style encoder** for personalized motion traits, and a new multimodal dataset, **HAGE**. In the paper’s framing, StyGazeTalk addresses limitations of prior talking-head systems that focus primarily on lip motion or head pose, rely on noisy vision-based annotations, or encode style through discrete speaker identities rather than transferable continuous traits [2509.17168].

## 1. Conceptual scope and problem setting

StyGazeTalk is situated within speech-driven facial animation, but its technical target is narrower and more specific than full talking-face synthesis. The model predicts motion in a space where each frame is a 7-dimensional vector containing **3D head rotation**—pitch, yaw, roll—and **binocular gaze**—left and right pitch and yaw. The stated problem is to generate realistic speech-driven motion that jointly models gaze and head dynamics, respects temporal structure, and preserves **speaker-specific motion style** [2509.17168].

The method is motivated by the observation that most existing audio-driven talking-head systems treat facial components in isolation. In the paper’s summary, such systems typically model **lip motion or lower-face expression**, **head pose**, or only one of these under weak supervision. StyGazeTalk instead emphasizes that natural human communication contains structured gaze behavior, including **fixations**, **saccades**, and **head-gaze compensation**, where head movement counterbalances eye motion to keep attention stable. This suggests that gaze is not an auxiliary signal but a central part of communicative motion organization.

A common misconception is to treat StyGazeTalk as a conventional talking-head model with an added gaze regressor. The paper argues against that interpretation. Its central novelty is joint generation: gaze and head motion are learned together because their coordination, rather than either stream alone, is the relevant target behavior. Another misconception is that style control here means a one-hot speaker label. The method explicitly replaces one-hot speaker embeddings with a continuous style representation extracted from motion windows [2509.17168].

## 2. Sequence formulation and generation pipeline

StyGazeTalk is implemented as a **sliding-window sequence generation** model. At each time step, it consumes **audio context**, **past gaze-head motion**, and a **style embedding** extracted from recent motion, then predicts the next segment of gaze-head motion. Using the notation in the paper, audio features are denoted by $\mathbf{A}_{1:T} = (\mathbf{a}_1,\ldots,\mathbf{a}_T)$ and motion by $\mathbf{X}_{1:T} = (\mathbf{x}_1,\ldots,\mathbf{x}_T)$, with $\mathbf{x}_t \in \mathbb{R}^7$ [2509.17168].

For each window starting at time $t$, the model uses an audio window $\mathbf{A}_{t:t+M-1}$, a past-motion window $\mathbf{X}_{t-1:t+M-1}$, and a style code $\mathbf{s}_{t-1}$, and predicts the next motion segment $\hat{\mathbf{X}}_{t:t+N-1}$. The audio frontend is **Wav2Vec 2.0**, which produces contextualized features later projected into a latent space. The past motion window is projected into the same latent dimension, and the style code is extracted by a style encoder from the preceding motion segment. These representations are concatenated into a fused latent sequence $\mathbf{Z}_t \in \mathbb{R}^{M \times (2d + d_s)}$, which is then passed to the motion generator [2509.17168].

A notable architectural decision is the use of a **multi-layer LSTM** rather than a Transformer. The paper attributes this choice to three conditions: the dataset is not huge, sequence length is moderate, and the task benefits from a strong temporal inductive bias. The generator is written as
$$
\hat{\mathbf{X}}_{t:t+N-1} = \text{MG}(\mathbf{Z}_t; \mathbf{h}_0, \mathbf{c}_0),
$$
where $(\mathbf{h}_0,\mathbf{c}_0)$ are **learnable initial hidden and cell states**. These initial states act as a trainable prior intended to improve **temporal continuity across windows**, **long-range motion trends**, and **head-gaze coordination** [2509.17168].

The paper distinguishes between **shared patterns** and **sample-specific patterns**. Shared patterns are described as universal behaviors such as fixation/saccade structure, head-gaze coupling, and conversational motion regularities. Sample-specific patterns include motion amplitude, velocity profile, and rhythmic tendencies for a particular speaker or session. In the proposed division of labor, the LSTM with learnable initial states models the former, while the style encoder models the latter.

## 3. Style representation and training objectives

A major component of StyGazeTalk is its **contrastive style encoder**, which extracts a continuous style representation from recent gaze-head motion. Given a motion window $\mathbf{X} \in \mathbb{R}^{M \times 7}$, the encoder first applies a linear projection and a temporal encoding module, producing
$$
\mathbf{X}' = \text{MTE}(\text{Linear}(\mathbf{X})) \in \mathbb{R}^{M \times d}.
$$
The encoded sequence is processed by a Transformer encoder, and average pooling yields a fixed-length vector
$$
\mathbf{s} = \frac{1}{M}\sum_{i=1}^{M}\mathcal{T}_\theta(\mathbf{X}')_i \in \mathbb{R}^{d_s}.
$$
The paper emphasizes that this embedding is intended to describe **how** a window moves rather than **what content** it expresses [2509.17168].

The encoder is trained with **NT-Xent contrastive loss**. Positive pairs are adjacent windows from the same speaker or session, while negative pairs are drawn from different speakers or temporally distant windows. Cosine similarity is defined as
$$
\text{sim}(\mathbf{s}_i,\mathbf{s}_j)=\frac{\mathbf{s}_i^\top\mathbf{s}_j}{\|\mathbf{s}_i\|\,\|\mathbf{s}_j\|},
$$
and the contrastive objective pulls nearby windows of the same speaker together while separating different styles. The paper presents this as the mechanism that supports **generalization**, **style transfer**, and encoding of subtle motion traits such as **gaze intensity**, **temporal rhythm**, and **head movement amplitude** [2509.17168].

Motion prediction is trained with a combined generation objective consisting of a frame-wise mean squared error term and a velocity-consistency term:
$$
\mathcal{L}_{\text{mse}} = \frac{1}{T}\sum_{t=1}^{T}\left\|\hat{\mathbf{x}}_t - \mathbf{x}_t\right\|_2^2,
$$
$$
\mathcal{L}_{\text{vel}} = \frac{1}{T-1}\sum_{t=2}^{T}\left\|(\hat{\mathbf{x}}_t-\hat{\mathbf{x}}_{t-1}) - (\mathbf{x}_t-\mathbf{x}_{t-1})\right\|_2^2,
$$
and
$$
\mathcal{L}_{\text{gen}} = \lambda \cdot \mathcal{L}_{\text{mse}} + (1-\lambda)\cdot \mathcal{L}_{\text{vel}}.
$$
This design reflects the paper’s concern that coordinate regression alone is insufficient for realistic dynamics. A plausible implication is that the velocity term is intended to discourage over-smoothed trajectories, a concern reinforced by the reported ablation results [2509.17168].

## 4. HAGE dataset and data quality assumptions

StyGazeTalk introduces **HAGE**, expanded as **Head Pose–Audio–Gaze–Expression**. The dataset is described as the **first high-precision face-to-face talking dataset** with professional eye-tracker-based gaze capture. Its stated purpose is to provide reliable supervision for fine-grained gaze-head coordination, which the paper argues is difficult to learn from vision-estimated signals because gaze dynamics are subtle and easily corrupted by low resolution, occlusion, drift, or estimation noise [2509.17168].

HAGE contains approximately **2.5 hours** of data from **8 subjects**, with **unscripted Chinese and English conversations**, **16 kHz audio**, **1080×1080 video**, and head pose and gaze aligned to **25 FPS**. Data were collected in a controlled indoor environment using a **Pupil Labs eye tracker**, microphone, camera, and a custom synchronization program built with the **Pupil Labs API** and **OpenCV**. Postprocessing included manual extraction of valid segments, alignment of gaze and head streams to 25 FPS, conversion to angular form, and removal of extreme-angle frames at $\pm 40^\circ$ [2509.17168].

The dataset’s significance in the paper is not merely scale but measurement quality. HAGE is explicitly contrasted with datasets based on vision-estimated head pose or gaze. The paper claims that direct sensor-based gaze measurements provide **lower noise and drift**, **less entanglement from occlusions or low resolution**, and **better ground truth for learning fine-grained gaze-head coordination**. This claim is consistent with one of the ablations, **MPGW**, which replaces precise gaze information with vision-estimated gaze input and is reported to degrade performance substantially [2509.17168].

At the same time, HAGE is not presented as exhaustive. The dataset is relatively small—about 2.5 hours and 8 subjects—and the paper identifies **cross-domain generalization** as an open problem. This constrains the claims that can be made about universal deployment beyond the specific recording conditions used in the study.

## 5. Evaluation methodology and empirical results

The evaluation uses both conventional motion-error measures and gaze-specific pattern metrics. Standard metrics include **MAE** (Mean Absolute Error), **Vel** (velocity error), **MEE** (motion energy error), **CE** (cosine error for style similarity), and **BAS** (Beat Alignment Score). The paper argues that these are insufficient for assessing whether gaze behavior appears human-like, so it introduces additional pattern-aware measures: **Fixation Ratio**, **Compensation Score**, and **SimScore** [2509.17168].

Fixation detection is based on an I-DT criterion with
$$
D = \max(x)-\min(x)+\max(y)-\min(y),
$$
where a window is treated as a fixation if $D \leq 3.5$ and lasts at least 3 frames. The fixation ratio is then
$$
\text{FixRatio} = \frac{N_{\text{fixation}}}{N_{\text{fixation}} + N_{\text{saccade}}}.
$$
The **Compensation Score** is defined from head velocity, eye velocity, and their combination, and is intended to quantify eye-head coordination. **SimScore** compares predicted and ground-truth fixation and compensation statistics through a composite similarity measure. These metrics formalize the paper’s claim that realistic gaze should be assessed as a dynamical pattern, not only as coordinate error [2509.17168].

The reported ablations compare **Base**, **SE-32**, **SE-64**, **SE-64-VEL**, **TFM-SE-64**, and **MPGW**. The main quantitative trends are directional rather than exhaustive. **SE-64** is described as giving the best overall balance on the full task and the **lowest MAE** among the ablations, while **SE-64-VEL** yields the best motion-beat alignment and gaze-pattern realism. The Transformer-based generator, **TFM-SE-64**, performs worse, especially for gaze, and **MPGW** shows that noisy vision-based gaze supervision significantly harms results [2509.17168].

Several concrete numbers anchor these trends. The paper reports a ground-truth fixation ratio of **61.13%**. The **Base** model yields **74.17%**, which the paper characterizes as too static; **SE-64** yields **69.22%**; **SE-64-VEL** yields **67.33%**; and **TFM-SE-64** yields **84.57%**, described as overly smoothed and unrealistic. **SimScore** improves from **0.8220** in **Base** to **0.9114** in **SE-64-VEL**. For style similarity, cosine error changes from **0.139** in **Base** to **0.138** in **SE-32**, **0.136** in **SE-64**, and **0.876** in **MPGW**, supporting the claim that precise supervision and continuous style encoding preserve speaker-specific traits [2509.17168].

The perceptual study involved **88 participants**, with **78 valid after consistency filtering**, each evaluating **12 videos** along **Naturalness**, **Realism**, and **Style Similarity**. **SE-64 (ours)** received the highest preference in all three dimensions: **48.3%** for Naturalness, **48.5%** for Realism, and **47.9%** for Style Similarity. The qualitative results reported in the paper align with these numbers: joint gaze-head motion appears more natural than head-only baselines, the style encoder produces more vivid and individualized behavior, velocity loss improves dynamics, Transformer-based generation is over-smoothed, and vision-based gaze initialization leads to unstable motion [2509.17168].

## 6. Relation to adjacent work, limitations, and interpretive context

StyGazeTalk belongs to a broader family of controllable talking-head systems, but its output space and control mechanism differ from nearby work. **CapTalk** is a **text-guided stylized talking-head / 3D head animation system** that takes **style text + emotion text + audio** and generates **FLAME**-based 3D head motions with separate control over style and emotion. Its emphasis is text-guided control and dynamic emotion updates across windows, rather than joint gaze-head generation from recent motion style [2605.29316]. **StyleTalk++** is a one-shot, style-controllable talking-head framework that transfers **facial expression style** and **head-pose style** from reference videos to a target identity through a 3DMM-based formulation. It uses style reference clips and style-aware decoders, whereas StyGazeTalk uses a continuous style encoder learned from gaze-head sequences and focuses on speech-driven head-gaze dynamics [2409.09292].

These comparisons help clarify what StyGazeTalk is and is not. It is not primarily a text-controlled system, unlike CapTalk, and it is not primarily a one-shot reference-video style-transfer framework, unlike StyleTalk++. It is also not a full facial synthesis pipeline in the narrow sense. The paper explicitly notes that the current model focuses on **dynamic gaze/head motion**, while full facial synthesis still requires external components such as **EMOCA** for shape, jaw, and expression visualization [2509.17168].

The limitations stated or implied in the paper are consequential. The dataset is relatively small, which constrains claims about broad generalization. Training is conducted in a single high-quality but limited domain, leaving **cross-domain adaptation** open. The model’s focus on dynamic gaze-head motion means that a full talking-face system must add other components. The authors identify future directions including **hybrid training**, **semantic integration**, and **cross-domain adaptation**. Another important finding is methodological rather than merely empirical: the Transformer ablation performs worse than the LSTM design, suggesting that higher-capacity sequence models are not automatically superior under small-data conditions and that task-specific temporal bias remains important [2509.17168].

In that sense, StyGazeTalk occupies a distinct position in the talking-head literature. It reframes gaze as a first-class structured motion variable, treats head and eye behavior as a coupled temporal process, and evaluates output with metrics intended to capture not only accuracy but also **pattern realism**. This suggests a broader shift from facial animation systems centered on lip synchronization alone toward models that represent speaking behavior as a coordinated multimodal dynamical system [2509.17168].

Source: https://www.emergentmind.com/topics/stygazetalk