GLARE: Generating Listening Heads with Appropriate Reactions

Abstract: While talking head generation has advanced rapidly, generating natural listener behavior in dyadic conversations, which know when to react, how to react, and with what type of response, remains underexplored. Existing dyadic datasets lack fine-grained listener reaction annotations, and prevailing evaluation metrics inherited from talking-head and video generation measure visual realism rather than whether a listener reacted appropriately. We address these gaps along three aspects. First, we curate a listening-head-specific dataset built from RealTalk and Seamless Interaction, comprising approximately 147 hours of paired speaker-listener videos with 64,557 event-level reaction annotations across six categories: nodding, head shaking, smiling, laughing, frowning, and surprised. Second, we introduce an audio-driven baseline built on a flow-matching transformer, namely GLARE, with prosody conditioning derived from Qwen2-Audio and a temporal reaction loss that explicitly supervises frame-wise reactions. Third, we propose a reaction-oriented evaluation protocol that jointly measures reaction occurrence (R-F1), temporal alignment (R-tIoU), asymmetric temporal deviation (R-ATD), and reaction-region visual quality (R-FID), giving a more behaviorally grounded assessment than visual-quality-only metrics. Experiment results show consistent gains over prior listening-head methods in both visual fidelity and reaction-level metrics, suggesting that reaction-aware data, modeling, and evaluation are critical for natural listening behavior.

• The paper develops the GLARE framework, which creates realistic listener reactions using enhanced data annotation, improved listening-head generator, and novel evaluation metrics.
• By applying reaction-centric metrics (R-F1, R-tIoU, R-ATD, and R-F1), the method achieves superior performance in generating accurate, timely, and engaged listener responses in dyadic conversations.
• Experimenting with the curated dataset from RealTalk and Seamless Interaction, GLARE demonstrates stronger listener dynamics on multiple reaction categories, highlighting the success of using reaction-specific losses and prosody conditioning.

Problem formulation and contribution

“GLARE: Generating Listening Heads with Appropriate Reactions” (2609.40317) addresses a specific deficiency in audio-driven facial animation: existing talking-head systems generally model the active speaker, whereas natural dyadic interaction also depends on the listener’s nonverbal feedback. A listener may nod, shake the head, smile, laugh, frown, or display surprise, but the communicative value of these behaviors depends jointly on reaction type, onset, duration, and temporal relation to the speaker’s behavior. A visually realistic listener can therefore remain behaviorally implausible if it reacts too late, reacts prematurely, produces the wrong reaction, or remains static.

The paper makes three interdependent contributions. First, it constructs a listening-head dataset from RealTalk and Seamless Interaction, yielding approximately 147 hours of paired speaker-audio/listener-video data, 107,149 aligned pairs, and 64,557 annotated reaction instances. Second, it introduces GLARE, a flow-matching listener generator augmented with prosody conditioning and an auxiliary frame-wise reaction objective. Third, it proposes reaction-centric evaluation metrics—R-F1, R-tIoU, R-ATD, and R-FID—that complement conventional image- and video-generation measures with explicit assessments of reaction occurrence, temporal localization, asymmetric timing error, and reaction-region quality.

The central claim is consequently broader than an architectural modification: reaction-aware data, supervision, and evaluation are necessary to distinguish a listener that merely looks realistic from one that behaves in a conversationally appropriate manner. The experiments support this claim, although the paper’s evaluation remains fundamentally reference-based and therefore does not establish that a generated response is socially appropriate in an absolute sense.

Dataset construction and reaction annotation

The data curation pipeline converts general-purpose dyadic conversation datasets into aligned listening-head examples. Raw videos are filtered for scene transitions, black frames, occlusions, unstable face crops, multiple visible participants, and other conditions that compromise facial-motion extraction. Each participant is cropped into a portrait-centered video. Speech activity is then estimated using source separation, denoising, and diarization for RealTalk, while Seamless Interaction provides cleaner streams and diarization information that is corrected when necessary. During a speaker-dominant interval, the active participant’s audio is paired with the temporally synchronized video of the other participant, who is treated as the listener.

Figure 1: GLARE’s data curation pipeline from dyadic audiovisual data to aligned speaker–listener clips and frame-wise reaction annotations.

The reaction taxonomy contains six classes: nodding, head shaking, smiling, laughing, frowning, and surprised. These categories are selected because they are visually recognizable, temporally localizable, and plausibly connected to conversational feedback. The distribution is uneven but not extremely concentrated: smiling is most frequent with 15,767 instances, followed by head shaking with 12,790, nodding with 10,230, frowning with 9,986, laughing with 8,320, and surprised with 7,464. This imbalance has direct methodological significance. The relatively sparse surprised class is also reported as the most difficult category, while smiling benefits from the largest number of examples.

Annotations are generated by a detector combining landmark dynamics, head-motion trajectories, facial expression cues, temporal smoothing, and class-specific post-processing. Nodding and head shaking are inferred primarily from normalized vertical and horizontal head trajectories, respectively. Smiling, laughing, frowning, and surprise rely on expression confidence, action-unit-like cues, mouth and eyebrow dynamics, and temporal persistence. Candidate intervals are thresholded, merged, and resolved into non-overlapping single-label events. Manual verification removes detections caused by tracking failure, occlusion, unstable crops, ambiguous expressions, or non-listening behavior.

This design provides both sparse event intervals for evaluation and dense frame-wise scores for training. However, the annotation procedure also imposes a strong assumption: overlapping or ambiguous behaviors are collapsed into one dominant class or discarded. Consequently, the dataset does not represent concurrent reactions—for example, a smile accompanying a nod—or uncertainty over reaction labels. The resulting supervision is operationally convenient but narrower than the underlying behavior space.

GLARE architecture

GLARE builds on a latent flow-matching transformer. The model receives a listener reference image, previous listener motion context, and speaker audio, and predicts a trajectory of listener motion latents that are subsequently decoded into video frames. A frozen LIA image autoencoder provides listener motion and identity latents; Wav2Vec 2.0 encodes the speaker audio; and a speech-emotion encoder supplies an additional emotional representation. The transformer predicts conditional vector fields, which are integrated with an Euler ODE solver using ten function evaluations.

The architecture addresses the weak coupling between speaker audio and listener motion by introducing two explicit mechanisms: a prosody-conditioning branch and a temporal reaction head.

Figure 2: GLARE combines latent flow matching, speaker prosody, listener context, and frame-wise reaction supervision to generate listener motion.

Prosody conditioning uses Qwen2-Audio-7B-Instruct (Chu et al., 2024) to identify localized prosodic fluctuations such as emphasis, pitch or loudness changes, altered speaking rate, hesitation, and affective variation. The resulting timestamped intervals are converted into a frame-aligned scalar intensity sequence and projected into a four-dimensional latent representation. This signal is concatenated with listener motion, acoustic, and emotional features. The design is deliberately low-dimensional: the model does not attempt to encode a full semantic interpretation of the utterance, but instead injects a temporally localized cue intended to identify moments at which listener feedback may be more likely.

The second mechanism is an auxiliary reaction predictor attached to the transformer’s hidden states. Each current-frame hidden token is projected into six class-specific 32-dimensional subspaces, and each subspace produces a sigmoid reaction score. Smooth-L1 loss compares these scores with the dense reaction annotations. The complete objective combines flow matching, velocity consistency, and reaction losses:

$$\mathcal{L} = \mathcal{L}_{\mathrm{fm}} + \lambda_{\mathrm{vel}}\mathcal{L}_{\mathrm{vel}} + \lambda_{\mathrm{react}}\mathcal{L}_{\mathrm{react}}.$$

The auxiliary head is not merely an evaluation component. Because it is attached to the internal motion representation during training, it pressures the latent trajectory to retain information about reaction type and timing. The paper’s ablations indicate that this explicit supervision is more consequential than prosody conditioning alone.

Reaction-centric evaluation

The proposed metrics formalize reaction generation as class-aware temporal event prediction. A generated video is processed by the same reaction detector used during annotation. A predicted event matches a ground-truth event only when the reaction classes agree and temporal IoU exceeds 0.5. Matching is one-to-one, preventing one generated event from explaining several reference events.

R-F1 measures event occurrence and reaction-type consistency. R-tIoU averages temporal overlap over matched events and therefore evaluates onset and offset localization. R-FID computes Fréchet distance only over frames inside reaction intervals, avoiding domination by neutral listening frames. The most distinctive metric is R-ATD, which penalizes temporal deviations asymmetrically. With the default parameters $\alpha=2$ and $\beta=0.5$, negative deviations—interpreted as premature, truncated, or too-short reactions—receive four times the penalty assigned to positive deviations of the same magnitude.

This asymmetry encodes a substantive behavioral preference rather than a neutral measurement principle. The paper argues that premature listener feedback is more disruptive than delayed or extended feedback. That assumption is plausible for some conversational contexts, but it is not universally valid: rapid backchannels can be appropriate, and delayed reactions can also be socially awkward. The metric therefore evaluates adherence to the authors’ temporal preference, not an uncontested ground truth about interaction quality.

The single-reference construction is an additional limitation. A generated listener can produce a socially appropriate smile when the recorded participant nodded, yet receive no credit for reaction-type mismatch. The authors acknowledge that the metrics measure consistency with the observed reference rather than the full set of acceptable reactions. This caveat is important when interpreting gains in R-F1 and R-tIoU.

Quantitative results

GLARE is evaluated against L2L, DIM, ViCo, ListenFormer, and DyStream, with all methods trained and tested using the curated data. On RealTalk, GLARE achieves the strongest result on nearly every reported metric. Its FID is 35.697, FVD is 142.454, LPIPS is 0.454, DI-Sync is 0.245, R-F1 is 0.594, R-tIoU is 0.704, R-ATD is 57.388, and R-FID is 15.192. Relative to DyStream, the strongest competing method on several conventional measures, GLARE improves FVD from 147.537 to 142.454, R-tIoU from 0.694 to 0.704, and R-FID from 15.337 to 15.192, while also attaining the highest R-F1.

On Seamless, which the paper characterizes as more diverse and difficult, the absolute performance decreases, but GLARE remains competitive. It obtains FID 27.288, FVD 193.281, LPIPS 0.371, R-F1 0.460, R-tIoU 0.582, R-ATD 102.113, and R-FID 28.857. It achieves the best FID, FVD, LPIPS, and R-FID among the listed methods, while DyStream has slightly higher motion diversity and lower R-ATD. Thus, the claim of consistent superiority should be interpreted carefully: GLARE provides the best overall balance, but it does not dominate every metric on every dataset.

Dataset	Method	R-F1	R-tIoU	R-ATD	R-FID
RealTalk	DyStream	0.535	0.694	63.171	15.337
RealTalk	GLARE	0.594	0.704	57.388	15.192
Seamless	DyStream	0.414	0.553	110.808	30.774
Seamless	GLARE	0.460	0.582	102.113	28.857

The per-class results expose the structure of the improvement. On RealTalk, GLARE improves R-F1 for all six categories, with the largest absolute values for laughing (0.656), smiling (0.633), and nodding (0.602). It also improves R-tIoU for smiling and laughing, while surprise remains problematic: its R-tIoU falls from 0.679 for DyStream to 0.658, and its R-FID rises from 15.397 to 16.007. On Seamless, GLARE improves R-F1, R-tIoU, and R-FID in every category, but all reaction metrics are substantially weaker than on RealTalk. Surprise remains the lowest-R-F1 category at 0.409, indicating that ambiguous, sparse facial events remain difficult despite the additional supervision.

Figure 3: Qualitative RealTalk comparisons showing GLARE’s stronger listener dynamics and closer temporal correspondence to annotated reactions.

The qualitative comparisons support the numerical results by showing that competing methods often produce static listeners or reactions with incorrect timing. GLARE more frequently generates visible nods, smiles, and other responses near the annotated intervals. Qualitative evidence is necessarily selective, however, and does not resolve the single-reference issue or establish robustness across unseen conversational styles.

Figure 4: Qualitative Seamless comparisons illustrating improved reaction expressiveness and temporal alignment under more diverse conversational conditions.

Human evaluation and ablations

The human study evaluates 100 RealTalk-generated videos containing 165 detected reaction events. Twenty-three participants provide 3,795 event-level judgments across naturalness, contextual appropriateness, and timing plausibility. The aggregate Human Agreement Rates are 0.899 for naturalness, 0.894 for contextual appropriateness, and 0.935 for timing plausibility. Timing receives the strongest score, consistent with the model’s emphasis on temporal reaction supervision. Surprise has the lowest naturalness score at 0.824, while laughter has the lowest contextual-appropriateness score at 0.836. These category-specific weaknesses align with the automated results and indicate that expressive reactions require more than prosodic triggering.

The study nevertheless evaluates only GLARE’s detected events, rather than comparing all systems under identical randomized conditions. It also uses a relatively small sample of clips and participants, and the binary criteria do not measure inter-rater reliability, graded appropriateness, or whether a reaction should have occurred but was absent. The reported rates are therefore supportive but not a complete perceptual validation.

The ablation results clarify the relative roles of the proposed components. Without either module, RealTalk performance is weak: R-F1 is 0.404, R-tIoU is 0.599, and R-ATD is 89.984. Adding prosody conditioning increases R-F1 to 0.481 and improves visual metrics, but worsens temporal deviation to 100.342. Adding the reaction loss without prosody produces a much larger improvement, reaching R-F1 0.579, R-tIoU 0.695, and R-ATD 55.657. Combining both modules produces the best overall visual and reaction-level balance.

This result yields a partly contradictory finding: prosody improves reaction occurrence but can harm temporal precision. In the per-class analysis, prosody increases R-F1 for every category, including a 29.31% gain for laughing and a 22.80% gain for smiling, yet it degrades R-tIoU and R-ATD for most classes. The implication is that speaker-side prosodic fluctuation is a useful trigger but an insufficient decision variable for determining whether, when, and how long a listener should react. Semantic content, discourse state, and interpersonal context remain implicit or absent.

The reaction-loss coefficient also exhibits a non-monotonic trade-off. The selected value $\lambda_{\mathrm{react}}=0.05$ yields R-F1 0.594, R-tIoU 0.704, and R-ATD 57.386. Increasing the coefficient to 0.1 raises R-F1 to 0.613 and R-tIoU to 0.711, but increases R-ATD to 64.790. At larger values, both visual quality and reaction metrics generally deteriorate. This sensitivity demonstrates that reaction supervision must remain balanced against the generative objective; stronger behavioral supervision is not uniformly better.

The audio-context study identifies ten seconds as a practical operating point. Increasing context from two to ten seconds improves R-F1 from 0.577 to 0.594 and R-tIoU from 0.678 to 0.704, while reducing R-ATD from 68.697 to 57.386. Longer contexts provide little consistent benefit and can degrade performance, suggesting that the model benefits from conversational context but is not simply improved by expanding the receptive field.

Limitations and open questions

The dataset and metrics encode a restricted reaction ontology. Micro-behaviors such as gaze shifts, blinking, subtle posture changes, and baseline head motion are excluded, although these behaviors contribute substantially to perceived listening. The single-label event representation also suppresses simultaneous or ambiguous reactions. Since annotations originate from an automatic detector followed by manual verification, detector bias can propagate into both training and evaluation: GLARE is rewarded for generating behaviors that the detector recognizes, not necessarily all behaviors that human observers would consider appropriate.

The model conditions primarily on audio prosody, audio embeddings, emotion features, and listener-side motion context. It does not explicitly model lexical semantics, dialogue acts, personality, relationship, cultural interaction norms, or longer conversational state. The observed degradation on Seamless and the weak results for surprise indicate that these omitted factors matter. The paper also evaluates speaker-dominant intervals rather than full-duplex interaction, and it does not address interruption, simultaneous speech, turn-taking decisions, or multi-party listener assignment.

Several methodological questions remain open within the paper’s own scope. The first is whether multi-reference or distributional evaluation can replace single-reference matching without sacrificing temporal specificity. The second is whether semantic and discourse representations can reduce premature prosody-triggered reactions. The third is whether human preference supervision can improve contextual appropriateness without collapsing reaction diversity. The authors mention reinforcement learning, but no reward design or preference-learning experiment is provided; such an extension would require avoiding rewards that merely reproduce the biases of the detector or a narrow annotator population.

Conclusion

GLARE reframes listening-head generation as reaction-aware event generation rather than generic facial-motion synthesis. Its principal empirical contribution is the demonstrated value of explicit frame-wise reaction supervision: the temporal reaction loss contributes more consistently to reaction accuracy and timing than prosody conditioning alone, while prosody supplies complementary cues for expressive reaction occurrence. Across RealTalk and Seamless, GLARE improves most reaction-centric and visual metrics, with RealTalk R-F1 of 0.594 and R-tIoU of 0.704, and human agreement rates of 89.9% for naturalness, 89.4% for contextual appropriateness, and 93.5% for timing plausibility. The work’s conclusions are strongest as evidence for reaction-aware dataset design and evaluation, while claims about absolute conversational appropriateness remain constrained by single-reference annotations, detector dependence, and a limited reaction taxonomy.