RealTalk: Realistic Conversation & Avatar Benchmarks

• RealTalk is a term representing diverse systems that benchmark affective face retrieval, long-term conversational memory, and realtime talking-head synthesis.
• The frameworks employ both retrieval‐based and generative methodologies to achieve social appropriateness, emotional expressivity, and accurate persona simulation.
• RealTalk underpins advances in realistic interaction by integrating multimodal conditioning, robust evaluation metrics, and controlled avatar behavior.

Searching arXiv for papers using the term “RealTalk” and closely related entries. RealTalk denotes several distinct resources and systems in recent arXiv literature: a dyadic affective-face dataset and retrieval framework for listener-video selection, a 21-day corpus for long-term conversation and memory evaluation, and two talking-head synthesis frameworks centered respectively on real-time audio-driven generation and controllable emotional generation. Because subsequent work reuses the same name across different subfields, the term is most accurately understood through its concrete instantiations rather than as a single canonical benchmark (Geng et al., 2023, Lee et al., 18 Feb 2025, Ji et al., 2024, Wang et al., 16 Aug 2025).

1. Terminological scope and disambiguation

The name “RealTalk” is attached to several non-equivalent artifacts. The most consequential distinction is between the 2023 dyadic-video resource used for listener modeling, the 2025 long-term messaging corpus for conversational memory, and two separate talking-head generation frameworks.

Referent	Primary role	Representative paper
RealTalk dyadic communication dataset/framework	Listener-video retrieval from unscripted two-person conversation	(Geng et al., 2023)
REALTALK long-term conversation corpus	Persona simulation and memory probing over 21-day chats	(Lee et al., 18 Feb 2025)
RealTalk audio-driven face generation	Real-time person-generic talking-face synthesis	(Ji et al., 2024)
RealTalk emotional talking-head synthesis	Emotion-controllable, identity-preserving talking heads	(Wang et al., 16 Aug 2025)

A common misconception is to treat these as successive versions of one benchmark. They are not. The 2023 RealTalk resource concerns dyadic listener behavior in video (Geng et al., 2023), whereas the 2025 REALTALK corpus concerns authentic messaging-app dialogues over 21 days (Lee et al., 18 Feb 2025). The two framework papers named RealTalk target talking-head generation rather than conversational retrieval or long-term memory (Ji et al., 2024, Wang et al., 16 Aug 2025).

2. RealTalk as a dyadic affective-face dataset and retrieval framework

The earliest prominent use of the name appears in “Affective Faces for Goal-Driven Dyadic Communication,” which introduces a video framework for modeling the association between verbal and non-verbal communication during dyadic conversation and a new video dataset of unscripted conversations covering diverse topics and demographics (Geng et al., 2023). The dataset is constructed from The Skin Deep and contains 692 videos, an average length of about 10 minutes per video, and 115 hours of raw conversation (Geng et al., 2023). For benchmarking, the paper isolates 1,896 clips in which only one person is speaking for at least 15.36 seconds, treats the other person as the listener to be predicted, uses 1,489 clips for training a visual prompt, and 407 clips for evaluation (Geng et al., 2023).

Methodologically, the framework is retrieval-based rather than generative. Given a speaker transcript $x$ and a listener goal $g$, GPT-3 produces a textual description $c=f(x,g)$ of the desired listener expression, and CLIP scores candidate listener videos with a similarity function of the form $\hat{y}=\arg\max_{y\in\mathcal{Y}}\phi(c,y)$ (Geng et al., 2023). Key frames are selected using EMOCA-derived facial-expression features, and a learned visual prompt $\delta$ adapts CLIP through a contrastive objective built from socially appropriate and socially inappropriate textual descriptions (Geng et al., 2023). The conditioning variables are natural-language goals such as “behave in a socially appropriate way” or “behave socially inappropriately,” and the same prompting mechanism can incorporate personality, relationship, or background information (Geng et al., 2023).

Evaluation in this original setting emphasizes social appropriateness. On 407 test clips with a retrieval bank of 1,896 listener videos, the social-response variant attains $R@500=0.31$, $R@1000=0.56$, and median rank $834$, outperforming Learning2Listen and random retrieval under the reported protocol (Geng et al., 2023). In a 2-alternative forced choice study, the socially conditioned RealTalk responses are preferred over Learning2Listen 71.7% of the time and over the paper’s “rude” responses 77.5% of the time, while still trailing the ground-truth listener videos (Geng et al., 2023). These results established RealTalk primarily as a benchmark for socially appropriate listener-face selection under goal conditioning.

3. RealTalk as a downstream benchmark for controllable listener generation

Subsequent listener-generation work repurposes the 2023 RealTalk dataset from retrieval into fully generative modeling. “CustomListener: Text-guided Responsive Interaction for User-friendly Listening Head Generation” extends RealTalk into a text-annotated listening-head dataset by re-extracting 3DMM coefficients with a 70-dimensional motion format for both speaker and listener, constructing $\mathcal{D}_{train}$ and $\mathcal{D}_{test}$, and generating segment-level text priors with LibreFace, Hopenet, FACS semantics, and GPT-based lexical diversification (Liu et al., 2024). In this formulation, RealTalk becomes both a training source and a benchmark for controllable listener motion. Static portrait tokens $g$0 derived from RoBERTa-encoded text are transformed by the Static to Dynamic Portrait module and then combined with Past Guided Generation for segment-to-segment coherence in 60-frame clips (Liu et al., 2024). The evaluation protocol on RealTalk includes FD, RTLCC, RWTLCC, $g$1, SND, V-D, plus rendered-image metrics such as SSIM, CPBD, PSNR, and FID, and the paper reports the best values across the listed motion metrics against RLHG, PCH, and L2L retrained on RealTalk (Liu et al., 2024).

“DiTaiListener: Controllable High Fidelity Listener Video Generation with Diffusion” uses RealTalk even more centrally as “the primary training source and the strongest benchmark” for long, emotionally rich listener-video generation (Siniukov et al., 5 Apr 2025). In this usage, RealTalk includes videos from The Skin Deep YouTube Podcast, 692 dyadic conversations, 115 hours, around 50,000 conversational turns of dyadic interactions with specified roles, and provided bounding boxes of participants’ faces as well as EMOCA 3DMM coefficients (Siniukov et al., 5 Apr 2025). Because raw RealTalk contains occlusions, shaky bounding boxes, extreme head poses, and overlays that caused model collapse, the paper applies stable cropping, LibreFace pose filtering with yaw and pitch thresholds of $g$2 degrees, and a CLIP-based artifact detector with 0.95 recall, after which 13.6% of data is removed (Siniukov et al., 5 Apr 2025).

RealTalk then serves as the testbed for both photorealism and motion evaluation. On the RealTalk test set, DiTaiListener reports $g$3, $g$4, $g$5, $g$6, and $g$7, compared with much worse scores for RLHG, L2L, and DIM (Siniukov et al., 5 Apr 2025). In EMOCA space, it reports $g$8, $g$9, $c=f(x,g)$0, $c=f(x,g)$1, and $c=f(x,g)$2, with SID and Var close to the reported ground-truth diversity values (Siniukov et al., 5 Apr 2025). Within this line of work, RealTalk is no longer just a retrieval corpus; it is a large-scale benchmark for responsive, controllable, and photorealistic listener-head generation.

4. REALTALK as a 21-day corpus for long-term conversation and memory

A separate line of work uses the name for a real-world dialogue benchmark rather than a video resource. “REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation” introduces a corpus of 10 authentic messaging-app conversations collected from 10 native English speakers in the United States, aged 18–25, with each participant appearing in two conversations and each pair instructed to converse for about 21 days and send at least 50 messages per day (Lee et al., 18 Feb 2025). The dataset reports approximately 894.4 turns, 21.9 sessions, and 17,109.8 tokens per conversation on average, along with asynchronous timing statistics such as a mean response time of 20.98 minutes, a median gap of 2.22 minutes, and a maximum gap of 27.90 hours (Lee et al., 18 Feb 2025).

The paper’s central analytic contribution is an emotional-intelligence and persona-consistency framework. Message-level attributes include reflectiveness, grounding acts, emotion, sentiment, intimacy, and empathy; speaker-level summaries aggregate these into reflective frequency, grounding frequency, emotion and sentiment diversity, intimacy progression, stability, alignment, and average empathy (Lee et al., 18 Feb 2025). Compared with synthetic long-term dialogue such as LoCoMo, RealTalk exhibits higher emotion and sentiment diversity, while LLM-generated dialogues show more uniform and often excessive empathy (Lee et al., 18 Feb 2025). Because each participant appears in two conversations, the paper can also measure how much EI attributes shift across partners, showing that some participants are stable while others are adaptive, with intimacy progression displaying the largest variability (Lee et al., 18 Feb 2025).

Two benchmark tasks formalize long-horizon conversational competence. Persona simulation asks a model to continue a conversation on behalf of a specific user from prior dialogue history; memory probing asks short-answer questions over full conversation history or event summaries (Lee et al., 18 Feb 2025). The memory benchmark contains 728 questions across Multi-hop, Temporal, and Commonsense categories, while a related description in D-Mem characterizes the same RealTalk benchmark as 728 questions across Multi-hop, Temporal, and Open-domain categories and emphasizes that each conversation averages over 16,000 words (You et al., 19 Mar 2026). In persona simulation, fine-tuning on another conversation from the same speaker improves EI-style matching: reflective accuracy rises from $c=f(x,g)$3 to $c=f(x,g)$4, grounding accuracy from $c=f(x,g)$5 to $c=f(x,g)$6, and empathy difference drops from $c=f(x,g)$7 to $c=f(x,g)$8, while ROUGE remains $c=f(x,g)$9 in both conditions (Lee et al., 18 Feb 2025). In memory probing, even gpt-4o on full conversation context reaches only $\hat{y}=\arg\max_{y\in\mathcal{Y}}\phi(c,y)$0 on Multi-hop, $\hat{y}=\arg\max_{y\in\mathcal{Y}}\phi(c,y)$1 on Temporal, and $\hat{y}=\arg\max_{y\in\mathcal{Y}}\phi(c,y)$2 on Commonsense, with substantial degradation when the input is reduced to event summaries (Lee et al., 18 Feb 2025).

The benchmark has already been adopted in LLM memory research. “D-Mem: A Dual-Process Memory System for LLM Agents” uses RealTalk as the real-world stress test for long-horizon conversational memory, pairing it with LoCoMo (You et al., 19 Mar 2026). On RealTalk with GPT-4o-mini, Mem0$\hat{y}=\arg\max_{y\in\mathcal{Y}}\phi(c,y)$3 achieves $\hat{y}=\arg\max_{y\in\mathcal{Y}}\phi(c,y)$4, Full Deliberation reaches $\hat{y}=\arg\max_{y\in\mathcal{Y}}\phi(c,y)$5, and Multi-dimensional Quality Gating reaches $\hat{y}=\arg\max_{y\in\mathcal{Y}}\phi(c,y)$6 with 16,786 tokens versus 48,772 for Full Deliberation, recovering 97.0% of Full Deliberation’s F1 at substantially lower computational cost (You et al., 19 Mar 2026). In this tradition, RealTalk functions as a benchmark for noisy, real-world, long-term conversational memory rather than audiovisual generation.

5. RealTalk as the name of two talking-head generation frameworks

Two additional papers use “RealTalk” as the name of an overview system. “RealTalk: Real-time and Realistic Audio-driven Face Generation with 3D Facial Prior-guided Identity Alignment Network” presents a person-generic audio-driven framework with an audio-to-expression transformer and a high-fidelity expression-to-face renderer (Ji et al., 2024). The first stage uses shape priors and historical expression coefficients in cross-modal attention to improve identity-aware lip motion; the second stage uses a lightweight Facial Identity Alignment module with lip-shape control and face-texture reference structures (Ji et al., 2024). The paper reports about 30 FPS, approximately 33.1 ms per frame total, and stronger lip-sync and image-quality metrics than Wav2Lip, TalkLip, DINet, IP-LAP, and StyleTalk on VoxCeleb1, MEAD, and HDTF, with a user study showing MOS 3.77 for visual quality and 3.72 for lip sync (Ji et al., 2024). This RealTalk is unrelated to the long-term chat corpus and only indirectly related to the 2023 dyadic-video dataset through the shared name.

“RealTalk: Realistic Emotion-Aware Lifelike Talking-Head Synthesis” proposes a different pipeline for emotional talking heads (Wang et al., 16 Aug 2025). It uses a VAE to generate neutral 3D facial landmarks from driving audio, a ResNet-plus-attention Landmark Deformation Model to combine those landmarks with one of 8 discrete emotion labels, and a tri-plane attention NeRF conditioned on emotional landmarks and facial blendshape coefficients (Wang et al., 16 Aug 2025). The framework is trained on VoxCeleb2 for the VAE, MEAD for the LDM, and MEAD plus Ad-NeRF videos for the NeRF, and the paper reports SSIM $\hat{y}=\arg\max_{y\in\mathcal{Y}}\phi(c,y)$7, PSNR $\hat{y}=\arg\max_{y\in\mathcal{Y}}\phi(c,y)$8, LPIPS $\hat{y}=\arg\max_{y\in\mathcal{Y}}\phi(c,y)$9, emotion score $\delta$0, M-LMD $\delta$1, and F-LMD $\delta$2 (Wang et al., 16 Aug 2025). Emotion labels cover angry, disgust, contempt, fear, happy, neutral, sad, and surprise, and the deformation scalar $\delta$3 is reported as the optimal intensity setting in ablation (Wang et al., 16 Aug 2025). Here again, “RealTalk” names an overview architecture rather than a dataset.

These two framework papers demonstrate that the label migrated from conversational corpora into talking-head generation. A plausible implication is that “RealTalk” had become semantically associated with realism, controllability, and conversational plausibility in avatar generation, even when the underlying tasks diverged.

6. Recurring themes, limitations, and research significance

Across these uses, several methodological themes recur. First, realism is tied to authentic conversational data: the 2023 RealTalk dataset emphasizes unscripted dyadic interaction with affective listener faces (Geng et al., 2023), while the 2025 REALTALK corpus emphasizes authentic messaging-app dialogue with asynchronous timing, revisited topics, and varied EI profiles (Lee et al., 18 Feb 2025). Second, controllability is central: goal-conditioned listener retrieval in the 2023 framework (Geng et al., 2023), free-form text-guided listener motion in CustomListener (Liu et al., 2024), multimodal causal conditioning in DiTaiListener (Siniukov et al., 5 Apr 2025), and explicit emotion labels in the 2025 emotional talking-head system (Wang et al., 16 Aug 2025). Third, several works expose the cost of realism: DiTaiListener requires aggressive cleaning because raw RealTalk artifacts caused model collapse (Siniukov et al., 5 Apr 2025), the 2023 listener-retrieval framework lacks multimodal speaker context and explicit temporal alignment (Geng et al., 2023), the 2025 long-term conversation corpus is demographically narrow and focused on chit-chat (Lee et al., 18 Feb 2025), and the emotional talking-head RealTalk remains limited to predefined emotion categories and per-subject NeRF training (Wang et al., 16 Aug 2025).

The name also sits within a broader research trajectory toward more natural spoken and embodied interaction. Adjacent systems pursue overlapping objectives without using the same title: RTTL-DG targets real-time textless dialogue generation with overlaps, backchannels, and an average gap of 393 ms (Mai et al., 8 Jan 2025); OmniTalker performs one-shot real-time text-driven talking audio-video generation with multimodal style mimicking at 25 FPS (Wang et al., 3 Apr 2025); TokTalk maps Audio-LLM tokens directly to expressive 3D facial motion with approximately 400 ms latency in the chatbot-avatar setting (Zhao et al., 29 May 2026). This suggests that the repeated reuse of “RealTalk” reflects a broader research program organized around realistic conversation, responsive listening, long-horizon memory, and controllable avatar behavior rather than a single unified benchmark.

In aggregate, RealTalk is best treated as a historically layered term. In dyadic-video research it names a benchmark for socially appropriate listener behavior (Geng et al., 2023); in long-term dialogue it names a corpus for persona simulation and memory probing under real human conversational noise (Lee et al., 18 Feb 2025); and in talking-head synthesis it names systems for real-time audio-driven rendering and emotion-aware facial generation (Ji et al., 2024, Wang et al., 16 Aug 2025). The shared thread is not ontology but aspiration: each instantiation uses the label to mark an attempt to close the gap between technically plausible interaction and conversation that appears socially, temporally, or affectively real.