LPM-Bench: Video Conversational Benchmark

• LPM-Bench is the first comprehensive benchmark designed to assess video-based generative models for full-duplex conversational character performance.
• It introduces standardized protocols and metrics across listening, speaking, and identity tests to tackle the performance trilemma of expressiveness, real-time inference, and long-horizon identity stability.
• The benchmark employs rigorous datasets and precise metrics like ICS, EXM, TSM, and LSE to guide advancements in interactive AI and embodied character systems.

LPM-Bench is the first comprehensive benchmark for the evaluation of full-duplex conversational character performance in video-based generative models. It targets the multifaceted notion of "performance"—the joint externalization of intent, emotion, and personality by a digital character—proposing standardized protocols, metrics, and data splits for rigorous, reproducible comparison of models that generate both listening and speaking behaviors in single-character dialogues. LPM-Bench is explicitly constructed to stress-test what is termed the "performance trilemma": achieving simultaneously high expressiveness, real-time inference, and long-horizon identity stability (Zeng et al., 9 Apr 2026).

1. Motivation and Conceptual Framework

LPM-Bench addresses a critical deficiency in the evaluation of character generation systems, where prior benchmarks focus on isolated modalities such as lip-sync or facial realism. The framework is motivated by the observation that no existing system robustly achieves all three properties of the performance trilemma:

• Expressiveness: The ability to produce emotionally and semantically aligned motions, encompassing nuanced expressions and gestures.
• Real-Time Inference: Generating outputs suitable for interactive, streaming applications, with low end-to-end latency.
• Long-Horizon Identity Stability: Maintaining a coherent character identity over multi-minute time scales, preventing visual "drift" or loss of identity features.

By evaluating systems jointly across these axes, LPM-Bench seeks to drive forward the development of models suitable for high-fidelity interactive applications (e.g., conversational agents, live stream characters, game NPCs). Each task and metric in LPM-Bench is designed to simultaneously probe these dimensions, moving beyond single-metric or single-modality evaluation (Zeng et al., 9 Apr 2026).

2. Benchmark Components and Evaluation Tasks

LPM-Bench consists of three primary evaluation tasks, each with specific inputs, generation requirements, and real-time constraints:

2.1 Listening Video Generation

• Inputs: Static reference frame $I_{\rm ref}$ (plus up to 4 identity references), user audio stream $\{a_t\}_{t=1}^T$, and optional text style prompt.
• Output: Real-time generation ($<30$ ms per frame) of video $\{ \hat I_t\}_{t=1}^T$ demonstrating natural attentive behaviors (e.g., head tilts, micro-nods, gaze) while preserving identity to $I_{\rm ref}$.

2.2 Speaking Video Generation

• Inputs: Same identity references as above; synthesizer-generated speech audio $\{s_t\}_{t=1}^T$ or text transcript; style prompt.
• Output: Streaming video depicting accurate lip-sync, upper-body gestures, and expressions synchronized to provided speech.

2.3 Identity Consistency Tests

• Protocol: Multi-minute continuous generation (up to 5 minutes), with periodic "re-anchoring" to identity references; quantification of identity drift using face-recognition embeddings.

Auxiliary protocols enforce strict separation of training and test identities and standardize streaming inference constraints (single NVIDIA A100 GPU, ≤12GB, latency cap).

3. Dataset Construction

LPM-Bench curates its dataset from publicly licensed interview/vlog repositories (TED-style talks, news anchors) and proprietary training videos (with legal clearance). Each example satisfies:

• Resolution: Minimum 720p.
• Speaker constraint: Single frontal speaker (≥30 seconds).
• Lighting: Controlled setup (no extreme shadows).
• Audio separation: Clear segmentation between speech and silence.

Segmentation into listening and speaking intervals is automatic, using robust voice-activity detection (VAD) with forced alignment ensuring 20 ms correspondence between audio and video labels. For identity anchoring, up to $K=4$ reference frames per session are obtained by K-means clustering over face embeddings and selecting the sharpest frame per cluster, providing diverse yet consistent identity conditioning (Zeng et al., 9 Apr 2026).

4. Metrics and Formal Definitions

LPM-Bench introduces four primary metrics:

• Identity Consistency Score (ICS):

$$d_t = \big\|\phi(\hat I_t) - \phi(I_{\rm ref})\big\|_2,\quad \mathrm{ICS} = 1 - \frac{1}{T}\sum_{t=1}^T \frac{d_t}{d_{\max}} \in [0,1]$$

with $\phi(\cdot)$ a pre-trained face-recognition embedding and $d_{\max}$ the 95th percentile normalization constant.

• Expressiveness Measure (EXM):

$\{a_t\}_{t=1}^T$0

quantifying the variance of facial action units or 3D head-pose parameters.

• Temporal Stability Metric (TSM):

$\{a_t\}_{t=1}^T$1

where a high TSM denotes smooth, stable motion.

• Lip-Sync Error (LSE):

$\{a_t\}_{t=1}^T$2

where $\{a_t\}_{t=1}^T$3 is the SyncNet correlation between predicted lips and audio; lower values are better (Zeng et al., 9 Apr 2026).

5. Protocols and Baselines

Training/validation/test splits consist of 80/10/10 identity-disjoint sessions of ~60 seconds each. Streaming decoding rates, multi-reference conditioning, and resource restrictions are strictly specified. Models are evaluated against established video-based character animation baselines:

• MakeItTalk (CVPR’19)
• Wav2Lip (ICCV’21)
• Pose2Talk (ECCV’20)
• Follow-The-Speaker (ECCV’22)
• Base LPM (17B, offline)
• Online LPM (distilled)

Quantitative test-set results for all metrics and latency are summarized as follows:

Model	ICS	EXM	TSM	LSE (↓)	Latency (ms)
MakeItTalk	0.52	0.11	0.62	0.38	45
Wav2Lip	0.57	0.09	0.65	0.28	50
Pose2Talk	0.61	0.14	0.58	0.31	60
Follow-The-Speaker	0.64	0.16	0.60	0.26	40
Base LPM (17B, offline)	0.78	0.22	0.72	0.18	120
Online LPM (distilled)	0.75	0.20	0.70	0.20	25

Base LPM exceeds previous baselines in expressiveness (EXM +37%) and identity stability (ICS +14%) but is not real-time; Online LPM sacrifices minimal accuracy to achieve 25 ms/frame inference (Zeng et al., 9 Apr 2026).

6. Analysis and Insights

LPM-Bench demonstrates that only a systems-level design—spanning dataset, multimodal conditioning, transformer-based diffusion models, and knowledge distillation—can overcome the performance trilemma for conversational character generation. The benchmark exposes trade-offs easily missed by evaluating single properties in isolation: for example, some models maintain short-term expressiveness but exhibit identity drift in multi-minute scenarios, while others achieve real-time speed but lack the rich expressiveness required for lifelike avatars.

Properly constructed benchmarks like LPM-Bench are essential for surfacing these multidimensional weaknesses and for guiding both architectural innovation and comparative evaluation (Zeng et al., 9 Apr 2026).

7. Future Directions

Suggested directions include extending LPM-Bench to multi-party conversational scenarios, incorporating longer-term discourse memory to track conversational coherence across minutes, and grounding generated performance in 3D environments for physically accurate gesture synthesis. Such extensions would further stress the bounds of the performance trilemma and provide fertile ground for next-generation video-based character models.

LPM-Bench, together with its associated Base and Online LPM models, represents a foundational suite for robustly benchmarking interactive video generation systems intended for conversational AI and embodied agent applications (Zeng et al., 9 Apr 2026).