WholeBodyBenchmark for Joint Audio-Video Generation

• WholeBodyBenchmark is a benchmark for joint whole-body talking avatar and speech generation, featuring multimodal evaluation across video, speech, and region-specific performance metrics.
• It combines a large-scale multimodal dataset of 10,000 unique identities and 2 million video samples with detailed annotations for body, face, hands, and audio to assess temporal and spatial coherence.
• The benchmark shows that pose guidance can improve digital avatar synthesis while highlighting current challenges in achieving consistent quality across entire body regions.

WholeBodyBenchmark is the public repository associated with JWB-DH-V1, short for “Joint Whole-Body Talking Avatar and Speech Generation Version I”, a benchmark for joint audio-video generation of whole-body animatable avatars (Di et al., 28 Jul 2025). It was introduced to address two deficits identified in contemporary digital-human generation: the absence of evaluation frameworks that jointly assess visual and audio quality, and the lack of region-specific analysis for whole-body generation. In its initial release, WholeBodyBenchmark combines a large-scale multimodal dataset with an evaluation protocol spanning video generation, co-speech avatar synthesis, and speech synthesis, while explicitly separating performance over the whole body, face, and hands (Di et al., 28 Jul 2025).

1. Identity, scope, and benchmark objective

WholeBodyBenchmark is not a generic label for all whole-body evaluation efforts; within the repository context, it refers specifically to the benchmark release underlying JWB-DH-V1, whose dataset and evaluation tools are publicly available at https://github.com/deepreasonings/WholeBodyBenchmark (Di et al., 28 Jul 2025). Its central target is joint whole-body talking avatar and speech generation, rather than 3D pose estimation, robotics, or medical whole-body parsing.

The benchmark is framed around a specific failure mode of current systems: methods can produce strong local realism, especially in face-centric or upper-body settings, yet still fail at multimodal consistency when speech, facial animation, hand motion, and full-body dynamics must be generated jointly. WholeBodyBenchmark therefore treats digital-human generation as a whole-body, multimodal synchronization problem. A recurring misconception is that a visually convincing face or a low speech error rate is sufficient evidence of strong whole-body generation. The benchmark is designed to reject that assumption by requiring simultaneous assessment of body-region fidelity, temporal stability, identity preservation, and speech quality (Di et al., 28 Jul 2025).

2. Dataset composition and annotation structure

The dataset scale reported for WholeBodyBenchmark is 10,000 unique identities and 2 million video samples, with each identity appearing in approximately 200 different scene configurations; 20,000 samples are used for evaluation (Di et al., 28 Jul 2025). The release is explicitly multimodal. Each sample includes video frames, body segmentation, landmarks, bounding box annotations for hands, legs, and whole body, motion text, speech transcription, a motion flag, ground-truth audio, and word boundaries (Di et al., 28 Jul 2025).

This annotation design suggests that the benchmark is intended to support more than full-frame realism scoring. It supports evaluation of alignment among body posture, hand gestures, leg stance, spoken language, and audio over time. The benchmark is thus structured for methods that synthesize or evaluate coordinated body motion and speech rather than isolated talking-head motion.

Some dataset properties remain intentionally under-specified in the release summary. Beyond the fact that 20,000 samples are used for evaluation, the text does not provide explicit train/validation/test split naming, exact clip-duration statistics, or a detailed collection and curation pipeline (Di et al., 28 Jul 2025). Likewise, demographic diversity, viewpoint diversity, and multilingual scope are not systematically tabulated in the provided description. This suggests a benchmark that is already operational at scale but whose Version I documentation emphasizes evaluation framing more than exhaustive corpus auditing.

3. Task formulation and evaluation protocol

WholeBodyBenchmark decomposes evaluation into three sub-protocols: video generation, co-speech avatar generation, and speech audio evaluation (Di et al., 28 Jul 2025). In the video-generation setting, models take an initial video frame and a text prompt to guide motion. In the talking-avatar setting, models are driven by audio, optionally with pose guidance. In the speech setting, models generate audio from benchmark utterances and prompting strategies.

The benchmark’s most distinctive technical feature is its region-specific evaluation. Visual metrics are computed independently for whole body, face, and hand regions rather than only on the full frame (Di et al., 28 Jul 2025). This design exposes cases where a system preserves coarse structure yet fails on articulated hands or face identity under motion.

For reference-free video assessment, the benchmark adopts six metrics: Subject Consistency (SC), Background Consistency (BC), Motion Smoothness (MS), Dynamic Degree (DD), Aesthetic Quality (AQ), and Imaging Quality (IQ) (Di et al., 28 Jul 2025). For reference-based visual assessment, it uses FID, FVD, SSIM, PSNR, E-FID, and CSIM (Di et al., 28 Jul 2025). For speech, it combines WER with a Large-Audio-LLM pairwise judgment protocol using Gemini 2.5 Pro. The explicit win-rate definition reported for a candidate system $T_i$ is

$$W(T_i) = \frac{\sum (\text{winner} = \text{index}_i) + 0.5 \cdot \sum (\text{winner} = 0)}{n}$$

where $\text{index}_i \in \{1,2\}$ is the randomized label assigned to $T_i$, $n$ is the total number of pairwise comparisons, and $\text{winner}=0$ denotes a tie (Di et al., 28 Jul 2025).

The release does not describe a separate human user study for scoring, and it does not report statistical significance testing or inter-rater agreement (Di et al., 28 Jul 2025). This suggests that Version I is primarily an instrumented benchmark protocol rather than a human-annotation-heavy evaluation campaign.

4. Baselines and principal empirical findings

WholeBodyBenchmark evaluates four groups of baselines: open-source video generation models (Step-Video-TI2V, HunyuanVideo, Wan, Open-Sora 2.0), open-source talking-avatar models (Hallo3 and EchoMimicV2, each with and without pose guidance), open-source TTS systems (Suno Bark, Tortoise-TTS, MiniCPM, Qwen2.5-Omni), and closed-source TTS systems (11Labs Eleven Multilingual v2 and gpt-4o-mini-audio-preview) (Di et al., 28 Jul 2025). Veo-3 is discussed but excluded because of instability for synchronized outputs from a single frame (Di et al., 28 Jul 2025).

The main visual finding is a persistent disparity between localized and whole-body performance. Among video-generation models, Wan is reported as one of the strongest performers, including whole-body SC = 96.83, MS = 99.69, FID = 92.76, FVD = 750.51, SSIM = 0.660, PSNR = 20.16, and CSIM = 0.896 (Di et al., 28 Jul 2025). Open-Sora 2.0 performs strongly on some global consistency metrics, including whole-body SC = 97.12 and BC = 98.00, but is weaker than Wan on several fidelity metrics (Di et al., 28 Jul 2025).

The talking-avatar baselines degrade sharply when evaluated as full-body generators. For example, reported whole-body FVD values include 2470.14 for Ha3/w, 4366.39 for Ha3/wo, 2519.77 for ECV2/w, and 3791.25 for ECV2/wo (Di et al., 28 Jul 2025). The benchmark interprets this as evidence that portrait- or semi-body-oriented systems do not transfer cleanly to full-body generation.

A second major result is that hands are particularly difficult. Reported hand-region FVD values remain extremely high for talking-avatar methods, including 7750.48, 10671.79, 8958.27, and 9472.28 across the four Hallo3 and EchoMimicV2 variants (Di et al., 28 Jul 2025). The face region also remains difficult: even stronger video models retain high face-region FID, including 254.03 for Wan and 275.50 for Open-Sora 2.0 (Di et al., 28 Jul 2025). This suggests that current systems do not yet provide uniformly strong generation across the face-hand-body hierarchy.

Pose guidance yields substantial gains. For Hallo3, reported whole-body SC rises from 10.74 without pose guidance to 20.20 with guidance, while whole-body FVD improves from 4366.39 to 2470.14 (Di et al., 28 Jul 2025). For EchoMimicV2, whole-body SC rises from 7.08 to 29.16, and whole-body FVD improves from 3791.25 to 2519.77 (Di et al., 28 Jul 2025). This suggests that pose conditioning remains a strong control signal even though it does not close the performance gap to leading general video generators.

The audio results indicate that low WER does not fully capture perceived speech quality. 11Labs multilingual v2 achieves the best reported WER on emotions at 4.68, while gpt-4o-mini-audio-preview achieves the highest reported win-rates, including 58.31% on emotions and 60.68% on paralinguistics (Di et al., 28 Jul 2025). WholeBodyBenchmark thereby treats speech evaluation as a perceptual and expressive problem, not solely an ASR-reconstruction problem.

5. Position within the broader landscape of whole-body benchmarks

WholeBodyBenchmark belongs to a broader but highly heterogeneous family of “whole-body” benchmarks. In H3WB, the term denotes 3D human whole-body pose estimation with 133 keypoints spanning body, face, hands, and feet, together with benchmark tasks for lifting and monocular estimation (Zhu et al., 2022). In M3Bench, it denotes whole-body motion generation for mobile manipulation in 3D scenes, where success depends on coordinated base-arm trajectories in household environments (Zhang et al., 2024). In HumanoidBench, it denotes simulated humanoid whole-body locomotion and manipulation across 27 tasks on a high-dimensional embodied platform (Sferrazza et al., 2024).

Against that backdrop, WholeBodyBenchmark occupies a distinct subdomain: whole-body digital human generation with synchronized speech. This suggests that the phrase “whole body” is not a single technical category but a family resemblance across benchmarks whose common property is that isolated subsystems—face only, arm only, locomotion only, audio only—are insufficient representations of the target problem.

6. Versioning, limitations, and prospective development

WholeBodyBenchmark is explicitly a Version I release (Di et al., 28 Jul 2025). Its current limitations are stated with unusual clarity. The release does not yet fully support stable end-to-end joint audio-video evaluation from a single initial frame, and current state-of-the-art joint generation models are described as still unstable, which is the reason Veo-3 is excluded (Di et al., 28 Jul 2025). The text also notes that the benchmark appears to focus on shorter clips in its present form.

Several infrastructural omissions remain salient. The release summary does not discuss dataset license, access approval requirements, privacy safeguards, consent procedures, or intended-use restrictions for a dataset containing 10,000 unique identities (Di et al., 28 Jul 2025). It also does not fully enumerate prompt templates, generation lengths, or compute-budget normalization across all evaluated models. These absences do not invalidate the benchmark, but they delimit the interpretability and governance maturity of Version I.

The stated future directions are concrete: later releases are intended to support 60-second clips, include refined audio-video annotations, and evaluate stable joint generation models from a single initial frame (Di et al., 28 Jul 2025). In that sense, WholeBodyBenchmark is best understood as a foundational benchmark release whose principal contribution is not the claim that whole-body talking-avatar generation is solved, but the demonstration that current methods remain substantially below a robust standard once evaluation becomes region-aware, multimodal, and explicitly whole-body.