---
title: WholeBodyBenchmark for Joint Audio-Video Generation
url: https://www.emergentmind.com/topics/wholebodybenchmark
type: topic
---

# WholeBodyBenchmark for Joint Audio-Video Generation

WholeBodyBenchmark is the public repository associated with **JWB-DH-V1**, short for **“Joint Whole-Body Talking Avatar and Speech Generation Version I”**, a benchmark for joint audio-video generation of whole-body animatable avatars [2507.20987]. It was introduced to address two deficits identified in contemporary digital-human generation: the absence of evaluation frameworks that jointly assess visual and audio quality, and the lack of region-specific analysis for whole-body generation. In its initial release, WholeBodyBenchmark combines a large-scale multimodal dataset with an evaluation protocol spanning video generation, co-speech avatar synthesis, and speech synthesis, while explicitly separating performance over the **whole body**, **face**, and **hands** [2507.20987].

## 1. Identity, scope, and benchmark objective

WholeBodyBenchmark is not a generic label for all whole-body evaluation efforts; within the repository context, it refers specifically to the benchmark release underlying JWB-DH-V1, whose dataset and evaluation tools are publicly available at `https://github.com/deepreasonings/WholeBodyBenchmark` [2507.20987]. Its central target is **joint whole-body talking avatar and speech generation**, rather than 3D pose estimation, robotics, or medical whole-body parsing.

The benchmark is framed around a specific failure mode of current systems: methods can produce strong local realism, especially in face-centric or upper-body settings, yet still fail at **multimodal consistency** when speech, facial animation, hand motion, and full-body dynamics must be generated jointly. WholeBodyBenchmark therefore treats digital-human generation as a **whole-body, multimodal synchronization problem**. A recurring misconception is that a visually convincing face or a low speech error rate is sufficient evidence of strong whole-body generation. The benchmark is designed to reject that assumption by requiring simultaneous assessment of body-region fidelity, temporal stability, identity preservation, and speech quality [2507.20987].

## 2. Dataset composition and annotation structure

The dataset scale reported for WholeBodyBenchmark is **10,000 unique identities** and **2 million video samples**, with each identity appearing in approximately **200 different scene configurations**; **20,000 samples are used for evaluation** [2507.20987]. The release is explicitly multimodal. Each sample includes **video frames**, **body segmentation**, **landmarks**, **bounding box annotations** for **hands**, **legs**, and **whole body**, **motion text**, **speech transcription**, a **motion flag**, **ground-truth audio**, and **word boundaries** [2507.20987].

This annotation design suggests that the benchmark is intended to support more than full-frame realism scoring. It supports evaluation of alignment among **body posture**, **hand gestures**, **leg stance**, **spoken language**, and **audio over time**. The benchmark is thus structured for methods that synthesize or evaluate coordinated body motion and speech rather than isolated talking-head motion.

Some dataset properties remain intentionally under-specified in the release summary. Beyond the fact that **20,000 samples are used for evaluation**, the text does not provide explicit train/validation/test split naming, exact clip-duration statistics, or a detailed collection and curation pipeline [2507.20987]. Likewise, demographic diversity, viewpoint diversity, and multilingual scope are not systematically tabulated in the provided description. This suggests a benchmark that is already operational at scale but whose Version I documentation emphasizes evaluation framing more than exhaustive corpus auditing.

## 3. Task formulation and evaluation protocol

WholeBodyBenchmark decomposes evaluation into three sub-protocols: **video generation**, **co-speech avatar generation**, and **speech audio evaluation** [2507.20987]. In the video-generation setting, models take an **initial video frame** and a **text prompt** to guide motion. In the talking-avatar setting, models are driven by **audio**, optionally with **pose guidance**. In the speech setting, models generate audio from benchmark utterances and prompting strategies.

The benchmark’s most distinctive technical feature is its **region-specific evaluation**. Visual metrics are computed independently for **whole body**, **face**, and **hand** regions rather than only on the full frame [2507.20987]. This design exposes cases where a system preserves coarse structure yet fails on articulated hands or face identity under motion.

For reference-free video assessment, the benchmark adopts six metrics: **Subject Consistency (SC)**, **Background Consistency (BC)**, **Motion Smoothness (MS)**, **Dynamic Degree (DD)**, **Aesthetic Quality (AQ)**, and **Imaging Quality (IQ)** [2507.20987]. For reference-based visual assessment, it uses **FID**, **FVD**, **SSIM**, **PSNR**, **E-FID**, and **CSIM** [2507.20987]. For speech, it combines **WER** with a **Large-Audio-Language Model** pairwise judgment protocol using **Gemini 2.5 Pro**. The explicit win-rate definition reported for a candidate system \(T_i\) is

$$
W(T_i) = \frac{\sum (\text{winner} = \text{index}_i) + 0.5 \cdot \sum (\text{winner} = 0)}{n}
$$

where \(\text{index}_i \in \{1,2\}\) is the randomized label assigned to \(T_i\), \(n\) is the total number of pairwise comparisons, and \(\text{winner}=0\) denotes a tie [2507.20987].

The release does not describe a separate human user study for scoring, and it does not report statistical significance testing or inter-rater agreement [2507.20987]. This suggests that Version I is primarily an instrumented benchmark protocol rather than a human-annotation-heavy evaluation campaign.

## 4. Baselines and principal empirical findings

WholeBodyBenchmark evaluates four groups of baselines: open-source video generation models (**Step-Video-TI2V**, **HunyuanVideo**, **Wan**, **Open-Sora 2.0**), open-source talking-avatar models (**Hallo3** and **EchoMimicV2**, each with and without pose guidance), open-source TTS systems (**Suno Bark**, **Tortoise-TTS**, **MiniCPM**, **Qwen2.5-Omni**), and closed-source TTS systems (**11Labs Eleven Multilingual v2** and **gpt-4o-mini-audio-preview**) [2507.20987]. **Veo-3** is discussed but excluded because of instability for synchronized outputs from a single frame [2507.20987].

The main visual finding is a persistent disparity between **localized** and **whole-body** performance. Among video-generation models, **Wan** is reported as one of the strongest performers, including **whole-body SC = 96.83**, **MS = 99.69**, **FID = 92.76**, **FVD = 750.51**, **SSIM = 0.660**, **PSNR = 20.16**, and **CSIM = 0.896** [2507.20987]. **Open-Sora 2.0** performs strongly on some global consistency metrics, including **whole-body SC = 97.12** and **BC = 98.00**, but is weaker than Wan on several fidelity metrics [2507.20987].

The talking-avatar baselines degrade sharply when evaluated as full-body generators. For example, reported **whole-body FVD** values include **2470.14** for **Ha3/w**, **4366.39** for **Ha3/wo**, **2519.77** for **ECV2/w**, and **3791.25** for **ECV2/wo** [2507.20987]. The benchmark interprets this as evidence that portrait- or semi-body-oriented systems do not transfer cleanly to full-body generation.

A second major result is that **hands are particularly difficult**. Reported hand-region FVD values remain extremely high for talking-avatar methods, including **7750.48**, **10671.79**, **8958.27**, and **9472.28** across the four Hallo3 and EchoMimicV2 variants [2507.20987]. The **face** region also remains difficult: even stronger video models retain high face-region FID, including **254.03** for Wan and **275.50** for Open-Sora 2.0 [2507.20987]. This suggests that current systems do not yet provide uniformly strong generation across the face-hand-body hierarchy.

Pose guidance yields substantial gains. For **Hallo3**, reported **whole-body SC** rises from **10.74** without pose guidance to **20.20** with guidance, while **whole-body FVD** improves from **4366.39** to **2470.14** [2507.20987]. For **EchoMimicV2**, **whole-body SC** rises from **7.08** to **29.16**, and **whole-body FVD** improves from **3791.25** to **2519.77** [2507.20987]. This suggests that pose conditioning remains a strong control signal even though it does not close the performance gap to leading general video generators.

The audio results indicate that **low WER does not fully capture perceived speech quality**. **11Labs multilingual v2** achieves the best reported **WER** on emotions at **4.68**, while **gpt-4o-mini-audio-preview** achieves the highest reported **win-rates**, including **58.31%** on emotions and **60.68%** on paralinguistics [2507.20987]. WholeBodyBenchmark thereby treats speech evaluation as a perceptual and expressive problem, not solely an ASR-reconstruction problem.

## 5. Position within the broader landscape of whole-body benchmarks

WholeBodyBenchmark belongs to a broader but highly heterogeneous family of “whole-body” benchmarks. In **H3WB**, the term denotes **3D human whole-body pose estimation** with **133 keypoints** spanning body, face, hands, and feet, together with benchmark tasks for lifting and monocular estimation [2211.15692]. In **M3Bench**, it denotes **whole-body motion generation for mobile manipulation in 3D scenes**, where success depends on coordinated base-arm trajectories in household environments [2410.06678]. In **HumanoidBench**, it denotes **simulated humanoid whole-body locomotion and manipulation** across **27 tasks** on a high-dimensional embodied platform [2403.10506].

Against that backdrop, WholeBodyBenchmark occupies a distinct subdomain: **whole-body digital human generation with synchronized speech**. This suggests that the phrase “whole body” is not a single technical category but a family resemblance across benchmarks whose common property is that isolated subsystems—face only, arm only, locomotion only, audio only—are insufficient representations of the target problem.

## 6. Versioning, limitations, and prospective development

WholeBodyBenchmark is explicitly a **Version I** release [2507.20987]. Its current limitations are stated with unusual clarity. The release does **not yet fully support stable end-to-end joint audio-video evaluation from a single initial frame**, and current state-of-the-art joint generation models are described as still unstable, which is the reason **Veo-3** is excluded [2507.20987]. The text also notes that the benchmark appears to focus on shorter clips in its present form.

Several infrastructural omissions remain salient. The release summary does not discuss **dataset license**, **access approval requirements**, **privacy safeguards**, **consent procedures**, or **intended-use restrictions** for a dataset containing **10,000 unique identities** [2507.20987]. It also does not fully enumerate prompt templates, generation lengths, or compute-budget normalization across all evaluated models. These absences do not invalidate the benchmark, but they delimit the interpretability and governance maturity of Version I.

The stated future directions are concrete: later releases are intended to support **60-second clips**, include **refined audio-video annotations**, and evaluate **stable joint generation models from a single initial frame** [2507.20987]. In that sense, WholeBodyBenchmark is best understood as a foundational benchmark release whose principal contribution is not the claim that whole-body talking-avatar generation is solved, but the demonstration that current methods remain substantially below a robust standard once evaluation becomes region-aware, multimodal, and explicitly whole-body.

Source: https://www.emergentmind.com/topics/wholebodybenchmark