CSG-405: Co-Speech Gesture Dataset

• CSG-405 is a large-scale co-speech gesture dataset featuring 405 hours of high-resolution video, 147,550 clips, and 133-keypoint full-body annotations.
• It employs synchronized 2D full-body skeletons derived via DWPose to align audio with gesture, effectively addressing the one-to-many ambiguity in gesture generation.
• The dataset supports both paired and unpaired evaluation protocols, enabling robust testing of audio-to-motion diffusion models and downstream human video synthesis methods.

CSG-405, short for Co-Speech Gesture-405, is a public dataset for high-fidelity co-speech gesture video generation introduced together with a lightweight generation framework that uses 2D full-body skeletons as an auxiliary condition between audio and video. It is defined by 405 hours of high-resolution videos across 71 speech types, with 147,550 clips, 17,787 speakers as estimated by a face understanding model, 512×512 cropped RGB video at 25 FPS, 16 kHz audio, and per-frame 133-keypoint COCO-WholeBody annotations with confidence scores (Yang et al., 9 Jul 2025). The dataset is intended to address the one-to-many ambiguity between speech and gesture while supporting research on expressive, synchronized, full-body motion across diverse speech scenarios and speaker demographics.

1. Corpus definition and scale

CSG-405 is positioned as a large-scale public corpus for co-speech gesture video generation. Its stated goal is to democratize research by providing a dataset that is substantially larger than prior public resources such as PATS at approximately 13 h, 4 speakers and TED-talks at approximately 3 h. The dataset is described as the first public dataset at this scale specifically curated for co-speech gesture video generation (Yang et al., 9 Jul 2025).

The corpus combines broad speech coverage with full-body pose supervision. The speech taxonomy contains 71 categories, including lectures, speeches, debates, chats, and vlogs, and the dataset spans both formal and casual contexts. The paper further states that the collection includes a wide range of genders, ethnicities, age groups, and emotions, with identities and attributes estimated using DeepFace.

Attribute	Value	Notes
Total duration	405 hours	Public dataset scale
Number of clips	147,550	Clip duration is 5–15 s
Speakers	17,787	Estimated via a face understanding model
Speech types	71	Enumerated via GPT-4o prompts
Video format	512×512, 25 FPS	Cropped around the speaker
Audio format	16 kHz	Aligned to video
Skeleton annotation	133 keypoints	COCO-WholeBody with confidence

The clip duration is 5–15 s, with mean duration 9.88 s. Source videos are downloaded at 1280×720, then cropped and resized around the speaker to 512×512. The annotation modality is dense: each frame contains 133 $(x, y, \text{confidence})$ entries. The paper does not specify/provide transcripts, and it does not state whether identity labels are released as metadata (Yang et al., 9 Jul 2025).

This scale and annotation density suggest that CSG-405 is intended not merely as a benchmark set, but as a training corpus for models that must jointly learn audio–gesture synchronization, speaker-conditioned body motion, and full-body spatiotemporal structure.

2. Collection, filtering, and preprocessing pipeline

The collection pipeline begins with a speech-type list enumerated via GPT-4o, followed by retrieval of public videos from YouTube at 1280×720. Raw videos are segmented into clips through shot-change detection, specifically by detecting camera motion/color changes and splitting them into 5–15 s segments (Yang et al., 9 Jul 2025).

Several filtering stages are then applied. PyAnnote is used for speaker diarization and overlap removal, excluding clips with simultaneous speakers. SyncNet is used for audio–lip sync filtering, removing misaligned clips. The pose extractor is DWPose, which outputs 133 keypoints per frame in COCO-WholeBody layout with per-keypoint confidence. When multiple persons appear, each person’s skeleton is processed independently, and clips with overlapping speech are removed. Additional filters exclude clips with missing upper-body keypoints, side/back views, very small figures, or mostly static poses (Yang et al., 9 Jul 2025).

The dataset then undergoes post-processing. The speaker is cropped with margins and resized to 512×512 at 25 FPS. Skeleton coordinates are transformed into a local motion representation, and temporal smoothing is applied to all keypoints except the mouth. The stated reason is to reduce jitter while preserving lip dynamics.

These design choices define the statistical character of the corpus. The paper explicitly notes that filtering for frontal visibility, single-speaker segments, and unobstructed upper bodies may bias the dataset toward frontal, single-speaker monologues. A plausible implication is that models trained exclusively on CSG-405 may require additional adaptation for conversational scenes with strong occlusion, profile views, or multi-speaker interaction.

3. Annotation schema and motion representation

The annotation core of CSG-405 is the 2D full-body skeleton sequence. The paper uses 133 COCO-WholeBody keypoints, covering face, hands, and body, with 2D image coordinates and a confidence score for each keypoint (Yang et al., 9 Jul 2025). The ordering follows the standard COCO-WholeBody layout.

A notable representational choice is the conversion from raw image coordinates to a local motion representation. Different body parts are expressed relative to different roots:

• Face: relative to the nose tip
• Each hand: relative to the corresponding wrist
• Remaining body parts: relative to the neck

This decouples global body motion from subtle local articulation. The paper uses temporal smoothing on all keypoints except the mouth, again to preserve lip motion while reducing jitter elsewhere.

The dataset also enforces temporal synchronization between modalities. Skeletons are aligned frame-by-frame to 25 FPS video, and audio–video sync is enforced during curation using SyncNet. The paper further notes that audio segments can be extracted with the same cadence, yielding $F$ audio segments for $F$ frames, which matches the conditioning strategy used by the associated model (Yang et al., 9 Jul 2025).

This representation is tightly coupled to the downstream formulation. The paper defines:

$S_g = G_s(s_r, A), \qquad V = G_v(S_g, R),$

where $A$ is the audio, $R$ is the reference image, $s_r \in \mathbb{R}^{1 \times 2K}$ is the reference skeleton, $S_g \in \mathbb{R}^{F \times 2K}$ is the generated skeletal sequence, and $\hat{A} \in \mathbb{R}^{F \times 768}$ denotes wav2vec 2.0 audio features. Here $K$ corresponds to the 2D coordinates of the skeleton representation, and $F$0 is the number of frames.

Because the corpus stores aligned audio and per-frame skeletons, it directly supports learning problems in which segment-level audio features are matched to frame-level gesture states. This suggests that CSG-405 is structured less like a generic video dataset and more like a synchronized multimodal supervision resource.

4. Benchmark protocols and reported empirical results

The paper uses a 90%/10% train-validation split, reported as approximately 364 h for training and approximately 41 h for validation, with a test set of 133 clips corresponding to approximately 0.5 h (Yang et al., 9 Jul 2025). Two evaluation regimes are defined. In the paired setting, audio and reference image come from the same video. In the unpaired setting, audio and reference image come from different videos. The paper describes the unpaired setting as more realistic for deployment.

Reported metrics include SSIM and PSNR for visual fidelity in paired evaluation; FID and FVD for image and video fidelity in unpaired evaluation; CSIM for identity consistency; and SyncNet-based Sync-C and Sync-D for audio–visual synchronization. The Fréchet formulation is stated as

$F$1

with Inception-v3 features for FID and I3D features for FVD.

On the unpaired benchmark, the reported rows are:

• S2G-MDDiffusion: FID 216.80, FVD 2538.70, Sync-C 2.48, Sync-D 11.54, SSIM 0.55, PSNR 14.02, CSIM 0.39
• DiffTED: FID 138.82, FVD 1852.10, Sync-C 0.66, Sync-D 13.06, SSIM 0.67, PSNR 15.98, CSIM 0.61
• MimicMotion: FID 115.65, FVD 1751.33, Sync-C 3.26, Sync-D 11.12, SSIM 0.75, PSNR 20.47, CSIM 0.60
• EchoMimicV2: FID 90.01, FVD 1444.00, Sync-C 6.53, Sync-D 8.09, SSIM 0.71, PSNR 17.88, CSIM 0.82
• StableAnimator: FID 94.94, FVD 1532.74, Sync-C 5.13, Sync-D 9.50, SSIM 0.83, PSNR 24.27, CSIM 0.67
• Ours (MimicMotion): FID 73.77, FVD 1264.22, Sync-C 4.43, Sync-D 10.36, SSIM 0.59, PSNR 14.98, CSIM 0.76
• Ours (EchoMimicV2): FID 70.03, FVD 1111.61, Sync-C 6.59, Sync-D 8.04, SSIM 0.69, PSNR 17.02, CSIM 0.82
• Ours (StableAnimator): FID 66.47, FVD 988.43, Sync-C 5.82, Sync-D 9.11, SSIM 0.65, PSNR 16.26, CSIM 0.81
• Ours† (StableAnimator fine-tuned on CSG-405): FID 62.93, FVD 984.13, Sync-C 6.28, Sync-D 8.68, SSIM 0.67, PSNR 16.62, CSIM 0.82

The paper’s stated takeaway is that CSG-405 enables strong unpaired performance and generalization, and that fine-tuning a general human video model on CSG-405 further improves FID/FVD and synchronization quality. It also states that EchoMimicV2 variants achieve the strongest raw lip-sync metrics, while the CSG-405-trained StableAnimator variant attains the best overall fidelity under FID/FVD (Yang et al., 9 Jul 2025).

5. Role in the associated diffusion framework

CSG-405 is introduced together with a two-stage generation framework. The first stage is an audio-to-skeleton diffusion model $F$2; the second stage is an off-the-shelf human video generator $F$3 that takes the generated skeletons and a reference image to synthesize video (Yang et al., 9 Jul 2025). The dataset provides the aligned triplets required by this decomposition: audio, per-frame skeletons, and speaker crops.

The diffusion objective is written in $F$4-prediction form:

$F$5

where $F$6 is the ground-truth skeleton sequence and $F$7 is its noisy version at diffusion step $F$8. The paper also provides the standard forward noising notation,

$F$9

$F$0

with $F$1.

Classifier-free guidance is defined as

$F$2

The conditioning design is specific. The reference skeleton $F$3 is concatenated with $F$4 along the frame dimension, so that self-attention can enforce body-shape consistency. Segment-level wav2vec 2.0 features $F$5 are concatenated with skeleton embeddings along the feature dimension before DiT blocks, which the paper reports as outperforming cross-attention. The stated rationale is that this creates a one-to-one correspondence between audio segments and gesture frames.

Implementation details reported for this framework are: 4×V100 (32 GB) GPUs, batch size 128, Adam optimizer with learning rate $F$6, and 2,000,000 steps. Downstream video generators include StableAnimator, EchoMimicV2, and MimicMotion. For EchoMimicV2, which expects only hand skeletons, the necessary hand keypoints are extracted from the full-body sequence $F$7 (Yang et al., 9 Jul 2025).

Within this setup, CSG-405 functions as both a dataset and a design constraint: its synchronized frame-level skeleton supervision and fixed 25 FPS / 16 kHz alignment directly determine the architecture’s segmentation and conditioning scheme.

6. Limitations, bias, access, and significance

The paper identifies several limitations. Because the videos are collected from public online content, the paper states that license/consent details are not provided and advises users to consult the repository and comply with platform TOS and local laws (Yang et al., 9 Jul 2025). It also notes that the filtering stages bias the corpus toward frontal, single-speaker, well-lit, and unobstructed upper-body scenes. Languages and accents are not labeled, and the paper reports no transcripts or semantic labels beyond speech-type categories.

There are also technical constraints. The dataset is standardized to 25 FPS and 16 kHz, which may require resampling for models with different I/O conventions. The annotations are 2D keypoints, which can miss depth and subtle articulations under occlusion. The paper further notes that small or texture-poor objects are not the issue here; rather, the corpus is limited by the visibility assumptions built into pose extraction and cleaning.

Access is mediated through the project page:

• https://mpi-lab.github.io/Democratizing-CSG/

The paper states that code, models, skeletons, tools, and links are provided there, but it does not specify the exact download size, directory structure, or on-disk file format in the manuscript itself.

In research terms, CSG-405 occupies a specific niche. It is a whole-body, audio-aligned, public co-speech gesture corpus designed for speaker-conditioned generation rather than merely pose estimation or talking-head synthesis. Its combination of 405 hours, 133-keypoint whole-body supervision, 71 speech types, and unpaired evaluation protocols suggests a benchmark oriented toward models that must generalize across speakers, contexts, and reference-image/audio mismatches. A plausible implication is that CSG-405 is most consequential for methods that decouple generation into audio-to-motion and motion-to-video stages, because the dataset is organized precisely around that factorization.