---
title: 'CSG-405: Co-Speech Gesture Dataset'
url: https://www.emergentmind.com/topics/csg-405
type: topic
---

# CSG-405: Co-Speech Gesture Dataset

CSG-405, short for **Co-Speech Gesture-405**, is a public dataset for **high-fidelity co-speech gesture video generation** introduced together with a lightweight generation framework that uses **2D full-body skeletons** as an auxiliary condition between audio and video. It is defined by **405 hours** of high-resolution videos across **71 speech types**, with **147,550 clips**, **17,787 speakers** as estimated by a face understanding model, **512×512** cropped RGB video at **25 FPS**, **16 kHz** audio, and per-frame **133-keypoint COCO-WholeBody** annotations with confidence scores [2507.06812]. The dataset is intended to address the one-to-many ambiguity between speech and gesture while supporting research on expressive, synchronized, full-body motion across diverse speech scenarios and speaker demographics.

## 1. Corpus definition and scale

CSG-405 is positioned as a large-scale public corpus for co-speech gesture video generation. Its stated goal is to **democratize research** by providing a dataset that is substantially larger than prior public resources such as **PATS** at approximately **13 h, 4 speakers** and **TED-talks** at approximately **3 h**. The dataset is described as the **first public dataset** at this scale specifically curated for co-speech gesture video generation [2507.06812].

The corpus combines broad speech coverage with full-body pose supervision. The speech taxonomy contains **71 categories**, including **lectures, speeches, debates, chats,** and **vlogs**, and the dataset spans both **formal** and **casual** contexts. The paper further states that the collection includes a wide range of **genders, ethnicities, age groups, and emotions**, with identities and attributes estimated using **DeepFace**.

| Attribute | Value | Notes |
|---|---:|---|
| Total duration | 405 hours | Public dataset scale |
| Number of clips | 147,550 | Clip duration is 5–15 s |
| Speakers | 17,787 | Estimated via a face understanding model |
| Speech types | 71 | Enumerated via GPT-4o prompts |
| Video format | 512×512, 25 FPS | Cropped around the speaker |
| Audio format | 16 kHz | Aligned to video |
| Skeleton annotation | 133 keypoints | COCO-WholeBody with confidence |

The clip duration is **5–15 s**, with mean duration **9.88 s**. Source videos are downloaded at **1280×720**, then cropped and resized around the speaker to **512×512**. The annotation modality is dense: each frame contains **133 \((x, y, \text{confidence})\)** entries. The paper does **not specify/provide transcripts**, and it does **not state whether identity labels are released as metadata** [2507.06812].

This scale and annotation density suggest that CSG-405 is intended not merely as a benchmark set, but as a training corpus for models that must jointly learn **audio–gesture synchronization**, **speaker-conditioned body motion**, and **full-body spatiotemporal structure**.

## 2. Collection, filtering, and preprocessing pipeline

The collection pipeline begins with a **speech-type list enumerated via GPT-4o**, followed by retrieval of **public videos** from **YouTube** at **1280×720**. Raw videos are segmented into clips through **shot-change detection**, specifically by detecting **camera motion/color changes** and splitting them into **5–15 s** segments [2507.06812].

Several filtering stages are then applied. **PyAnnote** is used for **speaker diarization and overlap removal**, excluding clips with simultaneous speakers. **SyncNet** is used for **audio–lip sync filtering**, removing misaligned clips. The pose extractor is **DWPose**, which outputs **133 keypoints per frame** in **COCO-WholeBody** layout with per-keypoint confidence. When multiple persons appear, **each person’s skeleton is processed independently**, and clips with **overlapping speech** are removed. Additional filters exclude clips with **missing upper-body keypoints**, **side/back views**, **very small figures**, or **mostly static poses** [2507.06812].

The dataset then undergoes post-processing. The speaker is **cropped with margins** and resized to **512×512** at **25 FPS**. Skeleton coordinates are transformed into a **local motion representation**, and temporal smoothing is applied to all keypoints **except the mouth**. The stated reason is to reduce jitter while preserving lip dynamics.

These design choices define the statistical character of the corpus. The paper explicitly notes that filtering for frontal visibility, single-speaker segments, and unobstructed upper bodies may bias the dataset toward **frontal, single-speaker monologues**. A plausible implication is that models trained exclusively on CSG-405 may require additional adaptation for conversational scenes with strong occlusion, profile views, or multi-speaker interaction.

## 3. Annotation schema and motion representation

The annotation core of CSG-405 is the **2D full-body skeleton** sequence. The paper uses **133 COCO-WholeBody keypoints**, covering **face, hands, and body**, with **2D image coordinates** and a **confidence score** for each keypoint [2507.06812]. The ordering follows the standard COCO-WholeBody layout.

A notable representational choice is the conversion from raw image coordinates to a **local motion representation**. Different body parts are expressed relative to different roots:

- **Face**: relative to the **nose tip**
- **Each hand**: relative to the corresponding **wrist**
- **Remaining body parts**: relative to the **neck**

This decouples **global body motion** from **subtle local articulation**. The paper uses temporal smoothing on all keypoints except the mouth, again to preserve lip motion while reducing jitter elsewhere.

The dataset also enforces temporal synchronization between modalities. Skeletons are aligned frame-by-frame to **25 FPS** video, and **audio–video sync is enforced** during curation using **SyncNet**. The paper further notes that **audio segments can be extracted with the same cadence**, yielding **\(F\)** audio segments for **\(F\)** frames, which matches the conditioning strategy used by the associated model [2507.06812].

This representation is tightly coupled to the downstream formulation. The paper defines:

\[
S_g = G_s(s_r, A), \qquad V = G_v(S_g, R),
\]

where \(A\) is the audio, \(R\) is the reference image, \(s_r \in \mathbb{R}^{1 \times 2K}\) is the reference skeleton, \(S_g \in \mathbb{R}^{F \times 2K}\) is the generated skeletal sequence, and \(\hat{A} \in \mathbb{R}^{F \times 768}\) denotes wav2vec 2.0 audio features. Here \(K\) corresponds to the 2D coordinates of the skeleton representation, and \(F\) is the number of frames.

Because the corpus stores aligned audio and per-frame skeletons, it directly supports learning problems in which **segment-level audio features** are matched to **frame-level gesture states**. This suggests that CSG-405 is structured less like a generic video dataset and more like a synchronized multimodal supervision resource.

## 4. Benchmark protocols and reported empirical results

The paper uses a **90%/10%** train-validation split, reported as approximately **364 h** for training and approximately **41 h** for validation, with a **test set of 133 clips** corresponding to approximately **0.5 h** [2507.06812]. Two evaluation regimes are defined. In the **paired** setting, audio and reference image come from the same video. In the **unpaired** setting, audio and reference image come from different videos. The paper describes the unpaired setting as more realistic for deployment.

Reported metrics include **SSIM** and **PSNR** for visual fidelity in paired evaluation; **FID** and **FVD** for image and video fidelity in unpaired evaluation; **CSIM** for identity consistency; and **SyncNet-based Sync-C** and **Sync-D** for audio–visual synchronization. The Fréchet formulation is stated as

\[
\mathrm{FID/FVD}^2 = \|\mu_r - \mu_g\|_2^2 + \mathrm{Tr}\left(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2}\right),
\]

with **Inception-v3** features for FID and **I3D** features for FVD.

On the unpaired benchmark, the reported rows are:

- **S2G-MDDiffusion**: **FID 216.80**, **FVD 2538.70**, **Sync-C 2.48**, **Sync-D 11.54**, **SSIM 0.55**, **PSNR 14.02**, **CSIM 0.39**
- **DiffTED**: **FID 138.82**, **FVD 1852.10**, **Sync-C 0.66**, **Sync-D 13.06**, **SSIM 0.67**, **PSNR 15.98**, **CSIM 0.61**
- **MimicMotion**: **FID 115.65**, **FVD 1751.33**, **Sync-C 3.26**, **Sync-D 11.12**, **SSIM 0.75**, **PSNR 20.47**, **CSIM 0.60**
- **EchoMimicV2**: **FID 90.01**, **FVD 1444.00**, **Sync-C 6.53**, **Sync-D 8.09**, **SSIM 0.71**, **PSNR 17.88**, **CSIM 0.82**
- **StableAnimator**: **FID 94.94**, **FVD 1532.74**, **Sync-C 5.13**, **Sync-D 9.50**, **SSIM 0.83**, **PSNR 24.27**, **CSIM 0.67**
- **Ours (MimicMotion)**: **FID 73.77**, **FVD 1264.22**, **Sync-C 4.43**, **Sync-D 10.36**, **SSIM 0.59**, **PSNR 14.98**, **CSIM 0.76**
- **Ours (EchoMimicV2)**: **FID 70.03**, **FVD 1111.61**, **Sync-C 6.59**, **Sync-D 8.04**, **SSIM 0.69**, **PSNR 17.02**, **CSIM 0.82**
- **Ours (StableAnimator)**: **FID 66.47**, **FVD 988.43**, **Sync-C 5.82**, **Sync-D 9.11**, **SSIM 0.65**, **PSNR 16.26**, **CSIM 0.81**
- **Ours† (StableAnimator fine-tuned on CSG-405)**: **FID 62.93**, **FVD 984.13**, **Sync-C 6.28**, **Sync-D 8.68**, **SSIM 0.67**, **PSNR 16.62**, **CSIM 0.82**

The paper’s stated takeaway is that CSG-405 enables strong **unpaired performance** and **generalization**, and that fine-tuning a general human video model on CSG-405 further improves **FID/FVD** and synchronization quality. It also states that **EchoMimicV2 variants achieve the strongest raw lip-sync metrics**, while the **CSG-405-trained StableAnimator** variant attains the **best overall fidelity** under **FID/FVD** [2507.06812].

## 5. Role in the associated diffusion framework

CSG-405 is introduced together with a two-stage generation framework. The first stage is an **audio-to-skeleton diffusion model** \(G_s\); the second stage is an **off-the-shelf human video generator** \(G_v\) that takes the generated skeletons and a reference image to synthesize video [2507.06812]. The dataset provides the aligned triplets required by this decomposition: audio, per-frame skeletons, and speaker crops.

The diffusion objective is written in **\(x_0\)-prediction** form:

\[
L = \mathbb{E}_{x_0, t}\big[\|x_0 - G_s(x_t, t, A)\|_2^2\big],
\]

where \(x_0 \in \mathbb{R}^{F \times 2K}\) is the ground-truth skeleton sequence and \(x_t\) is its noisy version at diffusion step \(t\). The paper also provides the standard forward noising notation,

\[
q(x_t \mid x_{t-1}) = \mathcal{N}(\sqrt{\alpha_t} x_{t-1}, (1-\alpha_t) I),
\]

\[
q(x_t \mid x_0) = \mathcal{N}(\sqrt{\bar{\alpha}_t} x_0, (1-\bar{\alpha}_t) I),
\]

with \(\bar{\alpha}_t = \prod_{i=1}^{t} \alpha_i\).

Classifier-free guidance is defined as

\[
G'_s(x_t, t, A) = G_s(x_t, t, \varnothing) + \alpha \big[G_s(x_t, t, A) - G_s(x_t, t, \varnothing)\big].
\]

The conditioning design is specific. The **reference skeleton** \(s_r\) is concatenated with \(x_t\) along the **frame dimension**, so that self-attention can enforce **body-shape consistency**. **Segment-level wav2vec 2.0 features** \(\hat A\) are concatenated with skeleton embeddings along the **feature dimension** before **DiT** blocks, which the paper reports as outperforming **cross-attention**. The stated rationale is that this creates a **one-to-one correspondence** between audio segments and gesture frames.

Implementation details reported for this framework are: **4×V100 (32 GB)** GPUs, **batch size 128**, **Adam** optimizer with **learning rate \(5 \times 10^{-5}\)**, and **2,000,000 steps**. Downstream video generators include **StableAnimator**, **EchoMimicV2**, and **MimicMotion**. For **EchoMimicV2**, which expects only hand skeletons, the necessary hand keypoints are extracted from the full-body sequence \(S_g\) [2507.06812].

Within this setup, CSG-405 functions as both a dataset and a design constraint: its synchronized frame-level skeleton supervision and fixed **25 FPS / 16 kHz** alignment directly determine the architecture’s segmentation and conditioning scheme.

## 6. Limitations, bias, access, and significance

The paper identifies several limitations. Because the videos are collected from **public online content**, the paper states that **license/consent details are not provided** and advises users to consult the repository and comply with **platform TOS** and **local laws** [2507.06812]. It also notes that the filtering stages bias the corpus toward **frontal**, **single-speaker**, **well-lit**, and **unobstructed upper-body** scenes. **Languages and accents are not labeled**, and the paper reports **no transcripts** or semantic labels beyond **speech-type categories**.

There are also technical constraints. The dataset is standardized to **25 FPS** and **16 kHz**, which may require resampling for models with different I/O conventions. The annotations are **2D keypoints**, which can miss **depth** and subtle articulations under occlusion. The paper further notes that small or texture-poor objects are not the issue here; rather, the corpus is limited by the visibility assumptions built into pose extraction and cleaning.

Access is mediated through the project page:

- **https://mpi-lab.github.io/Democratizing-CSG/**

The paper states that code, models, skeletons, tools, and links are provided there, but it does **not specify** the exact **download size**, **directory structure**, or **on-disk file format** in the manuscript itself.

In research terms, CSG-405 occupies a specific niche. It is a **whole-body, audio-aligned, public co-speech gesture corpus** designed for **speaker-conditioned generation** rather than merely pose estimation or talking-head synthesis. Its combination of **405 hours**, **133-keypoint whole-body supervision**, **71 speech types**, and **unpaired evaluation protocols** suggests a benchmark oriented toward models that must generalize across **speakers**, **contexts**, and **reference-image/audio mismatches**. A plausible implication is that CSG-405 is most consequential for methods that decouple generation into **audio-to-motion** and **motion-to-video** stages, because the dataset is organized precisely around that factorization.

Source: https://www.emergentmind.com/topics/csg-405