---
title: 'Ges-QA: Audio-to-3D Gesture Quality Dataset'
url: https://www.emergentmind.com/topics/ges-qa-dataset
type: topic
---

# Ges-QA: Audio-to-3D Gesture Quality Dataset

Ges-QA is a multidimensional quality assessment dataset for audio-to-3D gesture generation (A2G) that was introduced to measure human-perceived gesture quality, audio–gesture consistency, and emotion congruence more directly than conventional automatic metrics permit [2508.12020]. It contains 1,400 samples built from 200 audio clips and seven motion variants per clip, and it is paired with Ges-QAer, a tri-modal transformer-based assessor that operates on video, audio, and 3D skeleton modalities. The dataset is positioned around a specific methodological claim: existing A2G metrics such as Fréchet Gesture Distance and Beat Constancy do not adequately reflect human preference, especially for local temporal artifacts, motion naturalness, and speech–action synchronization.

## 1. Motivation and evaluation problem

Ges-QA was created in response to limitations in prevailing A2G evaluation practice [2508.12020]. The paper states that Fréchet Gesture Distance (FGD) “only computes the distributional similarity between the ground truth gestures and generated gestures,” while Beat Constancy (BC) “only uses the amplitude of changes in joints and audio to measure the audio-visual consistency.” On that characterization, both metrics omit perceptual properties that matter to human observers, including naturalness, realism, expressivity, unnatural limb movements across temporally adjacent frames, and speech–action desynchronization.

The dataset therefore reframes A2G evaluation as a human-centered, multidimensional quality assessment problem rather than a purely distributional or rhythm-alignment problem. This suggests a shift from proxy metrics toward supervised quality prediction from synchronized multimodal evidence. The paper further argues that objective assessment should distinguish at least two dimensions: perceived gesture visual quality and audio–gesture consistency, while also tracking whether the gesture matches the emotion conveyed by the speech.

## 2. Dataset composition and source material

Ges-QA contains 1,400 A2G samples, constructed as 200 audio clips multiplied by seven motion variants per clip: six generation approaches plus ground truth [2508.12020]. The motion-generation approaches included are EMAGE, MambaTalk, Syntalker, Language of Motion (LoM), MotionCraft, and GestureLSM; the seventh variant is the ground-truth (GT) motion from BEAT2.

The source audio comes from the BEAT2 dataset, described as containing 60 hours of English speech from 25 speakers. Ges-QA selects 25 sequences for each of eight emotions, with each sequence lasting 10 seconds. The paper notes a diversity control measure: “only four recordings come from the same two speakers.” The emotion taxonomy is not fully enumerated in the provided description, although happiness, contempt, sadness, and fear are explicitly mentioned as examples.

Each sample is multimodal. It includes the speech audio, a rendered A2G motion video, and retained 3D skeleton parameters represented as SMPL-X parameters. Details such as joint count, frame rate, coordinate system, units, audio sampling rate, and video resolution are not specified. Training and evaluation are conducted with 5-fold cross-validation over the full dataset; exact train/validation/test split files or ratios are not specified.

| Component | Description |
|---|---|
| Total samples | 1,400 |
| Construction | 200 audio clips × 7 motion variants |
| Audio source | BEAT2 |
| Source language | English |
| Speakers | 25 |
| Emotions | 8 categories |
| Clip duration | 10 seconds |
| Motion variants | EMAGE, MambaTalk, Syntalker, LoM, MotionCraft, GestureLSM, GT |
| Modalities | Audio, rendered video, SMPL-X motion |

The dataset’s composition makes it suitable for direct comparison among generated motions for the same speech input. Because GT motions are included alongside generated outputs, the benchmark can be used both for relative ranking of generators and for learning perceptual assessment models.

## 3. Annotation schema and score processing

Ges-QA provides two continuous Mean Opinion Score (MOS) dimensions and one binary emotion-congruence label [2508.12020]. The first MOS dimension is **gesture visual quality**, defined as perceived motion quality, including naturalness and resemblance to real-world human movement. The second is **audio–gesture consistency**, defined as the degree to which motion is consistent with the rhythm and timing of speech, with emphasis on synchronization rather than semantics. The binary label is **ESBA** (Subject Binary Annotation for Emotion), which indicates whether gestures and speech express the same emotion.

Eighteen subjects participated in the annotation process. The interface used two Likert scale sliders for the two MOS dimensions, although the exact Likert range is not stated. The raw Likert scores were converted to Z-scores and then rescaled to the interval $[0,100]$, after which MOS values were computed by averaging the normalized Z-scores. For ESBA, the final label was determined by majority voting after outlier exclusion. The paper does not specify the assignment scheme for raters per sample, and it does not report inter-rater reliability statistics such as Cohen’s $\kappa$, Fleiss’ $\kappa$, ICC, or confidence intervals.

This annotation design separates temporal alignment from visual plausibility, which is important because the two dimensions need not coincide. A motion can be rhythmically aligned with speech while still appearing unnatural, and conversely a visually plausible motion can be poorly synchronized. The ESBA label adds a third axis concerned with affective alignment rather than motion quality or timing alone.

## 4. Benchmark protocol and Ges-QAer architecture

The benchmark evaluates quality-prediction methods with SRCC, PLCC, KRCC, and RMSE under 5-fold cross-validation [2508.12020]. The paper gives the following standard definitions:
$$
r_{\text{PLCC}}=\frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i-\bar{x})^2}\sqrt{\sum_{i=1}^{n}(y_i-\bar{y})^2}}
$$

$$
\rho_{\text{SRCC}}=1-\frac{6\sum_{i=1}^{n} d_i^2}{n(n^2-1)}
$$

$$
\tau_{\text{KRCC}}=\frac{(\text{number of concordant pairs})-(\text{number of discordant pairs})}{\tfrac{1}{2}n(n-1)}
$$

$$
\mathrm{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}
$$

The evaluated baselines include AVID-CMA, VAST, ImageBind, DNN-RNT, DNN-SND, and GeneralAVQA. Ges-QAer is the paper’s proposed assessor. Its architecture uses three single-modality encoders: a Video Swin Transformer for video, an Audio Spectrogram Transformer (AST) pre-trained on AudioSet for audio, and a transformer-based motion encoder for temporally sequenced SMPL-X parameters. The video encoder produces features $F_v \in \mathbb{R}^{B \times N_v \times C}$ and uses 3D shifted-window multi-head self-attention. The audio encoder operates on Hamming-windowed log-Mel filterbanks segmented into 5-second clips and produces $F_a \in \mathbb{R}^{B \times N_a \times C}$. The motion branch introduces an extra learnable token, analogous to a classification token, and produces $F_m \in \mathbb{R}^{B \times 1 \times C}$.

Fusion is performed after pooling the vision and audio features to reduce temporal dimension and align them with the motion representation. The pooled features are projected into a common space and fused by a fully connected multilayer perceptron, yielding two continuous outputs: a gesture quality score and an audio–gesture consistency score. The paper also prints a PLCC-based learning objective,
$$
\mathcal{L}_{\text{PLCC} = \frac{1}{8}[ \text{MSE}(p, t) + \text{MSE}(\operatorname{Cov}(\mathbf{p}, \mathbf{t})\cdotp, t)]
$$
while noting, in effect, that the expression appears truncated or format-corrupted as printed. No corrected version is provided.

Training is performed in PyTorch on two NVIDIA RTX 3090 GPUs with learning rate $10^{-4}$, warm-up plus linear decay, batch size 10, and 20 epochs. Positional encodings, hidden sizes, attention heads, number of layers, data augmentation, and regularization are not specified.

## 5. Empirical results and ablation findings

Ges-QAer reports state-of-the-art performance on both annotated dimensions of Ges-QA [2508.12020]. On gesture quality, it achieves SRCC 0.9282, PLCC 0.9352, KRCC 0.7687, and RMSE 4.9838. On audio–gesture consistency, it achieves SRCC 0.8795, PLCC 0.9344, KRCC 0.7070, and RMSE 4.9749. The best baseline in both dimensions is GeneralAVQA, with gesture-quality metrics SRCC 0.8122, PLCC 0.8734, KRCC 0.6289, RMSE 6.8610, and audio–gesture-consistency metrics SRCC 0.7845, PLCC 0.8913, KRCC 0.6014, RMSE 6.3502. The authors report minimum SRCC improvements of 14.3% for gesture quality and 12.1% for audio–gesture consistency.

| Method / setting | Gesture quality | Audio–gesture consistency |
|---|---|---|
| Ges-QAer | SRCC 0.9282; PLCC 0.9352; KRCC 0.7687; RMSE 4.9838 | SRCC 0.8795; PLCC 0.9344; KRCC 0.7070; RMSE 4.9749 |
| Best baseline (GeneralAVQA) | SRCC 0.8122; PLCC 0.8734; KRCC 0.6289; RMSE 6.8610 | SRCC 0.7845; PLCC 0.8913; KRCC 0.6014; RMSE 6.3502 |
| Without video encoder | SRCC 0.0943; PLCC 0.1229; KRCC 0.0643; RMSE 13.9742 | SRCC 0.0863; PLCC 0.1278; KRCC 0.0587; RMSE 13.8898 |
| Without motion encoder | SRCC 0.9280; PLCC 0.9324; KRCC 0.7698; RMSE 5.0822 | SRCC 0.8760; PLCC 0.9310; KRCC 0.7015; RMSE 5.1044 |

The ablation study is especially informative. Removing the video encoder causes a severe degradation on both tasks, which the paper interprets as evidence that video-based perceptual cues remain central for quality assessment. Removing the motion encoder yields a much smaller decline, placing the system closer to conventional audio–video quality-assessment paradigms while still slightly harming performance. This suggests that explicit motion representation contributes additional but not dominant information beyond rendered video and audio.

The paper also discusses a qualitative pattern: ESBA accuracy is strongly positively correlated with MOS values across methods and emotions. It further notes that sadness and fear are harder to express, whereas happiness and contempt are easier to convey through body language. These observations are presented qualitatively rather than through a dedicated quantitative error breakdown.

## 6. Reproducibility, limitations, and naming context

Ges-QA is described as the first A2G-focused quality assessment dataset with 1,400 samples, but several reproducibility details remain unspecified in the paper [2508.12020]. No URL, versioned release location, or code repository is provided. License terms are not specified. File organization, naming conventions, video format, audio format, exact frame sampling for the video branch, and exact segmentation policy for 10-second clips are also not given. The dataset retains SMPL-X parameters, but the number of joints, frame rate, coordinate system, and normalization protocol are not specified. The paper does not report random seeds, detailed dependency versions, or full architectural hyperparameters for the encoders.

The annotation protocol likewise leaves open issues. The number of raters per sample is not reported, and inter-rater agreement statistics are absent. Ethical considerations, consent, privacy, and bias analysis are not discussed beyond noting that the audio originates from BEAT2 and that public or pretrained generation methods were used. These omissions do not negate the dataset’s value, but they constrain strict reproduction and secondary analysis.

The name “Ges-QA” should also be distinguished from several unrelated datasets with superficially similar abbreviations. These include the geography scenario-based QA benchmark GeoSQA [1908.07855], the graph reasoning dataset GRS-QA [2411.00369], the geospatial benchmark GS-QA [2605.22811], and the Gaussian Splatting quality dataset GScomp-QA [2605.26880]. In contrast, Ges-QA denotes the A2G quality-assessment resource introduced for multidimensional human evaluation of generated 3D gestures [2508.12020].

Source: https://www.emergentmind.com/topics/ges-qa-dataset