Ges-QA: Audio-to-3D Gesture Quality Dataset

• Ges-QA is a multidimensional quality assessment dataset for audio-to-3D gesture generation that captures gesture naturalness, audio–gesture consistency, and emotion congruence.
• It consists of 1,400 samples derived from 200 audio clips and seven motion variants per clip, enabling direct comparison between generated and ground-truth gestures.
• The dataset is paired with Ges-QAer, a tri-modal transformer-based assessor that fuses video, audio, and 3D motion data to predict perceptual quality scores.

Ges-QA is a multidimensional quality assessment dataset for audio-to-3D gesture generation (A2G) that was introduced to measure human-perceived gesture quality, audio–gesture consistency, and emotion congruence more directly than conventional automatic metrics permit (Gao et al., 16 Aug 2025). It contains 1,400 samples built from 200 audio clips and seven motion variants per clip, and it is paired with Ges-QAer, a tri-modal transformer-based assessor that operates on video, audio, and 3D skeleton modalities. The dataset is positioned around a specific methodological claim: existing A2G metrics such as Fréchet Gesture Distance and Beat Constancy do not adequately reflect human preference, especially for local temporal artifacts, motion naturalness, and speech–action synchronization.

1. Motivation and evaluation problem

Ges-QA was created in response to limitations in prevailing A2G evaluation practice (Gao et al., 16 Aug 2025). The paper states that Fréchet Gesture Distance (FGD) “only computes the distributional similarity between the ground truth gestures and generated gestures,” while Beat Constancy (BC) “only uses the amplitude of changes in joints and audio to measure the audio-visual consistency.” On that characterization, both metrics omit perceptual properties that matter to human observers, including naturalness, realism, expressivity, unnatural limb movements across temporally adjacent frames, and speech–action desynchronization.

The dataset therefore reframes A2G evaluation as a human-centered, multidimensional quality assessment problem rather than a purely distributional or rhythm-alignment problem. This suggests a shift from proxy metrics toward supervised quality prediction from synchronized multimodal evidence. The paper further argues that objective assessment should distinguish at least two dimensions: perceived gesture visual quality and audio–gesture consistency, while also tracking whether the gesture matches the emotion conveyed by the speech.

2. Dataset composition and source material

Ges-QA contains 1,400 A2G samples, constructed as 200 audio clips multiplied by seven motion variants per clip: six generation approaches plus ground truth (Gao et al., 16 Aug 2025). The motion-generation approaches included are EMAGE, MambaTalk, Syntalker, Language of Motion (LoM), MotionCraft, and GestureLSM; the seventh variant is the ground-truth (GT) motion from BEAT2.

The source audio comes from the BEAT2 dataset, described as containing 60 hours of English speech from 25 speakers. Ges-QA selects 25 sequences for each of eight emotions, with each sequence lasting 10 seconds. The paper notes a diversity control measure: “only four recordings come from the same two speakers.” The emotion taxonomy is not fully enumerated in the provided description, although happiness, contempt, sadness, and fear are explicitly mentioned as examples.

Each sample is multimodal. It includes the speech audio, a rendered A2G motion video, and retained 3D skeleton parameters represented as SMPL-X parameters. Details such as joint count, frame rate, coordinate system, units, audio sampling rate, and video resolution are not specified. Training and evaluation are conducted with 5-fold cross-validation over the full dataset; exact train/validation/test split files or ratios are not specified.

Component	Description
Total samples	1,400
Construction	200 audio clips × 7 motion variants
Audio source	BEAT2
Source language	English
Speakers	25
Emotions	8 categories
Clip duration	10 seconds
Motion variants	EMAGE, MambaTalk, Syntalker, LoM, MotionCraft, GestureLSM, GT
Modalities	Audio, rendered video, SMPL-X motion

The dataset’s composition makes it suitable for direct comparison among generated motions for the same speech input. Because GT motions are included alongside generated outputs, the benchmark can be used both for relative ranking of generators and for learning perceptual assessment models.

3. Annotation schema and score processing

Ges-QA provides two continuous Mean Opinion Score (MOS) dimensions and one binary emotion-congruence label (Gao et al., 16 Aug 2025). The first MOS dimension is gesture visual quality, defined as perceived motion quality, including naturalness and resemblance to real-world human movement. The second is audio–gesture consistency, defined as the degree to which motion is consistent with the rhythm and timing of speech, with emphasis on synchronization rather than semantics. The binary label is ESBA (Subject Binary Annotation for Emotion), which indicates whether gestures and speech express the same emotion.

Eighteen subjects participated in the annotation process. The interface used two Likert scale sliders for the two MOS dimensions, although the exact Likert range is not stated. The raw Likert scores were converted to Z-scores and then rescaled to the interval $[0,100]$, after which MOS values were computed by averaging the normalized Z-scores. For ESBA, the final label was determined by majority voting after outlier exclusion. The paper does not specify the assignment scheme for raters per sample, and it does not report inter-rater reliability statistics such as Cohen’s $\kappa$, Fleiss’ $\kappa$, ICC, or confidence intervals.

This annotation design separates temporal alignment from visual plausibility, which is important because the two dimensions need not coincide. A motion can be rhythmically aligned with speech while still appearing unnatural, and conversely a visually plausible motion can be poorly synchronized. The ESBA label adds a third axis concerned with affective alignment rather than motion quality or timing alone.

4. Benchmark protocol and Ges-QAer architecture

The benchmark evaluates quality-prediction methods with SRCC, PLCC, KRCC, and RMSE under 5-fold cross-validation (Gao et al., 16 Aug 2025). The paper gives the following standard definitions:

$$r_{\text{PLCC}}=\frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i-\bar{x})^2}\sqrt{\sum_{i=1}^{n}(y_i-\bar{y})^2}}$$

$$\rho_{\text{SRCC}}=1-\frac{6\sum_{i=1}^{n} d_i^2}{n(n^2-1)}$$

$$\tau_{\text{KRCC}}=\frac{(\text{number of concordant pairs})-(\text{number of discordant pairs})}{\tfrac{1}{2}n(n-1)}$$

$$\mathrm{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2}$$

The evaluated baselines include AVID-CMA, VAST, ImageBind, DNN-RNT, DNN-SND, and GeneralAVQA. Ges-QAer is the paper’s proposed assessor. Its architecture uses three single-modality encoders: a Video Swin Transformer for video, an Audio Spectrogram Transformer (AST) pre-trained on AudioSet for audio, and a transformer-based motion encoder for temporally sequenced SMPL-X parameters. The video encoder produces features $F_v \in \mathbb{R}^{B \times N_v \times C}$ and uses 3D shifted-window multi-head self-attention. The audio encoder operates on Hamming-windowed log-Mel filterbanks segmented into 5-second clips and produces $F_a \in \mathbb{R}^{B \times N_a \times C}$. The motion branch introduces an extra learnable token, analogous to a classification token, and produces $F_m \in \mathbb{R}^{B \times 1 \times C}$.

Fusion is performed after pooling the vision and audio features to reduce temporal dimension and align them with the motion representation. The pooled features are projected into a common space and fused by a fully connected multilayer perceptron, yielding two continuous outputs: a gesture quality score and an audio–gesture consistency score. The paper also prints a PLCC-based learning objective,

$\kappa$0

while noting, in effect, that the expression appears truncated or format-corrupted as printed. No corrected version is provided.

Training is performed in PyTorch on two NVIDIA RTX 3090 GPUs with learning rate $\kappa$1, warm-up plus linear decay, batch size 10, and 20 epochs. Positional encodings, hidden sizes, attention heads, number of layers, data augmentation, and regularization are not specified.

5. Empirical results and ablation findings

Ges-QAer reports state-of-the-art performance on both annotated dimensions of Ges-QA (Gao et al., 16 Aug 2025). On gesture quality, it achieves SRCC 0.9282, PLCC 0.9352, KRCC 0.7687, and RMSE 4.9838. On audio–gesture consistency, it achieves SRCC 0.8795, PLCC 0.9344, KRCC 0.7070, and RMSE 4.9749. The best baseline in both dimensions is GeneralAVQA, with gesture-quality metrics SRCC 0.8122, PLCC 0.8734, KRCC 0.6289, RMSE 6.8610, and audio–gesture-consistency metrics SRCC 0.7845, PLCC 0.8913, KRCC 0.6014, RMSE 6.3502. The authors report minimum SRCC improvements of 14.3% for gesture quality and 12.1% for audio–gesture consistency.

Method / setting	Gesture quality	Audio–gesture consistency
Ges-QAer	SRCC 0.9282; PLCC 0.9352; KRCC 0.7687; RMSE 4.9838	SRCC 0.8795; PLCC 0.9344; KRCC 0.7070; RMSE 4.9749
Best baseline (GeneralAVQA)	SRCC 0.8122; PLCC 0.8734; KRCC 0.6289; RMSE 6.8610	SRCC 0.7845; PLCC 0.8913; KRCC 0.6014; RMSE 6.3502
Without video encoder	SRCC 0.0943; PLCC 0.1229; KRCC 0.0643; RMSE 13.9742	SRCC 0.0863; PLCC 0.1278; KRCC 0.0587; RMSE 13.8898
Without motion encoder	SRCC 0.9280; PLCC 0.9324; KRCC 0.7698; RMSE 5.0822	SRCC 0.8760; PLCC 0.9310; KRCC 0.7015; RMSE 5.1044

The ablation study is especially informative. Removing the video encoder causes a severe degradation on both tasks, which the paper interprets as evidence that video-based perceptual cues remain central for quality assessment. Removing the motion encoder yields a much smaller decline, placing the system closer to conventional audio–video quality-assessment paradigms while still slightly harming performance. This suggests that explicit motion representation contributes additional but not dominant information beyond rendered video and audio.

The paper also discusses a qualitative pattern: ESBA accuracy is strongly positively correlated with MOS values across methods and emotions. It further notes that sadness and fear are harder to express, whereas happiness and contempt are easier to convey through body language. These observations are presented qualitatively rather than through a dedicated quantitative error breakdown.

6. Reproducibility, limitations, and naming context

Ges-QA is described as the first A2G-focused quality assessment dataset with 1,400 samples, but several reproducibility details remain unspecified in the paper (Gao et al., 16 Aug 2025). No URL, versioned release location, or code repository is provided. License terms are not specified. File organization, naming conventions, video format, audio format, exact frame sampling for the video branch, and exact segmentation policy for 10-second clips are also not given. The dataset retains SMPL-X parameters, but the number of joints, frame rate, coordinate system, and normalization protocol are not specified. The paper does not report random seeds, detailed dependency versions, or full architectural hyperparameters for the encoders.

The annotation protocol likewise leaves open issues. The number of raters per sample is not reported, and inter-rater agreement statistics are absent. Ethical considerations, consent, privacy, and bias analysis are not discussed beyond noting that the audio originates from BEAT2 and that public or pretrained generation methods were used. These omissions do not negate the dataset’s value, but they constrain strict reproduction and secondary analysis.

The name “Ges-QA” should also be distinguished from several unrelated datasets with superficially similar abbreviations. These include the geography scenario-based QA benchmark GeoSQA (Huang et al., 2019), the graph reasoning dataset GRS-QA (Pahilajani et al., 2024), the geospatial benchmark GS-QA (Saeedan et al., 21 May 2026), and the Gaussian Splatting quality dataset GScomp-QA (Martin et al., 26 May 2026). In contrast, Ges-QA denotes the A2G quality-assessment resource introduced for multidimensional human evaluation of generated 3D gestures (Gao et al., 16 Aug 2025).