Ges-QAer: Multimodal A2G Quality Assessment

• Ges-QAer is a multimodal transformer-based neural network that predicts human Mean Opinion Scores for gesture quality and audio–gesture consistency in A2G systems.
• It integrates video, audio, and 3D motion data (SMPL-X) through dedicated encoders and fuses these features with an MLP for robust perceptual evaluation.
• Empirical results demonstrate significant improvements over baseline methods, highlighting the critical role of video cues in assessing naturalness and alignment.

Searching arXiv for the Ges-QA / Ges-QAer paper and closely related A2G evaluation work. Calling arXiv search for "Ges-QA audio-to-3D gesture generation quality assessment". Ges-QAer is a multi-modal transformer-based neural network introduced for objective quality assessment in audio-to-3D-gesture generation (A2G). It was proposed together with the Ges-QA dataset to address a central limitation of earlier A2G evaluation protocols: metrics such as Fréchet Gesture Distance and Beat Constancy do not adequately reflect human preference for generated 3D gestures. In contrast, Ges-QAer is trained to predict human Mean Opinion Scores (MOS) along two perceptual dimensions—gesture quality and audio–gesture consistency—using synchronized video, audio, and 3D skeleton information derived from SMPL-X motion parameters (Gao et al., 16 Aug 2025).

1. Problem setting and conceptual scope

The system is situated in the evaluation layer of A2G pipelines rather than in gesture synthesis itself. The motivating claim is that existing automatic metrics are only partial proxies for perceptual quality. Fréchet Gesture Distance (FGD) measures distributional similarity between generated and ground-truth motions but ignores factors such as naturalness, realism, and semantic or emotional appropriateness. Its formulation is given as

$$\mathrm{FGD} = ||\mu_r - \mu_g||^2 + \mathrm{Tr}\!\left(C_r + C_g - 2(C_r C_g)^{1/2}\right),$$

where $\mu_r, C_r$ and $\mu_g, C_g$ are the mean and covariance of reference and generated gesture feature distributions, respectively. Beat Constancy evaluates audio-motion rhythm alignment via amplitude changes in joints and audio, but does not comprehensively capture perceptual quality, including unnatural limb transitions, motion realism, or semantic congruence.

Ges-QAer operationalizes a different target: direct prediction of human-scored perceptual judgments. The associated annotations distinguish between perceived motion naturalness and realism, and the alignment of motion with speech rhythm. A separate binary label set, ESBA, marks whether generated gestures match the emotions of the audio, but Ges-QAer itself is trained only for multidimensional MOS regression.

This design places Ges-QAer within a broader movement from proxy metric design toward learned evaluators. In this case, the evaluator is explicitly multimodal and does not treat gesture quality as recoverable from motion traces alone. That choice is structurally reflected in its three-branch architecture and empirically supported by the ablation results discussed below.

2. Ges-QA dataset and annotation protocol

Ges-QAer is trained and evaluated on Ges-QA, a dataset comprising 1,400 A2G samples with multidimensional human scores. The dataset is constructed as $(6$ A2G generators $+$ ground truth$)\times 200$ audio segments. Each sample retains three modalities: rendered video, the audio clip, and the 3D skeleton or motion parameters from the generation pipeline, specifically SMPL-X. The audio source is BEAT2 (English), with 25 speakers and eight emotional categories. The corpus construction uses 25 sequences of 10 seconds per emotion category, yielding 200 audio clips; to ensure diversity, only four recordings come from the same two speakers (Gao et al., 16 Aug 2025).

The six A2G generators represented are EMAGE, MambaTalk, Syntalker, LoM (Language of Motion), MotionCraft, and GestureLSM, augmented with the BEAT2 ground-truth motion. Annotation was performed by 18 subjects, each rating every sample with two Likert sliders. The first dimension, gesture quality, targets perceived motion naturalness and realism. The second, audio–gesture consistency, targets alignment of motion with speech rhythm. Scores were normalized to Z-scores and scaled to $[0,100]$, and MOS was computed as the averaged Z-score.

Emotion congruence is captured by ESBA, a binary match/non-match label indicating whether gestures express the same emotion as the speech. ESBA is determined via majority voting after outlier exclusion. The paper does not report Cronbach’s alpha, Kendall’s $W$, or other inter-rater reliability statistics; it states only that MOS distributions and ESBA accuracies are visualized and qualitatively discussed.

All experiments use 5-fold cross-validation. Specific train, validation, and test set sizes per fold are not enumerated, and the reported metrics are from the final weights after training per method. This leaves the benchmark explicitly positioned as an evaluation resource rather than a prescribed train/dev/test benchmark.

3. Architecture and multimodal representations

Ges-QAer is a transformer-based network with three branches for video, audio, and motion. Each branch encodes its own modality, after which the features are pooled to a common temporal shape and fused through a fully connected module to regress the two MOS dimensions (Gao et al., 16 Aug 2025).

The audio branch uses AST (Audio Spectrogram Transformer) pre-trained on AudioSet. Each input is segmented into $N_a$ clips of 5 seconds, and for each clip a log-Mel spectrogram is computed using a Hamming-windowed filterbank. AST tokenizes spectrogram patches and applies transformer encoder blocks, yielding

$F_a \in \mathbb{R}^{B \times N_a \times C}.$

The video branch uses Video Swin Transformer. It processes $\mu_r, C_r$0 sampled frames per segment, with frames patched and passed through 3D shifted-window multi-head self-attention transformer blocks, producing

$\mu_r, C_r$1

The motion branch is a transformer-based motion encoder that ingests temporally sequenced SMPL-X parameters. Inspired by ViT-style designs, it employs a learnable token that aggregates the sequence in a CLS-like manner, producing

$\mu_r, C_r$2

Tokenization is implicit in AST and Video Swin. For motion, the learnable pooling token is specified, but explicit positional encoding choices are not detailed. Likewise, the paper does not specify layer counts, hidden sizes $\mu_r, C_r$3, number of heads, or exact block configurations for the three encoders.

Fusion is performed by temporally pooling audio and video features to match the motion feature shape $\mu_r, C_r$4, then feeding all three features into an MLP-based feature fusion module. No cross-attention or late fusion is reported. The output layer contains two regression heads, producing scalar predictions for gesture quality and audio–gesture consistency. Although ESBA labels are part of the dataset, Ges-QAer does not include a binary emotion classification head and no emotion classification loss is used in the reported experiments.

4. Objectives, training procedure, and evaluation metrics

The training objective is described as a correlation-aware loss “based on PLCC” across the two MOS dimensions. The paper gives

$\mu_r, C_r$5

where $\mu_r, C_r$6 and $\mu_r, C_r$7 are the normalized predicted scores and normalized MOS, respectively. The paper itself notes that the formula contains a typographical error, but its intent is stated clearly: minimize MSE terms while encouraging high covariance. The total loss is

$\mu_r, C_r$8

For reference, the Pearson correlation conceptually underlying the loss is

$\mu_r, C_r$9

Evaluation uses four standard quality-prediction metrics. Spearman rank-order correlation is

$\mu_g, C_g$0

where $\mu_g, C_g$1 is the rank difference between prediction and ground truth for sample $\mu_g, C_g$2. Pearson linear correlation coefficient is

$\mu_g, C_g$3

Kendall rank-order correlation is

$\mu_g, C_g$4

where $\mu_g, C_g$5 and $\mu_g, C_g$6 are the numbers of concordant and discordant pairs. Root mean squared error is

$\mu_g, C_g$7

Implementation details are concise but specific. The framework is PyTorch. Training uses a learning rate of $\mu_g, C_g$8 with warm-up and linear decay, a batch size of 10, and 20 epochs, on two NVIDIA RTX 3090 GPUs. Preprocessing uses Hamming-windowed log-Mel filterbanks for audio, sampled and patched frames for video, and temporally ordered SMPL-X parameter sequences for motion. The paper does not specify regularization, augmentation, runtime, or code availability.

5. Empirical performance and ablation behavior

Comparative experiments are conducted against multi-modal alignment models—AVID-CMA, VAST, and ImageBind—and AVQA methods—DNN-RNT, DNN-SND, and GeneralAVQA. These baselines are re-trained or fine-tuned on Ges-QA. The reported conclusion is that multi-modal alignment models underperform, AVQA methods are stronger baselines, and Ges-QAer yields state-of-the-art performance on the Ges-QA dataset (Gao et al., 16 Aug 2025).

Task	Ges-QAer	Best baseline
Gesture quality SRCC	0.9282	0.8122
Gesture quality PLCC	0.9352	0.8734
Gesture quality KRCC	0.7687	0.6289
Gesture quality RMSE	4.9838	6.8610
Audio–gesture consistency SRCC	0.8795	0.7845
Audio–gesture consistency PLCC	0.9344	0.8913
Audio–gesture consistency KRCC	0.7070	0.6014
Audio–gesture consistency RMSE	4.9749	6.3502

For gesture quality, the best baseline is GeneralAVQA with SRCC 0.8122, PLCC 0.8734, KRCC 0.6289, and RMSE 6.8610, whereas Ges-QAer achieves SRCC 0.9282, PLCC 0.9352, KRCC 0.7687, and RMSE 4.9838. For audio–gesture consistency, GeneralAVQA attains SRCC 0.7845, PLCC 0.8913, KRCC 0.6014, and RMSE 6.3502, while Ges-QAer reaches SRCC 0.8795, PLCC 0.9344, KRCC 0.7070, and RMSE 4.9749. The paper reports a minimum SRCC improvement of 14.3% for gesture quality and 12.1% for audio–gesture consistency.

The ablation results are particularly informative about modality dependence. Removing the motion encoder produces only small degradations: gesture quality becomes SRCC 0.9280, PLCC 0.9324, KRCC 0.7698, RMSE 5.0822, and audio–gesture consistency becomes SRCC 0.8760, PLCC 0.9310, KRCC 0.7015, RMSE 5.1044. The paper interprets this as evidence that motion features add signal beyond video-only cues.

By contrast, removing the video encoder causes severe collapse: gesture quality falls to SRCC 0.0943, PLCC 0.1229, KRCC 0.0643, RMSE 13.9742, and audio–gesture consistency to SRCC 0.0863, PLCC 0.1278, KRCC 0.0587, RMSE 13.8898. The paper states that this severe degradation highlights video’s necessity for perceptual assessment and the benefits of mature video QA representations. That finding is central to understanding what Ges-QAer measures: the predictor is not reducible to kinematic consistency in SMPL-X parameter space.

6. Limitations, interpretation, and research directions

The paper frames several limitations directly. Generalization is restricted by dataset composition: Ges-QA is sourced from BEAT2, which is English and emotion-scripted, and from six specific generators. The reported performance may therefore not generalize to other languages, spontaneous speech, different styles, alternative camera settings, or unseen generators (Gao et al., 16 Aug 2025).

Annotation methodology is another limitation. No inter-rater reliability statistics are reported, so the stability of the MOS labels is not quantified through Cronbach’s alpha, Kendall’s $\mu_g, C_g$9, or related measures. ESBA majority voting may reduce outlier effects, but the paper does not claim that it removes annotator bias.

Architectural reproducibility is incomplete. Transformer depths, hidden sizes, attention heads, and exact token counts $(6$0 and $(6$1 are not specified. The loss function is described as PLCC-based, but the printed equation contains a typographical error. These omissions do not invalidate the benchmark results, but they constrain exact replication.

A further limitation is conceptual: although ESBA labels are collected, Ges-QAer does not use them as supervision. The paper notes that “sadness” and “fear” are difficult to express via body-only gestures and suggests that future work could incorporate additional modalities such as facial cues or text semantics, or directly add a binary emotion classification head with a cross-entropy term. This suggests a natural extension from pure MOS regression toward richer affect-aware evaluation.

The broader significance of Ges-QAer lies in the formulation of quality assessment as a multimodal perceptual prediction task. The system uses synchronized rendered video, audio, and SMPL-X motion, regresses two human-scored dimensions, and substantially outperforms AVQA and alignment baselines on Ges-QA. At the same time, the paper leaves open several technical directions: explicit emotion modeling, clearer correlation-maximizing objectives, broader domain coverage, and fuller architectural specification. In that sense, Ges-QAer is best understood not as a finalized standard metric, but as a benchmarked neural evaluator that redefines A2G quality assessment around human MOS rather than around distributional or rhythm-only surrogates.