VoxEmo: Pretrained Transformer & SER Benchmark

• VoxEmo is a dual-use term representing both a pretrained audio-visual Transformer and a benchmark toolkit for speech emotion recognition, each leveraging distinct methodologies.
• The 2022 Transformer employs cross-modal attention on audio and visual signals, pretrained on VoxCeleb2, to boost downstream performance by 5–7% on emotion classification tasks.
• The 2026 benchmark standardizes prompt-driven SER evaluation with both hard-label and soft-label metrics across 35 corpora in 15 languages, highlighting prompt sensitivity in speech LLMs.

VoxEmo denotes two distinct research artifacts in the recent literature on affective computing. In one usage, it is a pretrained audio-visual Transformer for emotion recognition built on the Multimodal Transformer (MulT) architecture and pretrained on VoxCeleb2 to learn from unlabeled human communicative behavior (Tran et al., 2022). In a later usage, it is a benchmark and evaluation toolkit for speech emotion recognition (SER) with speech LLMs, designed to standardize prompt-driven zero-shot evaluation and to preserve human disagreement through soft-label protocols (Zhang et al., 9 Mar 2026). The shared name reflects a common concern with emotion inference from speech-associated human behavior, but the two works differ in modality, methodology, and evaluation regime.

1. Nomenclature and scope

The term “VoxEmo” is not monosemous in the arXiv literature. It refers to two different systems with different research objectives.

Usage	Primary setting	Core contribution
VoxEmo (2022)	Audio-visual emotion recognition	A pretrained audio-visual MulT model for downstream classification and regression
VoxEmo (2026)	Speech LLM evaluation for SER	A benchmark spanning 35 emotion corpora across 15 languages

The 2022 work is a transfer-learning model: it pretrains on VoxCeleb2 and fine-tunes on CREMA-D and MSP-IMPROV (Tran et al., 2022). The 2026 work is a benchmark philosophy and toolkit: it reproduces inference-time conditions, exposes prompt sensitivity, preserves human disagreement through soft labels, and evaluates speech LLMs both as classifiers and as generators of subjective emotion distributions (Zhang et al., 9 Mar 2026).

A useful disambiguation is therefore to treat the earlier VoxEmo as a pretrained emotion-recognition architecture and the later VoxEmo as a benchmark for generative SER.

2. VoxEmo as a pretrained audio-visual Transformer

The 2022 VoxEmo is essentially a pretrained audio-visual MulT model. It was proposed because existing large-scale pretrained multimodal Transformers were not ideal for emotion recognition: they were pretrained on datasets like Kinetics-700 and AudioSet, which contain little direct information about human interaction or emotional expression (Tran et al., 2022). The proposal is to pretrain on VoxCeleb2, a large-scale dataset of celebrity interview videos, so the model can learn from human speech and facial behavior in a socially relevant setting.

Its architecture has four key components: temporal convolutions to project each modality into a common hidden size; sinusoidal positional encoding to preserve temporal order; cross-modal Transformers that let one modality attend to another; and self-attention Transformers that integrate the fused information. The paper emphasizes the Cross-modal Attention Block, where queries come from one modality and keys/values from the other. With two modalities, there are two cross-modal directions: $V \rightarrow A$ and $A \rightarrow V$. The outputs from both directions are concatenated and passed through a standard self-attention Transformer. Final outputs are projected back to audio and visual feature dimensions using two separate fully connected layers (Tran et al., 2022).

The model is designed to learn from unlabeled human communicative behavior rather than from action or object recognition data. This design choice suggests that the pretraining signal is intended to encode face-voice interaction patterns that are directly relevant to affect.

3. Pretraining corpus, masking objective, and implementation

The pretraining dataset is VoxCeleb2. The paper says VoxCeleb2 contains more than 1M utterances, more than 6,000 celebrities, and videos collected from about 150K YouTube videos. After preprocessing, the actual pretraining set used in the paper is 524K utterances from about 4K speakers, with average utterance length about 5 seconds (Tran et al., 2022).

For the audio modality, audio is segmented into 200 ms segments and then passed through TRILL for feature extraction. For the visual modality, frames are processed with OpenFace 2.0, which tracks 17 Facial Action Unit intensities. Video is originally sampled at 30 FPS; frames with OpenFace detection confidence below 80% are removed; and OpenFace outputs are downsampled to 5 FPS to match audio granularity. The preprocessing also discards utterances where audio and video are misaligned by more than 1 second. Although MulT can handle unaligned data, the paper states that it performs better when audio and video are aligned (Tran et al., 2022).

Pretraining follows a masked frame prediction objective, similar in spirit to BERT and Mockingjay. The masking strategy randomly selects 15% of frames and masks them in both audio and visual inputs. For selected frames, the model replaces them with zeros with probability 0.8, replaces them with random frames with probability 0.1, and keeps them unchanged with probability 0.1. Consecutive frames are also masked to prevent the model from exploiting local smoothness. Reconstruction uses $L1$-loss: $$\mathcal{L} = \mathcal{L}_{L1}^{audio} + \mathcal{L}_{L1}^{visual}$$ The training setup uses dynamic masking for training and static masking for validation (Tran et al., 2022).

Two pretrained configurations are reported.

Config	Main dimensions and depth	Parameters
BASE	hidden size 288; 6 $A \rightarrow V$, 6 $V \rightarrow A$, 6 self-attention blocks	38.3M
LARGE	hidden size 576; 8 $A \rightarrow V$, 8 $V \rightarrow A$, 8 self-attention blocks	89.2M

Both use 12 attention heads. Training uses Adam, learning rate $5 \times 10^{-4}$, a linear learning-rate scheduler, warmup ratio 0.1, batch size 64, and 30 epochs (Tran et al., 2022).

4. Downstream evaluation, empirical results, and limitations of the 2022 system

The pretrained model is fine-tuned for emotion recognition on two datasets. CREMA-D is used for emotion classification; it is an acted audiovisual dataset with 7,442 video clips, 91 actors, and 6 emotions: happy, sad, anger, fear, disgust, and neutral. The split is speaker-independent: 60% train, 20% validation, and 20% test. MSP-IMPROV is used for continuous emotion regression; it contains 8,450 video recordings, 12 actors, and 6 dyad sessions. The paper uses continuous annotations for valence and arousal, with Sessions 1–4 for train, Session 5 for validation, and Session 6 for test (Tran et al., 2022).

For downstream prediction, the last outputs of pretrained MulT are passed to a residual block, then a fully connected layer produces the final prediction. The comparison set includes EF-GRU, LF-GRU, TFN, and MulT without pretrained initialization (“WOP”). For fairness, EF-GRU and LF-GRU are made bidirectional, model sizes are matched as closely as possible, and MulT without pretraining is tested in both BASE and LARGE forms, with the better one reported (Tran et al., 2022).

The full-data results show consistent gains from pretraining.

Model	CREMA-D Accuracy	MSP-IMPROV Arousal CCC	MSP-IMPROV Valence CCC
TFN	63.09%	0.581	0.592
EF-GRU	57.06%	0.399	0.478
LF-GRU	58.53%	0.546	0.579
MulT WOP	63.93%	0.665	0.607
MulT BASE	68.87%	0.697	0.658
MulT LARGE	70.22%	0.693	0.692

Compared with the same model trained from scratch, fine-tuning the pretrained model improves emotion classification accuracy by 5–7% and Concordance Correlation Coefficients in continuous emotion recognition by 0.03–0.09. With only 10% of the original training set, fine-tuning the pretrained model leads to at least 10% better emotion recognition accuracy and a CCC score improvement by at least 0.1 for continuous emotion recognition (Tran et al., 2022).

The paper also reports a feature-selection study before the main Transformer experiments. Visual features tested are FaceNet embeddings, ResNet embeddings, and OpenFace Action Unit intensities; audio features tested are Mel-scale spectrogram, linear-scale spectrogram, and TRILL embeddings. OpenFace is reported as best for visual emotion representation and TRILL as best for acoustic representation. Reported gains include about 17% accuracy gain over ResNet for OpenFace on CREMA-D, more than 11% accuracy gain over linear spectrogram for TRILL on CREMA-D, at least 0.03 CCC improvement for TRILL over linear spectrogram on MSP-IMPROV, and at least 0.14 CCC improvement for OpenFace over the second-best baseline on MSP-IMPROV (Tran et al., 2022).

The main limitations are explicit. The approach is not end-to-end and depends on pre-extracted audio and visual features. It relies on alignment between modalities, even though MulT can handle unaligned sequences. Evaluation is restricted to two emotion datasets, and both are acted datasets. The paper therefore leaves open the question of generalization to spontaneous, real-world affect (Tran et al., 2022).

5. VoxEmo as a benchmark for speech LLMs

The 2026 VoxEmo addresses a different problem: the mismatch between classical SER benchmarks and the generative, prompt-driven nature of speech LLMs. The paper argues that classical SER benchmarks assume fixed closed-set classification, whereas speech LLMs generate text under prompts, making results highly sensitive to prompt wording, decoding, and parsing. It also argues that emotions in speech are inherently subjective and ambiguous, especially in naturalistic corpora, so collapsing multiple annotator opinions into a single hard label can hide important information (Zhang et al., 9 Mar 2026).

At a high level, VoxEmo covers 35 emotion corpora spanning 15 languages, with datasets released between 2006 and 2025. It includes 7 in-the-wild corpora and 28 acted corpora. The benchmark intentionally mixes perceived labels, expressed labels, and corpora with both, because label source is treated as a meaningful metadata dimension. The listed corpora include MSP-Podcast, BIIC-Podcast, M3ED, MELD, CREMA-D, IEMOCAP, RAVDESS, EmoDB, EmoV-DB, SAVEE, TESS, and multilingual resources in Mandarin, German, Italian, French, Urdu, Turkish, Persian, Bangla, Thai, and Polish (Zhang et al., 9 Mar 2026).

A central part of the benchmark is prompt design. The shared zero-shot template states: “You are a speech emotion recognition system. You are given an audio clip.” The prompt-complexity spectrum is built by adding instruction blocks: Direct decision, Distribution, ASR transcript, Acoustic caption, and Reasoning. The evaluated hard-label prompt variants are Direct, +T, +A, +T+A, and +T+A+R. For soft-label evaluation, the Distribution block is added so the model also outputs a probability distribution over emotion categories (Zhang et al., 9 Mar 2026).

The soft-label protocol is defined for the five multi-annotator corpora that expose annotator-level metadata: CREMA-D, IEMOCAP, MSP-Podcast, BIIC-Podcast, and EmotionTalk. Ground truth is represented as an unsmoothed count-based soft-label distribution: $$\mathbf{y} = \left[ \frac{n_1}{N}, \frac{n_2}{N}, \dots, \frac{n_C}{N} \right]$$ where $A \rightarrow V$0 is the count of votes for class $A \rightarrow V$1, $A \rightarrow V$2 is the number of annotations, and $A \rightarrow V$3 is the number of emotion categories. The prompt-ensemble strategy aggregates predictions from the five hard-label prompt variants into a distribution, treating differing prompt responses as a proxy for human annotator diversity. The equation in the provided paper text is noted as malformed, but its intended meaning is a 5-prompt vote distribution with uniform mass assigned to invalid outputs (Zhang et al., 9 Mar 2026).

Evaluation combines hard-label metrics and distribution-aware metrics. The hard-label setting uses Weighted Accuracy, Unweighted Accuracy, Micro-F1, and Macro-F1. Soft-label evaluation uses a dual-metric approach: Hard-decision Assessment with Macro-F1, Micro-F1, and Top-1 Accuracy, and Distribution-aware Assessment with Kullback-Leibler Divergence, Jensen-Shannon Divergence, Total Variation Distance, Cosine Similarity, and Mean Squared Error (Zhang et al., 9 Mar 2026).

6. Experimental protocol and findings of the 2026 benchmark

The benchmark evaluates two open-weight speech LLMs: Qwen2-Audio-7B-Instruct (Q2A) and Audio Flamingo 3 (AF3). Both use Whisper-large-v3 as the audio encoder. The zero-shot setup fixes greedy decoding, shared parsing rules, and allows only the prompt template to vary. For supervised contrast, both models are fine-tuned with LoRA adapters on the training sets, using $A \rightarrow V$4, $A \rightarrow V$5, dropout 0.05 on the language decoder attention projections, while freezing the audio encoder and most parameters. Training uses AdamW with learning rate $A \rightarrow V$6, 10% warmup, bf16, batch size 16, and 10 epochs (Zhang et al., 9 Mar 2026).

The principal empirical result is that zero-shot speech LLMs are highly prompt-sensitive and generally lag supervised baselines on hard-label classification, but they can better match human subjective label distributions. For Q2A, the gap between best and worst prompt variants exceeds 20 Macro-F1 points on 11 datasets and reaches 49 points on TESS. The best prompt differs by model and dataset: AF3 most often prefers the Direct prompt, while Q2A often benefits from the acoustic-caption prompt (+A). Combining transcript and acoustic caption often does not help, and the +T+A prompt does not outperform the best single-block prompt on any dataset. Adding reasoning (+T+A+R) can partially recover performance for Q2A but still rarely beats the best single-block prompt (Zhang et al., 9 Mar 2026).

The effect of prompts depends strongly on corpus type. Acoustic captions help more on acted corpora than in-the-wild corpora, because acted speech often keeps transcript content fixed across emotions, making paralinguistic cues the key discriminative signal. Transcript prompts help more in in-the-wild corpora for Q2A, especially in languages well represented in its training data, but they can hurt on acted corpora where lexical content is not informative. AF3 is less responsive to transcript prompts and often prefers the Direct prompt (Zhang et al., 9 Mar 2026).

In absolute terms, zero-shot performance remains modest. The paper states that best-prompt Macro-F1 remains below 20% on eight datasets for both models and is below the EmoBox supervised baselines on all 30 comparable datasets. The soft-label results are more distinctive: on the five datasets with annotator distributions, the ensemble often improves both hard-decision and distribution-aware metrics over single prompts. The paper highlights examples on BIIC-Podcast, CREMA-D, and EmotionTalk, and it notes that Q2A can exhibit catastrophic parse failure under certain prompts, especially +A, while AF3 is more robust in some cases. The ensemble reduces this brittleness by pooling across prompts (Zhang et al., 9 Mar 2026).

Supervised fine-tuning narrows the gap substantially. Q2A’s SFT improves over its best zero-shot prompt on 34 of 36 evaluation splits, with an average Macro-F1 gain of 23.7. It surpasses the EmoBox supervised reference on 10 of 30 comparable datasets and matches it closely on 5 more. AF3 also improves under SFT on many splits, but with smaller gains and never surpasses EmoBox on the comparable datasets under the single LoRA configuration used. Cross-corpus transfer experiments across CREMA-D, IEMOCAP, MELD, and MSP-Podcast show that Q2A benefits from cross-domain fine-tuning much more than AF3, with MELD described as a particularly strong source (Zhang et al., 9 Mar 2026).

The limitations are practical. VoxEmo evaluates only two models, both sharing the same Whisper-large-v3 audio encoder and similar scale. The SFT study uses only one LoRA configuration. Soft-label evaluation is limited to five corpora with per-annotator metadata. The benchmark also focuses on corpus-level aggregate metrics and does not analyze class imbalance, per-class behavior, or speaker effects in depth (Zhang et al., 9 Mar 2026).

7. Related nomenclature, neighboring systems, and common sources of confusion

The ambiguity of the name is reinforced by nearby work with similar labels or explicitly “VoxEmo-style” framing. VCEMO is a Chinese daily-conversation corpus for single-sentence emotion recognition with 7,477 textual samples or audio utterances, over 100 different speakers or users, and a multimodal benchmark that combines acoustic speech signal, transcript-level word embeddings, and BERT-encoded text embeddings using co-attention and supervised contrastive learning (Tang et al., 2024). Although its title is different, it occupies a neighboring space: realistic speech emotion recognition from everyday voice data.

The provided material also frames VoiceMoji as a “VoxEmo-style system”: an on-device pipeline that takes dictation output, performs emoji boundary detection, and predicts one of 64 emoji classes using an Attention-based Char Aware (ACA) LSTM, with approximately 4 MB ROM and 15 MB RAM for the full pipeline (Kumar et al., 2021). EmojiVoice is described as “a VoxEmo-like concept”: a real-time, offline, open-source TTS toolkit for robots that uses emoji prompting to control phrase-level expressivity with a Matcha-TTS backbone (Tuttösí et al., 18 Jun 2025). In a different direction, VoxStudio releases VoxEmoset, a large-scale paired emotional speech-image dataset with 82k images and 247k utterances for expressive speech-to-image generation (Lee et al., 5 Nov 2025).

These neighboring names do not denote the same artifact as either VoxEmo paper. A practical reading is therefore to reserve “VoxEmo” for the exact paper context in which it appears: either the 2022 pretrained audio-visual Transformer or the 2026 SER benchmark. In both cases, the common thread is the view that emotion-related information is not adequately captured by generic speech or video pipelines alone, and that evaluation or pretraining should be anchored more directly in human communicative behavior.