Test-Time Adaptation for Speech Emotion Recognition

Abstract: The practical utility of Speech Emotion Recognition (SER) systems is undermined by their fragility to domain shifts, such as speaker variability, the distinction between acted and naturalistic emotions, and cross-corpus variations. While domain adaptation and fine-tuning are widely studied, they require either source data or labelled target data, which are often unavailable or raise privacy concerns in SER. Test-time adaptation (TTA) bridges this gap by adapting models at inference using only unlabeled target data. Yet, having been predominantly designed for image classification and speech recognition, the efficacy of TTA for mitigating the unique domain shifts in SER has not been investigated. In this paper, we present the first systematic evaluation and comparison covering 11 TTA methods across three representative SER tasks. The results indicate that backpropagation-free TTA methods are the most promising. Conversely, entropy minimization and pseudo-labeling generally fail, as their core assumption of a single, confident ground-truth label is incompatible with the inherent ambiguity of emotional expression. Further, no single method universally excels, and its effectiveness is highly dependent on the distributional shifts and tasks.

• The paper systematically evaluates 11 test-time adaptation (TTA) methods for Speech Emotion Recognition (SER) across three tasks.
• Backpropagation-free (BP-free) methods, particularly T3A, show superior performance, highlighting methods beneficial for different distributional shifts while identifying the limited effectiveness of traditional entropy minimization (EM) and pseudo-labeling (PL) methods.
• The effectiveness of TTA in SER is primarily determined by the type and magnitude of the domain shift, rather than batch size or optimization details, as evidenced through a comprehensive analysis.

Motivation and problem setting

Speech Emotion Recognition (SER) systems degrade substantially under domain shift, and the shifts that matter in SER are distinctive: speaker variability arising from cultural and personality differences, the systematic gap between acted and naturalistic emotional expression, and cross-corpus variation induced by different recording conditions and collection protocols. Existing remedies are unsatisfactory in practice. Unsupervised domain adaptation requires access to source data, which is often blocked by privacy and data-sharing constraints; supervised adaptation and fine-tuning require labeled target data that does not exist at inference time. Test-time adaptation (TTA) sidesteps both requirements by adapting a source-trained model using only unlabeled target data at inference.

Despite a mature TTA literature for image classification (TENT, EATA, SAR, CoTTA, T3A, LAME, FOA) and growing interest in automatic speech recognition (SUTA, DSUTA, CEA, AWMC), no prior work had systematically evaluated whether these methods transfer to SER's particular distributional pathologies. This paper fills that gap with the first comprehensive evaluation of 11 TTA methods across three representative SER tasks: intra-corpus personalization, acted-to-natural adaptation, and cross-corpus generalization (2601.16240).

Methodology

The evaluation pipeline uses a wav2vec 2.0 base encoder fine-tuned on the source domain, followed by a single 256-unit fully connected classification layer over four emotion categories. Speech is segmented into 5-second windows for IEMOCAP and 8-second windows for RAVDESS; models are trained with AdamW (learning rate $3 \times 10^{-5}$) for 50 epochs with linear warm-up and decay. IEMOCAP contributes 5,531 utterances across happiness (with excitement merged), anger, sadness, and neutral from 10 speakers in dyadic sessions containing both scripted (acted) and improvised (natural) dialogues; RAVDESS contributes 1,440 speech utterances spanning eight emotions from 24 professional actors.

Three families of TTA methods are compared:

• Entropy minimization (EM): Tent, SAR, EATA, SUTA, DSUTA, and CEA update selected parameters (e.g., normalization layers) during inference by minimizing prediction entropy $H(p) = -\sum_c p^c \log p^c$, encouraging confident single-class decisions.
• Pseudo-labeling (PL): CoTTA and AWMC maintain a main model updated via cross-entropy against pseudo-labels produced by an anchor model whose parameters follow an exponential moving average of the main model.
• Backpropagation-free (BP-free): T3A recalibrates class prototypes from high-confidence target predictions; LAME performs maximum-likelihood correction of output probabilities with Laplacian regularization enforcing neighborhood consistency while remaining faithful to source predictions; FOA applies forward-only prompt tuning optimized with CMA-ES to jointly reduce entropy and align source/target feature statistics (means and standard deviations of latent embeddings).

Hyperparameters were selected by grid search, gradient-based methods use AdamW at learning rate $10^{-5}$, and headline results use batch size 32, with a dedicated batch-size analysis covering sizes 1–64.

Results

Intra-corpus personalization. BP-free methods achieve the best average performance on both corpora: 67.3% accuracy / 67.8% F1 on IEMOCAP and 72.2% / 69.2% on RAVDESS, versus source baselines of 67.1/67.4 and 72.5/69.5 respectively. Within this family, FOA is the strongest individual method, reaching 67.6% accuracy / 68.2% F1 on IEMOCAP (+0.5/+0.8 points) and 73.8% / 70.9% on RAVDESS (+1.3/+1.4 points). The authors attribute BP-free superiority to reduced catastrophic forgetting—freezing the backbone preserves source knowledge under subtle shifts—and note that EM and PL methods rest on an assumption poorly matched to emotion: that each sample has a single, confident ground-truth label, which conflicts with the inherent ambiguity of emotional expression. The gains here are nonetheless modest, indicating limited headroom for TTA when shifts are minor.

Acted-to-natural adaptation. This task proves largely resistant to TTA. The source model achieves 51.3% accuracy / 51.0% F1, and the best method (LAME, 51.5%/51.3%) improves it by only fractions of a point; several methods actively hurt performance (DSUTA drops to 46.1%). The paper argues that current TTA techniques excel at uniform, global shifts such as image corruptions, whereas the acted-to-natural shift is non-uniform and nuanced—acted emotions carry subtler acoustic cues tied to fixed transcripts, while improvised emotions are more intense and freely expressed—a pattern existing methods cannot capture.

Cross-corpus generalization. This is where TTA delivers its clearest benefit. With a weak source model (37.8% accuracy / 26.7% F1 adapting RAVDESS→IEMOCAP direction data; 50.0%/45.7% in the reverse), T3A yields the best results on both directions, with average gains of 3.1 points accuracy and 4.1 points F1 over the source baselines. The mechanism is instructive: when the source model is weak, most predictions are low-confidence noise, and T3A's confidence-filtered prototype recalibration prevents uncertain predictions from contaminating the adapted classifier. Notably, LAME collapses under this severe shift (28.7% accuracy, 12.3% F1 on RAVDESS→IEMOCAP), showing that even within the BP-free family, robustness varies sharply with shift severity. The contrast across tasks supports the paper's central claim that TTA effectiveness in SER is bounded primarily by the type and severity of the distributional shift rather than by batch reliability or optimization details.

Batch size analysis. Batch size matters mainly at the extremes. On IEMOCAP, PL and BP-free methods are stable across sizes, while EM degrades badly at batch size 1 due to unreliable statistics. On RAVDESS, EM and PL improve with larger batches, but BP-free performance decreases—an effect traced entirely to LAME's neighborhood graph construction, which amplifies misclassifications when large batches contain many errors under severe shift. From batch size 16 to 64 all methods plateau, implying that scaling batch size does not unlock further adaptation gains.

Limitations and open questions

The paper is candid about several constraints. The evaluation covers only two English-language corpora and four shared emotion categories for cross-corpus transfer, so conclusions about other languages, label spaces, and corpora remain untested. The acted-to-natural result is accompanied only by a hypothesis—that the shift is too complex and non-uniform for current TTA—rather than a demonstrated remedy, leaving open what class of adaptation objective could handle non-uniform, semantics-level shifts. The claimed superiority of BP-free methods rests partly on post-hoc reasoning about catastrophic forgetting and label ambiguity rather than direct measurement. Additionally, the single-label assumption pervades even BP-free approaches "to some extent," and the paper does not quantify how much multi-label or soft-label supervision at test time might change the picture. Whether confidence-based prototype filtering (T3A) remains beneficial when source models are moderately rather than severely degraded is also unresolved, since the two regimes behave very differently across methods.

Conclusion

This study establishes the first systematic benchmark of test-time adaptation for speech emotion recognition, evaluating 11 methods across personalization, acted-to-natural, and cross-corpus tasks. Its principal findings are that backpropagation-free methods are the most promising category overall, that entropy minimization and pseudo-labeling frequently fail because their single-confident-label assumption clashes with emotional ambiguity, and that no single method dominates—FOA leads on personalization, LAME marginally on acted-to-natural, and T3A decisively on cross-corpus transfer, where gains of roughly 3–4 points are largest precisely because the source model is weakest. The plateau observed across moderate batch sizes indicates that the binding constraint on TTA in SER is the character of the domain shift itself, not adaptation mechanics.