DeEAR: Evaluating Expressive Speech

• DeEAR is a framework that measures speech expressiveness by decomposing it into emotion, prosody, and spontaneity, providing a clear multi-dimensional evaluation.
• The system employs proxy models, including teacher-supervised regressions and LLM-based scoring, to align its scores with human expressiveness ratings (SRCC 0.86).
• It supports system benchmarking and data curation through a distilled model pipeline, yielding significant improvements in expressive speech generation.

Searching arXiv for the requested paper to ground the article in the current record. DeEAR, short for “Decoding the Expressive Preference of eAR,” is a framework for converting human preference for speech expressiveness into an objective score in speech-to-speech (S2S) evaluation and data curation. It was introduced to address the persistent gap between intelligibility and natural expressiveness in modern end-to-end S2S systems, which often sound “robotic” despite producing intelligible audio (Lin et al., 23 Oct 2025). The framework is grounded in phonetics and psychology and evaluates speech across three dimensions—Emotion, Prosody, and Spontaneity—before fusing them into a final expressiveness score on a 0–100 scale. In the reported experiments, DeEAR achieves strong alignment with human judgments of expressiveness, with Spearman’s Rank Correlation Coefficient (SRCC) of 0.86 using only $\sim 480$ human-annotated clips, and supports both system benchmarking and expressiveness-driven dataset construction (Lin et al., 23 Oct 2025).

1. Motivation and problem setting

Recent S2S models generate intelligible speech but still lack natural expressiveness, and the central obstacle identified for this deficiency is the absence of a reliable evaluation metric (Lin et al., 23 Oct 2025). Existing approaches to expressiveness evaluation are described as incomplete in different ways: subjective MOS tests are costly, low-level acoustic features such as pitch and energy are perceptually incomplete, and single-facet emotion classifiers are too narrow (Lin et al., 23 Oct 2025). Within this framing, DeEAR is designed not merely as an auxiliary scoring heuristic, but as a multi-dimensional metric intended to objectify human preference regarding expressiveness.

The framework has three stated high-level goals. First, it aims to provide an objective metric that correlates strongly with human judgments of expressiveness, reporting SRCC $=0.86$ in Section 4.1 (Lin et al., 23 Oct 2025). Second, it is explicitly data-efficient, requiring only $\sim 480$ human-annotated clips for alignment (Lin et al., 23 Oct 2025). Third, it is intended to enable downstream tasks beyond evaluation, including reliable benchmarking of S2S models and expressiveness-driven data curation (Lin et al., 23 Oct 2025).

This framing places DeEAR within a broader methodological shift from purely subjective evaluation toward structured preference modeling. A plausible implication is that DeEAR treats expressiveness as a latent construct that can be decomposed into operational subdimensions, each measurable through proxy models and then reassembled through a learned fusion layer.

2. Three-dimensional decomposition of expressiveness

DeEAR evaluates speech across three dimensions derived from phonetics, psychology, and paralinguistics: Emotion, Prosody, and Spontaneity (Lin et al., 23 Oct 2025). Each dimension is scored on a 0–100 scale, and the decomposition is intended to capture aspects of expressiveness that are not adequately represented by a single scalar proxy.

Emotion Intensity

Emotion Intensity is defined as emotion arousal (Lin et al., 23 Oct 2025). Its proxy model is a fine-tuned wav2vec2-large-robust model pretrained on MSP-IMPROV and further trained on 12 k Chinese (CNSCED) and 2 k English (IEMOCAP) data (Lin et al., 23 Oct 2025). The output is a scalar $S_{\text{emo}}\in[0,100]$ proportional to predicted arousal (Lin et al., 23 Oct 2025). When training DeEAR-Base, the corresponding regression loss is

$$\mathcal{L}_{\text{emo}} = \frac{1}{N}\sum_{i=1}^N \bigl(\hat S^{(i)}_{\text{emo}} - S^{(i)}_{\text{emo,teacher}}\bigr)^2.$$

This formulation makes Emotion a teacher-supervised regression target rather than a discrete categorical classifier, which is consistent with the paper’s preference-centric objective (Lin et al., 23 Oct 2025).

Prosodic Richness

Prosodic Richness is produced by a proxy annotator based on Gemini 2.5 Pro LMM with engineered prompts (Lin et al., 23 Oct 2025). The resulting scalar $S_{\text{pros}}\in[0,100]$ achieves SRCC $=0.73$ with respect to human judgments (Lin et al., 23 Oct 2025). The paper does not specify an explicit analytic formula for this subscore; instead, richness is directly represented by the LMM’s score. The DeEAR-Base supervision term is

$$\mathcal{L}_{\text{pros}} = \frac{1}{N}\sum_i \bigl(\hat S_{\text{pros}}^{(i)}-S_{\text{pros,teacher}}^{(i)}\bigr)^2.$$

The use of an LMM here is significant because prosodic richness is presented as a dimension that is difficult to reduce to low-level handcrafted descriptors alone. A plausible implication is that the prompt-engineered LMM acts as a high-level annotator for contours and variation patterns that are hard to specify analytically.

Spontaneity

Spontaneity is defined through the intuition that “perceptual incongruence” arises when acoustic quality is very high, described as hyper-clean, but base spontaneity is low (Lin et al., 23 Oct 2025). The mean DNSMOS quality metric is

$$M_{\text{avg}} = \tfrac{1}{4}\bigl(M_{\text{OVRL}} + M_{\text{SIG}} + M_{\text{BAK}} + M_{P.808}\bigr).$$

The base spontaneity level is $L_{\text{base}}\in\{1,3,5,7,9\}$, manually assigned per dataset (Lin et al., 23 Oct 2025). A clip is “hyper-clean” if all four DNSMOS scores exceed threshold $=0.86$0 (Lin et al., 23 Oct 2025). The heuristic spontaneity score is then defined as

$=0.86$1

where $=0.86$2 linearly scales $=0.86$3 to $=0.86$4, rewarding congruence, while $=0.86$5 reversely scales to a narrow punitive range, for example $=0.86$6 when $=0.86$7 (Lin et al., 23 Oct 2025). This heuristic labels $=0.86$8 k utterances, which are then used to fine-tune wav2vec2 with

$=0.86$9

Among the three dimensions, Spontaneity is the most overtly heuristic. The paper treats this not as a limitation to conceal, but as a formalization of the observation that over-processed speech can sound less natural precisely because its acoustic cleanliness is mismatched with conversational spontaneity (Lin et al., 23 Oct 2025).

3. Efficient alignment and model architecture

DeEAR uses a two-level construction: three proxy scorers first estimate the dimensional subscores, and a fusion model then aligns them to human judgments of overall expressiveness (Lin et al., 23 Oct 2025). After training the proxy models, the framework collects a human preference set of $\sim 480$0 clips, where each clip has proxy scores $\sim 480$1 and a human-annotated overall expressiveness label $\sim 480$2 (Lin et al., 23 Oct 2025).

The fusion model is an XGBoost regressor $\sim 480$3 trained by minimizing squared error:

$\sim 480$4

The final expressiveness score is

$\sim 480$5

To reduce deployment cost, the paper introduces DeEAR-Base, a distilled model based on a single wav2vec2-xlsr-53 backbone with three regression heads (Lin et al., 23 Oct 2025). The full “teacher” pipeline is applied to 20 k unlabeled utterances to obtain three proxy scores, and DeEAR-Base is trained to minimize the multi-task objective

$\sim 480$6

The deployed system is therefore a cascade: DeEAR-Base followed by XGBoost fusion (Lin et al., 23 Oct 2025). This architecture embodies the paper’s claim of efficient alignment: expensive or heuristic annotators are used to generate teachers, then distilled into a compact acoustic backbone aligned to a relatively small human preference set.

4. Evaluation protocol and empirical validity

To quantify alignment with human expressiveness ratings, the paper uses Spearman’s Rank Correlation Coefficient, computed as

$\sim 480$7

where $\sim 480$8 is the rank difference for sample $\sim 480$9 (Lin et al., 23 Oct 2025). The reported $S_{\text{emo}}\in[0,100]$0 for the fused expressiveness score is $S_{\text{emo}}\in[0,100]$1 (Lin et al., 23 Oct 2025). The evaluation also reports Pearson correlation and uses human annotations collected on four 100-utterance sets, with Krippendorff’s $S_{\text{emo}}\in[0,100]$2 (Lin et al., 23 Oct 2025).

The per-dimension and fused correlations are summarized below.

Dimension	PCC	SRCC
Emotion	0.72	0.65
Prosody	0.70	0.68
Spontaneity	0.84	0.84
Expressiveness	0.91	0.86

These results show that the fused expressiveness score exceeds the individual dimensions in alignment with human ratings (Lin et al., 23 Oct 2025). This suggests that no single component is sufficient to recover perceived expressiveness, and that the decomposition-plus-fusion strategy is central rather than incidental.

The framework is also evaluated at the system level on seven S2S systems and 20 prompts, where DeEAR’s system-level ranking attains SRCC $S_{\text{emo}}\in[0,100]$3 against four-listener human MOS on a 3-point scale (Lin et al., 23 Oct 2025). The score spread is approximately 60 points, with DouBao at 65.4 and Qwen2.5-Omni at 5.3 (Lin et al., 23 Oct 2025). In this setting, DeEAR is presented as a benchmarking instrument rather than solely a clip-level predictor.

5. ExpressiveSpeech dataset construction

One of DeEAR’s principal downstream uses is data curation. The paper constructs the ExpressiveSpeech dataset from five sources—Expresso, NCSSD, M$S_{\text{emo}}\in[0,100]$4ED, MultiDialog, and IEMOCAP—amounting to $S_{\text{emo}}\in[0,100]$5 hours of raw Chinese and English speech (Lin et al., 23 Oct 2025). The curation pipeline is described in five steps.

1. Audio standardization and noise removal to 16 kHz mono, using ClearerVoice for denoising and speaker separation (Lin et al., 23 Oct 2025).
2. Quality scoring via DNSMOS P.835, with average OVRL $S_{\text{emo}}\in[0,100]$6 (Lin et al., 23 Oct 2025).
3. Expressiveness scoring via DeEAR across Emotion, Prosody, Spontaneity, and the fused score (Lin et al., 23 Oct 2025).
4. Thresholding at $S_{\text{emo}}\in[0,100]$7, described as empirically aligned with “high expressiveness” in human tests (Lin et al., 23 Oct 2025).
5. ASR transcription added as metadata (Lin et al., 23 Oct 2025).

The final dataset contains 14 000 utterances, approximately 51 hours, balanced between English and Chinese (Lin et al., 23 Oct 2025). Its average expressiveness score is $S_{\text{emo}}\in[0,100]$8, which is reported as far above the source averages of 39.4–62.9 (Lin et al., 23 Oct 2025).

The significance of this curation pipeline lies in the fact that DeEAR is used not only to judge model outputs, but also to shape the training distribution. A plausible implication is that the framework functions as both an evaluator and a selection policy for collecting training examples with high expressive utility.

6. Applications, gains, and limitations

The paper identifies three principal applications for DeEAR: automated benchmarking of conversational S2S and TTS systems, data curation for expressiveness gains, and reward modeling for reinforcement-learning-based speech generation (Lin et al., 23 Oct 2025). Of these, the data-curation application is substantiated with the most detailed experimental results.

A baseline S2S system is compared with an Expressive-FT model fine-tuned on ExpressiveSpeech (Lin et al., 23 Oct 2025). The reported objective gains in overall expressiveness are substantial:

Setting	Baseline $S_{\text{emo}}\in[0,100]$9	Ours $$\mathcal{L}_{\text{emo}} = \frac{1}{N}\sum_{i=1}^N \bigl(\hat S^{(i)}_{\text{emo}} - S^{(i)}_{\text{emo,teacher}}\bigr)^2.$$0
In domain	2.3	24.0
Out of domain	1.8	23.0
Overall	2.0	23.4

The paper also reports component-level gains: Emotion from 5.7 to 15.9, Prosody from 35.7 to 36.7, and Spontaneity from 33.7 to 62.0, all with $$\mathcal{L}_{\text{emo}} = \frac{1}{N}\sum_{i=1}^N \bigl(\hat S^{(i)}_{\text{emo}} - S^{(i)}_{\text{emo,teacher}}\bigr)^2.$$1 (Lin et al., 23 Oct 2025). Subjectively, A/B preference by 10 native speakers favored the fine-tuned system 78.5% versus baseline 10.0%, with 11.5% ties and $$\mathcal{L}_{\text{emo}} = \frac{1}{N}\sum_{i=1}^N \bigl(\hat S^{(i)}_{\text{emo}} - S^{(i)}_{\text{emo,teacher}}\bigr)^2.$$2 (Lin et al., 23 Oct 2025).

The limitations are explicitly stated. Prosodic Richness depends on an LMM proxy and may inherit LMM biases or domain mismatches (Lin et al., 23 Oct 2025). The Spontaneity heuristic requires DNSMOS and may be sensitive to noise or artifact removal (Lin et al., 23 Oct 2025). Fusion alignment is based on only 480 clips, and the paper notes that future larger studies could refine the mapping (Lin et al., 23 Oct 2025). These limitations constrain any interpretation of DeEAR as a definitive theory of expressiveness; it is better understood as a practical alignment framework built from carefully selected proxies.

Potential extensions noted in the paper include integrating DeEAR as a reward in RLHF for end-to-end expressiveness control, expanding beyond bilingual English and Chinese, and exploring richer sub-dimension features such as linguistic and semantic features as well as multi-modal cues including video and text (Lin et al., 23 Oct 2025). These directions suggest that the current framework is intentionally modular.

7. Position within expressive speech evaluation

DeEAR is presented as an answer to a specific methodological gap: the lack of a reliable, objective, multi-dimensional metric for speech expressiveness (Lin et al., 23 Oct 2025). Its distinguishing characteristic is that it does not reduce expressiveness to a single low-level descriptor or to an isolated emotion label. Instead, it operationalizes human preference through three dimensions and an efficient alignment stage learned from a small human-rated set (Lin et al., 23 Oct 2025).

This design also helps clarify a common misconception in expressive speech evaluation: that expressiveness can be captured adequately either by MOS alone or by purely acoustic proxies. The DeEAR results indicate that subjective MOS is costly, low-level features are incomplete, and single-facet emotion modeling is too narrow for the task the paper targets (Lin et al., 23 Oct 2025). At the same time, DeEAR does not eliminate subjectivity; rather, it converts human preference into an objective score through proxy modeling and supervised fusion. This distinction is central to its formulation.

Within S2S research, DeEAR therefore occupies a dual role. It is both an evaluation framework and an instrument for dataset construction, enabling fair benchmarking and targeted curation of expressive utterances (Lin et al., 23 Oct 2025). The reported ability to distinguish expressiveness gaps across S2S models and to select 14K expressive utterances that materially improve downstream scores indicates that the framework is designed as infrastructure for iterative model development rather than as a standalone diagnostic (Lin et al., 23 Oct 2025).