---
title: 'DeEAR: Evaluating Expressive Speech'
url: https://www.emergentmind.com/topics/deear
type: topic
---

# DeEAR: Evaluating Expressive Speech

Searching arXiv for the requested paper to ground the article in the current record.
DeEAR, short for “Decoding the Expressive Preference of eAR,” is a framework for converting human preference for speech expressiveness into an objective score in speech-to-speech (S2S) evaluation and data curation. It was introduced to address the persistent gap between intelligibility and natural expressiveness in modern end-to-end S2S systems, which often sound “robotic” despite producing intelligible audio [2510.20513]. The framework is grounded in phonetics and psychology and evaluates speech across three dimensions—Emotion, Prosody, and Spontaneity—before fusing them into a final expressiveness score on a 0–100 scale. In the reported experiments, DeEAR achieves strong alignment with human judgments of expressiveness, with Spearman’s Rank Correlation Coefficient (SRCC) of 0.86 using only \(\sim 480\) human-annotated clips, and supports both system benchmarking and expressiveness-driven dataset construction [2510.20513].

## 1. Motivation and problem setting

Recent S2S models generate intelligible speech but still lack natural expressiveness, and the central obstacle identified for this deficiency is the absence of a reliable evaluation metric [2510.20513]. Existing approaches to expressiveness evaluation are described as incomplete in different ways: subjective MOS tests are costly, low-level acoustic features such as pitch and energy are perceptually incomplete, and single-facet emotion classifiers are too narrow [2510.20513]. Within this framing, DeEAR is designed not merely as an auxiliary scoring heuristic, but as a multi-dimensional metric intended to objectify human preference regarding expressiveness.

The framework has three stated high-level goals. First, it aims to provide an objective metric that correlates strongly with human judgments of expressiveness, reporting SRCC \(=0.86\) in Section 4.1 [2510.20513]. Second, it is explicitly data-efficient, requiring only \(\sim 480\) human-annotated clips for alignment [2510.20513]. Third, it is intended to enable downstream tasks beyond evaluation, including reliable benchmarking of S2S models and expressiveness-driven data curation [2510.20513].

This framing places DeEAR within a broader methodological shift from purely subjective evaluation toward structured preference modeling. A plausible implication is that DeEAR treats expressiveness as a latent construct that can be decomposed into operational subdimensions, each measurable through proxy models and then reassembled through a learned fusion layer.

## 2. Three-dimensional decomposition of expressiveness

DeEAR evaluates speech across three dimensions derived from phonetics, psychology, and paralinguistics: Emotion, Prosody, and Spontaneity [2510.20513]. Each dimension is scored on a 0–100 scale, and the decomposition is intended to capture aspects of expressiveness that are not adequately represented by a single scalar proxy.

### Emotion Intensity

Emotion Intensity is defined as emotion arousal [2510.20513]. Its proxy model is a fine-tuned wav2vec2-large-robust model pretrained on MSP-IMPROV and further trained on 12 k Chinese (CNSCED) and 2 k English (IEMOCAP) data [2510.20513]. The output is a scalar \(S_{\text{emo}}\in[0,100]\) proportional to predicted arousal [2510.20513]. When training DeEAR-Base, the corresponding regression loss is

$$
\mathcal{L}_{\text{emo}}
= \frac{1}{N}\sum_{i=1}^N
\bigl(\hat S^{(i)}_{\text{emo}} - S^{(i)}_{\text{emo,teacher}}\bigr)^2.
$$

This formulation makes Emotion a teacher-supervised regression target rather than a discrete categorical classifier, which is consistent with the paper’s preference-centric objective [2510.20513].

### Prosodic Richness

Prosodic Richness is produced by a proxy annotator based on Gemini 2.5 Pro LMM with engineered prompts [2510.20513]. The resulting scalar \(S_{\text{pros}}\in[0,100]\) achieves SRCC \(=0.73\) with respect to human judgments [2510.20513]. The paper does not specify an explicit analytic formula for this subscore; instead, richness is directly represented by the LMM’s score. The DeEAR-Base supervision term is

$$
\mathcal{L}_{\text{pros}}
= \frac{1}{N}\sum_i
\bigl(\hat S_{\text{pros}}^{(i)}-S_{\text{pros,teacher}}^{(i)}\bigr)^2.
$$

The use of an LMM here is significant because prosodic richness is presented as a dimension that is difficult to reduce to low-level handcrafted descriptors alone. A plausible implication is that the prompt-engineered LMM acts as a high-level annotator for contours and variation patterns that are hard to specify analytically.

### Spontaneity

Spontaneity is defined through the intuition that “perceptual incongruence” arises when acoustic quality is very high, described as hyper-clean, but base spontaneity is low [2510.20513]. The mean DNSMOS quality metric is

$$
M_{\text{avg}}
= \tfrac{1}{4}\bigl(M_{\text{OVRL}} + M_{\text{SIG}} + M_{\text{BAK}} + M_{P.808}\bigr).
$$

The base spontaneity level is \(L_{\text{base}}\in\{1,3,5,7,9\}\), manually assigned per dataset [2510.20513]. A clip is “hyper-clean” if all four DNSMOS scores exceed threshold \(T_q=3.5\) [2510.20513]. The heuristic spontaneity score is then defined as

$$
S_{\text{spon}} =
\begin{cases}
\mathrm{map}_{\text{penalty}}\bigl(M_{\text{avg}}\bigr)
& \text{if hyper-clean and } L_{\text{base}}<L_{\max},\\
\mathrm{map}_{\text{normal}}\bigl(M_{\text{avg}}\bigr)
& \text{otherwise},
\end{cases}
\tag{1}
$$

where \(\mathrm{map}_{\text{normal}}\) linearly scales \(M_{\text{avg}}\) to \([L_{\text{base}}-1,\,L_{\text{base}}+1]\), rewarding congruence, while \(\mathrm{map}_{\text{penalty}}\) reversely scales to a narrow punitive range, for example \([0,0.5]\) when \(L_{\text{base}}=1\) [2510.20513]. This heuristic labels \(\sim 20\) k utterances, which are then used to fine-tune wav2vec2 with

$$
\mathcal{L}_{\text{spon}}
= \frac{1}{N}\sum_i
\bigl(\hat S_{\text{spon}}^{(i)}-S_{\text{spon,heuristic}}^{(i)}\bigr)^2.
$$

Among the three dimensions, Spontaneity is the most overtly heuristic. The paper treats this not as a limitation to conceal, but as a formalization of the observation that over-processed speech can sound less natural precisely because its acoustic cleanliness is mismatched with conversational spontaneity [2510.20513].

## 3. Efficient alignment and model architecture

DeEAR uses a two-level construction: three proxy scorers first estimate the dimensional subscores, and a fusion model then aligns them to human judgments of overall expressiveness [2510.20513]. After training the proxy models, the framework collects a human preference set of \(n=480\) clips, where each clip has proxy scores \((S_{\text{emo}}, S_{\text{pros}}, S_{\text{spon}})\) and a human-annotated overall expressiveness label \(y\in[0,100]\) [2510.20513].

The fusion model is an XGBoost regressor \(f\) trained by minimizing squared error:

$$
\mathcal{L}_{\mathrm{fusion}}
= \frac{1}{n}\sum_{i=1}^n
\bigl(f(\mathbf{x}^{(i)}) - y^{(i)}\bigr)^2,
\qquad
\mathbf{x}^{(i)}=(S_{\text{emo}}^{(i)},S_{\text{pros}}^{(i)},S_{\text{spon}}^{(i)}).
$$

The final expressiveness score is

$$
S_{\text{expr}}
= f\bigl(S_{\text{emo}},S_{\text{pros}},S_{\text{spon}}\bigr)
\in[0,100].
$$

To reduce deployment cost, the paper introduces DeEAR-Base, a distilled model based on a single wav2vec2-xlsr-53 backbone with three regression heads [2510.20513]. The full “teacher” pipeline is applied to 20 k unlabeled utterances to obtain three proxy scores, and DeEAR-Base is trained to minimize the multi-task objective

$$
\mathcal{L}_{\text{DeEAR-Base}}
= \mathcal{L}_{\text{emo}} + \mathcal{L}_{\text{pros}} + \mathcal{L}_{\text{spon}}.
$$

The deployed system is therefore a cascade: DeEAR-Base followed by XGBoost fusion [2510.20513]. This architecture embodies the paper’s claim of efficient alignment: expensive or heuristic annotators are used to generate teachers, then distilled into a compact acoustic backbone aligned to a relatively small human preference set.

## 4. Evaluation protocol and empirical validity

To quantify alignment with human expressiveness ratings, the paper uses Spearman’s Rank Correlation Coefficient, computed as

$$
\rho
= 1 - \frac{6\sum_{i=1}^n d_i^2}{n(n^2 - 1)},
$$

where \(d_i\) is the rank difference for sample \(i\) [2510.20513]. The reported \(\rho\) for the fused expressiveness score is \(0.86\) [2510.20513]. The evaluation also reports Pearson correlation and uses human annotations collected on four 100-utterance sets, with Krippendorff’s \(\alpha=0.72\) [2510.20513].

The per-dimension and fused correlations are summarized below.

| Dimension | PCC | SRCC |
|---|---:|---:|
| Emotion | 0.72 | 0.65 |
| Prosody | 0.70 | 0.68 |
| Spontaneity | 0.84 | 0.84 |
| Expressiveness | 0.91 | 0.86 |

These results show that the fused expressiveness score exceeds the individual dimensions in alignment with human ratings [2510.20513]. This suggests that no single component is sufficient to recover perceived expressiveness, and that the decomposition-plus-fusion strategy is central rather than incidental.

The framework is also evaluated at the system level on seven S2S systems and 20 prompts, where DeEAR’s system-level ranking attains SRCC \(=0.96\) against four-listener human MOS on a 3-point scale [2510.20513]. The score spread is approximately 60 points, with DouBao at 65.4 and Qwen2.5-Omni at 5.3 [2510.20513]. In this setting, DeEAR is presented as a benchmarking instrument rather than solely a clip-level predictor.

## 5. ExpressiveSpeech dataset construction

One of DeEAR’s principal downstream uses is data curation. The paper constructs the ExpressiveSpeech dataset from five sources—Expresso, NCSSD, M\(^3\)ED, MultiDialog, and IEMOCAP—amounting to \(\sim 630\) hours of raw Chinese and English speech [2510.20513]. The curation pipeline is described in five steps.

1. Audio standardization and noise removal to 16 kHz mono, using ClearerVoice for denoising and speaker separation [2510.20513].  
2. Quality scoring via DNSMOS P.835, with average OVRL \(=3.17\) [2510.20513].  
3. Expressiveness scoring via DeEAR across Emotion, Prosody, Spontaneity, and the fused score [2510.20513].  
4. Thresholding at \(S_{\text{expr}}\ge 63.5\), described as empirically aligned with “high expressiveness” in human tests [2510.20513].  
5. ASR transcription added as metadata [2510.20513].

The final dataset contains 14 000 utterances, approximately 51 hours, balanced between English and Chinese [2510.20513]. Its average expressiveness score is \(S_{\text{expr}}=80.2\), which is reported as far above the source averages of 39.4–62.9 [2510.20513].

The significance of this curation pipeline lies in the fact that DeEAR is used not only to judge model outputs, but also to shape the training distribution. A plausible implication is that the framework functions as both an evaluator and a selection policy for collecting training examples with high expressive utility.

## 6. Applications, gains, and limitations

The paper identifies three principal applications for DeEAR: automated benchmarking of conversational S2S and TTS systems, data curation for expressiveness gains, and reward modeling for reinforcement-learning-based speech generation [2510.20513]. Of these, the data-curation application is substantiated with the most detailed experimental results.

A baseline S2S system is compared with an Expressive-FT model fine-tuned on ExpressiveSpeech [2510.20513]. The reported objective gains in overall expressiveness are substantial:

| Setting | Baseline \(S_{\text{expr}}\) | Ours \(S_{\text{expr}}\) |
|---|---:|---:|
| In domain | 2.3 | 24.0 |
| Out of domain | 1.8 | 23.0 |
| Overall | 2.0 | 23.4 |

The paper also reports component-level gains: Emotion from 5.7 to 15.9, Prosody from 35.7 to 36.7, and Spontaneity from 33.7 to 62.0, all with \(p<0.001\) [2510.20513]. Subjectively, A/B preference by 10 native speakers favored the fine-tuned system 78.5% versus baseline 10.0%, with 11.5% ties and \(p<0.001\) [2510.20513].

The limitations are explicitly stated. Prosodic Richness depends on an LMM proxy and may inherit LMM biases or domain mismatches [2510.20513]. The Spontaneity heuristic requires DNSMOS and may be sensitive to noise or artifact removal [2510.20513]. Fusion alignment is based on only 480 clips, and the paper notes that future larger studies could refine the mapping [2510.20513]. These limitations constrain any interpretation of DeEAR as a definitive theory of expressiveness; it is better understood as a practical alignment framework built from carefully selected proxies.

Potential extensions noted in the paper include integrating DeEAR as a reward in RLHF for end-to-end expressiveness control, expanding beyond bilingual English and Chinese, and exploring richer sub-dimension features such as linguistic and semantic features as well as multi-modal cues including video and text [2510.20513]. These directions suggest that the current framework is intentionally modular.

## 7. Position within expressive speech evaluation

DeEAR is presented as an answer to a specific methodological gap: the lack of a reliable, objective, multi-dimensional metric for speech expressiveness [2510.20513]. Its distinguishing characteristic is that it does not reduce expressiveness to a single low-level descriptor or to an isolated emotion label. Instead, it operationalizes human preference through three dimensions and an efficient alignment stage learned from a small human-rated set [2510.20513].

This design also helps clarify a common misconception in expressive speech evaluation: that expressiveness can be captured adequately either by MOS alone or by purely acoustic proxies. The DeEAR results indicate that subjective MOS is costly, low-level features are incomplete, and single-facet emotion modeling is too narrow for the task the paper targets [2510.20513]. At the same time, DeEAR does not eliminate subjectivity; rather, it converts human preference into an objective score through proxy modeling and supervised fusion. This distinction is central to its formulation.

Within S2S research, DeEAR therefore occupies a dual role. It is both an evaluation framework and an instrument for dataset construction, enabling fair benchmarking and targeted curation of expressive utterances [2510.20513]. The reported ability to distinguish expressiveness gaps across S2S models and to select 14K expressive utterances that materially improve downstream scores indicates that the framework is designed as infrastructure for iterative model development rather than as a standalone diagnostic [2510.20513].

Source: https://www.emergentmind.com/topics/deear