Relative Voice Impression Estimation (RIE)

• RIE is a comparative framework that models perceptual differences between two utterances along interpretable antonymic axes.
• It employs methods ranging from classical acoustic features to self-supervised deep learning to predict a low-dimensional vector of relative impressions.
• The approach informs applications in expressive synthesis, voice conversion, and clinical assessments while addressing challenges like annotation variability and dataset constraints.

Relative Voice Impression Estimation (RIE) is a supervised learning framework for predicting how the perceived impression of one utterance changes relative to another utterance, most explicitly for two utterances of the same speaker reading the same text in different speaking styles. In its canonical formulation, the model maps a pair of speech signals $(x_a, x_b)$ to a low-dimensional vector of perceptual differences $$\mathbf{r}_{\text{rel}} = f(x_a, x_b) \in \mathbb{R}^K$$, where each component encodes the direction and magnitude of change along an interpretable antonymic axis such as Dark–Bright or Cold–Warm; in the first systematic RIE study, $K=9$ (Fujita et al., 15 Feb 2026). Closely related work broadens this comparative perspective to pairwise preference learning for subjective voice descriptors, comparative timbre judgment, continuous dyadic impression recognition, and impression-conditioned speech synthesis, so the term now refers not only to a single benchmark setup but to a family of methods for modeling relative perceptual structure in speech (Kondo et al., 24 Jun 2025).

1. Conceptual definition and task boundaries

RIE differs from conventional absolute scoring in both input structure and target semantics. Absolute paralinguistic tasks map a single utterance to a scalar or vector label, such as an emotion class, a valence-arousal score, a MOS-like quality score, or an absolute speaker impression. RIE instead takes two utterances as input and predicts a relative perceptual shift, explicitly modeling “more $X$,” “less $Y$,” and the magnitude of that change rather than a fixed absolute level (Fujita et al., 15 Feb 2026).

In the most explicit formulation, the two utterances are constrained to come from the same speaker and the same text, so that speaker identity and lexical content are held constant and the remaining variance is primarily expressive and prosodic. This design isolates within-speaker style-induced variation as the source of impression change. The estimation target is a low-dimensional vector of subjective differences derived directly from pairwise judgments rather than from subtracting two independently collected absolute impression vectors (Fujita et al., 15 Feb 2026).

Comparative voice-impression research, however, is not restricted to this single setup. In personalized subjective voice descriptor assessment, the core object is a scalar scoring function $f_\theta(x)$ whose ordering over pairs is learned from Comparison Category Ratings (CCR); this is effectively a relative impression estimator specialized to one descriptor at a time (Kondo et al., 24 Jun 2025). In comparative timbre assessment, the task can be framed as a binary decision about whether voice $B$ is stronger than voice $A$ along a descriptor $v$, yielding a relative but non-metric formulation (Sheng et al., 14 May 2025). In continuous dyadic impression recognition, no explicit pairwise “$A$ is warmer than $$\mathbf{r}_{\text{rel}} = f(x_a, x_b) \in \mathbb{R}^K$$0” labels are collected, yet the architecture jointly models emitter and receiver signals and thereby learns latent relational structure that is directly informative for RIE (Li et al., 2022).

A useful boundary line is therefore the following. RIE is not synonymous with speech quality assessment, emotion recognition, or single-utterance impression regression. It is a comparative estimation problem in which the output is defined relative to another utterance, another listener state, or another point in impression space. This also implies that binary preference tasks are only one special case: some formulations estimate order only, whereas others estimate continuous relative displacement.

2. Impression spaces and annotation protocols

The first dedicated RIE study defines a 9-dimensional relative impression space using antonymic axes adopted from voice quality assessment literature: High–Low pitched, Clear–Hoarse, Calm–Restless, Powerful–Weak, Youthful–Elderly, Thick–Thin, Tense–Relaxed, Dark–Bright, and Cold–Warm. For each utterance pair, listeners judge how the impression of the second utterance differs from the first on a 7-point Likert scale, where the midpoint denotes no perceptual change and the endpoints denote strong movement toward one pole or the other. The final label is the mean rating per axis over at least 10 listeners, yielding a continuous 9-dimensional target vector $$\mathbf{r}_{\text{rel}} = f(x_a, x_b) \in \mathbb{R}^K$$1 (Fujita et al., 15 Feb 2026).

The data design in that study is tightly controlled. The corpus is an internal Japanese speech dataset recorded at 22 kHz and downsampled to 16 kHz for modeling. It contains 1,087 utterances and 814 utterance pairs, all produced by a single professional female voice actor reading the same text in 52 speaking styles derived from the “Guideline for TTS Speaking Style Classification” (JEITA). The subjective evaluation involved 3,920 crowdworkers, and both presentation orders $$\mathbf{r}_{\text{rel}} = f(x_a, x_b) \in \mathbb{R}^K$$2 and $$\mathbf{r}_{\text{rel}} = f(x_a, x_b) \in \mathbb{R}^K$$3 were collected to mitigate order effects (Fujita et al., 15 Feb 2026).

Other comparative label geometries coexist with this vector-valued formulation. In CCR-based subjective voice descriptor learning, raters choose among four ordered categories—“$$\mathbf{r}_{\text{rel}} = f(x_a, x_b) \in \mathbb{R}^K$$4 is more so,” “$$\mathbf{r}_{\text{rel}} = f(x_a, x_b) \in \mathbb{R}^K$$5 is a little more so,” “$$\mathbf{r}_{\text{rel}} = f(x_a, x_b) \in \mathbb{R}^K$$6 is a little more so,” and “$$\mathbf{r}_{\text{rel}} = f(x_a, x_b) \in \mathbb{R}^K$$7 is more so”—which are then mapped to soft preference probabilities $$\mathbf{r}_{\text{rel}} = f(x_a, x_b) \in \mathbb{R}^K$$8 for pairwise ranking. This setup is scalar per descriptor, but its supervision is explicitly relative and encodes both direction and strength (Kondo et al., 24 Jun 2025). In the Voice Timbre Attribute Detection challenge, labels are binary hypotheses over ordered speaker pairs and descriptor dimensions, with “speaker $$\mathbf{r}_{\text{rel}} = f(x_a, x_b) \in \mathbb{R}^K$$9 is stronger than speaker $K=9$0 in descriptor $K=9$1” as the atomic supervisory unit (Sheng et al., 14 May 2025).

A related but broader impression representation appears in impression-controllable TTS. There, voice impression is encoded as an 11-dimensional vector spanning Low–High, Masculine–Feminine, Clear–Hoarse, Calm–Restless, Powerful–Weak, Youthful–Elderly or Youthful–Aged, Thick–Thin, Tense–Relaxed or Firm–Relaxed, Dark–Bright, Cold–Warm, and Slow–Fast. In Japanese zero-shot TTS this vector is trained from crowd-sourced ratings plus standardized speech rate (Fujita et al., 6 Jun 2025), while LibriTTS-VI provides a public English corpus with manual 11-dimensional voice-impression labels, expert ratings, and a corresponding voice impression estimator (Ohmura et al., 19 Sep 2025).

These annotation schemes imply that RIE does not depend on a single psychometric template. It can be built from direct relative vector ratings, soft pairwise preferences, binary comparative decisions, or continuous trajectories. What unifies them is that impression is treated as a perceptual relation rather than as a purely intrinsic property of one isolated utterance.

3. Modeling paradigms

One major modeling family treats RIE as direct regression from a pair of utterances to a relative impression vector. In the canonical 9-dimensional setup, classical acoustic baselines use openSMILE with eGeMAPSv02 on voiced segments. For each utterance $K=9$2, an utterance-level feature vector $K=9$3 is extracted, and the relative input is the feature difference $K=9$4. Feature–target correlations are computed per axis, the top-8 features are selected, and standard regressors including Linear Regression, Ridge Regression, PLS2, Random Forest, Gradient Boosted Decision Trees, and SVR are trained. A neural variant concatenates $K=9$5 and predicts the 9-dimensional target vector with a 3-layer MLP trained by mean squared error (Fujita et al., 15 Feb 2026).

A second family uses self-supervised speech representations. The dedicated RIE investigation employs rinna/japanese-hubert-base, producing 768-dimensional frame-level embeddings that are passed through a BiLSTM and attention pooling to obtain a 128-dimensional utterance embedding $K=9$6. The pairwise representation is then $K=9$7, followed by a three-layer MLP that outputs $K=9$8. The model is trained with MSE over the 9 dimensions using AdamW with learning rate 0.002 and batch size 8, while HuBERT is used as a feature extractor rather than explicitly fine-tuned (Fujita et al., 15 Feb 2026).

A third family casts relative impression as pairwise ranking over scalar impression scores. In subjective voice descriptor assessment, both a CNN-BLSTM-FC model operating on 257-dimensional magnitude spectrograms and a fixedSSL-FC model based on frozen wav2vec2.0 features output a scalar score $K=9$9. RankNet then converts score differences into preference probabilities

$X$0

with training targets $X$1 derived from the four CCR categories. This directly optimizes relative ordering rather than indirect regression to absolute ratings (Kondo et al., 24 Jun 2025).

A fourth family introduces cross-domain relational modeling. Continuous dyadic impression recognition uses a BLSTM encoder over emitter and receiver sequences, followed by intra-domain attention and inter-domain attention: $X$2 with knowledge-distillation and similarity-enhancement regularization. The primary task is regression of continuous warmth and competence trajectories, but the architecture is explicitly designed to capture relationships between emitter and receiver signals. This makes it directly informative for RIE, especially for formulations in which impression depends not only on vocal acoustics but also on listener reactions (Li et al., 2022).

Current multimodal LLMs constitute a fifth, exploratory family. In zero-shot settings, GPT-5 and Gemini 2.5 Pro were prompted with two audio files and asked to output nine numerical scores for the same antonymic axes used in RIE. This established that audio-capable MLLMs can be evaluated within the same pairwise framework, but the results also showed that prompt-based inference is not yet competitive with dedicated acoustic or SSL models for fine-grained comparative regression (Fujita et al., 15 Feb 2026).

4. Evaluation metrics and empirical findings

The dedicated RIE benchmark evaluates each impression axis with Pearson correlation and Concordance Correlation Coefficient (CCC) under 10-fold cross-validation. Across all nine axes, SSL consistently outperforms classical acoustic features. Representative gains include Dark–Bright, where the best classical neural model reaches Pearson $X$3 and CCC $X$4 while the HuBERT-based SSL model reaches $X$5 and CCC $X$6, and Cold–Warm, where performance improves from $X$7, CCC $X$8 to $X$9, CCC $Y$0. The strongest improvements are reported for “complex and dynamic” impressions such as Calm–Restless, Powerful–Weak, and Cold–Warm (Fujita et al., 15 Feb 2026).

The same study also reports a one-fold comparison against audio-capable MLLMs. SSL remains best on every axis, classical acoustic models come next, Gemini is weaker, and GPT-5 is near zero or very small on most dimensions; for example, GPT-5 yields $Y$1 on Dark–Bright and a negative correlation on Powerful–Weak, whereas Gemini reaches approximately $Y$2 on Dark–Bright and $Y$3 on Cold–Warm. The conclusion is that current MLLMs are not yet effective for comparative two-utterance tasks such as relative impression estimation in zero-shot settings (Fujita et al., 15 Feb 2026).

CCR-based subjective voice descriptor learning evaluates ranking accuracy with $Y$4, the fraction of test pairs for which the model’s predicted ordering matches the ground-truth preference. Two variants are used: $Y$5-strong for “is more so” comparisons and $Y$6-weak for “is a little more so” comparisons. Estimated human-consistency upper bounds show that the task is intrinsically noisy: for example, $Y$7 upper bounds are 0.922 for youthfulF, 0.891 for youthfulM, and 0.907 for resonantM, while $Y$8 upper bounds are 0.792, 0.726, and 0.756, respectively. Empirically, CCR-based RankNet training yields higher $Y$9-strong than ACR-based MSE training across descriptors and model classes, particularly in very small-data regimes (Kondo et al., 24 Jun 2025).

Comparative findings from adjacent tasks reinforce the same pattern. In continuous dyadic impression recognition, a cross-domain model with attention and regularization reaches CCC 0.770 for competence and 0.748 for warmth, and ablating emitter audio reduces competence from 0.770 to 0.727 and warmth from 0.748 to 0.699, indicating that vocal information remains crucial even when receiver-side visual and physiological cues are available (Li et al., 2022). In voice impression control, improved methods reduce objective 11-dimensional impression-vector MSE from 0.61 to 0.41 and subjective MSE from 1.15 to 0.92 while maintaining high fidelity, showing that relative movement in impression space can be both measurable and controllable (Ohmura et al., 19 Sep 2025).

Taken together, these results support three empirical regularities. First, relative voice impression is predictable with nontrivial accuracy. Second, self-supervised representations are currently the strongest general-purpose front end for fine-grained pairwise impression regression. Third, relative supervision—whether as direct difference vectors or as pairwise comparisons—aligns better with the task than absolute single-sample scoring alone.

5. Relation to adjacent research areas and practical uses

RIE is closely related to, but distinct from, several neighboring research programs. Comparative timbre assessment asks whether one speaker is stronger than another in a descriptor such as bright, coarse, soft, or magnetic. This is relative impression estimation specialized to timbre descriptors and binary directionality, with evaluation by EER and accuracy rather than continuous regression (Sheng et al., 14 May 2025). Subjective voice descriptor learning uses scalar impression functions trained from ACR or CCR and is particularly aimed at personalized descriptors such as “my favorite voice,” making it an instance of RIE in a limited-data, per-descriptor regime (Kondo et al., 24 Jun 2025).

Clinical voice-quality assessment provides another adjacent formulation. CAPE-V prediction is an absolute regression problem for overall severity, roughness, breathiness, strain, pitch, and loudness, but the resulting scalar scores can be converted into relative judgments by comparing two utterances or their predicted differences. That work also shows that lightly weighted interpretable parameters such as jitter, shimmer, HNR, and zero-crossing rate can outperform several pretrained models on noise-related voice-quality scorings, especially roughness and strain (Lin et al., 2023). This suggests that RIE need not always favor large latent representations; feature choice depends on the perceptual axis under study.

In speech synthesis, voice impression control operationalizes the inverse problem of RIE. Instead of estimating $f_\theta(x)$0 from two utterances, the system manipulates an impression vector so that the synthesized utterance moves to a desired point or offset in impression space. Zero-shot TTS studies use 11-dimensional impression vectors and show monotonic changes under modulation levels such as $f_\theta(x)$1, while LibriTTS-VI introduces public annotation standards and explicit impression-leakage metrics based on the mismatch between target and realized impression vectors (Fujita et al., 6 Jun 2025, Ohmura et al., 19 Sep 2025).

Several application domains follow directly from these formulations. Expressive TTS and style control can use RIE outputs as target perceptual deltas, so that a new synthesis is “0.5 more Powerful and 0.3 more Warm” relative to a baseline. Voice conversion and style transformation can use relative impression vectors as training losses or evaluation criteria. Voice coaching can compare current and previous attempts and report directional changes rather than absolute traits. Comparative retrieval can search a library for utterances that are slightly brighter, calmer, or more youthful than a given reference (Fujita et al., 15 Feb 2026, Fujita et al., 6 Jun 2025).

Two recurrent misconceptions are contradicted by the current evidence. One is that RIE is merely pairwise MOS; the literature instead treats it as a structured perceptual estimation problem with interpretable axes, continuous magnitudes, and sometimes transitive scalar score functions (Fujita et al., 15 Feb 2026, Kondo et al., 24 Jun 2025). Another is that classical acoustic features are uniformly obsolete. They are outperformed by SSL on the main 9-axis RIE benchmark, but they remain reasonably effective for more directly acoustic dimensions such as High–Low or Dark–Bright, and compact interpretable parameters can even surpass pretrained models for clinically relevant roughness and strain (Fujita et al., 15 Feb 2026, Lin et al., 2023).

6. Limitations, controversies, and open directions

The present RIE literature remains methodologically narrow in several respects. The first dedicated benchmark uses a single professional Japanese female voice actor, one text, 52 speaking styles, and 814 utterance pairs. This was an intentional design choice to isolate expressive and prosodic variation, but it limits immediate generalization across speakers, genders, accents, and languages (Fujita et al., 15 Feb 2026). The same issue appears in voice impression control, where early systems are trained entirely on Japanese or audiobook English corpora and rely on relatively small manually annotated subsets for estimator training (Fujita et al., 6 Jun 2025, Ohmura et al., 19 Sep 2025).

A second limitation is annotator inconsistency. In subjective voice descriptor learning, $f_\theta(x)$2 upper bounds remain well below 1.0, especially for weak-preference pairs, showing that disagreement is intrinsic rather than merely algorithmic noise (Kondo et al., 24 Jun 2025). In LibriTTS-VI, Krippendorff’s alpha ranges from around 0.19 for Cold–Warm to 0.875 for Masculine–Feminine, with average $f_\theta(x)$3, which is comparable to other subjective audio tasks but still implies substantial uncertainty in some dimensions (Ohmura et al., 19 Sep 2025). A plausible implication is that future RIE systems will benefit from explicit uncertainty modeling rather than point prediction alone.

A third limitation concerns descriptor coupling. Inter-scale correlations are strong in both impression-control and public-corpus settings; for example, Low–High and Masculine–Feminine are heavily correlated in LibriTTS-VI, and zero-shot TTS work reports strong correlations such as High–Low pitched versus Thick–Thin and Dark–Bright versus Tense–Relaxed. This suggests that independent per-dimension manipulation or regression is only an approximation and that multivariate models may be preferable when relative changes along one axis propagate into others (Fujita et al., 6 Jun 2025, Ohmura et al., 19 Sep 2025).

A fourth unresolved issue is model robustness outside tightly controlled settings. The dedicated RIE study reports only exploratory zero-shot MLLM experiments on one representative fold and explicitly notes that conclusions about MLLMs are preliminary (Fujita et al., 15 Feb 2026). Personalized descriptor learning highlights label noise in “a little more so” cases and leaves active pair selection unexplored (Kondo et al., 24 Jun 2025). Comparative timbre benchmarks expose only final binary relations rather than raw rater uncertainty and remain restricted to a fixed descriptor inventory (Sheng et al., 14 May 2025). Dyadic impression recognition shows how listener reactions can enrich the signal, but it does not collect explicit relative voice labels and therefore supports RIE only indirectly (Li et al., 2022).

The main directions already identified in the literature are consistent. These include extending RIE to multi-speaker and cross-gender settings, moving beyond Japanese and beyond single-text controlled corpora, applying the framework to spontaneous speech and additional impression axes, exploring few-shot or fine-tuned MLLMs rather than prompt-only inference, and using pairwise or ranking objectives more directly in impression-control pipelines (Fujita et al., 15 Feb 2026, Kondo et al., 24 Jun 2025, Ohmura et al., 19 Sep 2025). More broadly, the field is converging on a view in which voice impression is a structured perceptual space, relative change is the most operationally meaningful target, and the central technical challenge is to estimate or control that change while disentangling it from speaker identity, lexical content, and contextual confounds.