VCTK-RVA: Annotated Corpus for Voice Attributes

• VCTK-RVA is an annotated corpus that extends the VCTK dataset with expert pairwise voice descriptor comparisons for controlled attribute modeling.
• The dataset comprises 6,038 annotated comparisons using 18 curated descriptors, validated by a ~91.78% agreement among listeners.
• It underpins research in voice conversion, timbre synthesis, and attribute detection, serving as a benchmark for advanced speech processing methods.

The VCTK-RVA dataset is a specialized annotated corpus that builds upon the original VCTK speech corpus to advance research in voice attribute editing, detection, and explainability. By providing expert-generated, fine-grained, pairwise annotations of relative voice attribute differences, VCTK-RVA enables rigorous supervised learning and benchmarking of perceptual voice timbre modeling in both generative and discriminative speech applications.

1. Origin, Structure, and Annotation Process

VCTK-RVA is constructed by extending the widely used VCTK corpus, which itself contains recordings from 110 speakers (62 females, 48 males) with approximately 400 sentences per speaker. VCTK-RVA adds a layer of expert manual annotations describing relative voice characteristic differences among speakers, making it unique in its capacity to support attribute-based modeling.

The creation process involves several stages:

• Descriptor Set Development: Speech specialists curate a set of concise and frequently used voice descriptors (e.g., “Bright,” “Coarse,” “Low,” “Thin,” “Magnetic”), merging synonyms and identifying those most common in describing speaker characteristics. The final descriptor set includes approximately 18 descriptors spanning modalities such as auditory, visual, and tactile perception. Some descriptors are gender-specific (e.g., “Shrill” for females, “Husky” for males).
• Pairwise Expert Annotation: Experts conduct systematic, pairwise comparisons among same-gender speaker pairs, listening to reference utterances and annotating each comparison with one or more descriptors expressing relative prominence. The annotation tuple takes the form {SpeakerA, SpeakerB, v}, where v may contain one or more descriptors (or the label “Similar” for indistinguishable voices). The dataset comprises 6,038 such annotated data points (computed via 62×61 + 48×47).
• Validation: To ensure reliability, 200 annotated comparisons are cross-verified by 40 independent listeners via crowdsourcing (Amazon Mechanical Turk), achieving ~91.78% agreement.

2. Annotation Format and Statistical Properties

Annotations are structured as ordered tuples:

SpeakerA	SpeakerB	Descriptor(s)
X	Y	"Bright"
X	Y	"Magnetic, Thin"
X	Y	"Similar"

• ~71.19% of tuples feature a single descriptor, ~26.84% contain two, and ~1.97% feature three descriptors. The “Similar” label is present in ~6.8% of samples, signaling indistinguishability in given attributes.
• Each comparison highlights which attributes of SpeakerB are stronger than those of SpeakerA, facilitating fine-grained, supervised learning of attribute mappings and contrasts.

3. Roles in Voice Attribute Editing and Timbre Detection

VCTK-RVA is pivotal in several research directions:

• Voice Attribute Editing: The dataset underpins architectures such as VoxEditor (Sheng et al., 2024). By mapping natural language attribute prompts (e.g., “more magnetic and bright”) to relative changes in speaker embeddings, models learn to interpolate and modify timbre along specific dimensions. Modules such as the Residual Memory (ResMem) block and Voice Attribute Degree Prediction (VADP) block are trained using these reference tuples to achieve attribute-controllable synthesis.
• Voice Timbre Attribute Detection: VCTK-RVA serves as the gold standard for the vTAD task (He et al., 14 May 2025, Wu et al., 21 Aug 2025, Chen et al., 8 Sep 2025), where the goal is to explain voice timbre by comparing intensities of sensory attributes in speech utterances. Labels are recast into binary vectors per descriptor, where each dimension $l_n \in \{1, 0, -1\}$ encodes whether SpeakerB is stronger in descriptor $n$ relative to SpeakerA. Supervised frameworks extract speaker embeddings (e.g., via ECAPA-TDNN or FACodec) and apply discriminative modules (Diff-Net, RTSA$^2$) to infer relative intensities.

4. Methodologies Leveraging VCTK-RVA

Multiple advanced methodologies exploit the relative annotation structure and richness of VCTK-RVA:

• Speaker Embedding Comparison: Embeddings from two utterances are concatenated and processed by a network (e.g., Diff-Net) yielding a vector $\hat{y} = \sigma(\text{Diff-Net}(e_P))$, estimating the probability $P(\text{SpeakerB} > \text{SpeakerA}|v_n)$ for each descriptor. Loss is computed via binary cross-entropy on labeled dimensions.
• Graph-Based Data Augmentation (QvTAD): A Directed Acyclic Graph (DAG) is built where nodes represent “Speaker, Attribute” pairs and edges reflect annotated comparisons. Disjoint-Set Union (DSU) transitivity is assumed to mine additional pseudo-annotated pairs, increasing the effective training set from 6,038 to 166,409 and balancing rare attributes (Wu et al., 21 Aug 2025).
• Differential Attention Mechanisms: Relative Timbre Shift-Aware Differential Attention (RTSA$^2$) amplifies perceptual differences using equations such as

$$\Delta = e_b^{(\text{att})} - e_a^{(\text{att})}, \quad \hat{\Delta} = \tanh(\Delta) \cdot \|\Delta\|_2 \cdot \gamma$$

where attention maps suppress shared features and $\gamma$ is contrast strength (Wu et al., 21 Aug 2025).

5. Benchmarks, Challenge Results, and Encoder Insights

VCTK-RVA provides a rigorous testbed for the first voice timbre attribute detection challenge (Chen et al., 8 Sep 2025). Evaluation splits speakers into “seen” (present during training) and “unseen” (not present), with each speaker pair assessed across 18 descriptors using metrics such as Accuracy (ACC) and Equal Error Rate (EER). Approaches include:

• Speaker Encoder Selection: ECAPA-TDNN performs better in seen scenarios, FACodec generalizes better to unseen speakers, with FACodec yielding unseen accuracies up to ~91.8% for males and ~89.7% for females (He et al., 14 May 2025).
• Innovations: Teams apply varied strategies—enhancing Diff-Net, using SiamAM-ResNet or WavLM for encoding, and employing graph-based augmentation. Best performance is typically linked to robust encoder choice and differential attribute comparison modules (Chen et al., 8 Sep 2025).

6. Applications, Limitations, and Future Directions

VCTK-RVA enables quantitative controllability and explainability in voice conversion, speaker synthesis, and perceptual timbre editing. Its design supports direct training of text-guided modification systems (VoxEditor, DreamVoiceDB (Hai et al., 2024)) and benchmarking of attribute detection frameworks. However, core challenges persist:

• Label Imbalance: Despite augmentation, rare descriptors remain underrepresented, requiring continual methodological innovations.
• Subjectivity of Annotations: Relative perceptual judgments, even when crowdsourced and validated, inherently limit objective ground-truth consistency across all use-cases.

A plausible implication is that ongoing research will leverage the fine-grained structure of VCTK-RVA for multi-domain adaptation, improved generalization, and enhanced attribute balancing.

7. Comparative Advantages and Research Impact

Compared to earlier speech corpora, VCTK-RVA’s explicit, relative annotation layer provides unique supervision for learning attribute-conditioned synthesis and explainable timbre modeling. This positions it as a primary resource for challenges in the fine-grained control and detection of voice qualities, especially in zero-shot, low-resource, or personalized generation settings. Its role is foundational for the current generation of text-guided voice conversion (DreamVoiceDB (Hai et al., 2024)), attribute editing (VoxEditor (Sheng et al., 2024)), and benchmarking in explainability-driven challenges (Chen et al., 8 Sep 2025).

In summary, VCTK-RVA stands as a key annotated corpus for the development of controllable, explainable, and perceptually grounded voice attribute modeling in advanced speech research.