VERSA Evaluation Toolkit

• VERSA Evaluation Toolkit is a unified evaluation system designed for comprehensive assessment of speech, audio, and music signals using both reference-based and independent metrics.
• It integrates over 63 specialized metrics with modular architecture, flexible YAML/JSON configuration, and command-line and Python interfaces for seamless research workflows.
• The toolkit is applied in real-world scenarios such as speech enhancement, TTS, music generation, and challenge benchmarking, ensuring reproducibility and standardization.

VERSA is a unified evaluation toolkit designed to provide comprehensive, modular, and extensible assessment for speech, audio, and music signals. It serves both as a standardized Python package encompassing a broad suite of metrics and as a platform for constructing advanced evaluation systems and research benchmarks. VERSA provides reference-independent and reference-based automatic evaluation across numerous downstream tasks, with an architecture and configuration system intended for transparent integration into research pipelines and challenge evaluations (Shi et al., 2024).

1. Architecture and Design Principles

VERSA is organized as a Python package with two primary entry points: scorer.py for metric execution and aggregate_result.py for summarizing results. Each metric is encapsulated in an "evaluator" class within versa/metrics/. Input/output abstraction layers support diverse file formats, including soundfile, flat directories, and Kaldi-style wav.scp lists. All metric configuration is unified under a YAML/JSON-based Config object, ensuring only user-requested evaluators are instantiated. Modularity is expressed both in path structure (independent, dependent, non-match, and distributional metric directories) and in dependency management.

Design goals prioritize:

• Flexibility & Extensibility: New evaluators are registered via subclassing the base Metric class; configuration enables modular backend selection for metrics such as PESQ and WER.
• Dependency Control: The core library (pip install versa) avoids heavy dependencies, bringing in only essential packages. Heavyweight metrics, e.g., DNSMOS or Whisper-ASR, are opt-in via versa[full] or explicit scripts.
• Consistency: All evaluators adhere to unified resampling and I/O rules. Numerical correctness is maintained by internal forks of key backend libraries.
• Reproducibility: Example configuration and evaluation scripts enable consistent experimental workflows (Shi et al., 2024).

2. Supported Metrics and Task Coverage

VERSA provides 63 automatic evaluation metrics (historically cited as 65), partitioned as follows:

Category	Example Metrics	Reference Requirement
Independent	DNSMOS, NISQA, Torch-SQuIM PESQ	None
Dependent	MCD, PESQ, STOI, F0-CORR, SDR	Matching reference required
Non-Matching Reference	Noresqa, CLAP Score, Speaker SIM	Non-matching/cross-modal
Distributional	FAD, KID, KLD, Embedding Coverage	Dataset-level, often none

By varying configurations (e.g., reference modality, sampling rate, model tag), VERSA exposes up to 711 (or 729 legacy) distinct variants. Notable metrics include:

• Speech Quality: DNSMOS P.835/P.808, NISQA (all variants)
• Speech Enhancement and Coding: PESQ, SI-SNR, STOI, SDR, MCD, F0-correlation
• Recognition Alignment: WER via OpenAI Whisper/ESPnet-ASR/OWSM
• Speaker/Emotion Similarity: Embedding-based cosine similarity, AASIST anti-spoofing
• Music/Singing Evaluation: SingMOS, SHEET-SSQA, APA, CLAP Score
• Distributional: Fréchet Audio Distance, Kernel Inception Distance, KL-divergence

All metrics are implemented as self-contained Pythonic classes and are activated only when specified in the configuration (Shi et al., 2024).

3. Interfaces, Configuration, and Extensibility

VERSA exposes both a command-line interface and a native Python API. The configuration system utilizes YAML/JSON files, enabling users to select, parameterize, and chain metrics. Each metric block may specify task-specific options—for instance, mcep_dim for MCD or beam_size for WER via Whisper.

Examples include:

from versa.scorer import Scorer scorer = Scorer(score_config="egs/codec.yaml", gt_list="data/librispeech.lst", pred_list="data/encodec_outputs.lst") results = scorer.run() print("SDR:", results["sdr"].mean())

python versa/bin/scorer.py \ --score_config egs/tts_demo.yaml \ --gt speech_synth/ref.wav.lst \ --pred speech_synth/sys.wav.lst \ --output_file tts_eval.json

Dependency control is granular: core Pythonic metrics are always present; model-based or resource-intense metrics are opt-in. Pretrained models are automatically downloaded to cache. Example scripts are provided for corpus preparation and for integrating new external models (Shi et al., 2024).

4. Methodological Integration and Benchmarking

VERSA is designed to standardize metric computation in research and challenge settings. Evaluation protocols are exemplified in standardized benchmarks such as the URGENT24 Challenge and AudioMOS Challenge. In these workflows, VERSA is used both for metric computation and as a feature-extraction layer for higher-level regression models:

• Direct metric evaluation: For each candidate-prediction pair (with optional reference/transcription/captions), VERSA computes the selected metrics, which are then summarized via scripts (aggregate_result.py) into tabular or JSON reports for downstream analysis.
• Meta-evaluation as feature generator: In the T12 AudioMOS Challenge system, VERSA supplied 28 reference-free metrics per audio clip, serving as feature vectors for gradient-boosted regression models predicting Audio Aesthetics Score (AES) dimensions: Production Quality, Production Complexity, Content Enjoyment, Content Usefulness. Model performance reached utterance-level SRCC up to 0.908 (Yamamoto et al., 5 Dec 2025).
• Standalone deep regression: The Uni-VERSA framework jointly predicts multiple profiled dimensions (noise, naturalness, intelligibility, speaker similarity) through a single multi-head network, formalized as

$$\hat{Y}^i = \text{UniVERSA}(S^i, S_\text{ref}^i, T^i)$$

and trained via a multi-task $L_1$ loss on available metric labels per utterance (Shi et al., 27 May 2025).

5. Applications and Use Cases

VERSA underpins varied evaluation scenarios:

• Speech Enhancement: Used to quantify SI-SNR, PESQ, DNSMOS, and other enhancement metrics on challenge submissions (e.g., URGENT24), facilitating fast, standardized ranking of system outputs (Shi et al., 27 May 2025).
• Speech Synthesis and TTS: Offers reference-free naturalness and quality estimates (UTMOS, DNSMOS, SHEET-SSQA) and reference-based intelligibility/accuracy (WER, MCD), validated against VoiceMOS and other human-MOS datasets (Shi et al., 2024).
• Music Generation and Singing Synthesis: Evaluates both fidelity (e.g., Fréchet Audio Distance, CLAP Score) and diversity (embedding coverage) in generative tasks.
• Distributed/Embedded Quality Monitoring: Supports real-time dashboards by streaming frames of conversational audio into VERSA models to monitor DNSMOS/UTMOS/SPEAKERSIM.
• Challenge System Integration: In ensemble pipelines such as the T12 AudioMOS Challenge system, VERSA metrics are stacked with deep-learned representations to improve robustness across diverse subjective axes (Yamamoto et al., 5 Dec 2025).

6. Implementation, Deployment, and Reproducibility

VERSA is open source, hosted at https://github.com/wavlab-speech/versa. Typical installation proceeds via:

pip install . # Minimal install pip install .[full] # All available metrics

Optional dependencies are managed via platform-agnostic scripts in tools/. Evaluation runs require YAML configuration and list files for (candidate, reference, transcription, or caption) inputs. Results may be visualized interactively or processed into reports via Pandas or Matplotlib.

All metrics adhere to consistent sample-rate and normalization semantics (via librosa), and external pretrained weights are downloaded and cached on first use. For standardized benchmarks, example scripts and data-preparation steps are provided for reproducibility—e.g., Voicebank-DEMAND and URGENT24 recipes.

7. Limitations and Ongoing Developments

VERSA is subject to several limitations:

• Dependence on External Models: Performance of certain metrics is tied to the accuracy, coverage, and bias of their underlying pretrained models or datasets, yielding potential cross-linguistic or cross-domain discrepancies.
• Inability to Fully Substitute Human Judgement: Objective metrics, even when highly correlated with human listeners, are not guaranteed to capture listener preferences or nuanced quality dimensions.
• Metric Evolution Lag: With the rapid progress in generative audio and speech, new perceptual metrics are continually needed to address style, prompt alignment, or diverse genre-specific criteria.
• Planned Extensions: Future directions include multilingual and dialect-aware resources, integration with multi-modal (audio-visual) metrics, a plugin system for community-driven extension, and interactive web dashboards for result exploration (Shi et al., 2024).

VERSA remains a central and rapidly evolving toolkit for reproducible, extensible, and standardized audio and speech evaluation in academic and applied settings.