VocBench: Ontology & Neural Vocoder Benchmarks

• VocBench is a dual-artifact platform, offering multilingual ontology editing via VocBench 3 and standardized neural vocoder benchmarking for speech synthesis.
• The ontology management tool features advanced lexicon support, SKOS mapping, versioning, and role-based collaboration compliant with FAIR principles.
• The neural vocoder benchmark provides unified datasets, identical preprocessing, and comprehensive evaluation metrics to reveal clear performance trade-offs across architectures.

VocBench is a term reflecting two distinct, technically rigorous research artifacts in the academic literature, each foundational in its original domain: (1) VocBench 3 for multilingual ontology management, and (2) VocBench (Neural Vocoder Benchmark) for standardized evaluation of neural vocoders in speech synthesis pipelines. The following entry documents both, as they represent leading platforms in their respective fields and are both encountered under the term "VocBench."

1. VocBench 3: Multilingual Ontology Editing and Management

VocBench 3 is an open-source, web-based ontology editor developed to address the requirements of collaborative, multilingual ontology development. It is distinctive in formal ontology engineering for supporting not only class/property hierarchies but also lexica, linguistic models, and rich mapping across vocabularies, with features targeted toward FAIR-compliant, distributed, role-segregated workflows (Gillis-Webber et al., 2022).

1.1. Functional Coverage of Multilingual Ontology Editing

Based on a systematic assessment comparing seven ontology editors, VocBench 3 is nearest to meeting nine established requirements for multilingual ontology management:

Requirement	VocBench 3 Support	Notes
Linguistic Model	✓	Full OntoLex-Lemon plugin for lexicon–ontology linkage
Lexicons	✓	Creation, import, and management in UI/API
Mappings	✓	SKOS-based (exactMatch, closeMatch, relatedMatch, etc.)
Localisation & Viewing	–	No per-language “switch”; all labels visible concurrently
Verbalisation	–	No controlled-language NLG
Synchronisation	–	No automatic sync; manual update of imports
Multi-Role Collaboration	✓	Fine-grained RBAC, language-specific roles
Versioning	✓	State and change-based, provenance-aware
FAIR Principles	✓ (8/8)	Persistent IDs, metadata, protocols, licensing

Legend: ✓ = full support, – = not supported.

A key asset is the OntoLex-Lemon plugin enabling complex lexicalization patterns, allowing entries in multiple languages to be attached, edited, and queried natively through the UI. SKOS mapping support, lexicon management, and robust collaboration workflows are implemented at a level unmatched by other tools (e.g., Protégé, MoKi, NeOn Toolkit).

1.2. System Limitations and Gaps

VocBench 3’s documented limitations include:

• Only partial coverage of OWL 2-DL constructs: highly expressive ontologies may not import losslessly.
• Absence of dedicated UI for single-language browsing or switching.
• Lack of built-in verbalisation/NLG for axioms.
• No automated synchronization when external ontologies are updated; all updates to lexica/mappings require explicit user action.

These gaps delimit the use of VocBench 3 in projects needing full DL reasoning, advanced natural language generation, or automated translation synchronization.

1.3. FAIR Compliance and Role Differentiation

VocBench 3 provides comprehensive support for the FAIR sub-principles (F1–F3, A1, I1–I3, R1) via its metadata, namespace management, content negotiation, and licensing mechanisms. Its role-based collaboration model allows distinct privileges for ontology engineers, domain experts, translation specialists, and validators, facilitating distributed internationalization workflows.

2. VocBench (Neural Vocoder Benchmark): Fair Benchmarking in Speech Synthesis

VocBench, as introduced in neural speech synthesis, is an open-source framework for benchmarking state-of-the-art neural vocoders under strictly controlled, reproducible experimental conditions (AlBadawy et al., 2021). It resolves the longstanding barrier to “apples-to-apples” comparison across diverse vocoder architectures by unifying datasets, preprocessing, model pipelines, and evaluation strategies.

2.1. Systematic Benchmarking Architecture

VocBench’s core architecture consists of:

• Feature Extractor: Standardizes raw audio by converting to 80-dim Mel-spectrograms.
• Vocoder Trainer: Supports six canonical architectures: WaveNet, WaveRNN (autoregressive); MelGAN, Parallel WaveGAN (GAN-based); WaveGrad, DiffWave (diffusion-based). All are trained from scratch using identical data splits.
• Synthesizer: Generates waveforms from held-out Mel-spectrograms with fixed protocols.
• Evaluator: Computes both objective (SSIM, FAD, LS-MSE, PSNR, MCD, SNR) and subjective (MOS) scores.

2.2. Dataset and Preprocessing Standardization

All models are benchmarked against three publicly available datasets:

Corpus	Type	Size/Info
LJ Speech	Single-speaker	~24 h, 13,100 utt.
LibriTTS	Multi-speaker	~585 h, ~1,150 spkrs
VCTK	Multi-speaker	110 spkrs, ~44 h

Preprocessing is identical: audio is resampled to 24 kHz, Mel-extracted (Hanning 40 ms/12.5 ms, 1,024-pt FFT, 0–12 kHz cutoff), and then log-compressed and min–max normalized.

2.3. Evaluation Metrics and Experimental Methodology

VocBench employs both subjective and objective measures, notably:

• Mel-Cepstral Distortion (MCD):

$$\mathrm{MCD} = \frac{10}{\ln 10}\,\sqrt{2\,\sum_{d=1}^D (c_d - \hat c_d)^2}$$

• Signal-to-Noise Ratio (SNR):

$$\mathrm{SNR} = 10 \log_{10} \frac{\sum_{n} s[n]^2}{\sum_{n}(s[n] - \hat s[n])^2}$$

• Structural Similarity Index Measure (SSIM), Fréchet Audio Distance (FAD), Log-Mel Spectrogram MSE (LS-MSE), Peak SNR (PSNR).
• Subjective 5-point Mean Opinion Score (MOS): Collected on 20 sentences per dataset, with 95% confidence intervals.

2.4. Comparative Results: Quality, Speed, Trade-offs

Benchmark outcomes highlight architectural trade-offs:

Model	Quality (best)	Speed (RTF, GPU/CPU)	Parameters
MelGAN	RTF ≈ 0.001/0.029	3 M, 3 GFLOPS	Fastest
Parallel WaveGAN	RTF ≈ 0.002/0.576	1.3 M, 31 GFLOPS
WaveGrad	RTF ≈ 0.38/9.86	Diffusion, 50 steps
DiffWave	Highest MOS (4.07)	RTF ≈ 0.07/4.45
WaveNet	Top FAD (0.99, VCTK)	Slow, sample-wise AR
WaveRNN	Topline fidelity (offline)	Slow

Key findings:

• Diffusion-based vocoders reach or surpass GANs in subjective quality, notably in single-speaker settings, but have higher inference latency.
• GAN-based models (MelGAN, Parallel WaveGAN) offer best speed–quality ratio in multi-speaker scenarios.
• Autoregressive architectures (WaveNet, WaveRNN) still provide the highest offline fidelity but are computationally impractical for low-latency deployment.

2.5. Impact, Community Use, and Limitations

VocBench’s rigor in standardization reveals performance and trade-offs that are obscured in non-uniform evaluation regimes. Minor differences in architecture, imperceptible under heterogenous pipelines, are resolved into clear comparative patterns. VocBench is extensible: the open-sourced codebase allows integration of novel vocoder architectures and community-contributed tasks.

By publicizing all preprocessing, data splits, and metrics, VocBench establishes a baseline for continuous progress tracking in neural vocoder research. Its methodology is referenced as a benchmark standard in subsequent neural speech synthesis literature.

3. Comparative Assessment and Position within Tooling Ecosystems

In ontology management, only VocBench 3 and NeOn Toolkit deliver a full suite of linguistic-model integration, lexicon management, and mapping support. When compared against seven leading editors, only VocBench 3 provides robust multi-role access controls, complete versioning (snapshot and change-based), and support for all assessed FAIR principles. NaturalOWL is unique in its verbalisation feature but lacks linguistic model support. No evaluated tool offers true per-language user-interface switching or automated change-propagation for localization; these remain open development challenges (Gillis-Webber et al., 2022).

For speech synthesis benchmarking, VocBench’s scope—objective/subjective synthesis assessment under unified conditions—contrasts with domain-specific expressive TTS benchmarks such as NV-Bench (nonverbal vocalization synthesis) or conversational speech-agent benchmarks like VocalBench / VocalBench-zh that encompass linguistic, paralinguistic, and robustness axes. VocBench is specifically focused on waveform fidelity, naturalness, and computational metrics of the vocoder component.

4. Methodological Innovations and Benchmarking Principles

Both VocBench platforms reflect a strict adherence to standardized, reproducible experimental design:

• VocBench 3: Uniquely supports complex lexicon-linguistic models, formal mappings (SKOS, OWL), modular role management, and FAIR principle realization, but with known feature gaps regarding full expressivity, NLG, and per-language presentation.
• VocBench (Speech): Implements fixed data splits, identical preprocessing, and exhaustive hyperparameter search, publishing all scripts/configurations for reproducibility, enabling fair cross-architecture and cross-publication comparison.

5. Future Directions

Anticipated advances for VocBench 3 include:

• Full OWL 2-DL support for highly expressive ontologies.
• Implementation of single-language UI switching for enhanced localization and usability.
• Integration of natural language verbalisation modules.
• Development of synchronization mechanisms for external ontology and lexicon updates.

For VocBench (Speech), continued expansion is expected in incorporating new neural vocoder paradigms, extending support for more diverse languages and acoustic conditions, and integration with upstream/downstream end-to-end speech pipelines.

• VocBench: A Neural Vocoder Benchmark for Speech Synthesis (AlBadawy et al., 2021)
• A Review of Multilingualism in and for Ontologies (Gillis-Webber et al., 2022)