Benchmarking Speech-to-Speech Translation Models

Abstract: Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol: studies report non-overlapping metric subsets, preventing direct comparisons. We introduce COMPASS, a unified and reproducible benchmarking framework integrating 46 metrics across eight dimensions, and deploy it on 1,248 model-language configurations from FLEURS and CVSS, spanning cascaded and end-to-end architectures over ten language pairs. Architectures exhibit complementary strengths: best-vs-worst gaps exceed 30\% on naturalness and speaker preservation but remain within a few points on translation quality, so single-metric rankings systematically misrepresent system quality. Correlation filtering reduces 46 metrics to 10 per direction, with three axes requiring different metrics across X$\to$EN and EN$\to$X (e.g., TER/UTMOS vs. ChrF++/NISQA-MOS); these subsets preserve rankings (Spearman's $ρ>0.80$) while cutting evaluation time by $\approx 2.5\times$. Human validation across dubbing, podcasts, and medical domains shows standalone MOS predictors fail to predict listener preference, while top domain-specific metrics correlate with human judgment ($ρ\geq 0.90$). We release COMPASS as a foundation for domain-aware S2ST evaluation.

• The paper introduces COMPASS, a unified toolkit employing 46 metrics across eight evaluation axes for comprehensive S2ST analysis.
• It benchmarks 1,248 configurations across datasets, revealing trade-offs with up to 30% differences in naturalness and speaker preservation.
• The study demonstrates effective metric selection with high Spearman correlations (>0.80), highlighting architecture and language-specific challenges.

Unified Benchmarking of Speech-to-Speech Translation: The COMPASS Framework

Introduction

The paper "Benchmarking Speech-to-Speech Translation Models" (2606.03241) addresses the absence of standardized, multidimensional evaluation protocols for speech-to-speech translation (S2ST) systems. While S2ST is critical for applications ranging from video dubbing to medical conversations, prior research has relied on disparate, non-overlapping metric subsets, limiting direct cross-paper model comparisons and obscuring architectural trade-offs. This work introduces COMPASS, an integrated toolkit encompassing 46 metrics across eight conceptual axes, offering both modularity and reproducibility. The study benchmarks 1,248 configurations across FLEURS and CVSS datasets, covering end-to-end, S2TT+TTS, and cascaded ASR+MT+TTS architectures over ten language pairs.

Multi-Dimensional Evaluation Taxonomy and Metric Integration

COMPASS formalizes S2ST evaluation into eight axes—translation quality (text/ASR), audio naturalness, speaker consistency, prosody/emotion preservation, isochrony, isometry, and lipsync—mirroring diverse real-world application constraints. This architecture supports both text-based and audio-grounded analyses, incorporating metrics such as COMET-DA, TER, ChrF++, NISQA-MOS, UTMOS, WavLM speaker similarity, AutoPCP prosody, emotion embedding, and duration compliance. Importantly, the toolkit provides a single interface for deterministic and domain-aware benchmarking, facilitating precise metric selection per deployment scenario.

Figure 1: The COMPASS pipeline applies correlation-driven filtering to reduce metric redundancy and identify a compact, empirically distinct subset for each translation direction.

Data-Driven Metric Subset Selection and Redundancy Analysis

The study implements a rigorous filtering pipeline, applying system-level Spearman correlation analysis, hierarchical clustering, and multi-criteria ranking (discriminability, cross-language stability, independence, literature adoption) to distill 46 metrics down to 10 non-redundant representatives per translation direction. The resulting compact subsets preserve system-level rankings (Spearman $\rho > 0.80$) and reduce computation time by $\approx 2.5\times$, maintaining performance while maximizing throughput and interpretability.

Figure 2: t-SNE visualization reveals distinct clusters for each evaluation dimension, confirming successful minimization of intra-dimension redundancy.

Figure 3: Spearman correlation matrices for compact metric subsets demonstrate low cross-dimensional correlations and independent signal extraction.

Architectural Trade-Offs and Directional Asymmetries

Benchmarking across paradigms reveals no dominant architecture. Best-vs-worst gaps exceed 30% on naturalness and speaker preservation, but translation quality differences remain within a few points. End-to-end models (e.g., SeamlessM4T, Qwen3-Omni) deliver superior audio naturalness and cross-lingual stability; cascades (ASR+MT+TTS) provide optimal speaker identity retention and timing control. Notably, X$\to$EN and EN$\to$X require distinct metric sets: TER and UTMOS reflect English-centric bottlenecks, while ChrF++ and NISQA-MOS are critical for morphologically complex and CJK targets.

Figure 4: System radar plots on core metrics illustrate multidimensional performance stratification between architectures across benchmark datasets.

Language Proficiency Profiles and Cross-Lingual Variance

COMPASS’s composite language proficiency mapping shows end-to-end architectures generalize more symmetrically across language families, driven by joint encoder-decoder pretraining. Cascades display directional instability, with performance bottlenecks often traced to weakest pipeline components, especially in low-resource or morphologically divergent languages. Evaluation variance (coefficient of variation) is tightly correlated with corpus structure and architecture.

Figure 5: Language proficiency profiles highlight stable cross-lingual performance for end-to-end systems and pronounced directional collapses in cascaded pipelines.

Figure 6: Radar plots for FLEURS X$\rightarrow$EN detail per-language system behavior, isolating specific cross-lingual challenges such as character compliance and speaker preservation.

Figure 7: EN$\rightarrow$X profiles on FLEURS confirm metric reliability and architecture-dependent variance for Romance and CJK targets.

Figure 8: CVSS X$\rightarrow$EN radar plots validate structural generalization of metric subsets under varying corpus conditions.

Human Evaluation: Domain Dependency and Metric-Human Alignment

Human evaluations (dubbing, podcasts, medical) confirm domain-specific metric requirements. Automatic MOS predictors (UTMOS, NISQA-MOS) fail universally to align with human judgment, with negative or near-zero correlation, especially for emotionally expressive content. In contrast, translation quality (COMET-DA, Semantic Score), prosody (AutoPCP, CPS Ratio), and isochrony (duration compliance) metrics show strong alignment with human ratings (Spearman $\rho \geq 0.90$), reinforcing that S2ST evaluation must be deployment-context conditional.

Implications and Future Directions

Practically, COMPASS provides the field’s first reproducible, multidimensional benchmarking protocol, supporting fair cross-architecture and cross-domain comparisons, and enabling rapid iteration via compact metric sets. Theoretically, the observed empirical collapse of eight axes into six non-redundant dimensions depends on benchmark corpus properties and current architectural profiles; future expansion to spontaneous-speech corpora and decoupled prosodic architectures may require re-estimation of metric groupings.

COMPASS’s modular design allows continuous integration of new metrics and benchmarks, addressing observed limitations such as spontaneous-speech bias, ASR self-evaluation loops, downstream usability gaps (especially in medical contexts), and long-form phenomena. The framework will remain relevant as architectures evolve, and its domain-aware approach is critical for reliable deployment in sensitive applications.

Conclusion

This work establishes COMPASS as the definitive framework for offline S2ST evaluation, integrating deep structural taxonomy, empirical metric selection, and extensive system benchmarking. Findings underscore multidimensional trade-offs, direction-specific metric requirements, and the necessity of context adaptation for metric suite construction. Human validation solidifies the claim that effective metrics must reflect deployment scenario priorities, with MOS predictors insufficient as global proxies. COMPASS’s release will drive more transparent and accountable evaluation practice in the S2ST community, serving as both a diagnostic tool and as a foundation for future research into metric refinement, spontaneous-speech evaluation, and real-world application alignment.