OpenSTBench: Unified Speech Translation Benchmark

• OpenSTBench is a comprehensive benchmark that standardizes evaluation across speech-to-text and speech-to-speech translation modalities.
• It employs a unified record format and conditional metric selection to assess translation quality, speech naturalness, speaker preservation, and latency.
• The framework facilitates balanced comparisons between offline and streaming systems, enabling deployment-oriented performance analysis.

OpenSTBench is a benchmark and evaluation framework for speech translation that addresses a specific comparability problem: contemporary speech translation systems differ not only in model architecture but also in modality and generation regime, spanning speech-to-text translation (S2TT), speech-to-speech translation (S2ST), offline generation, and streaming generation. Its central design is a unified multidimensional evaluation framework that places heterogeneous outputs into a shared evaluation format, defined by a shared sample record, a common evaluator interface, and a consistent output schema, with metrics selected according to available output modality and timing information (An et al., 29 May 2026). Within that protocol, OpenSTBench jointly evaluates translation quality, speech quality, speaker preservation, emotion fidelity, paralinguistic fidelity, temporal consistency, and latency, while avoiding forced comparisons for metrics that do not apply to a given output regime.

1. Problem setting and unifying design

OpenSTBench is motivated by the fragmentation of existing speech translation evaluation practice. Translation quality is commonly measured with machine-translation metrics, speech-side quality with separate evaluators for S2ST naturalness or similarity, and streaming behavior with SimulEval-style latency metrics. OpenSTBench consolidates these into one extensible benchmark, while retaining regime-specific validity: text-only systems can be scored on translation quality and, when streaming, timing metrics; systems with target speech can additionally be scored on speech quality, speaker preservation, emotion fidelity, paralinguistic fidelity, and temporal consistency; systems with streaming timing information can be scored on latency; and offline local systems are evaluated for efficiency via RTF instead of incremental latency (An et al., 29 May 2026).

The framework supports S2TT systems, S2ST systems, offline generation, and streaming generation. This support is not merely taxonomic. It determines what information appears in the shared record and which evaluators are invoked. Depending on the system, the record can contain source speech, target text hypothesis, generated target speech waveform, synchronized text for generated speech, and timing information needed for latency evaluation. For offline systems, complete speech segments are processed. For streaming systems, source speech is fed incrementally according to the system’s native online interface and native input granularity, explicitly to reduce segmentation-induced latency bias.

A notable design choice is that OpenSTBench does not collapse all systems into a single homogeneous notion of “speech translation.” The paper highlights Baidu Realtime ST as a representative edge case: it receives streaming audio but emits sentence-final text, so it is treated as a streaming-input S2TT baseline rather than as a fully incremental S2ST system. This distinction is important because it keeps the benchmark unified without erasing architectural differences that materially affect latency, modality, and speech-side evaluation.

2. Evaluation dimensions and metric structure

OpenSTBench organizes evaluation into three groups—translation quality, speech quality, and temporal quality—while covering seven operational dimensions. The benchmark’s metric structure is conditional rather than universal: each metric is reported only when the corresponding modality or timing signal exists.

Dimension	Metrics	Inputs required
Translation quality	sacreBLEU, chrF++, COMET, BLEURT	system text output or text associated with speech output, reference translations
Speech quality proper	UTMOS	generated target speech waveform
Realization fidelity / intelligibility	CER for Chinese, Japanese, and Korean targets; WER otherwise	generated speech waveform, generated target text
Speaker preservation	Resemblyzer, WavLM	generated target speech, target-language reference speech from the same speaker
Emotion fidelity	Emotion2Vec cosine similarity (E2V), Emotion classification accuracy	generated target speech, reference emotional target speech or labels
Paralinguistic fidelity	Event Content F1, Event Timing F1	source event annotations or estimated locations, generated speech
Temporal consistency and latency	SLC 0.2, SLC 0.4, Start Offset, ATD, Custom ATD, RTF	duration information, timing logs, aligned textual representations, runtime and source duration

Translation quality remains MT-style. OpenSTBench reports sacreBLEU, chrF++, COMET, and BLEURT, using sacreBLEU for BLEU, chrF++, Unbabel/wmt22-comet-da for COMET, and a PyTorch implementation of BLEURT using the BLEURT-20 checkpoint. These metrics apply to both S2TT and S2ST; for S2ST, target speech can be evaluated through associated generated text or transcription when needed (An et al., 29 May 2026).

The framework separates naturalness from realization fidelity. Naturalness is measured with UTMOS. Realization fidelity is measured by transcribing generated speech with Whisper medium and comparing that transcription with the system’s generated target text using CER for Chinese, Japanese, and Korean targets and WER otherwise. This distinction is methodologically important because a system may produce semantically strong text while failing to realize that text faithfully in speech, or conversely may sound natural while exhibiting textual realization errors.

OpenSTBench further extends beyond semantic fidelity to expressive and temporal behavior. Speaker preservation is measured with Resemblyzer and WavLM. Emotion fidelity is measured with Emotion2Vec cosine similarity and emotion classification accuracy. Paralinguistic fidelity is measured with Event Content F1 and Event Timing F1. Temporal consistency is measured with Speech Length Compliant at two tolerances, SLC 0.2 and SLC 0.4. Latency is measured with Start Offset, ATD, and Custom ATD in streaming settings, while RTF is used for offline local systems.

3. Evaluation protocol for speech, expression, and time

Several protocol choices distinguish OpenSTBench from a simple aggregation of existing metrics. The speaker-preservation protocol uses a same-language speaker anchor, motivated by the observation that cross-language comparison can conflate speaker identity with language mismatch. To support this, the benchmark constructs a LibriTTS-based paired speaker set. The paper reports that for same-speaker pairs, EN–EN comparisons produce substantially higher similarity than EN–ZH comparisons: EN–EN yields Resemblyzer 0.9073 and WavLM 0.7721, whereas EN–ZH yields Resemblyzer 0.8462 and WavLM 0.4556 (An et al., 29 May 2026). This motivates the use of target-language reference speech rather than source-language speech for the main benchmark. For EN→ZH, source transcripts are translated into Chinese and Chinese target reference speech is synthesized with Qwen3-TTS using the source utterance as speaker prompt; for ZH→EN, outputs are compared to original LibriTTS English speech.

Paralinguistic fidelity is handled explicitly rather than being absorbed into speech quality. Target events are detected and localized from generated speech using CLAP under the same candidate label set. Event Content F1 measures preservation of event label or type and count, while Event Timing F1 measures preservation of relative event timing in addition to event content. The benchmark notes that source-side event locations are approximate: for NonverbalTTS, event locations are estimated by mapping positions of nonverbal text tags to utterance duration, and for SynParaSpeech, event locations are derived by aligning text-level event tags with VAD-based segment timing.

Temporal consistency is formalized through Speech Length Compliant. The paper defines $\mathrm{SLC}_p$ as the percentage of samples whose duration ratio lies within $[1-p, 1+p]$, with ratio

$$\mathrm{ratio} = \frac{\sum_{i=0}^{T_{y'}} d_i}{\sum_{j=0}^{T_x} d_j}.$$

Here, $T_{y'}$ is the translated speech token sequence, $T_x$ is the source speech token sequence, and $d_i$, $d_j$ are token durations. OpenSTBench reports $\mathrm{SLC}_{0.2}$ and $\mathrm{SLC}_{0.4}$, corresponding to duration-ratio intervals $[0.8,1.2]$ and $[1-p, 1+p]$0.

Latency evaluation follows the SimulEval interface, but the scoring pipeline is extended to support speech-to-speech outputs. Start Offset measures how long the system waits before emitting output. ATD measures content-level delay between source and aligned generated output. For speech outputs, ATD is computed through aligned textual representations of generated speech. Custom ATD further subtracts target-audio playback duration from measured delay to better isolate generation-side latency. This suggests that OpenSTBench treats streaming behavior as a property of both response timing and output realization, rather than as a single scalar delay.

4. Datasets, language coverage, and systems evaluated

OpenSTBench does not derive all dimensions from one corpus. Instead, it uses different datasets for different dimensions, each aligned to a specific evaluation problem. The main datasets are MSLT dev with EN→ZH 1,000 samples and ZH→EN 1,000 samples for translation quality, speech quality, temporal consistency, and latency; a LibriTTS-based paired speaker set with EN→ZH 300 and ZH→EN 300 for speaker preservation; RAVDESS with EN→ZH 1,440 for emotion preservation; MCAE-SPPS with ZH→EN 1,029 for emotion preservation; NonverbalTTS test with EN→ZH 359 for paralinguistic fidelity; and SynParaSpeech with ZH→EN 500 for paralinguistic fidelity (An et al., 29 May 2026). The experimental language coverage is English↔Chinese.

The evaluated systems span both streaming and offline regimes. The streaming systems are Qwen3-LiveTranslate, Doubao AST 2.0, GPT Realtime Translate, and Baidu Realtime ST. Among these, Qwen3-LiveTranslate, Doubao AST 2.0, and GPT Realtime Translate are treated as streaming S2ST systems, whereas Baidu Realtime ST is treated as streaming-input S2TT because it emits sentence-final text rather than incremental target speech. The offline systems are SeamlessM4T-v2-Large and UniSS, both evaluated on translation quality and on the speech-side dimensions other than streaming latency.

The evaluation conditions are matched to each regime. Streaming systems are fed source speech incrementally using each system’s native input granularity. Offline systems process full speech segments. Speech metrics are applied only to systems with target speech. RTF is measured on a single NVIDIA GeForce RTX 3090 GPU for local models. UniSS uses its official VAD-based chunked pipeline, and SeamlessM4T-v2-Large uses full-utterance generation. API systems do not report RTF or model size because provider-side infrastructure is opaque.

5. Reporting, empirical findings, and benchmark interpretation

OpenSTBench’s main quantitative outputs are the original metrics, but the paper also introduces bilingual-average radar plots based on fixed-range normalization. For a metric value $[1-p, 1+p]$1, the normalized score is

$[1-p, 1+p]$2

when higher is better, and

$[1-p, 1+p]$3

when lower is better. The fixed ranges include, for example, BLEU $[1-p, 1+p]$4, chrF++ $[1-p, 1+p]$5, COMET $[1-p, 1+p]$6, BLEURT $[1-p, 1+p]$7, UTMOS $[1-p, 1+p]$8, CER/WER $[1-p, 1+p]$9, latency metrics $$\mathrm{ratio} = \frac{\sum_{i=0}^{T_{y'}} d_i}{\sum_{j=0}^{T_x} d_j}.$$0 ms, and RTF $$\mathrm{ratio} = \frac{\sum_{i=0}^{T_{y'}} d_i}{\sum_{j=0}^{T_x} d_j}.$$1. The paper is explicit that these normalized scores are for visualization only; the main result tables use original metric values (An et al., 29 May 2026).

The central empirical result is that no single system dominates across all dimensions. Text-side metrics produce relatively clear rankings: Qwen3-LiveTranslate is strongest overall in translation quality across both directions, while Doubao AST 2.0 and UniSS are strong alternatives in streaming and offline settings, respectively. However, these rankings do not carry over uniformly to speech quality or temporal behavior. Systems with similar translation quality can differ substantially in UTMOS, CER/WER realization fidelity, speaker preservation, and emotion preservation. The paper explicitly notes that UTMOS and CER/WER do not always favor the same system, showing that naturalness and faithful realization are distinct properties.

Speaker and emotion preservation vary independently of translation quality. UniSS and Doubao AST 2.0 show consistently strong speaker preservation. Emotion results differ depending on whether evaluation uses embedding similarity or classification accuracy, which supports the use of multiple emotion metrics rather than a single proxy. Paralinguistic fidelity is weak across all systems: Event Content F1 and Event Timing F1 remain low in absolute terms, suggesting that preservation of acoustic events and nonverbal timing remains difficult and is not exposed by conventional translation metrics.

Time-related behavior is also multidimensional. The benchmark emphasizes that latency and temporal consistency are not interchangeable: a system can be responsive in streaming delay metrics yet fail to preserve duration structure well, while an offline model can be efficient in RTF but differ substantially in temporal consistency. Baidu Realtime ST illustrates this regime sensitivity particularly clearly, since a streaming-input system that emits sentence-final text occupies a different point in the design space from incremental S2ST systems. OpenSTBench therefore argues against a single global leaderboard and instead supports application-oriented comparison. This suggests a “profile-based evaluator” interpretation: a deployment that prioritizes semantic fidelity, natural target speech, voice preservation, expressive translation, real-time interaction, dubbing-style timing control, or local efficiency will rank systems differently.

6. Limitations, cautions, and conceptual boundaries

The benchmark’s reported experiments are limited to EN↔ZH, so broader cross-lingual validation remains open (An et al., 29 May 2026). Several dimensions rely on pretrained automatic evaluators, and the paper notes that their alignment with human judgment is not fully validated. Comparisons across streaming S2ST, streaming S2TT, and offline S2ST are necessarily only partially matched because the systems differ fundamentally in output form and generation regime. Speaker-preservation evaluation is especially delicate because scores are sensitive to language mismatch, the choice of source versus target anchor, and natural versus synthesized anchor speech. Paralinguistic timing labels are approximate rather than fully frame-annotated, which introduces noise. The framework also depends on multiple pretrained evaluators and processing stages, implying nontrivial computational overhead.

The future directions stated in the paper are correspondingly conservative: extend to more language pairs, evaluate more realistic interactive settings, better calibrate automatic speech-side metrics against human judgments, and further refine the unified multidimensional protocol as speech translation systems evolve. These are extensions of the benchmark’s present logic rather than departures from it.

A common naming confusion is that OpenSTBench should not be conflated with STBench, which evaluates LLMs on spatio-temporal analysis tasks rather than speech translation (Li et al., 2024). OpenSTBench is specifically a speech translation benchmark centered on heterogeneous S2TT and S2ST evaluation across offline and streaming regimes. Its distinct contribution is not a new single metric, but the benchmarking protocol itself: a shared data organization, a common evaluator interface, a consistent reporting schema, modality-aware metric selection, and a multidimensional view of system behavior that treats semantic, acoustic, expressive, temporal, and responsiveness properties as jointly relevant.