---
title: 'SpeechEditBench: Bilingual Speech Editing Benchmark'
url: https://www.emergentmind.com/topics/speecheditbench
type: topic
---

# SpeechEditBench: Bilingual Speech Editing Benchmark

Searching arXiv for the named benchmark and closely related speech-editing work.
SpeechEditBench is a bilingual multi-attribute benchmark for instruction-guided speech editing that evaluates whether a model can modify specified speech attributes while preserving unrelated characteristics [2606.01804]. It is designed for a setting in which a model receives a source speech clip, sometimes an additional reference speech, and a natural-language instruction describing how the audio should be edited. The benchmark addresses a central evaluation difficulty in speech editing: valid outputs are one-to-many at the waveform level, while existing evaluations are fragmented across isolated tasks or rely on waveform similarity that is too rigid for creative edits [2606.01804]. Within this landscape, SpeechEditBench provides a unified framework spanning seven atomic editing tasks and compositional editing tasks, and it operationalizes evaluation through an anchor-based protocol that separately measures target success, preservation success, and joint success [2606.01804]. The benchmark’s design is closely related to the multi-dimensional requirements already emphasized in text-based speech editing work such as FluentEditor2, where content correctness, boundary smoothness, speaker preservation, and global prosody are treated as distinct dimensions of performance [2410.03719].

## 1. Definition and task formulation

SpeechEditBench is specifically constructed for instruction-guided speech editing, a task in which a model must modify only the requested attributes and preserve all other attributes [2606.01804]. This imposes dual constraints: precise, attribute-specific modification and rigorous preservation of everything else. The benchmark frames this as a systematic, unified evaluation problem rather than as a collection of unrelated sub-tasks.

The benchmark is bilingual, covering English and Mandarin Chinese, and multi-attribute, covering seven distinct editing dimensions: content, speaker, emotion, style, prosody, paralinguistic, and acoustic [2606.01804]. In addition to atomic editing tasks, it includes compositional editing tasks that combine multiple operations within a single instruction. This design allows direct comparison across heterogeneous editing capabilities under a single evaluation framework.

Each sample contains the source audio, a natural-language instruction, and one or more anchors used for evaluation [2606.01804]. These anchors are task-specific labels or references such as target transcripts, reference speakers, acoustic conditions, edit spans, or target words. The benchmark therefore evaluates semantic or attribute-level constraints rather than waveform identity. This suggests that SpeechEditBench treats editing as controlled transformation under invariance constraints, not as waveform reconstruction.

## 2. Dataset construction and coverage

SpeechEditBench contains 4,700 samples: 4,300 atomic samples and 400 compositional samples [2606.01804]. The language split is 2,400 English samples and 2,300 Chinese samples. Audio is sourced from a wide range of public corpora spanning clean read speech, emotional or expressive speech, non-verbal or paralinguistic material, and acoustic resources for controlled noise and reverberation synthesis [2606.01804].

The sources explicitly include LibriTTS, AISHELL-3, WenetSpeech, VCTK, LibriQuote, IEMOCAP, CSEMOTIONS, NaturalVoices, StoryTTS, NonverbalTTS, DisfluencySpeech, Emilia-NV, MUSAN, and RIRS_NOISES [2606.01804]. MUSAN and RIRS_NOISES are used for controlled noise and reverb synthesis. GPT-4o and Gemini-2.5-Pro are used during data construction to propose edits, filter and annotate emotional, style, and paralinguistic labels, and generate or template instructions that are single-goal and aligned with the anchors [2606.01804].

The overall construction is summarized below.

| Component | Coverage | Details |
|---|---|---|
| Total samples | 4,700 | 4,300 atomic, 400 compositional |
| Languages | English and Chinese | 2,400 English, 2,300 Chinese |
| Data sources | Multi-corpus | Clean, expressive, paralinguistic, and acoustic resources |

This breadth is significant because instruction-guided speech editing is not reducible to transcript modification alone. Earlier text-based speech editing systems such as FluentEditor2 focus primarily on edited-region fluency, local acoustic consistency, and global prosody consistency within utterance-level editing [2410.03719]. SpeechEditBench generalizes the evaluation problem to a broader instruction-following regime that includes reference-based speaker transfer, non-verbal event manipulation, and acoustic environment control [2606.01804].

## 3. Atomic tasks and their anchors

SpeechEditBench defines seven atomic editing tasks, each targeting one attribute while requiring preservation of non-target attributes, especially the linguistic transcript when the task is non-content [2606.01804].

**Content editing** targets lexical content through word or phrase replacement, insertion, or deletion. Its anchors are the original transcript, target transcript, and edit type and span [2606.01804]. The expected output must match the target transcript exactly, with the specified span edited and all other words preserved. Diagnostics use WER/CER between predicted and target transcript. Because the target transcript itself is the desired end state, content editing’s target success and joint success are identical [2606.01804].

**Speaker editing** targets speaker identity in a voice conversion setting from source to reference speaker. Inputs include source audio and reference speaker audio of 3–8 seconds. The anchor is the target speaker identity and reference clip [2606.01804]. The expected output must match the reference speaker while preserving the source transcript.

**Emotion editing** targets vocal emotion while preserving the transcript exactly. It includes a standard subset, where text is emotionally neutral, and a challenging subset, where the text lexically expresses the source emotion but the instruction requests a different target emotion [2606.01804]. The anchors are source and target emotion labels from a taxonomy including angry, fearful, happy, sad, surprise, excited, frustrated, and playfulness.

**Style editing** targets speaking style rather than emotion or content. The style taxonomy contains six styles: public-broadcast, intimate, dramatic, restrained-flat, storytelling, and conversational [2606.01804]. The anchors are source and target style labels, with source styles assigned via Gemini-based style annotation with strict thresholds.

**Prosody editing** targets speaking speed, pitch, or stress on specific words. Anchors specify prosody type and direction, and for stress tasks the list of target words in the transcript [2606.01804].

**Paralinguistic editing** targets non-verbal vocal events: breath, laugh, cough, and sigh. Operations are add and remove. Anchors specify event category and operation [2606.01804].

**Acoustic editing** includes speech enhancement and environment transfer. Anchors specify degradation type, environment type and subtype, or target RT60 range [2606.01804]. The expected output either removes degradation while preserving content or adds the desired environment while preserving content.

The atomic tasks show that the benchmark treats speech as a multi-attribute object whose content, identity, delivery, non-verbal events, and recording conditions can each be edited independently. This suggests that SpeechEditBench is designed as a disentanglement-oriented evaluation suite, even though the paper formulates it operationally through anchors and success indicators rather than through latent-factor language.

## 4. Anchor-based evaluation and success metrics

Anchor-based evaluation is the central design principle of SpeechEditBench [2606.01804]. An anchor is an explicit, task-specific specification of what must change or remain. Examples include the target transcript and edited span for content editing, a reference speaker clip for speaker editing, target labels for emotion and style, direction and target words for prosody, event type and operation for paralinguistic editing, and environment type or RT60 range for acoustic editing [2606.01804].

For a set of samples $\mathcal{D}$, the benchmark defines the target success indicator $t_i \in \{0,1\}$, the preservation success indicator $p_i \in \{0,1\}$, and the joint success indicator
$$
j_i = t_i p_i.
$$
Aggregate scores are
$$
\mathrm{TS} = \frac{1}{|\mathcal{D}|} \sum_i t_i,
$$
$$
\mathrm{PS} = \frac{1}{|\mathcal{D}|} \sum_i p_i,
$$
$$
\mathrm{JS} = \frac{1}{|\mathcal{D}|} \sum_i j_i.
$$
These are reported as percentages [2606.01804].

For non-content tasks, preservation success is enforced via an ASR gate. Output audio is transcribed using Whisper large-v3 for English and Paraformer (FunASR) for Chinese, text is normalized, and English uses WER while Chinese uses CER [2606.01804]. Preservation success is
$$
p_i = \mathbf{1}[e_i \le 0.10],
$$
where $e_i$ is WER/CER against the expected transcript [2606.01804]. For non-content atomic tasks, the expected transcript is the source transcript; for compositional tasks with content editing, it is the content-target transcript.

Target success is task-specific. Speaker editing succeeds if cosine similarity between WavLM-large plus ECAPA-TDNN embeddings of output and reference is at least $0.50$ [2606.01804]. Emotion editing succeeds if a Gemini-based audio emotion classifier predicts the target label. Style editing succeeds if a Gemini style judge returns `target_style_success = true` and `target_style_score \ge 3`, or if the boolean is absent, `target_style_score \ge 3` [2606.01804]. Prosody editing uses duration ratio, median F0 shift in semitones, and stress prominence criteria. For speed, with duration ratio $r = d_\text{out}/d_\text{src}$, “faster” succeeds if $r \le 0.95$ and “slower” if $r \ge 1.05$ [2606.01804]. For pitch, “higher” succeeds if $\Delta f \ge +0.3$ and “lower” if $\Delta f \le -0.3$ [2606.01804]. Paralinguistic editing uses Gemini event scores from 0 to 3, with addition succeeding at score $\ge 2$ and removal at score $\le 1$ [2606.01804]. Acoustic editing uses DNSMOS gains for enhancement, RT60 constraints for reverb transfer, and PANNs scene classification plus score thresholds for noise or scene transfer [2606.01804].

This evaluation design is notable because it separates edit effectiveness from preservation fidelity rather than collapsing them into a single score. Related text-based speech editing work already highlighted that speech editing performance is inherently multi-dimensional, involving content correctness, boundary smoothness, identity preservation, and global prosody [2410.03719]. SpeechEditBench extends this logic into a unified benchmark formalism.

## 5. Compositional editing and formalization of multi-intent success

SpeechEditBench includes 400 compositional samples involving two or three atomic operations within a single instruction [2606.01804]. The benchmark groups attributes into four high-level categories: semantic content, speaker identity, expressive delivery, and acoustic environment [2606.01804]. Two-component combinations total 320 samples, with eight pairings: content + speaker, content + emotion, content + prosody, content + acoustic, speaker + emotion, speaker + acoustic, emotion + acoustic, and prosody + acoustic [2606.01804]. Three-component combinations total 80 samples, with four triplets: content + speaker + emotion, content + speaker + acoustic, content + emotion + acoustic, and speaker + emotion + acoustic [2606.01804].

For compositional sample $i$, let $C_i$ be the set of components, and let $a_{i,c} \in \{0,1\}$ be the atomic evaluator’s success indicator for component $c \in C_i$ [2606.01804]. The benchmark defines component success as
$$
\mathrm{CS} =
\frac{\sum_i \sum_{c \in C_i} a_{i,c}}{\sum_i |C_i|}.
$$
It defines all-component success indicator
$$
q_i = \prod_{c \in C_i} a_{i,c},
$$
content preservation indicator $p_i \in \{0,1\}$, and compositional joint success indicator
$$
j_i = q_i p_i.
$$
These definitions explicitly separate success on individual requested edits, simultaneous success on all requested edits, and simultaneous success under content preservation [2606.01804].

The importance of this distinction is empirical. The reported results show that models can often satisfy individual components substantially more often than they satisfy all requested constraints simultaneously [2606.01804]. A plausible implication is that compositional instruction following exposes attribute entanglement and instruction competition more sharply than atomic editing. This interpretation aligns with the paper’s failure analysis, which documents partial success, last-mentioned-instruction dominance, and content corruption during multi-attribute editing [2606.01804].

## 6. Experimental setup, results, and failure patterns

The benchmark evaluates two model categories: Speech LLMs and specialized speech editing or audio systems [2606.01804]. Open-source Speech LLMs include Ming-UniAudio-16B-A3B-Edit, Step-Audio-EditX (3B), Qwen3-Omni (30B), Kimi-Audio (7B), Mimo-Audio-Base (7B), and Mimo-Audio-Instruction (7B). Closed-source models are Gemini-Live and GPT-Realtime [2606.01804]. Specialized systems include VoiceCraft-X for content editing, Seed-VC for speaker editing, VoxCPM2 for emotion/style/prosody, Chatterbox-TTS plus AudioSep for paralinguistic editing, and DeepFilterNet3 plus DSP with RIR convolution for acoustic editing [2606.01804]. The benchmark focuses on evaluation rather than training; most models are evaluated as-is, with default generation settings and no reported per-model prompt engineering beyond feeding the provided natural-language instructions [2606.01804].

The main findings are threefold. First, no single model performs well across all editing dimensions [2606.01804]. For example, Ming-UniAudio is the best open-source content editor at 76.46% joint success but weak on expressive tasks such as emotion, where it reaches 3.43% joint success [2606.01804]. Step-Audio-EditX is stronger on expressive editing, with 49.67% style and 31.25% paralinguistic joint success, but low on content at 16.50% [2606.01804]. Qwen3-Omni is strong on prosody and acoustic editing, with 38.17% and 37.80% joint success respectively, and also reaches 72.00% on content [2606.01804].

Second, closed-source Speech LLMs generally outperform open-source models [2606.01804]. GPT-Realtime achieves 96.67% joint success on content, 68.67% on style, and 47.00% on paralinguistic editing [2606.01804]. Gemini-Live achieves 65.17% joint success on prosody, 27.79% on emotion, 63.67% on style, and 93.17% on content [2606.01804]. Third, compositional editing remains highly challenging. In two-component tasks, Gemini-Live has 50.25% component success but only 21.50% all-component and 21.50% joint success; GPT-Realtime has 49.75% component success, 20.50% all-component success, and 20.00% joint success [2606.01804]. In three-component tasks, almost all models have 0.00% joint success, with Mimo-Audio-Base as the only reported model showing 5.00% joint success [2606.01804].

The benchmark also reveals that target success alone is misleading. Step-Audio-EditX, for example, reaches 51.51% target success on prosody but only 39.60% preservation and 20.13% joint success, whereas Ming-UniAudio reaches 28.00% target success, 84.50% preservation, and 26.50% joint success [2606.01804]. This shows that aggressive editing can improve target attainment while damaging transcript preservation.

The appendix’s failure taxonomy includes target not applied, wrong expressive label, local content edit with global drift, speaker target miss, prosody too weak or wrong direction, paralinguistic add/remove miss, acoustic enhancement without preservation, environment target miss, and compositional partial success [2606.01804]. These failure modes indicate that instruction-guided speech editing is constrained not only by edit controllability but also by the ability to preserve untargeted attributes under distributional and linguistic variation.

## 7. Position in the field, limitations, and implications

SpeechEditBench is positioned against prior speech or audio benchmarks and task-specific speech editing datasets that focus on single tasks, use non-comparable metrics, or omit the simultaneous evaluation of edit success and preservation success [2606.01804]. The benchmark is presented as a first-of-its-kind diagnostic suite for instruction-guided multi-attribute speech editing because it combines bilingual coverage, seven atomic tasks, compositional instructions, and a unified anchor-based evaluation framework [2606.01804].

Its broader significance becomes clearer when contrasted with specialized editing systems such as FluentEditor2. FluentEditor2 addresses text-based speech editing with explicit local acoustic consistency and global prosody consistency objectives, and its evaluation centers on fluency-sensitive metrics such as FMOS and IMOS together with objective measures like MCD, STOI, and PESQ [2410.03719]. SpeechEditBench shifts the focus from utterance-local editing quality to instruction-conditioned attribute control and preservation across a wider task space [2606.01804]. This suggests a division in the field between systems optimized for narrow editing regimes and benchmarks intended to diagnose general-purpose editing competence.

The benchmark has several explicit limitations. It covers only English and Chinese, with no low-resource languages or broader typological coverage [2606.01804]. Its evaluation is heavily reliant on automatic metrics and LLM-based judges such as Gemini, ASR, and DNSMOS, and the paper does not report separate human listening tests [2606.01804]. It focuses on single-turn editing rather than multi-turn interactive refinement, and its attribute scope does not include code-switching, dialect or accent conversion, fine-grained voice-style interpolation, or longer conversational context [2606.01804].

The authors suggest extending the benchmark to more languages and richer attribute sets, incorporating human-in-the-loop evaluation, and exploring multi-turn editing and interactive correction scenarios [2606.01804]. A plausible implication is that SpeechEditBench may function less as a final evaluation standard than as a scaffolding for a broader family of speech editing benchmarks. Even in its current form, however, it establishes a rigorous diagnostic framework in which the central question is not merely whether a model can edit speech, but whether it can do so precisely, preserve untargeted attributes, and satisfy multiple simultaneous constraints under natural-language instructions [2606.01804].

Source: https://www.emergentmind.com/topics/speecheditbench