SpeechEditBench: Bilingual Speech Editing Benchmark

• SpeechEditBench is a bilingual multi-attribute evaluation framework for instruction-guided speech editing that defines both atomic and compositional tasks using anchor-based metrics.
• It covers seven editing dimensions—content, speaker, emotion, style, prosody, paralinguistic, and acoustic—with specific anchors to ensure precise modifications while preserving unrelated attributes.
• Featuring 4,700 samples from diverse corpora in English and Mandarin, the benchmark highlights the potential and current challenges of controlled, multi-attribute speech editing.

Searching arXiv for the named benchmark and closely related speech-editing work. SpeechEditBench is a bilingual multi-attribute benchmark for instruction-guided speech editing that evaluates whether a model can modify specified speech attributes while preserving unrelated characteristics (Zhang et al., 1 Jun 2026). It is designed for a setting in which a model receives a source speech clip, sometimes an additional reference speech, and a natural-language instruction describing how the audio should be edited. The benchmark addresses a central evaluation difficulty in speech editing: valid outputs are one-to-many at the waveform level, while existing evaluations are fragmented across isolated tasks or rely on waveform similarity that is too rigid for creative edits (Zhang et al., 1 Jun 2026). Within this landscape, SpeechEditBench provides a unified framework spanning seven atomic editing tasks and compositional editing tasks, and it operationalizes evaluation through an anchor-based protocol that separately measures target success, preservation success, and joint success (Zhang et al., 1 Jun 2026). The benchmark’s design is closely related to the multi-dimensional requirements already emphasized in text-based speech editing work such as FluentEditor2, where content correctness, boundary smoothness, speaker preservation, and global prosody are treated as distinct dimensions of performance (Liu et al., 2024).

1. Definition and task formulation

SpeechEditBench is specifically constructed for instruction-guided speech editing, a task in which a model must modify only the requested attributes and preserve all other attributes (Zhang et al., 1 Jun 2026). This imposes dual constraints: precise, attribute-specific modification and rigorous preservation of everything else. The benchmark frames this as a systematic, unified evaluation problem rather than as a collection of unrelated sub-tasks.

The benchmark is bilingual, covering English and Mandarin Chinese, and multi-attribute, covering seven distinct editing dimensions: content, speaker, emotion, style, prosody, paralinguistic, and acoustic (Zhang et al., 1 Jun 2026). In addition to atomic editing tasks, it includes compositional editing tasks that combine multiple operations within a single instruction. This design allows direct comparison across heterogeneous editing capabilities under a single evaluation framework.

Each sample contains the source audio, a natural-language instruction, and one or more anchors used for evaluation (Zhang et al., 1 Jun 2026). These anchors are task-specific labels or references such as target transcripts, reference speakers, acoustic conditions, edit spans, or target words. The benchmark therefore evaluates semantic or attribute-level constraints rather than waveform identity. This suggests that SpeechEditBench treats editing as controlled transformation under invariance constraints, not as waveform reconstruction.

2. Dataset construction and coverage

SpeechEditBench contains 4,700 samples: 4,300 atomic samples and 400 compositional samples (Zhang et al., 1 Jun 2026). The language split is 2,400 English samples and 2,300 Chinese samples. Audio is sourced from a wide range of public corpora spanning clean read speech, emotional or expressive speech, non-verbal or paralinguistic material, and acoustic resources for controlled noise and reverberation synthesis (Zhang et al., 1 Jun 2026).

The sources explicitly include LibriTTS, AISHELL-3, WenetSpeech, VCTK, LibriQuote, IEMOCAP, CSEMOTIONS, NaturalVoices, StoryTTS, NonverbalTTS, DisfluencySpeech, Emilia-NV, MUSAN, and RIRS_NOISES (Zhang et al., 1 Jun 2026). MUSAN and RIRS_NOISES are used for controlled noise and reverb synthesis. GPT-4o and Gemini-2.5-Pro are used during data construction to propose edits, filter and annotate emotional, style, and paralinguistic labels, and generate or template instructions that are single-goal and aligned with the anchors (Zhang et al., 1 Jun 2026).

The overall construction is summarized below.

Component	Coverage	Details
Total samples	4,700	4,300 atomic, 400 compositional
Languages	English and Chinese	2,400 English, 2,300 Chinese
Data sources	Multi-corpus	Clean, expressive, paralinguistic, and acoustic resources

This breadth is significant because instruction-guided speech editing is not reducible to transcript modification alone. Earlier text-based speech editing systems such as FluentEditor2 focus primarily on edited-region fluency, local acoustic consistency, and global prosody consistency within utterance-level editing (Liu et al., 2024). SpeechEditBench generalizes the evaluation problem to a broader instruction-following regime that includes reference-based speaker transfer, non-verbal event manipulation, and acoustic environment control (Zhang et al., 1 Jun 2026).

3. Atomic tasks and their anchors

SpeechEditBench defines seven atomic editing tasks, each targeting one attribute while requiring preservation of non-target attributes, especially the linguistic transcript when the task is non-content (Zhang et al., 1 Jun 2026).

Content editing targets lexical content through word or phrase replacement, insertion, or deletion. Its anchors are the original transcript, target transcript, and edit type and span (Zhang et al., 1 Jun 2026). The expected output must match the target transcript exactly, with the specified span edited and all other words preserved. Diagnostics use WER/CER between predicted and target transcript. Because the target transcript itself is the desired end state, content editing’s target success and joint success are identical (Zhang et al., 1 Jun 2026).

Speaker editing targets speaker identity in a voice conversion setting from source to reference speaker. Inputs include source audio and reference speaker audio of 3–8 seconds. The anchor is the target speaker identity and reference clip (Zhang et al., 1 Jun 2026). The expected output must match the reference speaker while preserving the source transcript.

Emotion editing targets vocal emotion while preserving the transcript exactly. It includes a standard subset, where text is emotionally neutral, and a challenging subset, where the text lexically expresses the source emotion but the instruction requests a different target emotion (Zhang et al., 1 Jun 2026). The anchors are source and target emotion labels from a taxonomy including angry, fearful, happy, sad, surprise, excited, frustrated, and playfulness.

Style editing targets speaking style rather than emotion or content. The style taxonomy contains six styles: public-broadcast, intimate, dramatic, restrained-flat, storytelling, and conversational (Zhang et al., 1 Jun 2026). The anchors are source and target style labels, with source styles assigned via Gemini-based style annotation with strict thresholds.

Prosody editing targets speaking speed, pitch, or stress on specific words. Anchors specify prosody type and direction, and for stress tasks the list of target words in the transcript (Zhang et al., 1 Jun 2026).

Paralinguistic editing targets non-verbal vocal events: breath, laugh, cough, and sigh. Operations are add and remove. Anchors specify event category and operation (Zhang et al., 1 Jun 2026).

Acoustic editing includes speech enhancement and environment transfer. Anchors specify degradation type, environment type and subtype, or target RT60 range (Zhang et al., 1 Jun 2026). The expected output either removes degradation while preserving content or adds the desired environment while preserving content.

The atomic tasks show that the benchmark treats speech as a multi-attribute object whose content, identity, delivery, non-verbal events, and recording conditions can each be edited independently. This suggests that SpeechEditBench is designed as a disentanglement-oriented evaluation suite, even though the paper formulates it operationally through anchors and success indicators rather than through latent-factor language.

4. Anchor-based evaluation and success metrics

Anchor-based evaluation is the central design principle of SpeechEditBench (Zhang et al., 1 Jun 2026). An anchor is an explicit, task-specific specification of what must change or remain. Examples include the target transcript and edited span for content editing, a reference speaker clip for speaker editing, target labels for emotion and style, direction and target words for prosody, event type and operation for paralinguistic editing, and environment type or RT60 range for acoustic editing (Zhang et al., 1 Jun 2026).

For a set of samples $\mathcal{D}$, the benchmark defines the target success indicator $t_i \in \{0,1\}$, the preservation success indicator $p_i \in \{0,1\}$, and the joint success indicator

$j_i = t_i p_i.$

Aggregate scores are

$\mathrm{TS} = \frac{1}{|\mathcal{D}|} \sum_i t_i,$

$\mathrm{PS} = \frac{1}{|\mathcal{D}|} \sum_i p_i,$

$\mathrm{JS} = \frac{1}{|\mathcal{D}|} \sum_i j_i.$

These are reported as percentages (Zhang et al., 1 Jun 2026).

For non-content tasks, preservation success is enforced via an ASR gate. Output audio is transcribed using Whisper large-v3 for English and Paraformer (FunASR) for Chinese, text is normalized, and English uses WER while Chinese uses CER (Zhang et al., 1 Jun 2026). Preservation success is

$p_i = \mathbf{1}[e_i \le 0.10],$

where $e_i$ is WER/CER against the expected transcript (Zhang et al., 1 Jun 2026). For non-content atomic tasks, the expected transcript is the source transcript; for compositional tasks with content editing, it is the content-target transcript.

Target success is task-specific. Speaker editing succeeds if cosine similarity between WavLM-large plus ECAPA-TDNN embeddings of output and reference is at least $0.50$ (Zhang et al., 1 Jun 2026). Emotion editing succeeds if a Gemini-based audio emotion classifier predicts the target label. Style editing succeeds if a Gemini style judge returns target_style_success = true and target_style_score \ge 3, or if the boolean is absent, target_style_score \ge 3 (Zhang et al., 1 Jun 2026). Prosody editing uses duration ratio, median F0 shift in semitones, and stress prominence criteria. For speed, with duration ratio $t_i \in \{0,1\}$0, “faster” succeeds if $t_i \in \{0,1\}$1 and “slower” if $t_i \in \{0,1\}$2 (Zhang et al., 1 Jun 2026). For pitch, “higher” succeeds if $t_i \in \{0,1\}$3 and “lower” if $t_i \in \{0,1\}$4 (Zhang et al., 1 Jun 2026). Paralinguistic editing uses Gemini event scores from 0 to 3, with addition succeeding at score $t_i \in \{0,1\}$5 and removal at score $t_i \in \{0,1\}$6 (Zhang et al., 1 Jun 2026). Acoustic editing uses DNSMOS gains for enhancement, RT60 constraints for reverb transfer, and PANNs scene classification plus score thresholds for noise or scene transfer (Zhang et al., 1 Jun 2026).

This evaluation design is notable because it separates edit effectiveness from preservation fidelity rather than collapsing them into a single score. Related text-based speech editing work already highlighted that speech editing performance is inherently multi-dimensional, involving content correctness, boundary smoothness, identity preservation, and global prosody (Liu et al., 2024). SpeechEditBench extends this logic into a unified benchmark formalism.

5. Compositional editing and formalization of multi-intent success

SpeechEditBench includes 400 compositional samples involving two or three atomic operations within a single instruction (Zhang et al., 1 Jun 2026). The benchmark groups attributes into four high-level categories: semantic content, speaker identity, expressive delivery, and acoustic environment (Zhang et al., 1 Jun 2026). Two-component combinations total 320 samples, with eight pairings: content + speaker, content + emotion, content + prosody, content + acoustic, speaker + emotion, speaker + acoustic, emotion + acoustic, and prosody + acoustic (Zhang et al., 1 Jun 2026). Three-component combinations total 80 samples, with four triplets: content + speaker + emotion, content + speaker + acoustic, content + emotion + acoustic, and speaker + emotion + acoustic (Zhang et al., 1 Jun 2026).

For compositional sample $t_i \in \{0,1\}$7, let $t_i \in \{0,1\}$8 be the set of components, and let $t_i \in \{0,1\}$9 be the atomic evaluator’s success indicator for component $p_i \in \{0,1\}$0 (Zhang et al., 1 Jun 2026). The benchmark defines component success as

$p_i \in \{0,1\}$1

It defines all-component success indicator

$p_i \in \{0,1\}$2

content preservation indicator $p_i \in \{0,1\}$3, and compositional joint success indicator

$p_i \in \{0,1\}$4

These definitions explicitly separate success on individual requested edits, simultaneous success on all requested edits, and simultaneous success under content preservation (Zhang et al., 1 Jun 2026).

The importance of this distinction is empirical. The reported results show that models can often satisfy individual components substantially more often than they satisfy all requested constraints simultaneously (Zhang et al., 1 Jun 2026). A plausible implication is that compositional instruction following exposes attribute entanglement and instruction competition more sharply than atomic editing. This interpretation aligns with the paper’s failure analysis, which documents partial success, last-mentioned-instruction dominance, and content corruption during multi-attribute editing (Zhang et al., 1 Jun 2026).

6. Experimental setup, results, and failure patterns

The benchmark evaluates two model categories: Speech LLMs and specialized speech editing or audio systems (Zhang et al., 1 Jun 2026). Open-source Speech LLMs include Ming-UniAudio-16B-A3B-Edit, Step-Audio-EditX (3B), Qwen3-Omni (30B), Kimi-Audio (7B), Mimo-Audio-Base (7B), and Mimo-Audio-Instruction (7B). Closed-source models are Gemini-Live and GPT-Realtime (Zhang et al., 1 Jun 2026). Specialized systems include VoiceCraft-X for content editing, Seed-VC for speaker editing, VoxCPM2 for emotion/style/prosody, Chatterbox-TTS plus AudioSep for paralinguistic editing, and DeepFilterNet3 plus DSP with RIR convolution for acoustic editing (Zhang et al., 1 Jun 2026). The benchmark focuses on evaluation rather than training; most models are evaluated as-is, with default generation settings and no reported per-model prompt engineering beyond feeding the provided natural-language instructions (Zhang et al., 1 Jun 2026).

The main findings are threefold. First, no single model performs well across all editing dimensions (Zhang et al., 1 Jun 2026). For example, Ming-UniAudio is the best open-source content editor at 76.46% joint success but weak on expressive tasks such as emotion, where it reaches 3.43% joint success (Zhang et al., 1 Jun 2026). Step-Audio-EditX is stronger on expressive editing, with 49.67% style and 31.25% paralinguistic joint success, but low on content at 16.50% (Zhang et al., 1 Jun 2026). Qwen3-Omni is strong on prosody and acoustic editing, with 38.17% and 37.80% joint success respectively, and also reaches 72.00% on content (Zhang et al., 1 Jun 2026).

Second, closed-source Speech LLMs generally outperform open-source models (Zhang et al., 1 Jun 2026). GPT-Realtime achieves 96.67% joint success on content, 68.67% on style, and 47.00% on paralinguistic editing (Zhang et al., 1 Jun 2026). Gemini-Live achieves 65.17% joint success on prosody, 27.79% on emotion, 63.67% on style, and 93.17% on content (Zhang et al., 1 Jun 2026). Third, compositional editing remains highly challenging. In two-component tasks, Gemini-Live has 50.25% component success but only 21.50% all-component and 21.50% joint success; GPT-Realtime has 49.75% component success, 20.50% all-component success, and 20.00% joint success (Zhang et al., 1 Jun 2026). In three-component tasks, almost all models have 0.00% joint success, with Mimo-Audio-Base as the only reported model showing 5.00% joint success (Zhang et al., 1 Jun 2026).

The benchmark also reveals that target success alone is misleading. Step-Audio-EditX, for example, reaches 51.51% target success on prosody but only 39.60% preservation and 20.13% joint success, whereas Ming-UniAudio reaches 28.00% target success, 84.50% preservation, and 26.50% joint success (Zhang et al., 1 Jun 2026). This shows that aggressive editing can improve target attainment while damaging transcript preservation.

The appendix’s failure taxonomy includes target not applied, wrong expressive label, local content edit with global drift, speaker target miss, prosody too weak or wrong direction, paralinguistic add/remove miss, acoustic enhancement without preservation, environment target miss, and compositional partial success (Zhang et al., 1 Jun 2026). These failure modes indicate that instruction-guided speech editing is constrained not only by edit controllability but also by the ability to preserve untargeted attributes under distributional and linguistic variation.

7. Position in the field, limitations, and implications

SpeechEditBench is positioned against prior speech or audio benchmarks and task-specific speech editing datasets that focus on single tasks, use non-comparable metrics, or omit the simultaneous evaluation of edit success and preservation success (Zhang et al., 1 Jun 2026). The benchmark is presented as a first-of-its-kind diagnostic suite for instruction-guided multi-attribute speech editing because it combines bilingual coverage, seven atomic tasks, compositional instructions, and a unified anchor-based evaluation framework (Zhang et al., 1 Jun 2026).

Its broader significance becomes clearer when contrasted with specialized editing systems such as FluentEditor2. FluentEditor2 addresses text-based speech editing with explicit local acoustic consistency and global prosody consistency objectives, and its evaluation centers on fluency-sensitive metrics such as FMOS and IMOS together with objective measures like MCD, STOI, and PESQ (Liu et al., 2024). SpeechEditBench shifts the focus from utterance-local editing quality to instruction-conditioned attribute control and preservation across a wider task space (Zhang et al., 1 Jun 2026). This suggests a division in the field between systems optimized for narrow editing regimes and benchmarks intended to diagnose general-purpose editing competence.

The benchmark has several explicit limitations. It covers only English and Chinese, with no low-resource languages or broader typological coverage (Zhang et al., 1 Jun 2026). Its evaluation is heavily reliant on automatic metrics and LLM-based judges such as Gemini, ASR, and DNSMOS, and the paper does not report separate human listening tests (Zhang et al., 1 Jun 2026). It focuses on single-turn editing rather than multi-turn interactive refinement, and its attribute scope does not include code-switching, dialect or accent conversion, fine-grained voice-style interpolation, or longer conversational context (Zhang et al., 1 Jun 2026).

The authors suggest extending the benchmark to more languages and richer attribute sets, incorporating human-in-the-loop evaluation, and exploring multi-turn editing and interactive correction scenarios (Zhang et al., 1 Jun 2026). A plausible implication is that SpeechEditBench may function less as a final evaluation standard than as a scaffolding for a broader family of speech editing benchmarks. Even in its current form, however, it establishes a rigorous diagnostic framework in which the central question is not merely whether a model can edit speech, but whether it can do so precisely, preserve untargeted attributes, and satisfy multiple simultaneous constraints under natural-language instructions (Zhang et al., 1 Jun 2026).