SpeechEditBench: A Bilingual Multi-Attribute Benchmark for Instruction-Guided Speech Editing

Abstract: Instruction-guided speech editing requires a model to modify specified speech attributes while preserving unrelated characteristics. Despite rapid progress in Speech LLMs (Speech LLMs), systematic evaluation of this capability remains challenging, as existing benchmarks are fragmented across isolated editing tasks. To bridge this gap, we introduce \textbf{SpeechEditBench}, a bilingual multi-attribute benchmark for instruction-guided speech editing. SpeechEditBench encompasses seven atomic editing tasks, as well as compositional editing tasks that integrate multiple operations within a single instruction. We propose an anchor-based evaluation protocol that separately assesses the edit success of target attributes and the preservation of untargeted attributes, leading to three metrics: target success, preservation success, and joint success. Using this benchmark, we evaluate mainstream Speech LLMs and specialized speech editing systems. The results reveal three key findings: (1) no single model performs well across all editing dimensions; (2) closed-source Speech LLMs generally outperform open-source models; (3) compositional editing remains highly challenging, with even the most advanced models struggling to achieve high joint success. SpeechEditBench provides a rigorous diagnostic framework to identify bottlenecks in Speech LLMs, thereby facilitating the development of next-generation Speech LLMs with more robust and precise instruction-guided editing capabilities. Data and code will be released upon acceptance.

• The paper introduces SpeechEditBench, a unified bilingual benchmark organizing seven atomic and compositional speech editing tasks.
• It employs an innovative anchor-based evaluation protocol to measure both target attribute modifications and the preservation of non-target features.
• Results reveal significant performance trade-offs and compositional editing challenges, underscoring the need for more robust SpeechLLMs.

SpeechEditBench: A Systematic Benchmark for Instruction-Guided Speech Editing

Introduction and Motivation

Instruction-guided speech editing presents unique challenges for Speech LLMs (SpeechLLMs): a model must not only precisely modify specified attributes (e.g., speaker, emotion, background acoustics) in response to natural language instructions but also maintain all unrelated aspects of the input signal. Existing benchmarks either focus on isolated editing tasks or lack unified definitions and evaluation criteria, making it difficult to comprehensively evaluate or compare SpeechLLM capabilities across multiple, composable dimensions.

SpeechEditBench introduces a unified, bilingual (English/Chinese) multi-attribute benchmark specifically targeting instruction-guided speech editing, filling a critical gap in robust performance diagnostics and methodological rigor for SpeechLLMs.

Figure 1: Overall framework of the proposed SpeechEditBench.

Benchmark Design

Task Hierarchy and Dataset Construction

SpeechEditBench comprises seven atomic editing tasks—content, speaker, emotion, style, prosody, paralinguistic, and acoustic—and compositional tasks combining multiple editing operations per instruction. Each sample consists of a source speech, a natural language instruction, and, when needed, reference anchors (e.g., a reference speaker waveform for timbre conversion). Cross-lingual generalization is supported by balanced task splits in both English and Chinese.

Figure 2: Hierarchical sample distribution of SpeechEditBench (sunburst chart).

Dataset construction leverages high-quality, publicly available corpora, with additional data curation using LLMs (e.g., GPT-4o for semantic edits and Gemini-2.5-pro for style/event annotation). The final dataset contains 4,700 samples, including both atomic and compositional editing tasks.

Anchor-Based Evaluation Protocol

A key innovation is the anchor-based evaluation, replacing rigid waveform matching protocols that are ill-suited for the one-to-many nature of valid speech edits. Evaluation is decomposed into:

• Target success: Whether the requested attribute modification is achieved.
• Preservation success: Whether all non-target (unmodified) aspects are preserved.
• Joint success: Success in both target modification and preservation.

For example, content edits require precise transcript alignment (WER/CER), speaker edits require a minimum cosine similarity threshold in embedding space, and expressive/audio environment edits rely on classifier or regression metrics (e.g., DNSMOS for enhancement). For compositional tasks, evaluation is conducted at component, all-component, and joint levels.

Model Evaluation and Analysis

Model Coverage

SpeechEditBench systematically evaluates:

• Generalist SpeechLLMs: Six open-source (Ming-UniAudio, Step-Audio-EditX, Qwen3-Omni, Kimi-Audio, Mimo-Audio-Base, Mimo-Audio-Instruction) and two closed-source (Gemini-Live, GPT-Realtime) models. Of note, current SpeechLLMs do not accept both source and reference audio, precluding direct speaker-editing evaluation in that category.
• Task-specialized systems: Domain-specific pipelines (VoiceCraft-X, Seed-VC, VoxCPM2, DeepFilterNet, etc.) serve as performance reference points.

Quantitative Findings

• Capability fragmentation: No SpeechLLM achieves consistently high accuracy across all editing axes. Ming-UniAudio is strong in content, Step-Audio-EditX in expressive attributes, and Qwen3-Omni in prosody/acoustic, but all models demonstrate significant trade-offs.
• Closed-source superiority: Gemini-Live and GPT-Realtime outperform open-source models, particularly on joint success, indicating that proprietary models have better integrated attribute control with preservation. For instance, GPT-Realtime achieves 96.67% joint success in content editing and 68.67% in style, with markedly smaller gaps between target and joint success.
• Compositional edit bottleneck: All models—including the best closed-source LLMs—struggle with compositional edits. Joint success for two-component edits never exceeds 21.50%, and is near-zero for three-component combinations, illustrating a core limitation in current SpeechLLMs’ ability to jointly satisfy multiple constraints.
• Preservation is central: Many systems can apply the desired attribute change (high target success) but fail to reliably preserve non-target information, leading to sharp drops in joint success.

  Figure 3: Task-wise capability profiles of six representative SpeechLLMs, showing (left) target success and (right) much lower joint success when preservation is enforced.

• Specialized vs. generalist comparison: Specialized systems (e.g., VoiceCraft-X for English content editing, Seed-VC for speaker editing) outperform generalist SpeechLLMs on their target tasks (e.g., 84.00% vs. 76.46% joint success on English content editing). However, the specialized models do not provide general-purpose instruction following and cannot match open benchmarks for other tasks.

Language Effects and Fine-Grained Analyses

• Language bias: Several models display language-dependent performance, with some favoring English for preservation and Mandarin for editing accuracy on lexical content (notably Ming-UniAudio and Step-Audio-EditX).
• Emotion task design: The benchmark incorporates both standard and challenging emotion-editing splits, the latter involving lexical-affective conflict. Most models’ performance decreases on the challenging set, but the effect is model-specific, indicating divergent ability to override textual-affective content when modulating prosody/emotion.

Implications and Theoretical Impact

SpeechEditBench exposes structural weaknesses in even the strongest SpeechLLMs, notably in compositional controllability and non-target attribute preservation. This carries both methodological and practical implications:

• Research directions: The findings suggest future SpeechLLMs must incorporate more advanced mechanisms for explicit attribute disentanglement, robust preservation gates, and compositional reasoning across audio attributes. The anchor-based evaluation can serve as a diagnostic tool for model ablations and architecture search in SpeechLLM development.
• Practical deployment: In production scenarios (e.g., editing spoken content for media, accessibility, or voice UIs), existing generalist SpeechLLMs cannot yet guarantee attribute-specific modifications without risk of degrading other speech properties.
• Benchmark extensibility: While the current SpeechEditBench focuses on English and Mandarin and seven atomic attributes, the protocol and methodology generalize to low-resource languages, additional paralinguistic styles, code-switching, and multi-turn (conversational) edit scenarios.

Conclusion

SpeechEditBench represents a rigorous, large-scale, bilingual diagnostic suite for instruction-guided speech editing. The anchor-based evaluation paradigm enables precise quantification of both edit effectiveness and preservation fidelity, establishing new standards for holistic SpeechLLM assessment. Analysis indicates that while closed-source models outperform open-source models in joint controllability, all models currently exhibit severe fragmentation and low compositional edit reliability. Integrating the strengths of specialized pipelines into generalist SpeechLLMs and advancing compositional control remain open challenges. SpeechEditBench provides both the dataset and methodological infrastructure to accelerate research closure on these gaps.

For further methodological details and full evaluation metrics, see "SpeechEditBench: A Bilingual Multi-Attribute Benchmark for Instruction-Guided Speech Editing" (2606.01804).