ISSE: Speech Style Editing Dataset

• The paper introduces ISSE as a paired speech style editing dataset that transforms source utterances to match desired styles while preserving content and speaker identity.
• It employs a three-stage data generation and quality control pipeline using metrics like WER, style similarity, and speaker similarity to ensure precise edits.
• Empirical results indicate that the benchmark model LlasaEdit trained on ISSE outperforms competitors in both in-domain and cross-domain evaluations.

Searching arXiv for the cited papers and closely related benchmarks to ground the article. Instruction-guided Speech Style Editing Dataset (ISSE) is an open-source dataset and benchmark for speech style editing in which a model modifies stylistic properties of speech while preserving the original linguistic content and speaker identity (Chen et al., 29 Sep 2025). It was introduced to address the limitations of label-driven and reference-audio-driven editing, as well as the coarse, template-like instructions of earlier natural-language approaches, by providing approximately 382 hours of speech, more than 100,000 source-target pairs, and diverse fine-grained textual editing instructions aligned with paired speech examples (Chen et al., 29 Sep 2025). Within the recent literature, ISSE occupies the specific niche of paired source–target speech editing under natural-language control, distinct from spoken-instruction style benchmarks, instruction-following TTS evaluations, descriptive speech corpora, bilingual editing benchmarks, joint audio-visual editing datasets, and dialogue-native style-control resources (Zhan et al., 9 Sep 2025, Huang et al., 19 Jun 2025, Jin et al., 2024, Zhang et al., 1 Jun 2026, Chen et al., 2 Jun 2026, Tu et al., 26 Oct 2025).

1. Task formulation and research motivation

ISSE addresses speech style editing as the transformation of a source utterance into a target utterance that matches a desired style while keeping transcript and timbre unchanged (Chen et al., 29 Sep 2025). The task is defined as follows: given source speech $x$ and an instruction $c$ describing the style change, generate $y$ that preserves the content and speaker identity of $x$ and reflects $c$. The style space explicitly includes emotion, speaking rate, prosody, pitch, energy or loudness, articulation or clarity, emphasis, pausing, monotonicity, expressivity, and timbre-preserving change specifications.

The central motivation is that prior approaches depend on explicit categorical labels or reference audio, which limits flexibility and scalability. More recent natural-language-guided methods remain constrained by oversimplified instructions and coarse style control. ISSE is intended to overcome those constraints through multi-attribute natural-language instructions generated from style descriptions and through a paired data construction process that enforces content and timbre preservation.

A defining property of ISSE is that it treats instruction-guided control as an editing problem rather than only an overview problem. Each example is a triplet $(x, y, c)$: source speech $x$, target speech $y$ with the same transcript and speaker but different style, and instruction $c$ describing the edit from $x$ to $c$0. This makes the dataset suitable for training models that must preserve untargeted attributes rather than simply synthesize expressive speech from scratch.

2. Dataset composition and annotation structure

ISSE contains approximately 382 hours of speech, comprising approximately 90 hours of “Real” recorded human speech and approximately 292 hours of generated speech, with over 100,000 source-target pairs and 28 distinct style categories (Chen et al., 29 Sep 2025). It is built from Ears and Expresso, which provide speech, transcriptions, and style descriptions. Non-stylistic words such as “he” and “she” were removed from style descriptions to minimize speaker bias.

Aspect	ISSE
Total duration	≈382 hours
Real / generated	≈90 h / ≈292 h
Paired examples	over 100,000
Style coverage	28 styles
Instance format	$c$1
Language	English only
Source corpora	Ears and Expresso

The paper describes the audio length distribution as short-to-medium with balanced coverage, and the style distribution as broad and balanced across the 28 styles. The dataset is English only, and the paper does not report sample rate, bit depth, format, number of speakers, accents, or detailed gender and age distributions. Split sizes and seen-versus-unseen protocols are also not explicitly reported. Instead, evaluation is conducted under in-domain and cross-domain settings.

The instruction schema is centered on natural-language editing directives derived by an LLM, Qwen3-8B, which compares source and target style descriptions and articulates the transformation. The default template is “Convert the source speech to {target style},” but multiple paraphrases are generated to avoid repetition. Examples given in the paper include “Convert the source speech to a whispering style with a high-pitched voice,” “Convert the source speech into a high-pitched whisper,” and the multi-constraint instruction “Convert the source speech to a medium-pitched, raspy monotone that conveys awe.”

3. Data generation pipeline and quality control

ISSE is constructed through a three-stage instructed speech data generation pipeline (Chen et al., 29 Sep 2025). In the first stage, an anchor speech is sampled as the source $c$2, fixing transcript and speaker identity. Then $c$3 speeches with distinct styles are sampled as style references, and their style descriptions define the target styles. EmoCapTTS synthesizes target speech using the anchor transcript and a style reference description. Chatterbox voice conversion then adjusts the stylized speech to match the anchor’s timbre while retaining the target style.

In the second stage, the synthesized data are filtered using automatic quality criteria. Whisper-large-v3 is used for ASR-based content verification; emotion2vec is used for style similarity; and ECAPA-TDNN embeddings are used for speaker similarity. Samples are retained only when all three constraints are satisfied: $c$4, $c$5, and $c$6. The speaker similarity is computed by cosine similarity,

$c$7

and the paper states that the same cosine form is used for style similarity with emotion2vec embeddings.

In the third stage, Qwen3-8B compares source and target style descriptions, extracts the differences, and generates fine-grained natural-language instructions. This instruction design step is crucial because it converts raw paired style differences into editable text controls. The paper attributes content and timbre alignment to the combination of anchor-based synthesis, Chatterbox voice conversion, and the thresholded filtering heuristics.

A plausible implication is that ISSE’s main contribution is not only scale, but the coupling of paired speech generation with explicit automatic retention criteria. Unlike datasets built from free-form captions alone, ISSE uses the same metrics employed later in evaluation—WER, style similarity, and speaker similarity—as construction-time filters, which tightly constrains the admissible edit space.

4. Baseline model, objective, and evaluation protocol

The benchmark model trained on ISSE is LlasaEdit, an instruction-guided autoregressive speech model (Chen et al., 29 Sep 2025). Waveforms are encoded into discrete audio tokens by XCodec2, instructions are tokenized as text, and a transformer-based generator initialized from Llasa autoregressively predicts target tokens conditioned on source tokens and instruction tokens. The conditional factorization is

$c$8

with negative log-likelihood training

$c$9

The paper does not specify auxiliary style, content, or speaker losses beyond NLL. Training uses LoRA fine-tuning with rank $y$0 and $y$1 on attention and MLP layers, AdamW with learning rate $y$2 and learning-rate decay, 5 epochs, batch size 2, and 8 NVIDIA RTX A100 GPUs.

Evaluation is organized around four objective metrics. Content consistency is measured by WER using Whisper-large-v3,

$y$3

style similarity is measured with emotion2vec cosine similarity,

$y$4

speaker identity is measured with ECAPA-TDNN cosine similarity,

$y$5

and naturalness is measured by UTMOS. The same metrics are used in both in-domain and cross-domain evaluation, which makes the benchmark internally consistent with the data construction pipeline.

5. Empirical results and generalization behavior

The reported results show that LlasaEdit trained on ISSE outperforms the same model trained on ESD in both in-domain and cross-domain settings (Chen et al., 29 Sep 2025). In-domain, LlasaEdit-ISSE on ISSE achieves WER 8.06, SIMsty 0.68, SIMspk 0.58, and UTMOS 4.29, while LlasaEdit-ESD on ESD achieves WER 10.07, SIMsty 0.64, SIMspk 0.49, and UTMOS 4.01. The paper summarizes this as better content fidelity, style adherence, timbre preservation, and naturalness for ISSE.

Evaluation setting	WER	SIMsty	SIMspk	UTMOS
LlasaEdit-ISSE on ISSE	8.06	0.68	0.58	4.29
LlasaEdit-ESD on ESD	10.07	0.64	0.49	4.01
LlasaEdit-ISSE on ESD	7.69	0.59	0.51	4.31
LlasaEdit-ESD on ISSE	68.17	0.54	0.49	4.12

The cross-domain results are especially notable. LlasaEdit-ISSE tested on ESD attains WER 7.69, SIMsty 0.59, SIMspk 0.51, and UTMOS 4.31, surpassing ESD in-domain on most metrics. By contrast, LlasaEdit-ESD tested on ISSE degrades severely to WER 68.17, SIMsty 0.54, SIMspk 0.49, and UTMOS 4.12. The paper attributes this difference to ISSE’s scale, diversity, and fine-grained instructions, and concludes that these substantially improve controllability and generalization.

The paper also reports synthesis-oriented extensions. For TTS with Llasa-1B, finetuning on ISSE reduces WER from 5.48 to 3.74 and improves UTMOS from 3.93 to 4.03, while SIMsty changes from 0.48 to 0.49. For ESS, the reported numbers are WER 5.25, SIMsty 0.60, Gender 0.96, and UTMOS 3.94. These results indicate that the dataset is usable beyond direct editing, although the paper’s primary framing remains instruction-guided speech style editing.

6. Position within the broader literature, limitations, and ethical considerations

ISSE is part of a rapidly expanding cluster of instruction-guided speech style resources, but its formulation is narrower and more explicitly edit-oriented than several adjacent datasets and benchmarks. VStyle introduces Voice Style Adaptation with spoken instructions and a bilingual benchmark covering acoustic attributes, natural-language instruction, role play, and implicit empathy; it is designed precisely for instruction-guided speech style editing in practice, but it functions primarily as a test benchmark rather than a paired editing corpus (Zhan et al., 9 Sep 2025). InstructTTSEval provides a multilingual benchmark for complex natural-language instruction following in TTS through Acoustic-Parameter Specification, Descriptive-Style Directive, and Role-Play, with Gemini used as an automatic judge; it effectively fulfills an ISSE-like benchmarking role without being a source–target editing dataset (Huang et al., 19 Jun 2025). SpeechCraft supplies a much larger bilingual corpus with Description and Instruction modalities, rich style captions, and emphasis-oriented parallel subsets, making it a strong foundation for instruction-guided editing even though most of its supervision is not organized as explicit source–instruction–target triplets (Jin et al., 2024).

More recent benchmarks sharpen the evaluation landscape. SpeechEditBench is a bilingual multi-attribute benchmark explicitly designed for instruction-guided speech editing, including style editing, emotion editing, prosody editing, paralinguistic editing, acoustic editing, content editing, speaker editing, and compositional tasks, with anchor-based metrics that separate target success from preservation success (Zhang et al., 1 Jun 2026). JAVEdit-100k extends the problem to human-centric joint audio-visual editing, explicitly separating speech content editing from speech style editing and supporting voice timbre and style changes, identity-preserving content replacement, and lip-sync consistency; the paper does not mention a dedicated ISSE, but presents JAVEdit-100k as a practical substitute within joint audio-visual editing (Chen et al., 2 Jun 2026). UltraVoice, by contrast, is described as an instruction-guided, style-controlled speech dialogue dataset rather than an editing dataset in the strict sense, because it focuses on instruction-driven generation in dialogue rather than source-audio-to-target-audio transformation (Tu et al., 26 Oct 2025).

The main documented limitations of ISSE are its English-only coverage, the absence of reported speaker demographics, and incomplete reporting of audio specifications and explicit split protocols (Chen et al., 29 Sep 2025). The paper also notes broader misuse risks: speech style editing and voice conversion can be used for impersonation or deepfake generation. Suggested safeguards include watermarking, provenance tracking, and usage restrictions aligned with ethical guidelines. This suggests that ISSE should be understood not only as a data resource, but also as a benchmark that foregrounds the unresolved tension between controllable voice manipulation and responsible deployment.