--- title: 'ISSE: Speech Style Editing Dataset' url: https://www.emergentmind.com/topics/instruction-guided-speech-style-editing-dataset-isse type: topic --- # ISSE: Speech Style Editing Dataset Searching arXiv for the cited papers and closely related benchmarks to ground the article. Instruction-guided Speech Style Editing Dataset (ISSE) is an open-source dataset and benchmark for speech style editing in which a model modifies stylistic properties of speech while preserving the original linguistic content and speaker identity [2509.24570]. It was introduced to address the limitations of label-driven and reference-audio-driven editing, as well as the coarse, template-like instructions of earlier natural-language approaches, by providing approximately 382 hours of speech, more than 100,000 source-target pairs, and diverse fine-grained textual editing instructions aligned with paired speech examples [2509.24570]. Within the recent literature, ISSE occupies the specific niche of paired source–target speech editing under natural-language control, distinct from spoken-instruction style benchmarks, instruction-following TTS evaluations, descriptive speech corpora, bilingual editing benchmarks, joint audio-visual editing datasets, and dialogue-native style-control resources [2509.09716] [2506.16381] [2408.13608] [2606.01804] [2606.03168] [2510.22588]. ## 1. Task formulation and research motivation ISSE addresses speech style editing as the transformation of a source utterance into a target utterance that matches a desired style while keeping transcript and timbre unchanged [2509.24570]. The task is defined as follows: given source speech $x$ and an instruction $c$ describing the style change, generate $y$ that preserves the content and speaker identity of $x$ and reflects $c$. The style space explicitly includes emotion, speaking rate, prosody, pitch, energy or loudness, articulation or clarity, emphasis, pausing, monotonicity, expressivity, and timbre-preserving change specifications. The central motivation is that prior approaches depend on explicit categorical labels or reference audio, which limits flexibility and scalability. More recent natural-language-guided methods remain constrained by oversimplified instructions and coarse style control. ISSE is intended to overcome those constraints through multi-attribute natural-language instructions generated from style descriptions and through a paired data construction process that enforces content and timbre preservation. A defining property of ISSE is that it treats instruction-guided control as an editing problem rather than only a synthesis problem. Each example is a triplet $(x, y, c)$: source speech $x$, target speech $y$ with the same transcript and speaker but different style, and instruction $c$ describing the edit from $x$ to $y$. This makes the dataset suitable for training models that must preserve untargeted attributes rather than simply synthesize expressive speech from scratch. ## 2. Dataset composition and annotation structure ISSE contains approximately 382 hours of speech, comprising approximately 90 hours of “Real” recorded human speech and approximately 292 hours of generated speech, with over 100,000 source-target pairs and 28 distinct style categories [2509.24570]. It is built from Ears and Expresso, which provide speech, transcriptions, and style descriptions. Non-stylistic words such as “he” and “she” were removed from style descriptions to minimize speaker bias. | Aspect | ISSE | |---|---| | Total duration | ≈382 hours | | Real / generated | ≈90 h / ≈292 h | | Paired examples | over 100,000 | | Style coverage | 28 styles | | Instance format | $(x, y, c)$ | | Language | English only | | Source corpora | Ears and Expresso | The paper describes the audio length distribution as short-to-medium with balanced coverage, and the style distribution as broad and balanced across the 28 styles. The dataset is English only, and the paper does not report sample rate, bit depth, format, number of speakers, accents, or detailed gender and age distributions. Split sizes and seen-versus-unseen protocols are also not explicitly reported. Instead, evaluation is conducted under in-domain and cross-domain settings. The instruction schema is centered on natural-language editing directives derived by an LLM, Qwen3-8B, which compares source and target style descriptions and articulates the transformation. The default template is “Convert the source speech to {target style},” but multiple paraphrases are generated to avoid repetition. Examples given in the paper include “Convert the source speech to a whispering style with a high-pitched voice,” “Convert the source speech into a high-pitched whisper,” and the multi-constraint instruction “Convert the source speech to a medium-pitched, raspy monotone that conveys awe.” ## 3. Data generation pipeline and quality control ISSE is constructed through a three-stage instructed speech data generation pipeline [2509.24570]. In the first stage, an anchor speech is sampled as the source $x$, fixing transcript and speaker identity. Then $N$ speeches with distinct styles are sampled as style references, and their style descriptions define the target styles. EmoCapTTS synthesizes target speech using the anchor transcript and a style reference description. Chatterbox voice conversion then adjusts the stylized speech to match the anchor’s timbre while retaining the target style. In the second stage, the synthesized data are filtered using automatic quality criteria. Whisper-large-v3 is used for ASR-based content verification; emotion2vec is used for style similarity; and ECAPA-TDNN embeddings are used for speaker similarity. Samples are retained only when all three constraints are satisfied: $\mathrm{WER} < 10$, $\mathrm{SIM}_{sty} > 0.5$, and $\mathrm{SIM}_{spk} > 0.5$. The speaker similarity is computed by cosine similarity, $$ s = \frac{u \cdot v}{\|u\| \|v\|}, $$ and the paper states that the same cosine form is used for style similarity with emotion2vec embeddings. In the third stage, Qwen3-8B compares source and target style descriptions, extracts the differences, and generates fine-grained natural-language instructions. This instruction design step is crucial because it converts raw paired style differences into editable text controls. The paper attributes content and timbre alignment to the combination of anchor-based synthesis, Chatterbox voice conversion, and the thresholded filtering heuristics. A plausible implication is that ISSE’s main contribution is not only scale, but the coupling of paired speech generation with explicit automatic retention criteria. Unlike datasets built from free-form captions alone, ISSE uses the same metrics employed later in evaluation—WER, style similarity, and speaker similarity—as construction-time filters, which tightly constrains the admissible edit space. ## 4. Baseline model, objective, and evaluation protocol The benchmark model trained on ISSE is LlasaEdit, an instruction-guided autoregressive speech model [2509.24570]. Waveforms are encoded into discrete audio tokens by XCodec2, instructions are tokenized as text, and a transformer-based generator initialized from Llasa autoregressively predicts target tokens conditioned on source tokens and instruction tokens. The conditional factorization is $$ p(y \mid x, c) = \prod_{t=1}^{T} p(y_t \mid y_{