AST: Adaptive, Seamless, and Training-Free Precise Speech Editing

Abstract: Text-based speech editing aims to modify specific segments while preserving speaker identity and acoustic context. Existing methods rely on task-specific training, which incurs high data costs and struggles with temporal fidelity in unedited regions. Meanwhile, adapting Text-to-Speech (TTS) models often faces a trade-off between editing quality and consistency. To address these issues, we propose AST, an Adaptive, Seamless, and Training-free precise speech editing framework. Leveraging a pre-trained autoregressive TTS model, AST introduces Latent Recomposition to selectively stitch preserved source segments with newly synthesized targets. Furthermore, AST extends this latent manipulation to enable precise style editing for specific speech segments. To prevent artifacts at these edit boundaries, the framework incorporates Adaptive Weak Fact Guidance (AWFG). AWFG dynamically modulates a mel-space guidance signal, enforcing structural constraints only where necessary without disrupting the generative manifold. To fill the gap of publicly accessible benchmarks, we introduce LibriSpeech-Edit, a new and larger speech editing dataset. As existing metrics poorly evaluate temporal consistency in unedited regions, we propose Word-level Dynamic Time Warping (WDTW). Extensive experiments demonstrate that AST resolves the controllability-quality trade-off without extra training. Compared to the previous most temporally consistent baseline, AST improves consistency while reducing Word Error Rate by nearly 70%. Moreover, applying AST to a foundation TTS model reduces WDTW by 27%, achieving state-of-the-art speaker preservation and temporal fidelity.

• The paper introduces AST, a training-free framework that precisely edits speech by leveraging latent inversion and recomposition techniques.
• It demonstrates significant improvements with a 27% reduction in temporal errors and a 70% decrease in WER while preserving speaker identity and prosody.
• Methodologies like Adaptive Weak Fact Guidance (AWFG) ensure smooth transitions at edit boundaries, enabling localized style and emotion editing without retraining.

AST: Adaptive, Seamless, and Training-Free Precise Speech Editing

Introduction

The field of neural TTS has reached high-fidelity, expressive outputs through AR (autoregressive) semantic models and AM-FM (Autoregressive Model - Flow Matching) decoders. However, text-based speech editing—targeting replacement or modification of specific segments while perfectly preserving speaker identity and the acoustic context—remains a severely under-solved problem. Most prior work relies on task-specific training, leading to high data cost and domain generalization issues, or leverages TTS backbones that induce artifacts and prosodic drift in unedited speech segments.

The authors introduce AST, a training-free, adaptive, and seamless text-based speech editing framework designed for AM-FM TTS models. AST leverages flow-matching latent inversion and a novel Latent Recomposition mechanism to enable high-precision segmental editing, while Adaptive Weak Fact Guidance (AWFG) ensures that artifacts at edit boundaries are suppressed and speaker identity, prosody, and local rhythm are robustly preserved. The introduction of the LibriSpeech-Edit benchmark and the Word-level Dynamic Time Warping (WDTW) metric target a major gap in evaluation protocols for this domain.

Figure 1: TTS models cannot preserve the rhythmic characteristics of the original speech, but AST can.

Methodology

Flow Matching Latent Inversion and Latent Recomposition

AST operates by exploiting the invertible generative process of AM-FM TTS models. First, the input mel-spectrogram is inverted into the model's continuous latent space using an inverse ODE solver. Target text prompts are processed by a GPT-style AR semantic model to extract desired semantic conditions. Crucially, source and target transcripts are aligned at the word level via LCS to precisely isolate unchanged and edited regions.

For each region, AST constructs factored mel and semantic sequences by concatenating inverted source latents (unchanged segments) and freshly initialized noise (edited; segments), facilitating highly precise boundary-aware speech edits.

Figure 2: Overview of the AST framework, showing input inversion, alignment and recomposition, and fact-guided flow-matching generation.

The latent recomposition strategy critically enables source region invariance—unedited regions directly re-use the original latent and conditioning, acting as a 'weak fact', while new regions are synthesized. This approach outperforms TTS models that merely regenerate the entire utterance for edit instruction, thus preserving fine rhythmic and prosodic structure.

Figure 3: Illustration of the latent recomposition strategy: source latents are retained in unchanged regions and noise is inserted only at locally edited regions based on alignment masks.

Adaptive Weak Fact Guidance (AWFG)

A naïve latent recomposition approach introduces artifacts at edit boundaries due to mismatches between inverted and generatively sampled latent trajectories. AWFG remedies this by modulating the flow's velocity field with a mel-space guidance vector, computed as the difference between the constructed fact mel and the current latent, weighted adaptively based on local deviation magnitude. This mechanism is governed by a framewise weight $\gamma$ with empirically robust results for $0.2 \leq \lambda \leq 0.9$. AWFG thus enforces structural constraints exactly where required, preventing both excessive correction (which could destroy naturalness) and boundary artifacts.

Figure 4: Qualitative comparison of speech synthesis at edit boundaries; AWFG eliminates boundary artifacts and preserves smoothness.

Support for Localized Style Editing

AST fundamentally supports not only content but also style, emotion, and timbre control at the segment level, owing to the design of the parallel recomposition in the latent and semantic space. By injecting different style conditions into the target semantics of specific segments and maintaining original latents and conditions elsewhere, AST can modulate prosody or style within arbitrary temporal intervals—surpassing utterance-level approaches in locality and expressiveness.

Figure 5: Mel-spectrogram visualization of controllable emotion editing shows fine-grained control over localized style while preserving context and speaker consistency.

Experimental Evaluation

Dataset and Benchmarks

The authors release LibriSpeech-Edit, a 2000-pair corpus derived from LibriSpeech's test-clean subset and augmented with diverse, text-guided edits via LLM augmentation. The dataset surpasses prior real-edit datasets in both scale and reproducible access.

Metrics

Evaluation employs WER (transcript fidelity), DNSMOS (acoustic quality), Speaker Similarity (SpkSim via WavLM embedding cosine distance), and the proposed WDTW, which accurately captures word-level temporal consistency through forced alignment and segmental DTW.

Results

AST demonstrates the following strong results on LibriSpeech-Edit:

• WDTW (Temporal Consistency): 0.2025 (best), reducing the previous best by 27% and the TTS-model baseline by over 27%, confirming exact preservation of temporal structure in unedited segments.
• Speaker Preservation: SpkSim = 0.986 (best), outperforming both task-specific and zero-shot TTS-trained competitors.
• Textual Accuracy: WER = 2.91%, with a nearly 70% reduction over task-specific models.
• Acoustic Quality: DNSMOS comparable with state-of-the-art, despite local constraints from AWFG.

Ablation studies establish that without AWFG, boundary artifacts increase, and WER inflates by nearly 58%, affirming that guidance is critical for high-fidelity edits with robust preservation.

Implications and Future Directions

AST fundamentally redefines the design space for speech editing in AM-FM TTS architectures by:

• Enabling high-precision, locality-preserving edits without any task-specific finetuning.
• Allowing flexible extension to style and emotion editing with segment-level granularity, which can drive new use cases in media post-production, personalized TTS assistants, and downstream prosody modeling research.
• Proposing LibriSpeech-Edit and WDTW as reproducible community standards for quantifying edit fidelity and temporal consistency.

Limitations include potential reliance on high-quality alignment and forced alignment models, and the framework's current coupling to AM-FM architectures. Future directions may extend AST editing to other generative frameworks (e.g., purely diffusion-based, non-AR models), investigate hierarchical or cross-utterance consistency editing, and explore more sophisticated or neural-guided weighting for fact guidance. Additionally, the method opens the path for plug-and-play editing in increasingly large multi-lingual, multi-style foundational speech models.

Conclusion

AST offers an effective, domain-agnostic solution to text-based speech editing, surpassing existing task-trained models in all controllability metrics while remaining entirely training-free. Its core contributions—latent recomposition and AWFG—set new benchmarks for fine-grained content and style editing in neural TTS. The work underscores the value of structured latent-space manipulation for precise, controllable, and robust neural generative modeling in speech processing.