AudioEdit: Instruction-Based Audio Editing

• AudioEdit is the modification of recordings to meet user intent while preserving edit-irrelevant content in various modalities.
• It employs instruction-conditioned generation using models like latent diffusion and attention mechanisms to precisely localize and execute edits.
• Evaluation relies on objective and subjective metrics, ensuring both faithful instruction adherence and preservation of original audio qualities.

AudioEdit denotes the modification of a pre-existing recording so that it satisfies a user’s intent while preserving edit-irrelevant content. In the recent foundation-model literature, the term refers both to the broader problem of instruction-based audio editing across speech, music, and general sound, and, in a narrower usage, to specific datasets and benchmarks built for that problem. The central formulation is conditional generation of an edited waveform $a'$ from an original audio clip $a$ and a user instruction $u$, as in $a' \sim P_{\theta}(\cdot \mid a,u)$, with modern systems differing primarily in how they represent the source audio, localize the edit, and maintain fidelity outside the edited region (Pan et al., 22 Jun 2026, Gong et al., 19 Jun 2026).

1. Concept and problem formulation

Audio editing is defined as modifying a pre-existing recording so that it meets a user’s intent, whether by changing its acoustic character, semantic content, or the presence of particular sound sources, while keeping everything else unchanged. A narrower operational definition, used in benchmark construction, treats audio editing as an operation that takes an existing audio clip and a textual edit instruction and produces a modified clip in which one may add a new sound event, remove a particular event, or replace one event with another while preserving the rest of the audio (Pan et al., 22 Jun 2026, Jia et al., 16 Aug 2025).

The literature increasingly formulates the task as instruction-conditioned generation rather than direct signal manipulation. Bagpiper-Edit makes this explicit by casting open-ended audio editing as conditional generation of an edited waveform given an “anchor” clip and a free-form instruction, then reframing the problem through a “rich caption” $c$, a detailed natural-language description of all perceptual events and attributes in the source. In that formulation, the system first extracts $c$ from $a$, rewrites $c$ into an edited caption $c'$ using the user request $u$, and then generates $a$0 conditioned on the original audio, the original caption, and the edited caption (Gong et al., 19 Jun 2026).

This framing separates three requirements that recur throughout the field. First, the system must execute the requested edit. Second, it must preserve non-target content such as speaker identity, background ambience, temporal structure, or musical phrasing. Third, it must do so under open-domain conditions that may involve speech, music, sound events, or mixtures of these modalities. The survey literature treats these requirements as the defining constraints of AudioEdit in the era of foundation models (Pan et al., 22 Jun 2026).

2. Taxonomy of operations and task complexity

A unified taxonomy divides audio editing into three high-level categories: acoustic editing, semantic editing, and instance editing. Acoustic editing includes loudness or volume editing, denoising and restoration, reverberation editing, and audio equalization. Semantic editing includes linguistic editing, expressive editing, and stylistic editing. Instance editing includes instance replacement, instance deletion or extraction, instance insertion, and instance overlay. These twelve fine-grained tasks are organized by the survey under three information layers: signal-level, semantic, and source-level editing (Pan et al., 22 Jun 2026).

Recent systems extend this taxonomy beyond the early add-delete-replace triad. MMEdit formalizes six representative operations: addition, removal, replacement, reordering, loudness adjustment, and speed modification, viewing each as a transformation from a source clip $a$1 to a target edited clip $a$2 conditioned on an instruction $a$3. Audio-Omni’s AudioEdit dataset covers add, remove, extract, and style transfer across environmental sounds, music passages, and speech utterances. SmartDJ further emphasizes declarative editing, in which the user specifies a desired outcome and the system decomposes it into atomic operations such as adding, removing, or spatially relocating events (Tao et al., 23 Dec 2025, Tian et al., 12 Apr 2026, Lan et al., 25 Sep 2025).

Benchmark design has also expanded from operation type to task complexity. MMAE organizes AudioEdit evaluation across seven modalities—sound, music, speech, and all mixed combinations—eight operation types, two levels of granularity, and six levels of complexity: single, multi-part, multi-instruction, multi-audio, multi-round, and multi-hop. Local edits are confined to a time-segment or event; global edits affect the entire clip or its overall characteristics. This suggests that operation type alone is insufficient for characterizing difficulty, because the same nominal edit may become substantially harder when embedded in mixed-modality, multi-hop, or iterative contexts (Ma et al., 5 Jun 2026).

3. Model paradigms and architectural strategies

The survey literature groups modern AudioEdit methods into training-based and training-free paradigms. Training-based methods include task-specific training, reference- and attribute-based training, and instructional training. Training-free methods include inversion-based editing, attention modification, and mask-guided editing. Across both paradigms, current systems largely rely on latent diffusion models, diffusion transformers, autoregressive codec LLMs, or multimodal audio-LLMs for instruction grounding (Pan et al., 22 Jun 2026).

Training-based systems generally learn from paired or pseudo-paired edit data. AUDIT constructs triplets of instruction, input audio, and edited output audio for adding, dropping, replacement, inpainting, and super-resolution, and conditions a latent diffusion U-Net on both the noisy target latent and the clean input-audio latent so that the model can learn “where to change” without an explicit mask. Bagpiper-Edit instead avoids paired audio-editing training data by reformulating editing as rich-caption rewriting, using Qwen3-8B-Base with X-Codec as the audio-language backbone and Qwen3-235B-Instruct for caption rewriting at inference time. SmartDJ separates planning from execution: an Audio LLM attends jointly to stereo audio and text to produce a sequence of atomic editing steps, and a Latent Diffusion Model executes those steps one by one. MMEdit integrates a Qwen2-Audio encoder with an MMDiT-based generator, while the hybrid rectified-flow framework performs joint attention over audio and text at a low-resolution stage and alternating joint-attention and cross-attention refinement at a high-resolution stage to balance semantic alignment and efficiency (Wang et al., 2023, Gong et al., 19 Jun 2026, Lan et al., 25 Sep 2025, Tao et al., 23 Dec 2025, Gao et al., 18 Jun 2026).

Training-free systems use pretrained generative backbones without audio-editing fine-tuning. AudioEditor builds on a pretrained latent-diffusion text-to-audio model and combines DDIM inversion, Null-Text Optimization, and EOT-suppression so that edited generations preserve original audio features while executing prompt changes. PPAE is a training-free controller that plugs into latent diffusion models by extracting and fusing cross-attention maps from the source prompt and the edited prompt through a hierarchical local-global pipeline. DirectAudioEdit removes the inversion step entirely and constructs an editing path through “diffusion prediction contrast,” re-noising source and edited states under shared Gaussian noise and updating the clean state by contrasting reverse-diffusion displacements. AudioMorphix operates directly on spectrograms, uses morphing and demorphing in latent space, adds energy-based guidance functions, and injects cached self-attention keys and values from the original source inversion to preserve fine detail. EditGen applies Prompt-to-Prompt-like cross-attention control to a frozen auto-regressive MUSICGEN backbone and defines replacement, reweighting, and refinement mechanisms over attention scores (Jia et al., 2024, Xu et al., 2024, Ge et al., 5 Jun 2026, Liang et al., 21 May 2025, Sioros et al., 15 Jul 2025).

These strategies differ mainly in how they localize the edit. Some methods concatenate source and target latents, some rewrite textual semantic representations, some manipulate cross-attention, and some introduce explicit or implicit masks. A plausible implication is that AudioEdit has become less a single algorithmic problem than a design space defined by trade-offs among instruction grounding, locality, and source preservation.

4. Data construction, benchmarks, and evaluation protocols

Data scarcity is a recurring bottleneck. Bagpiper-Edit explicitly circumvents paired audio-editing training data by treating editing as caption rewriting, but most training-based systems synthesize or curate large triplet corpora. Audio-Omni introduces AudioEdit, a dataset comprising over one million editing pairs, built from a real-data branch and an overview branch, with 1.1 M training samples covering add, remove, extract, and style transfer across sound, speech, and music. MMEdit likewise constructs over one million $a$4 pairs through a Scaper-backed pipeline with event-level metadata and reports 2,500 hours total. RFM-Editing builds a dataset from AudioCaps2 and single-event clips, retaining only pairs whose CLAP similarity to their caption is at least 0.35 and producing 234,639 training samples, 26,103 validation samples, and 26,103 test samples. SmartDJ generates 50 K complex-instruction trajectories for training and expands them to 500 K single-step pairs for the diffusion editor (Tian et al., 12 Apr 2026, Tao et al., 23 Dec 2025, Gao et al., 17 Sep 2025, Lan et al., 25 Sep 2025).

Evaluation protocols mix objective metrics, subjective listening studies, and, increasingly, edit-aware judges. Common objective metrics include Fréchet Audio Distance, Fréchet Distance in embedding space, Log Spectral Distance, KL divergence, Inception Score, CLAP similarity, speaker similarity, WER or CER, DNSMOS, and task-specific measures such as GCC MSE, CRW MSE, FSAD, Rhythm F1, or edit-span accuracy. The survey emphasizes that audio editing must be judged both on instruction adherence and on preservation of non-target content, and not solely on generic generation quality (Pan et al., 22 Jun 2026).

AuditScore was introduced as a dedicated subjective-evaluation dataset with 6,360 edited samples generated from 7 representative frameworks and 23 system configurations, each rated by five professional raters on Quality, Relevance, and Faithfulness. AuditEval then trains three regression heads, one per dimension, to predict MOS-style scores from original audio, edited audio, original caption, and editing instruction; on AuditScore, system-level LCC reaches 0.6101 for Quality, 0.6196 for Relevance, and 0.8460 for Faithfulness. This is one of the clearest demonstrations that edit-aware scoring has become a research topic in its own right (Jia et al., 16 Aug 2025).

MMAE pushes evaluation further by decomposing each sample into rubric-level checks. For task $a$5, with instruction-following rubric set $a$6 and consistency rubric set $a$7, it defines

$a$8

$a$9

and

$u$0

MMAE contains 2,000 high-fidelity samples and 17,741 rubrics, providing a more stringent view of exact task completion than average embedding or MOS scores (Ma et al., 5 Jun 2026).

5. Representative systems and empirical trends

AUDIT established an early instruction-guided latent-diffusion baseline for general audio editing by training directly on triplets rather than relying on generation-time inversion. On its reported tasks, it achieves, for example, LSD $u$1 versus a best baseline of $u$2 on adding, FD $u$3 versus $u$4, and KL $u$5 versus $u$6; on dropping, it reports Quality $u$7 versus $u$8 and Relevance $u$9 versus $a' \sim P_{\theta}(\cdot \mid a,u)$0 in subjective tests. These results established the practicality of instruction-only conditioning without requiring a full target caption (Wang et al., 2023).

Later systems broaden the scope of what counts as editable audio. Bagpiper-Edit reports that its multi-turn pattern achieves 14.01% WER, 79.8% span accuracy, and SpkSIM 0.83 on speech editing, while the single-turn pattern preserves speaker identity best at SpkSIM 0.86 and the base model without anchoring drops to SpkSIM 0.58. On free-form editing, the multi-turn pattern reaches CapSIM $a' \sim P_{\theta}(\cdot \mid a,u)$1 with FAD $a' \sim P_{\theta}(\cdot \mid a,u)$2. SmartDJ, designed for declarative stereo editing, reports FD $a' \sim P_{\theta}(\cdot \mid a,u)$3, FAD $a' \sim P_{\theta}(\cdot \mid a,u)$4, IS $a' \sim P_{\theta}(\cdot \mid a,u)$5, and CLAP $a' \sim P_{\theta}(\cdot \mid a,u)$6 on high-level instruction editing, and is chosen approximately $a' \sim P_{\theta}(\cdot \mid a,u)$7–$a' \sim P_{\theta}(\cdot \mid a,u)$8 of the time in human preference tests. Audio-Omni reports AE_FAD $a' \sim P_{\theta}(\cdot \mid a,u)$9, AE_LSD $c$0, and CLAP $c$1, outperforming ZETA and SDEdit on its AudioEdit benchmark (Gong et al., 19 Jun 2026, Lan et al., 25 Sep 2025, Tian et al., 12 Apr 2026).

A parallel trend is the reduction of inference cost. AudioEditor demonstrates the viability of training-free inversion-based editing but remains computationally heavy. DirectAudioEdit removes inversion and reports a macro-averaged 15.9% reduction in FAD and 15.8% reduction in KL compared with DDPM inversion, while achieving up to 64.5% editing speedup. The hybrid rectified-flow diffusion transformer reports average editing time $c$2 s versus AudioEditor’s $c$3 s, with a compact $c$4 M-parameter model. This suggests that editing quality and editing latency are no longer treated as separable concerns: inference efficiency is becoming a first-order benchmark axis alongside fidelity and instruction adherence (Ge et al., 5 Jun 2026, Gao et al., 18 Jun 2026, Jia et al., 2024).

The empirical picture, however, is not uniformly monotonic. Bagpiper-Edit observes that single-turn prompting can over-anchor and under-edit, whereas multi-turn prompting better balances consistency and edit strength. MMAE reports that Exact Match Rate remains below 5% for all evaluated models and falls to 0% in the hardest mixed-modality settings. A plausible implication is that many systems already produce convincing local improvements while still failing exact, end-to-end task execution under compositional instructions (Gong et al., 19 Jun 2026, Ma et al., 5 Jun 2026).

6. Limitations, misconceptions, and future directions

The survey literature identifies three persistent challenges: complex entangled scenes, open-domain robustness, and faithful evaluation. Real-world audio interweaves speech, music, effects, ambience, and reverberation in a single waveform; models remain brittle under unseen genres, heavy noise, long contexts, or overlapping sources; and no single benchmark yet covers the full space of speech, music, and general-audio edits with complete preservation annotations. This indicates that AudioEdit remains constrained as much by representation and evaluation as by raw generation quality (Pan et al., 22 Jun 2026).

System-specific failure modes reinforce that diagnosis. Bagpiper-Edit notes occasional failure on large semantic jumps and incomplete support for highly complex multi-speaker separation. SAO-Instruct identifies instruction-phrasing sensitivity, artifacts in speech reconstructions, unnatural placement of added sounds, and confusion in scenes with more than 2–3 elements. MMEdit reports difficulties with heavily overlapping events and very fine temporal shifts below 50 ms. AV-Edit notes failure when visual cues are ambiguous or when overlapping simultaneous sources have similar correlation scores. AudioEditor highlights limits on much longer audio and the need for manual marking of edit versus preserve tokens (Gong et al., 19 Jun 2026, Ungersböck et al., 26 Oct 2025, Tao et al., 23 Dec 2025, Guo et al., 26 Nov 2025, Jia et al., 2024).

A recurrent misconception is that lower-level distributional metrics are sufficient proxies for editing quality. AuditEval’s filtering experiment shows the opposite: semantic alignment and predicted human scores improve after filtering, while FAD and FD worsen. MMAE reaches a related conclusion from a different angle: average instruction-following and consistency scores can remain moderate even when exact task completion is effectively absent. Together, these results show that AudioEdit evaluation is intrinsically multi-objective and that “good generation” is not equivalent to “correct editing” (Jia et al., 16 Aug 2025, Ma et al., 5 Jun 2026).

Future directions in the literature are relatively consistent. The survey advocates object-aware and disentangled representations, structured control signals such as region or source masks, unified editing frameworks across acoustic, semantic, and source levels, robust instruction grounding with iterative refinement, and richer benchmarks. Concrete proposals include end-to-end alignment of instruction-to-caption rewriting with an audio-aware LLM, explicit cross-modal contrastive objectives, hierarchical diffusion schedules for sub-event resolution, more real-world triplets, multi-step and chained edits, multilingual instruction understanding, and universal generative audio intelligence spanning understanding, generation, and editing (Pan et al., 22 Jun 2026, Gong et al., 19 Jun 2026, Tao et al., 23 Dec 2025, Tian et al., 12 Apr 2026).