Bagpiper-Edit: Zero-Shot Audio Editing

• Bagpiper-Edit is a zero-shot audio editor that formulates editing as rich-caption rewriting, enabling precise semantic transformations.
• It reinterprets audio through detailed natural language captions and uses LLMs to modify and regenerate audio while preserving key acoustic elements.
• It leverages self-supervised acoustic anchoring and multi-turn training to balance semantic edits with consistency in speaker identity and background sound.

Searching arXiv for the specified Bagpiper-Edit paper and closely related Bagpiper work. Bagpiper-Edit is a zero-shot, open-ended audio editor that turns free-form text instructions into concrete edits on an existing audio clip—speech, music, or general sounds—without ever being trained on paired “before/after + instruction” editing data (Gong et al., 19 Jun 2026). It reformulates audio editing as a rich-caption rewriting task by treating a rich caption as the semantic representation of an audio clip, translating the user request into an edited caption, and then generating the target edited audio with the original audio as contextual acoustic anchor (Gong et al., 19 Jun 2026). The system is built on Bagpiper-Base, an 8B audio foundation model that interprets physical audio via rich captions and establishes a bidirectional mapping between raw audio and a high-level conceptual space (Tian et al., 5 Feb 2026).

1. Definition and problem formulation

The paper considers audio editing as

$a' \sim P_\theta(\cdot \mid a, u)$

where $a$ is the original audio, $u$ is a free-form user edit request in natural language, and $a'$ is the edited audio (Gong et al., 19 Jun 2026). The model must change specific aspects requested in $u$, keep everything else in $a$ intact, and maintain high perceptual realism (Gong et al., 19 Jun 2026).

“Open-ended” means no closed set of operation types, no template restrictions on instructions, and coverage of speech, music, and arbitrary soundscapes within a unified pipeline (Gong et al., 19 Jun 2026). This definition distinguishes Bagpiper-Edit from earlier text-guided audio editing systems that often define a small set of atomic operations and templates, or decompose flexible instructions into chains of template-based edits (Gong et al., 19 Jun 2026). The cited limitations are a heavy reliance on paired editing datasets, rigid operation templates, and modality-specific pipelines (Gong et al., 19 Jun 2026).

Within the broader Bagpiper framework, this formulation inherits the “caption-then-process” philosophy: represent audio in a conceptual space, reason over that representation, and then materialize audio from the modified description (Tian et al., 5 Feb 2026). This suggests that editing is treated not as direct waveform manipulation but as controlled transformation in a language-mediated semantic space.

2. Rich captions as semantic representation

Bagpiper-Edit builds on Bagpiper-Base, an autoregressive audio foundation model that tightly couples audio and rich captions through two directions: $a \rightarrow c$ for audio understanding and $c \rightarrow a$ for generation (Gong et al., 19 Jun 2026). A rich caption $c$ is a detailed natural-language description of the audio clip, including events, sources, speaker attributes, environment, background noises, style, timbre, emotion, and possibly multiple events in sequence (Gong et al., 19 Jun 2026).

Formally,

$c = f_\theta(a), \qquad a \sim g_\theta(c)$

and in Bagpiper-Base both mappings are realized within a decoder-only LLM that interleaves text tokens and audio tokens (Gong et al., 19 Jun 2026). Bagpiper describes rich captions as a universal conceptual representation of audio and a high-bandwidth intermediate representation for speech content, speaker attributes, prosody, sound events, acoustic scenes, music, mixtures, and meta context (Tian et al., 5 Feb 2026).

Bagpiper-Edit defines editing as caption rewriting: $a$0 where $a$1 is the user instruction and $a$2 is the edited caption (Gong et al., 19 Jun 2026). The original caption is treated as the canonical semantic representation of the source audio, the user request modifies that representation, and the resulting caption guides synthesis of the edited waveform (Gong et al., 19 Jun 2026). This reframing avoids explicit, handcrafted editing operators and converts the core problem into language understanding, language rewriting, and audio generation conditioned on both text and the original audio (Gong et al., 19 Jun 2026).

The same abstraction appears elsewhere in the Bagpiper family. Bagpiper-TTS first reasons over the user’s intent to derive a rich caption and then uses that caption to generate speech, positioning the rich caption as an internal canonical representation and a textual blueprint (Tian et al., 22 Jun 2026). A plausible implication is that Bagpiper-Edit generalizes this blueprint mechanism from unconditional generation to constrained transformation of existing audio.

3. Architecture and inference pipeline

At inference time, given original audio $a$3 and a free-form instruction $a$4, Bagpiper-Edit performs three stages (Gong et al., 19 Jun 2026).

First, it extracts a rich caption: $a$5 using audio-to-caption mode (Gong et al., 19 Jun 2026).

Second, it rewrites the caption with a strong text-only LLM, specifically Qwen3-235B-A22B-Instruct-2507-FP8, producing

$a$6

with prompting that retains parts of $a$7 that should stay unchanged and only modifies pieces according to $a$8 (Gong et al., 19 Jun 2026).

Third, it generates edited audio from $a$9, conditioned on the original audio $u$0: $u$1 where $u$2 specifies what should be in the edited audio and $u$3 serves as an acoustic anchor to preserve speaker identity, global timbre, and background acoustics (Gong et al., 19 Jun 2026).

The underlying backbone is Bagpiper-Base, described as a decoder-only LLM based on Qwen3-8B-Base that operates over text tokens and discrete audio tokens from multi-stream X-Codec at 50 Hz (Gong et al., 19 Jun 2026). Bagpiper presents the same backbone as a unified audio foundation model with an external continuous audio encoder, an MLP adaptor, and autoregressive prediction of multi-stream X-Codec tokens, with each audio frame encoded by 8 discrete codec tokens arranged with delay interleaving (Tian et al., 5 Feb 2026). Training in Bagpiper-Base includes text-only LM data, caption-to-audio generation, and audio-to-caption understanding (Gong et al., 19 Jun 2026).

Bagpiper-Edit does not change the basic architecture; instead, it changes the input-output format, training objectives via multi-turn packaging, and the conditioning pattern used to teach the model to respect previous audio while generating new audio (Gong et al., 19 Jun 2026). Inference uses the same decoding strategy as Bagpiper-Base, including classifier-free guidance style sampling and gumbel-top-k (Gong et al., 19 Jun 2026). Bagpiper reports audio generation with CFG scale 3 and sampling parameters of text temperature 0.6, top-k 20, and audio temperature 0.8, top-k 20 (Tian et al., 5 Feb 2026).

4. Self-supervised acoustic anchoring

The central technical problem identified by the paper is that the base model, when asked to generate conditioned on both caption and audio, tends to ignore the original audio and just regenerate from text, causing style and identity drift (Gong et al., 19 Jun 2026). To address this without paired editing data, Bagpiper-Edit introduces a self-supervised training paradigm based on acoustic anchoring (Gong et al., 19 Jun 2026).

Training data are constructed from raw long audio with captions by forming two audio segments $u$4 and corresponding captions $u$5 (Gong et al., 19 Jun 2026). Two strategies are used.

In audio repetition,

$u$6

and the task is essentially to regenerate the same audio twice, teaching that when semantic description stays the same, acoustic identity and timbre should stay the same (Gong et al., 19 Jun 2026).

In audio segmentation, a continuous recording is segmented into two adjacent clips, $u$7 and $u$8, with separate captions $u$9 and $a'$0 (Gong et al., 19 Jun 2026). Because the clips are contiguous in time, they share the same speaker, background environment, and room acoustics, and the model is trained to maintain acoustic continuity when generating the second (Gong et al., 19 Jun 2026). The paper describes this as a self-supervised surrogate for “edit with consistent environment and identity” without ever seeing explicit edits (Gong et al., 19 Jun 2026).

The constructed data are packaged in two dialogue patterns (Gong et al., 19 Jun 2026). In the Single-Turn (ST) pattern, the user turn contains $a'$1 and the assistant turn contains $a'$2, so the model receives both semantic descriptions before generating both audio segments in a single long sequence (Gong et al., 19 Jun 2026). In the Multi-Turn (MT) pattern, the sequence is explicitly divided into two turns: $a'$3 followed by

$a'$4

so that when generating $a'$5, the model has already generated $a'$6 and seen it in context (Gong et al., 19 Jun 2026). This encourages in-context audio-to-audio conditioning and teaches the model to generate $a'$7 consistent with the same environment and identity while semantically matching $a'$8 (Gong et al., 19 Jun 2026).

Training uses 500k samples constructed from YODAS, LAION-Audio, Emilia-En, AudioSet, WavCaps, and AudioCaps, with no explicit editing triplets, global batch 128k tokens, and learning rate $a'$9 (Gong et al., 19 Jun 2026). The resulting variants are Bagpiper-Edit (ST) and Bagpiper-Edit (MT) (Gong et al., 19 Jun 2026).

5. Editing operations and empirical performance

Because editing is expressed in natural language through $u$0, the system is described as inherently open-ended (Gong et al., 19 Jun 2026). The reported operation types include speech transcription editing, full sentence replacement, speech style and emotion editing, sound event insertion, sound event removal, and free-form compositional edits that combine speech, sound, and music (Gong et al., 19 Jun 2026). Examples include “Add a dog bark in the background,” “Remove the car horn,” and “Turn this podcast clip into a phone call with slight static noise, same speaker, same words” (Gong et al., 19 Jun 2026).

Evaluation spans speech editing, audio-event editing, and free-form rich-caption editing (Gong et al., 19 Jun 2026). On LibriSpeech test-clean, Bagpiper-Base exhibits very high WER at 72.19%, low SpkSIM at 0.58, poor DNSMOS at 2.23, and low LLM scores, indicating severe style and identity drift and poor editing reliability (Gong et al., 19 Jun 2026). Bagpiper-Edit (ST) reaches SpkSIM 0.86 but has transcription edit accuracy 47.11% and WER 19.62%, which the authors attribute to over-anchoring to the original audio (Gong et al., 19 Jun 2026). Bagpiper-Edit (MT) provides the best tradeoff, with WER 14.01%, edit accuracy 79.76%, SpkSIM 0.83, and DNSMOS 3.15 (Gong et al., 19 Jun 2026).

For emotion editing, both ST and MT achieve emotion classification accuracy comparable to baselines while significantly outperforming them in speaker similarity (Gong et al., 19 Jun 2026). For style editing, Bagpiper-Edit lags specialized speech models, which the paper attributes to the limitation of the underlying Bagpiper-Base’s expressiveness for nuanced speaking styles (Gong et al., 19 Jun 2026).

On AudioSet-based audio-event editing, metrics include FAD, conCLAP, editCLAP, and LLM-based quality rating (Gong et al., 19 Jun 2026). For addition, Bagpiper-Edit (ST) yields FAD 3.26 and conCLAP 0.74 but editCLAP 0.08, showing strong consistency and weak insertion capability; Bagpiper-Edit (MT) yields FAD 3.29, conCLAP 0.51, and editCLAP 0.18, with the highest editCLAP and best LLM overall score (Gong et al., 19 Jun 2026). For removal, Bagpiper-Edit (MT) records FAD 4.35, conCLAP 0.52, and editCLAP 0.07, together with the best LLM overall score (Gong et al., 19 Jun 2026).

For free-form rich-caption editing, Bagpiper-Base shows FAD 7.62, CapSIM 0.4636, and low LLM scores; Bagpiper-Edit (ST) shows FAD 0.91 and CapSIM 0.5355; Bagpiper-Edit (MT) shows FAD 2.85, CapSIM 0.5961, Qwen3 2.75, and Gemini 3.95 (Gong et al., 19 Jun 2026). The reported pattern is consistent across tasks: ST over-anchors acoustics and under-edits semantics, while MT provides a better balance for open-ended editing (Gong et al., 19 Jun 2026).

The paper’s summary claim is that evaluations across speech, audio, and free-form editing show Bagpiper-Edit maintains good consistency to the original audio and achieves similar performance to other expert models in most cases (Gong et al., 19 Jun 2026). This suggests that the caption-rewriting formulation is competitive even without paired editing supervision.

6. Relation to Bagpiper and to prior editing systems

Bagpiper-Edit is explicitly rooted in Bagpiper, which is described as an 8B audio foundation model trained on a massive corpus of 600B tokens and 422M audio–rich-caption pairs, with unified understanding and generation for general audio (Tian et al., 5 Feb 2026). Bagpiper adopts a caption-then-process workflow during fine-tuning, simulating an intermediate cognitive reasoning step to solve diverse tasks without task-specific priors (Tian et al., 5 Feb 2026). For editing, Bagpiper’s details already describe the conceptual pattern: interpret existing audio into a rich caption, modify the caption according to an instruction, and generate audio from the modified conceptual description (Tian et al., 5 Feb 2026).

Against prior editing methods, the conceptual differences emphasized by Bagpiper-Edit are threefold (Gong et al., 19 Jun 2026). First, it uses no fixed editing operators, whereas prior works such as AUDIT, Prompt-Guided-Edit, ZETA, and Step-Audio-EditX define atomic operations and train specific modules or conditionings for each (Gong et al., 19 Jun 2026). Second, it uses no paired edit data, whereas many systems train on synthetic or curated $u$1 pairs (Gong et al., 19 Jun 2026). Third, it is unified across modalities, in contrast to speech editors such as VoiceCraft, CosyVoice-3, and Ming-UniAudio-Edit, or diffusion-based audio editors such as AudioLDM2 and AudioEditor that often focus on music or generic audio rather than speech identity preservation (Gong et al., 19 Jun 2026).

The paper further distinguishes Bagpiper-Edit as text-space editing rather than signal-space editing (Gong et al., 19 Jun 2026). Diffusion and inversion-based methods manipulate latent or waveform representations directly, whereas Bagpiper-Edit works purely in language space first and then re-materializes audio (Gong et al., 19 Jun 2026). Its stated technical novelty is the formulation of audio editing as rich-caption rewriting plus acoustic anchoring, self-supervised dialogue-based training to teach audio-to-audio consistency without edited pairs, and a demonstration that such a zero-shot setup can approach domain-specific models on many metrics (Gong et al., 19 Jun 2026).

Bagpiper-TTS provides a closely related speech-synthesis instance of the same design pattern. It accepts free-form natural language prompts, reasons over user intent to produce a rich caption, and uses that caption to generate speech (Tian et al., 22 Jun 2026). Within that family resemblance, Bagpiper-Edit can be understood as applying the same rich-caption mediation to editing rather than pure synthesis.

7. Limitations, implications, and future directions

The paper identifies several limitations and failure modes (Gong et al., 19 Jun 2026). For style editing of speech, Bagpiper-Edit remains behind specialized models, reflecting limitations of Bagpiper-Base’s style modeling (Gong et al., 19 Jun 2026). For complex acoustic scenes, such as multi-speaker separation, performance is constrained by base model capacity (Gong et al., 19 Jun 2026). The WER histogram for full-sentence replacement shows a tail of extreme failures, especially for full-sentence replacements, because the target caption $u$2 is purely LLM-generated without real audio supervision, so errors there propagate (Gong et al., 19 Jun 2026).

Bagpiper more generally notes a limitation on descriptive bias and inference latency: explicitly generating rich captions and thinking traces adds computational overhead and increases inference latency versus direct TTS or TTA models (Tian et al., 5 Feb 2026). It also notes that the model struggles to simultaneously satisfy multiple conflicting or complex constraints within a single instruction (Tian et al., 5 Feb 2026). These constraints are directly relevant to editing, particularly for long, highly compositional requests.

Future directions mentioned in Bagpiper-Edit are scaling the base model to improve understanding of complex acoustic scenes and fine-grained temporal control, and end-to-end alignment strategies that better connect user instructions to caption rewriting and audio generation, possibly integrating the rich-caption rewriting into a single multimodal model (Gong et al., 19 Jun 2026). Bagpiper also points toward richer caption schemas with explicit time tags or layers and more robust compositional reasoning over conflicting constraints (Tian et al., 5 Feb 2026).

The broader impact discussion is implicit but clear: like any high-quality editing tool, Bagpiper-Edit raises concerns about voice mimicry, deepfakes, and deceptive manipulation, together with a need for watermarking, consent, and usage controls (Gong et al., 19 Jun 2026). A plausible implication is that the same generality that enables unified open-ended editing across speech, music, and environmental sound also heightens the importance of governance mechanisms.

In the Bagpiper lineage, Bagpiper-Edit marks the point at which the rich-caption paradigm becomes an explicit editing framework rather than only an understanding and generation framework. Its distinctive claim is that zero-shot editing can be induced from generic audio-caption data by combining caption rewriting with self-supervised acoustic anchoring, producing a single model that operates across speech, music, and sound events without modality-specific editing pipelines (Gong et al., 19 Jun 2026).