Disentangled Audio Infilling

• Disentangled audio infilling is a generative approach that reconstructs masked audio segments by separating conditioning factors such as speaker timbre, environmental noise, and musical style.
• The framework employs diverse techniques including flow matching, structured conditioning, and token-based prediction to enhance controllability and synthesis quality.
• Empirical evidence across TTS, singing conversion, symbolic music infilling, and bandwidth extension demonstrates improved performance in both objective metrics and perceptual quality.

Disentangled audio infilling is a family of generative formulations in which a model reconstructs masked, missing, or underspecified audio content while separating the conditioning factors that govern the reconstruction. In the most explicit usage of the term, DAIEN-TTS formulates environment-aware zero-shot text-to-speech as the masked reconstruction of an environmental speech mel using text and disentangled prompt conditions for speaker timbre and background environment (Lu et al., 18 Sep 2025). Closely related formulations appear in singing style conversion, where a masked segment of a target mel-spectrogram is predicted from complementary mel context and disentangled acoustic features (Violeta et al., 16 Mar 2025); in symbolic music infilling with scope-separated control tokens (Guo et al., 2022); and in bandwidth-extension systems that treat missing high-frequency content as a form of token or band infilling under harmonic–percussive or multi-band disentanglement (Giniès et al., 26 Nov 2025, Ginies et al., 4 Oct 2025). Across these variants, the central premise is that infilling quality and controllability improve when speaker identity, environment, style, melody, texture, or frequency-band structure are represented separately rather than being left entangled in a single prompt or latent stream.

1. Definition and task scope

Disentangled audio infilling covers several task definitions rather than a single canonical problem. In DAIEN-TTS, the inputs are a text sequence $T$ split into a reference transcript $z_{\text{ref}}$ and generation transcript $z_{\text{gen}}$, a speaker prompt $A_{\text{spk}}$, and an environment prompt $A_{\text{env}}$; the output is a waveform $y$ whose speaker timbre and background environment can be controlled independently by $A_{\text{spk}}$ and $A_{\text{env}}$ (Lu et al., 18 Sep 2025). The model seeks to synthesize $y$ with mel-spectrogram $x_1$ that completes masked regions of the target environmental speech mel by conditioning on text $z_{\text{ref}}$0 and the unmasked spans of $z_{\text{ref}}$1 and $z_{\text{ref}}$2, while maintaining timbre identity from $z_{\text{ref}}$3, environment characteristics and SNR from $z_{\text{ref}}$4, and accurate linguistic content from $z_{\text{ref}}$5.

In Serenade, the task is singing style conversion rather than TTS. Given a source singing waveform whose melody should be retained and a reference singing waveform whose style should be copied, the goal is to generate a converted waveform that preserves the source melody and content while expressing the target style (Violeta et al., 16 Mar 2025). The core acoustic domain is again the mel-spectrogram, but the conditioning factors are disentangled acoustic features rather than text plus prompts: ContentVec linguistic features, a mel prior from a prior encoder, MIDI-based melody features, loudness, and a global style encoder.

A broader interpretation of the term includes tasks in which the “missing content” is not a masked time span in a mel-spectrogram. MusIAC treats infilling as reconstructing missing musical material in symbolic multi-track MIDI, with user-specified controls for texture and tonal tension (Guo et al., 2022). The soft-disentanglement codec treats high-frequency restoration as band-limited infilling: the model receives a 16 kHz version of the signal and its upsampled copy as inputs, with no high-band information provided to the 32 kHz branch, which must infer and fill the high-frequency band using the low-frequency content as a prior (Ginies et al., 4 Oct 2025). HP-codecX similarly recasts bandwidth extension as conditional audio token prediction, that is, infilling of missing high-frequency tokens given low-frequency tokens under harmonic, percussive, and residual decomposition (Giniès et al., 26 Nov 2025).

This diversity suggests that “infilling” in this literature is not restricted to literal time-gap completion. It includes masked-span mel reconstruction, continuation of environmental speech, symbolic track/bar reconstruction, and conditional prediction of absent spectral or codec-token content.

2. Generative formulations and masking regimes

A unifying feature of these systems is masked or conditional reconstruction, but the operational details differ substantially across domains.

DAIEN-TTS builds on F5-TTS, which frames zero-shot TTS as text-guided speech infilling under conditional flow matching and uses a Diffusion Transformer backbone (Lu et al., 18 Sep 2025). Two random span masks of varying lengths are sampled independently for the speech and environment mel-spectrograms. The conditioning signals are the visible parts,

$z_{\text{ref}}$6

and the model predicts the masked region of the environmental speech mel $z_{\text{ref}}$7 under the speech mask $z_{\text{ref}}$8. Its conditional flow-matching objective is

$z_{\text{ref}}$9

At inference, the model solves the ODE $z_{\text{gen}}$0, discards the reference portion, and vocodes the generated mel to waveform.

Serenade also uses flow matching, but with a distinct masking scheme and conditioning topology. The paper defines target mel-spectrogram features $z_{\text{gen}}$1 and creates a mask $z_{\text{gen}}$2 where a segment from $z_{\text{gen}}$3 to $z_{\text{gen}}$4 is set to zeros (Violeta et al., 16 Mar 2025). The complement mask $z_{\text{gen}}$5 is applied to form the conditioning feature $z_{\text{gen}}$6, and the model predicts the masked segment $z_{\text{gen}}$7. The decoder is a 1D UNet based on the 1D UNet architecture proposed in Matcha-TTS, modified to improve style modeling by conditioning the style embeddings after every residual block and increasing attention head dimension to 256 and channels to 512. The generative objective is conditional flow matching with optimal transport, and inference uses the first-order Euler forward ODE-solver.

MusIAC uses a different masked-sequence regime. Pretraining masks spans up to three tokens, with up to 15% of encoder tokens masked, and optimizes

$z_{\text{gen}}$8

in a MASS-style masked seq2seq setup (Guo et al., 2022). Finetuning then switches to larger, musically meaningful masks: all tracks in a randomly selected bar, all bars in a randomly selected track, or randomly selected tracks in randomly selected bars.

In bandwidth extension, masking is implicit rather than explicit. HP-codecX predicts the missing high-frequency token streams $z_{\text{gen}}$9, $A_{\text{spk}}$0, and $A_{\text{spk}}$1 from low-frequency tokens, formalized as

$A_{\text{spk}}$2

with cross-entropy training over autoregressive transformers (Giniès et al., 26 Nov 2025). The soft frequency-disentanglement codec uses a residual cascade in which the 32 kHz branch receives no genuine $A_{\text{spk}}$3 kHz content and must infer plausible high-frequency structure from low-band cues (Ginies et al., 4 Oct 2025).

Taken together, these formulations show that disentangled audio infilling is less a single architecture than a design pattern: missing content is reconstructed under structured conditions, and the masking or absence mechanism can be temporal, spectral, symbolic, or token-based.

3. Mechanisms of disentanglement and control

The defining property of the area is not infilling alone, but infilling under factor separation.

In DAIEN-TTS, disentanglement begins with a pretrained speech–environment separation module. Let environmental speech be $A_{\text{spk}}$4, where $A_{\text{spk}}$5 is speech and $A_{\text{spk}}$6 is background environment. A Transformer-based masking net predicts speech and environment masks from the magnitude spectrogram $A_{\text{spk}}$7, producing separated magnitudes and then mel-spectrograms $A_{\text{spk}}$8 and $A_{\text{spk}}$9 (Lu et al., 18 Sep 2025). Speaker mel and text are concatenated in the input pathway, while environment is injected through per-block multi-head cross-attention. During inference, DAIEN-TTS applies Dual Class-Free Guidance separately to speech and environment branches:

$A_{\text{env}}$0

The paper uses $A_{\text{env}}$1. It also introduces SNR adaptation, scaling the environment condition derived from $A_{\text{env}}$2 so the output speech–environment balance matches the SNR of $A_{\text{env}}$3.

Serenade disentangles style through structured acoustic conditioning and cyclic training. Its conditioning features are not masked and include ContentVec, MIDI, loudness, GST, and a mel prior (Violeta et al., 16 Mar 2025). The cyclic training procedure fine-tunes the model on synthetic converted samples while reconstructing the original source mel-spectrogram as both masked conditioning feature and target feature. This forces the network to reconstruct the original source mel from conditioning features that come from a converted sample with target style semantics, thereby making the network more robust to disentangling source style factors. Melody retention is further isolated through SiFiGAN post-processing, which resynthesizes converted waveforms using the original F0 patterns after WORLD analysis-synthesis.

MusIAC separates control factors by scope rather than by explicit latent penalties. Track-level controls are note density rate, polyphony rate, and occupation rate; bar-level controls are tensile strain and cloud diameter (Guo et al., 2022). These tokens are inserted at dedicated positions in the sequence, with global header positions for track controls and per-bar positions for tonal tension controls. The paper states that this clear structural separation helps keep controls separable and interpretable.

The codec-based systems implement disentanglement in the representation itself. HP-codec divides both low-frequency and high-frequency branches into harmonic, percussive, and residual RVQ sections, with HPSS-informed training used to specialize the sections (Giniès et al., 26 Nov 2025). The soft frequency-disentanglement codec uses a 16 kHz branch and a residual 32 kHz branch:

$A_{\text{env}}$4

so that the higher-rate branch is encouraged to focus on the residual high-frequency structure while remaining free to refine lower bands and correct boundary artifacts (Ginies et al., 4 Oct 2025).

A common misconception is that disentanglement in audio generation necessarily requires explicit adversarial, contrastive, or mutual-information objectives. These papers show several alternatives: separation modules, scope-specific tokens, residual branch design, HPSS-guided specialization, and cyclic reconstruction can all function as disentanglement mechanisms without introducing a dedicated mutual-information penalty or adversarial disentanglement loss.

4. Representative systems and empirical evidence

The literature spans speech synthesis, singing style conversion, symbolic music infilling, and bandwidth extension.

System	Domain	Disentangled factors
DAIEN-TTS (Lu et al., 18 Sep 2025)	Zero-shot, environment-aware TTS	Speaker timbre, background environment, linguistic content
Serenade (Violeta et al., 16 Mar 2025)	Singing style conversion	Style, melody, content, loudness
MusIAC (Guo et al., 2022)	Symbolic multi-track music infilling	Track texture controls, bar-level tonal tension
Soft multi-branch codec (Ginies et al., 4 Oct 2025)	High-band audio infilling	Low-band base signal, residual high-band content
HP-codecX (Giniès et al., 26 Nov 2025)	Bandwidth extension	Harmonic, percussive, residual token streams

DAIEN-TTS reports results on SeedTTS test-en for both clean-speech synthesis and environmental speech synthesis (Lu et al., 18 Sep 2025). With a silence environment prompt, DAIEN-TTS reaches WER 1.93, SIM-o 0.60, MOS $A_{\text{env}}$5, and SSMOS $A_{\text{env}}$6, compared with F5-TTS (env spk prompt) at WER 2.87, SIM-o 0.49, MOS $A_{\text{env}}$7, and SSMOS $A_{\text{env}}$8. With a background environment prompt, DAIEN-TTS reaches WER 2.83, SIM-o 0.55, MOS $A_{\text{env}}$9, SSMOS $y$0, and ESMOS $y$1. The variant without cross-attention is worse on MOS and ESMOS, and the paper identifies SES as crucial for intelligibility and similarity under environmental prompts.

Serenade evaluates naturalness with 5-point MOS and style similarity with a 4-scale XAB test (Violeta et al., 16 Mar 2025). Serenade without SiFiGAN reports N-MOS $y$2 and SIM $y$3; Serenade SiFiGAN reports N-MOS $y$4 and SIM $y$5; NU-SVC reports N-MOS $y$6 and SIM $y$7. The paper attributes Serenade’s similarity advantage especially to breathy and mixed styles, while also observing that SiFiGAN substantially improves naturalness by alleviating out-of-tune singing.

MusIAC’s evidence is objective rather than perceptual. Adding control tokens lowers differences between generated and original infilled segments across pitch- and rhythm-related metrics (Guo et al., 2022). For melody-track infilling, note number difference drops from 0.82 to 0.29, pitch range from 0.59 to 0.49, duration histogram from 0.55 to 0.42, and onset interval histogram from 0.45 to 0.35. For accompaniment-track infilling, pitch number drops from 0.57 to 0.41 and duration histogram from 1.03 to 0.74. These results support the claim that scope-specific controls increase stylistic similarity to the original infilled material.

The codec-based infilling papers quantify gains in both objective and subjective terms. In the soft frequency-disentanglement codec, 32 kHz reconstruction improves over DAC from mel loss 0.90 to 0.80, STFT loss 2.28 to 2.14, waveform loss 0.060 to 0.050, SI-SDR 5.00 dB to 6.05 dB, and ViSQOL 3.97 to 4.22; MUSHRA rises from $y$8 to $y$9 (Ginies et al., 4 Oct 2025). In its high-band infilling toy experiment, $A_{\text{spk}}$0 reaches mel 2.04, STFT 5.37, waveform 0.053, SI-SDR 6.22 dB, and ViSQOL 2.06, improving over the upsampled 16 kHz baseline. HP-codecX reports, in the 8–24 kHz band, Mel $A_{\text{spk}}$1, STFT $A_{\text{spk}}$2, and ViSQOL $A_{\text{spk}}$3, outperforming Apollo and AudioSR; in MUSHRA it reaches $A_{\text{spk}}$4 versus $A_{\text{spk}}$5 for AudioSR and $A_{\text{spk}}$6 for Apollo (Giniès et al., 26 Nov 2025).

5. Comparative positioning and conceptual boundaries

Disentangled audio infilling is distinct from simple post-mixing or unconstrained conditional generation. DAIEN-TTS explicitly contrasts its approach with post-mix methods, which do not condition the TTS model on environment structure and therefore lead to poorer speech–environment alignment and limited control over time-varying characteristics (Lu et al., 18 Sep 2025). It also distinguishes itself from joint modeling without separation, such as F5-TTS conditioned on environmental prompts, which preserves background unintentionally and entangles it with timbre, reducing speaker similarity and intelligibility under noisy prompts.

Serenade positions infilling against direct full-mel prediction. NU-SVC directly predicts the full target mel without infilling, whereas Serenade models the target style through an audio infilling task and then strengthens disentanglement via cyclic fine-tuning (Violeta et al., 16 Mar 2025). The comparative results suggest that this is especially beneficial for breathy and mixed singing styles, where the paper states that NU-SVC has difficulties in modeling breath features.

MusIAC clarifies another conceptual boundary: controllable infilling need not rely on latent-variable disentanglement if controls are typed, scoped, and inserted at structurally meaningful locations (Guo et al., 2022). Its track-level and bar-level token placement provides disentangled modulation by design, though the paper does not report mutual-information or adversarial disentanglement losses.

The codec literature broadens the term still further. In HP-codecX, infilling is conditional prediction of missing high-frequency codec tokens from low-frequency tokens, not reconstruction of a visible temporal gap (Giniès et al., 26 Nov 2025). In the soft multi-branch codec, the high-band branch can infer plausible high-frequency content from low-band cues without specialized inpainting training (Ginies et al., 4 Oct 2025). A plausible implication is that the term “disentangled audio infilling” has become more architectural than task-specific: it names a strategy in which structured representations lower modeling entropy and increase controllability when some part of the signal must be generated rather than copied.

6. Limitations, failure modes, and future directions

The reported limitations are domain-specific, but several recur across the literature.

DAIEN-TTS notes that highly nonstationary or rapidly changing environments may still challenge precise reconstruction; long-range coherence of complex backgrounds can degrade, particularly when prompt segments are short (Lu et al., 18 Sep 2025). Prompt mismatch, such as a speaker prompt with severe noise or reverberation not captured by SES, can reduce speaker similarity and intelligibility. Artifacts may arise in boundary regions where the model transitions from prompt-conditioned spans to generated spans, or when environment SNR scaling is extreme. The SES loss is not detailed, and the paper states that suboptimal separation quality would directly impact conditioning fidelity.

Serenade identifies a persistent tradeoff between melody preservation and style similarity (Violeta et al., 16 Mar 2025). Fixing F0 to the source improves naturalness but can reduce perceived style similarity for styles in which F0 patterns are style-defining, including falsetto and pharyngeal styles. The paper also notes unspecified training details, such as masking coverage ratios, batch sizes, and learning rates.

MusIAC is limited to symbolic MIDI, up to three tracks, and 16 bars per sample (Guo et al., 2022). It does not include explicit disentanglement objectives, and the authors note that unseen or implicit musical attributes such as idiomatic phrasing and groove are harder to control. Human listening studies are identified as important future work.

HP-codecX requires a coupled two-model pipeline, fixed input/output sampling rates in the main model, and can invent high-frequency detail when content is absent in the high-frequency band (Giniès et al., 26 Nov 2025). The soft multi-branch codec notes that complex transients and percussive onsets may be under-represented without explicit transient modeling, and that no explicit disentanglement regularizers are used (Ginies et al., 4 Oct 2025).

Across these systems, a plausible implication is that future work will continue to move in two directions simultaneously. One direction is stronger factor separation: better separation modules, more explicit loss formulations for disentanglement, dynamic band allocation, or additional semantic partitions. The other is broader infilling scope: longer-range temporal dependencies, improved objective metrics validated across many systems, and hybrid designs that combine disentangled codecs with sequence models or diffusion-style generators. The common technical challenge is to preserve controllability without sacrificing coherence at the boundaries between observed and generated content.