DAIEN-TTS: Environment-Aware Zero-Shot TTS

• The paper introduces a novel disentangled audio infilling formulation that separately controls speaker timbre and background environment via dual prompts.
• It leverages a pretrained SES module and a conditional flow-matching DiT framework to reconstruct masked mel-spectrogram regions in environmental sounds.
• Inference-time dual classifier-free guidance and SNR adaptation enable robust zero-shot control over both speech quality and environmental fidelity.

Searching arXiv for DAIEN-TTS and its backbone paper to ground the article in current papers. arxiv_search(query="DAIEN-TTS Disentangled Audio Infilling for Environment-Aware Text-to-Speech Synthesis", max_results=5) {"query":"DAIEN-TTS Disentangled Audio Infilling for Environment-Aware Text-to-Speech Synthesis","max_results":5} DAIEN-TTS is a zero-shot, environment-aware text-to-speech framework that enables independent control over speaker identity and background acoustic environment by using separate speaker and environment prompts. It is formulated as “Disentangled Audio Infilling” on top of F5-TTS: a pretrained speech-environment separation module first decomposes environmental speech into clean-speech and environment streams, and a conditional flow-matching model then infills masked regions of environmental speech in the mel-spectrogram domain while conditioning on text, speaker context, and environment context. The design is intended to synthesize speech with the prompt speaker’s timbre and the prompt environment’s time-varying background audio, while retaining inference-time control through dual classifier-free guidance and signal-to-noise-ratio adaptation (Lu et al., 18 Sep 2025).

1. Concept and problem formulation

DAIEN-TTS defines environment-aware TTS as the use of separate speaker and environment prompts to independently control synthesized speech timbre and acoustic background. Its central motivation is that ordinary prompt-conditioned zero-shot TTS entangles these factors inside a single prompt waveform: an environmental speaker prompt contains both speaker identity and background noise or reverberation, so a model may either preserve unwanted environmental contamination or fail to preserve the speaker under noisy conditions (Lu et al., 18 Sep 2025).

In the paper’s zero-shot setting, the target speaker is unseen during training, and inference uses only short reference prompts. The speaker prompt controls speaker-related characteristics, especially timbre, while the environment prompt controls the background environment, including time-varying environmental audio and its signal-to-noise characteristics. This factorization is the defining distinction of DAIEN-TTS relative to standard prompt-based zero-shot TTS, which can inherit the acoustic environment of the reference but does not expose speaker and environment as independently controllable variables (Lu et al., 18 Sep 2025).

The key reformulation is to cast environment-aware TTS as a disentangled audio infilling problem. Rather than treating the prompt as a monolithic condition, DAIEN-TTS separates environmental speech into speech and environment streams, masks portions of both, and trains a conditional generator to reconstruct the missing target region of the original environmental speech. This suggests that the model is not merely transferring a global style embedding; it is learning to continue both speech content and the time-varying acoustic scene in a synchronized manner.

2. Disentanglement and masked infilling objective

The foundational module is a pretrained speech-environment separation (SES) model. Given an environmental speech waveform $\boldsymbol{y} \in \mathbb{R}^T$, DAIEN-TTS converts it into a magnitude spectrogram $|\boldsymbol{Y}| \in \mathbb{R}^{F \times L}$ using STFT. A Transformer-based masking network predicts two masks, $|\boldsymbol{M}^{S}|$ and $|\boldsymbol{M}^{E}| \in \mathbb{R}^{F \times L}$, which are applied to produce separated spectrograms:

$$\vert \boldsymbol{Y}^{S} \vert = \vert \boldsymbol{Y} \vert \odot \vert \boldsymbol{M}^{E} \vert \in \mathbb{R}^{F \times L}$$

$$\vert \boldsymbol{Y}^{E} \vert = \vert \boldsymbol{Y} \vert \odot \vert \boldsymbol{M}^{S} \vert \in \mathbb{R}^{F \times L}.$$

The superscript notation and verbal description are slightly awkward in the manuscript, but the intended effect is explicit: one branch extracts the speech component and the other extracts the background environment component. After mel filtering, the system obtains a clean-speech mel-spectrogram $\boldsymbol{c}_{spk} \in \mathbb{R}^{M \times L}$ and an environment mel-spectrogram $\boldsymbol{c}_{env} \in \mathbb{R}^{M \times L}$, which become the disentangled conditions of the TTS model (Lu et al., 18 Sep 2025).

Training uses two independently sampled binary random span masks, $\boldsymbol{m}_{spk}$ and $\boldsymbol{m}_{env}$, with varying lengths. The visible conditions are

$|\boldsymbol{Y}| \in \mathbb{R}^{F \times L}$0

while the reconstruction target is the masked environmental mel-spectrogram of the original mixed speech. This dual masking strategy converts disentanglement into a conditional infilling problem: the model must continue personalized speech and background environment simultaneously, conditioned on partial speaker context, partial environment context, and text (Lu et al., 18 Sep 2025).

The generative formulation follows conditional flow matching. Let the original environmental speech mel-spectrogram be $|\boldsymbol{Y}| \in \mathbb{R}^{F \times L}$1, let $|\boldsymbol{Y}| \in \mathbb{R}^{F \times L}$2 be Gaussian noise, and let $|\boldsymbol{Y}| \in \mathbb{R}^{F \times L}$3 be a sampled flow time. The noisy interpolation is

$|\boldsymbol{Y}| \in \mathbb{R}^{F \times L}$4

Text is converted into an extended text sequence $|\boldsymbol{Y}| \in \mathbb{R}^{F \times L}$5 aligned to mel length as in F5-TTS. The model predicts a velocity field conditioned on $|\boldsymbol{Y}| \in \mathbb{R}^{F \times L}$6, $|\boldsymbol{Y}| \in \mathbb{R}^{F \times L}$7, and the partially observed speech and environment conditions. The manuscript’s loss formula is imperfectly typeset, but its operational meaning is clear: the model is trained to predict the flow-matching target $|\boldsymbol{Y}| \in \mathbb{R}^{F \times L}$8 only over the masked region selected by $|\boldsymbol{Y}| \in \mathbb{R}^{F \times L}$9. In plain terms, DAIEN-TTS learns to infill missing environmental speech conditioned on text plus disentangled visible context (Lu et al., 18 Sep 2025).

3. Backbone architecture and conditioning design

DAIEN-TTS inherits its backbone from F5-TTS, described in the paper as a flow-matching-based speech infilling model using Diffusion Transformer (DiT) blocks. The model operates in the mel-spectrogram domain rather than predicting discrete speech tokens or direct waveform samples, and generation proceeds through a learned velocity field integrated by an ODE solver rather than a standard denoising diffusion objective (Lu et al., 18 Sep 2025).

Its main architectural additions relative to F5-TTS are threefold. First, it introduces the pretrained SES module, which provides the clean speech and environment streams. Second, it changes the environment conditioning pathway. The paper emphasizes that, unlike UmbraTTS, which concatenates all conditions to the DiT input, DAIEN-TTS injects the environment stream through a multi-head cross-attention layer inserted into each DiT block. Specifically, the unmasked environment mel-spectrogram $|\boldsymbol{M}^{S}|$0 is fed through cross-attention, while the unmasked speech mel-spectrogram and text embedding are concatenated and used as the standard conditioning signal. This dedicated routing is intended to better model time-varying environmental background as a distinct conditional stream (Lu et al., 18 Sep 2025).

Third, the paper adds two inference-time control mechanisms. The first is dual classifier-free guidance (DCFG), which separately amplifies speech-side and environment-side conditioning:

$|\boldsymbol{M}^{S}|$1

where $|\boldsymbol{M}^{S}|$2 denotes a null condition, and $|\boldsymbol{M}^{S}|$3 and $|\boldsymbol{M}^{S}|$4 are the speech and environment guidance strengths. In experiments, both were set to $|\boldsymbol{M}^{S}|$5. The paper presents DCFG as a controllability mechanism, although it does not provide a dedicated ablation table isolating its effect (Lu et al., 18 Sep 2025).

At inference, the model receives a speaker prompt speech $|\boldsymbol{M}^{S}|$6, an environment prompt speech $|\boldsymbol{M}^{S}|$7, the transcription of the speaker prompt $|\boldsymbol{M}^{S}|$8, and the target text $|\boldsymbol{M}^{S}|$9. The SES module extracts the speech component from $|\boldsymbol{M}^{E}| \in \mathbb{R}^{F \times L}$0 and the environment component from $|\boldsymbol{M}^{E}| \in \mathbb{R}^{F \times L}$1, the texts are concatenated into $|\boldsymbol{M}^{E}| \in \mathbb{R}^{F \times L}$2, and an ODE solver integrates

$|\boldsymbol{M}^{E}| \in \mathbb{R}^{F \times L}$3

After generation, the $|\boldsymbol{M}^{E}| \in \mathbb{R}^{F \times L}$4 portion is discarded and a vocoder reconstructs the final waveform. The paper does not specify the vocoder architecture beyond stating that a vocoder is used (Lu et al., 18 Sep 2025).

4. SNR adaptation, data construction, and training protocol

The second inference-time control mechanism is SNR adaptation, introduced to preserve not only environment type but also the relative loudness of environment and speech. The paper models environmental speech as

$|\boldsymbol{M}^{E}| \in \mathbb{R}^{F \times L}$5

with speech component $|\boldsymbol{M}^{E}| \in \mathbb{R}^{F \times L}$6 and environment component $|\boldsymbol{M}^{E}| \in \mathbb{R}^{F \times L}$7. Using Parseval’s theorem, the SNR is

$|\boldsymbol{M}^{E}| \in \mathbb{R}^{F \times L}$8

To align the synthesized mixture with the environment prompt, the environment magnitude spectrogram from the environment prompt is scaled so that, when paired with the speech magnitude from the speaker prompt, the resulting ratio matches the prompt SNR. The derived scaling factor is

$|\boldsymbol{M}^{E}| \in \mathbb{R}^{F \times L}$9

The scaled environment spectrogram is then converted to mel and used as the final environment condition. The intended effect is improved alignment between generated speech and desired environmental loudness (Lu et al., 18 Sep 2025).

Training uses LibriTTS for speech and DNS-Challenge for environmental sounds. The paper specifies 580 hours of LibriTTS and 68,000 environmental clips from DNS-Challenge, whose sources include AudioSet and FreeSound. Training mixtures use SNRs uniformly sampled from $$\vert \boldsymbol{Y}^{S} \vert = \vert \boldsymbol{Y} \vert \odot \vert \boldsymbol{M}^{E} \vert \in \mathbb{R}^{F \times L}$$0 dB to $$\vert \boldsymbol{Y}^{S} \vert = \vert \boldsymbol{Y} \vert \odot \vert \boldsymbol{M}^{E} \vert \in \mathbb{R}^{F \times L}$$1 dB. Evaluation uses the SeedTTS test-en set mixed with environment clips from SoundBible, with SNRs uniformly sampled from $$\vert \boldsymbol{Y}^{S} \vert = \vert \boldsymbol{Y} \vert \odot \vert \boldsymbol{M}^{E} \vert \in \mathbb{R}^{F \times L}$$2 dB to $$\vert \boldsymbol{Y}^{S} \vert = \vert \boldsymbol{Y} \vert \odot \vert \boldsymbol{M}^{E} \vert \in \mathbb{R}^{F \times L}$$3 dB. All audio is at 24 kHz (Lu et al., 18 Sep 2025).

A notable procedural detail is the dynamic mixing strategy. The SES module is pretrained on fully mixed speech-environment data. During TTS training, each speech sample is mixed with environmental audio with probability 50%, and with silent segments otherwise. The authors state that this is intended to improve text-speech alignment while still exposing the model to environmental variation. Both SES and TTS are trained for 600k steps with batch size corresponding to 102,800 audio frames, on 24 NVIDIA V100 32GB GPUs. The SES module uses 8 Transformer layers, 16 attention heads, embedding dimension 1024, and FFN dimension 2048. Each DiT block in the TTS module has an added cross-attention layer with 16 heads. The safest reading of the paper is that the SES module is pretrained separately rather than jointly optimized end to end with the TTS module (Lu et al., 18 Sep 2025).

5. Evaluation scenarios and empirical results

The evaluation is organized around two scenarios: a silence environment prompt, which tests whether the model can remove environment while preserving the speaker from a noisy prompt, and a background environment prompt, which tests explicit environment reconstruction and control. The baselines are F5-TTS (Clean Spk. Prompt), F5-TTS, DAIEN-TTS (w/o CA), and the full DAIEN-TTS. The metrics are WER using Whisper-large-v3, SIM-o using WavLM-large speaker verification embeddings, MOS, SSMOS, and ESMOS (Lu et al., 18 Sep 2025).

Scenario	Full DAIEN-TTS	Key comparison
Silence environment prompt	WER 1.93, SIM-o 0.60, MOS 3.84, SSMOS 3.64	F5-TTS: 2.87, 0.49, 3.09, 2.92
Background environment prompt	WER 2.83, SIM-o 0.55, MOS 3.78, SSMOS 3.73, ESMOS 3.65	DAIEN-TTS (w/o CA): 2.93, 0.54, 3.68, 3.70, 3.49

In the silence-environment setting, the reported values are: Human: WER 2.14, SIM-o 0.73, MOS 3.91, SSMOS 3.72; Vocoder: WER 2.18, SIM-o 0.70; F5-TTS (Clean Spk. Prompt): WER 2.30, SIM-o 0.58, MOS 3.80, SSMOS 3.60; F5-TTS: WER 2.87, SIM-o 0.49, MOS 3.09, SSMOS 2.92; DAIEN-TTS (w/o CA): WER 2.03, SIM-o 0.59, MOS 3.81, SSMOS 3.60; DAIEN-TTS: WER 1.93, SIM-o 0.60, MOS 3.84, SSMOS 3.64. These results support two claims made in the paper: SES-based disentanglement improves robustness to noisy prompts relative to standard F5-TTS, and the system can synthesize clean personalized speech from environmental prompts while suppressing unwanted environment (Lu et al., 18 Sep 2025).

In the background-environment setting, the results are: Human + Environment: WER 2.80, SIM-o 0.70, MOS 3.86, SSMOS 3.81, ESMOS 3.72; Vocoder: WER 3.03, SIM-o 0.65; DAIEN-TTS (w/o CA): WER 2.93, SIM-o 0.54, MOS 3.68, SSMOS 3.70, ESMOS 3.49; DAIEN-TTS: WER 2.83, SIM-o 0.55, MOS 3.78, SSMOS 3.73, ESMOS 3.65. The reported interpretation is that cross-attention is a better way to inject environment information than direct concatenation, especially when the task requires reconstruction of a real time-varying environment rather than merely suppressing it (Lu et al., 18 Sep 2025).

The cross-attention ablation is the main explicit architectural ablation. When the environment prompt is silent, cross-attention matters little, suggesting that it is not simply a generic quality booster. When actual environment reconstruction is required, however, it improves environmental fidelity and naturalness. By contrast, DCFG and SNR adaptation are described as important controllability mechanisms but are not isolated in separate quantitative ablation tables.

6. Position within TTS research, strengths, and limitations

The novelty of DAIEN-TTS is specific rather than generic. Relative to F5-TTS, it adds a pretrained SES module, disentangled speaker and environment conditioning, per-block environment cross-attention, and inference-time DCFG plus SNR adaptation. Relative to earlier environment-aware methods, the paper argues that prior approaches often used global embeddings, focused on time-invariant effects such as reverberation, or assumed separate clean and pure-environment prompts of equal length. DAIEN-TTS instead supports zero-shot speaker generalization, independent timbre and time-varying environment control, prompting with ordinary environmental speech, environment modeling through disentangled audio infilling, and inference-time control through DCFG and SNR adaptation (Lu et al., 18 Sep 2025).

Within the broader TTS literature represented in nearby work, other systems separate or specialize control along different axes. DSE-TTS separates linguistic speaking style from target timbre for cross-lingual synthesis (Liu et al., 2023). DEX-TTS separates time-invariant and time-variant style in reference-based expressive TTS (Park et al., 2024). IndexTTS emphasizes industrial zero-shot controllability and efficiency (Deng et al., 8 Feb 2025). DiaMoE-TTS targets dialect-aware synthesis through IPA-based input and Mixture-of-Experts (Chen et al., 25 Sep 2025). DAIEN-TTS differs by making the background acoustic environment itself an independently controllable prompt factor.

Its implied application space is any setting where explicit control over background acoustics matters, including audiobook postproduction, immersive media, AR/VR, or synthetic speech that must match a target scene. The reported experiments support the claim that the model can decouple timbre from background acoustics while maintaining intelligibility, naturalness, and environmental fidelity (Lu et al., 18 Sep 2025).

Several limitations and caveats are explicit or strongly implied. The method depends heavily on the quality of the SES module; poor separation will degrade downstream disentanglement and controllability. The paper does not provide a dedicated failure analysis. It does not report model parameter count, latency, inference speed, exact masking-length distribution, ODE solver choice, or exact vocoder configuration. It also does not report dedicated ablations for DCFG or SNR adaptation. A plausible implication is that the system is computationally nontrivial, since it requires large-scale training and flow-matching inference with an ODE solver. The authors also point to future work on the mutual influence between speech and background environment, which suggests that the current formulation still treats the two streams somewhat separately even though real recordings couple them acoustically (Lu et al., 18 Sep 2025).

Taken together, DAIEN-TTS is best understood as a disentangled extension of F5-TTS for environment-aware zero-shot TTS: it separates environmental speech into clean speech and environment streams, performs dual-stream masked mel infilling under a conditional flow-matching DiT framework, routes environment information through dedicated cross-attention, and exposes inference-time controls over speech fidelity, environment fidelity, and SNR. Its contribution is not merely robustness to noisy prompts, but explicit and independent control over who is speaking and what acoustic scene surrounds the speech (Lu et al., 18 Sep 2025).