UniFlow-Audio: Unified Flow Matching in Audio

• UniFlow-Audio is a universal audio generation framework that unifies time-aligned (TA) and non-time-aligned (NTA) tasks using a single non-autoregressive flow matching model.
• It employs a dual-fusion conditioning mechanism to integrate modality-specific embeddings with explicit temporal alignment for robust performance across diverse tasks such as TTS, SVS, and speech enhancement.
• The model leverages a waveform VAE and flow-matching Transformer backbone to achieve competitive performance while efficiently handling tasks across text, audio, and video modalities.

UniFlow-Audio is a universal audio generation framework based on flow matching that targets general audio generation across both time-aligned and non-time-aligned tasks within a single non-autoregressive model. It is designed to support omni-modalities, including text, audio, and video, and to unify tasks that have traditionally been modeled with different architectural assumptions, such as text-to-speech, singing voice synthesis, speech enhancement, audio super-resolution, video-to-audio, text-to-audio, and text-to-music (Xu et al., 29 Sep 2025).

1. Conceptual scope and task formulation

UniFlow-Audio organizes audio generation around the distinction between time-aligned (TA) and non-time-aligned (NTA) tasks. In TA tasks, the input sequence has explicit temporal correspondence with the output audio. The paper lists text-to-speech, singing voice synthesis, speech enhancement, audio super-resolution, and video-to-audio as TA tasks. In NTA tasks, the input acts as a global condition for the whole audio, and no explicit frame-level alignment is available or used; text-to-audio and text-to-music are the representative cases (Xu et al., 29 Sep 2025).

This distinction is used to explain why earlier systems developed along separate trajectories. TA tasks often rely on duration or alignment modules and frame-level conditioning, whereas NTA tasks typically rely on cross-attention from audio latents to global text or other embeddings. UniFlow-Audio treats this separation as a property of task formulation rather than of audio itself. Its stated objective is therefore a single non-autoregressive model that supports both TA and NTA tasks, supports multiple input modalities, and benefits from the generative modeling capabilities of flow matching (Xu et al., 29 Sep 2025).

A central implication is that UniFlow-Audio is positioned as a unified alternative to two major prior tendencies. Relative to autoregressive unifiers such as UniAudio, it avoids sequential decoding and discrete-token dependence. Relative to non-autoregressive models limited to NTA settings, it introduces explicit mechanisms for temporal alignment. This suggests that UniFlow-Audio is best understood as an attempt to turn heterogenous audio generation problems into instances of a single conditional latent flow model rather than as a task-specific architecture reused across domains (Xu et al., 29 Sep 2025).

2. Core architecture and latent generative model

At a high level, UniFlow-Audio consists of four components: a waveform VAE, content encoding with task instruction, a duration adapter, and a flow-matching Transformer backbone. The waveform VAE encodes raw 24 kHz waveforms into a short continuous latent sequence and reconstructs waveform from those latents. The VAE follows Stable Audio Open, uses a fully convolutional 1D encoder and decoder with residual blocks and Snake activations, and is trained on about 6000 hours of mixed speech, singing, music, and general audio to be universal across domains (Xu et al., 29 Sep 2025).

The generative model operates in this latent space with a rectified-flow-style formulation. Let $\mathbf{z}_0 \sim p_\text{data}$ denote a real audio latent and $\mathbf{z}_1 \sim \mathcal{N}(0, I)$ denote Gaussian noise. UniFlow-Audio uses the linear interpolation

$$\mathbf{z}_\tau = (1 - \tau)\,\mathbf{z}_0 + \tau\,\mathbf{z}_1, \quad \tau \in [0,1].$$

A neural network $$v_\theta(\mathbf{z}_\tau, \tau, \mathbf{C}^I, \mathbf{C}^I_T)$$ predicts a velocity field satisfying

$$\frac{d \mathbf{z}_\tau}{d\tau} = v_\theta(\mathbf{z}_\tau, \tau, \mathbf{C}^I, \mathbf{C}^I_T),$$

and training minimizes

$$\mathcal{L}_{\mathrm{FM}} = \mathbb{E}_{\tau, \mathbf{z}_0, \mathbf{z}_1} \big\| v_\theta(\mathbf{z}_\tau, \tau, \mathbf{C}^I, \mathbf{C}^I_T) - (\mathbf{z}_1 - \mathbf{z}_0) \big\|^2.$$

The paper’s interpretation is that the model is trained to predict the true derivative of the linear path between data and noise (Xu et al., 29 Sep 2025).

During inference, conditioning inputs are encoded, duration is predicted, a latent $\mathbf{z}_1 \sim \mathcal{N}(0, I)$ is initialized, and the probability flow ODE is integrated backward from $\tau=1$ to $\tau=0$ using 25 steps by default. The final latent is then decoded to waveform by the VAE decoder. UniFlow-Audio also uses classifier-free guidance during inference for most tasks, with guidance scale $w=5.0$ for TTS, SVS, T2A, T2M, and V2A, and no guidance for SE and SR because CFG degrades those tasks (Xu et al., 29 Sep 2025).

3. Dual-fusion conditioning and temporal alignment

The most distinctive architectural mechanism in UniFlow-Audio is the dual-fusion mechanism, which is intended to unify TA and NTA conditioning within the same Transformer. Content encoders first produce modality-specific embeddings $\mathbf{z}_1 \sim \mathcal{N}(0, I)$0, and a task instruction is encoded and fused with content through a content adapter:

$\mathbf{z}_1 \sim \mathcal{N}(0, I)$1

The resulting task-aware content embeddings are then used in two different forms: the original sequence $\mathbf{z}_1 \sim \mathcal{N}(0, I)$2 for global conditioning, and a duration-expanded sequence $\mathbf{z}_1 \sim \mathcal{N}(0, I)$3 for time-aligned conditioning (Xu et al., 29 Sep 2025).

For TA tasks, UniFlow-Audio predicts a sequence duration vector $\mathbf{z}_1 \sim \mathcal{N}(0, I)$4, where $\mathbf{z}_1 \sim \mathcal{N}(0, I)$5 is the number of audio latents assigned to content unit $\mathbf{z}_1 \sim \mathcal{N}(0, I)$6. The duration adapter expands content by repetition:

$\mathbf{z}_1 \sim \mathcal{N}(0, I)$7

This yields a sequence with the same length as the audio latent sequence. For TTS and SVS, $\mathbf{z}_1 \sim \mathcal{N}(0, I)$8 corresponds to the number of audio latents per phoneme or per phoneme-note pair. For SE, SR, and V2A, the relation is fixed by frame correspondence. For NTA tasks, $\mathbf{z}_1 \sim \mathcal{N}(0, I)$9 is not used meaningfully and no $$\mathbf{z}_\tau = (1 - \tau)\,\mathbf{z}_0 + \tau\,\mathbf{z}_1, \quad \tau \in [0,1].$$0 is applied (Xu et al., 29 Sep 2025).

Each Transformer block then combines the two forms of conditioning. The paper describes the block as follows. First, self-attention with AdaLN and time conditioning is applied. Second, time-aligned content is injected by position-wise addition,

$$\mathbf{z}_\tau = (1 - \tau)\,\mathbf{z}_0 + \tau\,\mathbf{z}_1, \quad \tau \in [0,1].$$1

Third, non-aligned content is integrated by cross-attention,

$$\mathbf{z}_\tau = (1 - \tau)\,\mathbf{z}_0 + \tau\,\mathbf{z}_1, \quad \tau \in [0,1].$$2

Fourth, a feed-forward layer with AdaLN is applied. This block-wise dual fusion is meant to preserve the strength of TA conditioning throughout depth while keeping NTA cross-attention available in every block (Xu et al., 29 Sep 2025).

A further refinement is the use of dummy embeddings to avoid interference between TA and NTA paths. For TA tasks, $$\mathbf{z}_\tau = (1 - \tau)\,\mathbf{z}_0 + \tau\,\mathbf{z}_1, \quad \tau \in [0,1].$$3 is replaced with a learnable dummy vector so that only $$\mathbf{z}_\tau = (1 - \tau)\,\mathbf{z}_0 + \tau\,\mathbf{z}_1, \quad \tau \in [0,1].$$4 carries content. For NTA tasks, $$\mathbf{z}_\tau = (1 - \tau)\,\mathbf{z}_0 + \tau\,\mathbf{z}_1, \quad \tau \in [0,1].$$5 is replaced with a learnable dummy sequence so that only $$\mathbf{z}_\tau = (1 - \tau)\,\mathbf{z}_0 + \tau\,\mathbf{z}_1, \quad \tau \in [0,1].$$6 carries content. The paper reports that this strict separation outperforms both pure cross-attention and “double fusion” baselines. It also reports that if aligned content is only added at the input layer rather than in every block, TA performance degrades sharply, because repeatedly applied cross-attention dominates the representation (Xu et al., 29 Sep 2025).

This dual-path design has an evident relation to later specialized systems. Foley-Flow, for example, uses temporally varying video features as dynamic conditions for segment-level generation in a video-to-audio setting, but it is specialized to masked audio-visual alignment and coordinated V2A rather than a seven-task omni-modal backbone (Mo et al., 9 Mar 2026). UniFlow-Audio instead generalizes across modalities by separating aligned and non-aligned conditioning inside a single latent flow architecture (Xu et al., 29 Sep 2025).

4. Modalities, tasks, and training regime

UniFlow-Audio supports text, audio, and video through specialized frozen encoders mapped into a shared content space. For TTS and SVS, text is converted to phonemes, speaker information is represented with x-vectors, and SVS additionally incorporates MIDI information; the content encoder is FastSpeech2-style. For T2A and T2M, free-form text is encoded with a frozen Flan-T5 encoder. For SE and SR, audio input is encoded with the same VAE encoder used for audio representation. For V2A, a frozen CLIP visual encoder provides visual features (Xu et al., 29 Sep 2025).

The system is trained and evaluated on seven tasks:

Category	Tasks
TA	TTS, SVS, SE, SR, V2A
NTA	T2A, T2M

The total training regime uses about 7,717 hours of public data. The task-wise breakdown given in the paper is: LibriTTS for TTS at about 555 hours; M4Singer for SVS at about 30 hours; AudioCaps train for T2A at about 253 hours; synthetic SE data totaling about 538 hours; SR data totaling about 316 hours from MUSDB, MoisesDB, HQ-TTS, and FreeSound; MSD plus LP-MusicCaps-MSD for T2M at about 5789 hours; and VisualSound for V2A at about 236 hours (Xu et al., 29 Sep 2025).

Because the datasets are highly imbalanced, UniFlow-Audio uses task-balanced sampling. A plain task-based round robin would still bias the model toward TA tasks because there are five TA tasks and only two NTA tasks. To counter this, NTA tasks are explicitly upsampled: T2M is sampled 3 times more frequently and T2A 2 times more frequently. The reported ablation indicates that removing task-balanced sampling degrades NTA metrics while leaving TA metrics nearly unchanged, which the paper interprets as evidence that NTA tasks are more sensitive to imbalance (Xu et al., 29 Sep 2025).

Training minimizes

$$\mathbf{z}_\tau = (1 - \tau)\,\mathbf{z}_0 + \tau\,\mathbf{z}_1, \quad \tau \in [0,1].$$7

The clip-duration loss for all tasks is

$$\mathbf{z}_\tau = (1 - \tau)\,\mathbf{z}_0 + \tau\,\mathbf{z}_1, \quad \tau \in [0,1].$$8

and the sequence-duration loss for TA tasks is

$$\mathbf{z}_\tau = (1 - \tau)\,\mathbf{z}_0 + \tau\,\mathbf{z}_1, \quad \tau \in [0,1].$$9

The paper states that durations are implemented in log-domain frame counts following FastSpeech2, and that a gradient multiplier is applied to the duration predictor using

$$v_\theta(\mathbf{z}_\tau, \tau, \mathbf{C}^I, \mathbf{C}^I_T)$$0

with $$v_\theta(\mathbf{z}_\tau, \tau, \mathbf{C}^I, \mathbf{C}^I_T)$$1 (Xu et al., 29 Sep 2025).

Three model variants are reported. The small variant has depth 12, embedding size 512, 8 heads, 593M total parameters, and 208M trainable parameters. The medium variant has depth 16, embedding size 768, 12 heads, 780M total parameters, and 395M trainable parameters. The large variant has depth 24, embedding size 1024, 16 heads, 1.2B total parameters, and 847M trainable parameters (Xu et al., 29 Sep 2025).

5. Empirical performance and ablations

UniFlow-Audio is reported to achieve strong results across all seven tasks using fewer than 8K hours of public training data and under 1B trainable parameters, with the small variant showing competitive performance. The paper’s comparison against single-task baselines emphasizes especially strong outcomes on TTS, SE, SR, and V2A, near-parity outcomes on T2A and T2M, and a small remaining gap on SVS (Xu et al., 29 Sep 2025).

For the large model, the reported metrics are as follows. On TTS, UniFlow-Audio achieves WER 3.09 and SIM 55.8, compared with NaturalSpeech2 at WER 9.94 and SIM 34.8. Subjectively, NaturalSpeech2 has MOS 2.72 and SMOS 3.43, while UniFlow-Audio has MOS 3.79 and SMOS 3.21. On SVS, UniFlow-Audio reports F0 RMSE 0.147 and SA 59.9, compared with DiffSinger at F0 RMSE 0.144 and SA 58.0; subjectively DiffSinger has MOS 4.26 and SMOS 4.43, and UniFlow-Audio has MOS 4.05 and SMOS 4.31. On T2A, UniFlow-Audio reports FD 17.2 versus AudioLDM2 at 21.8, with CLAP equal at 0.476; subjectively AudioLDM2 has OVL 3.57 and REL 3.48, while UniFlow-Audio has OVL 3.41 and REL 3.54. On T2M, UniFlow-Audio reports FD 27.1 versus MusicGen at 29.5 and CLAP 0.241 versus 0.245; subjective OVL and REL are 3.37 and 3.09 for UniFlow-Audio, versus 3.45 and 3.08 for MusicGen. On SE, UniFlow-Audio reports PESQ 2.91 and STOI 0.944 versus DOSE at PESQ 2.50 and STOI 0.931, with subjective MOS 4.76 versus 3.43. On SR, UniFlow-Audio reports LSD 1.49 versus AudioSR at 1.75, with subjective MOS 4.19 versus 3.58. On V2A, UniFlow-Audio reports IB 28.6 versus DiffFoley at 22.7 and SYNC 1145 versus 922, while subjective OVL and SYNC are 3.61 and 3.55 for UniFlow-Audio versus 2.80 and 2.94 for DiffFoley (Xu et al., 29 Sep 2025).

The model-size study indicates that even the small model, with 208M trainable parameters, already outperforms or matches strong specialized baselines on many tasks. The paper lists, for UniFlow-Audio small, TTS WER 3.23, T2A FD 19.7, T2M FD 26.2, SE PESQ 2.60, SR LSD 1.58, and V2A IB 25.5. Scaling to medium and large improves many metrics, though the gains are moderate rather than drastic. The authors interpret this as evidence of parameter efficiency and multi-task transfer (Xu et al., 29 Sep 2025).

The ablation study is central to the paper’s technical argument. Cross-attention only causes severe degradation, with TTS WER increasing from 3.23 to 27.6, SE PESQ falling from 2.60 to 1.10, T2A FD worsening from 19.7 to 30.1, and T2M FD worsening from 26.2 to 37.2. Double fusion, in which both $$v_\theta(\mathbf{z}_\tau, \tau, \mathbf{C}^I, \mathbf{C}^I_T)$$2 and $$v_\theta(\mathbf{z}_\tau, \tau, \mathbf{C}^I, \mathbf{C}^I_T)$$3 are always active, preserves some TA performance but degrades NTA performance, with T2A FD 22.3 and T2M FD 30.5. Input fusion, in which aligned content is added only once at the input, produces especially poor TA results, including TTS WER 42.0, SVS SA 41.8, and SE PESQ 1.07. These numbers are used to support the claim that block-wise dual fusion with dummy embeddings is the effective unifying mechanism (Xu et al., 29 Sep 2025).

Inference hyperparameters are also task-sensitive. The paper states that larger guidance scale and more inference steps improve T2A and T2M. By contrast, increasing guidance scale is harmful for SE and causes PESQ to drop from about 2.9 to about 1.75; increasing step count also slightly reduces SE PESQ. The proposed explanation is that stronger guidance amplifies noise components in tasks where the condition already contains noisy signal (Xu et al., 29 Sep 2025).

6. Relation to adjacent research, limitations, and significance

UniFlow-Audio is best read as one branch within a broader movement toward unified audio systems, but it is distinct in being explicitly non-autoregressive and centered on flow matching. AudioCALM pursues unification from the opposite direction: it extends autoregressive next-token prediction to continuous audio latents, uses a thin flow-matching head in a Qwen3-based LM, and targets speech, sound, and music generation with block-causal AR-Flow attention (Liu et al., 22 Jun 2026). UniFlow, despite the similar name, addresses speech front-end tasks such as SE, TSE, AEC, and LASS in a shared continuous latent space using a waveform VAE and DiT under DDPM, FM, and mean flow objectives rather than a seven-task omni-modal generation setting (Wang et al., 11 Aug 2025). Foley-Flow shows that masked alignment and time-varying conditioning are especially effective in coordinated video-to-audio generation, which can be viewed as a specialized refinement of one UniFlow-Audio task class rather than a competing universal framework (Mo et al., 9 Mar 2026).

A second adjacent line concerns representation and tokenization. UniCodec proposes a unified audio codec with a single codebook over speech, music, and sound, while LoSATok proposes a low-dimensional semantic-acoustic tokenizer intended to reduce DiT modeling burden in cross-domain understanding and generation (Jiang et al., 27 Feb 2025, Zhang et al., 27 May 2026). UniFlow-Audio differs in operating directly on continuous VAE latents rather than on a single discrete token stream or a compressed semantic-acoustic latent. This suggests a division of labor across the literature: tokenization-focused systems optimize representation for language-model-like decoding, whereas UniFlow-Audio optimizes the conditional generative backbone itself (Xu et al., 29 Sep 2025).

The paper explicitly notes several limitations. It does not cover multi-input tasks such as voice conversion or TTS with background music; it does not systematically evaluate zero-shot generalization to unseen tasks or modalities; it does not explore large-scale data or model scaling beyond the reported regime; it is focused on single-stream audio; and it still shows some quality gaps, particularly for SVS and in subjective OVL for T2A and T2M relative to certain baselines (Xu et al., 29 Sep 2025). A plausible implication is that the current framework is strongest when a task can be expressed as a single TA or NTA conditioning problem on a universal VAE latent, and less developed for multi-source or strongly structured generation settings.

Within the broader audio-generation literature, UniFlow-Audio’s main significance lies in showing that a single flow-matching Transformer with explicit separation between aligned and non-aligned conditioning can cover text, audio, and video inputs across seven tasks, using relatively modest public data and under 1B trainable parameters (Xu et al., 29 Sep 2025). This suggests that unified non-autoregressive audio foundation models need not choose between temporal alignment and general conditionality, provided those two regimes are architecturally separated rather than naively merged.