Dragon-FM: Hybrid AR and Flow TTS

• Dragon-FM is a TTS approach that unifies autoregression for chunk-level coherence with parallel flow-matching for local refinement.
• It compresses 48 kHz audio into 12.5 Hz codec tokens, reducing sequential decisions to one per chunk and improving efficiency.
• Empirical evaluations demonstrate lower FAD scores and reduced TNFE, highlighting its advantage over conventional AR and diffusion-based systems.

Dragon-FM is a text-to-speech design introduced in “Next Tokens Denoising for Speech Synthesis” that unifies autoregression and flow-matching for speech generation (Liu et al., 30 Jul 2025). It operates on 48 kHz audio codec tokens arranged into moderate-length chunks, using chunk-level autoregressive prediction to preserve global coherence and key-value cache efficiency, while applying parallel within-chunk denoising to exploit bidirectional context. In the reported formulation, 2 s of audio are represented by 25 codec tokens at 12.5 tokens per second, so the sequential process is shifted from token-by-token generation to chunk-by-chunk generation, with only a small number of ODE-style refinement steps inside each chunk. The model is presented as a mechanism for bridging continuous and discrete feature modeling, including prediction of discrete tokens through finite scalar quantization, and is evaluated primarily for high-quality zero-shot podcast synthesis.

1. Hybridization of autoregression and flow-matching

Dragon-FM is motivated by a complementary failure mode analysis of two dominant generative paradigms (Liu et al., 30 Jul 2025). Autoregressive models, exemplified as GPT-style next-token predictors, rely on causal attention. This gives them efficient key-value caching and low first-byte latency, but prevents access to future context and makes long-sequence generation intrinsically sequential. Diffusion and flow-matching models instead generate in parallel through iterative denoising under bidirectional attention, which improves fine-detail modeling and future-context access, but precludes KV caching because each denoising step recomputes full attention.

The central design choice is to divide the audio-token sequence into chunks and to assign different generative roles to different temporal scales. Across chunks, Dragon-FM uses autoregression, so chunk $k$ is conditioned on preceding chunks. Within each chunk, it uses flow-matching to denoise all token positions in parallel. The stated consequence is that the model retains global coherence and KV-cache utilization across chunks, while incorporating future context within each chunk through full self-attention (Liu et al., 30 Jul 2025).

A common misconception would be to classify Dragon-FM as either a conventional autoregressive model or a conventional diffusion model. The architecture is neither in the narrow sense used in the paper. It is autoregressive only at chunk granularity and flow-based only within the chunk. This distinction matters because the claimed efficiency gains derive from reducing the number of sequential decisions from one per token to one per chunk, rather than from eliminating iterative refinement altogether.

2. Audio representation and chunked architecture

Dragon-FM relies on a learned neural codec that compresses 48 kHz waveforms into discrete tokens at 12.5 Hz, corresponding to one token every 80 ms (Liu et al., 30 Jul 2025). Raw audio $x(t)$ is processed by a bidirectional Transformer encoder together with a downsampling stack to produce 12.5 Hz continuous features $a \in \mathbb{R}^D$. A Finite Scalar Quantizer then quantizes each scalar component of $a$ independently to the nearest center in a small one-dimensional codebook $C \subset \mathbb{R}$. The decoder re-upsamples the resulting discrete tokens back to 48 kHz through a causal Transformer and ISTFT.

The generative pipeline is two-stage. First, every 25 consecutive codec tokens, corresponding to 2 s of speech, are grouped into a chunk. Chunk-level autoregression predicts each chunk conditioned on all previous chunks. Because a whole chunk is treated as the autoregressive “next token,” the model requires one autoregressive forward pass per chunk rather than 25 passes. Second, once the chunk-level conditioning vector $H_k$ is available, the 25 token positions of chunk $k$ are initialized as noise and refined in parallel by a small number of flow-matching steps.

The cache and attention structure are split accordingly. Across chunks, causal decoder-only attention stores keys and values from prior chunks, so each new chunk only updates the final chunk state, with $O(1)$ KV updates per chunk. Within a chunk, flow-matching blocks use bidirectional self-attention over all 25 noisy embeddings $z_i^{(1:25)}$, so the model can condition each token position on future positions inside the same chunk. This separation of temporal scopes is the paper’s core architectural claim.

3. Continuous flow objective and discrete token recovery

The within-chunk objective is formulated in continuous embedding space, even though the ultimate outputs are discrete FSQ token identities (Liu et al., 30 Jul 2025). For a single chunk, let $z_0 \in \mathbb{R}^{25 \times d}$ denote the continuous embeddings of the true 25 FSQ tokens. A perturbed sample at time $x(t)$0 is defined by the linear schedule

$x(t)$1

with $x(t)$2, $x(t)$3, and $x(t)$4.

The ideal velocity field $x(t)$5 satisfies the ODE

$x(t)$6

which transports $x(t)$7 back to $x(t)$8 as $x(t)$9. Dragon-FM parameterizes a neural vector field $a \in \mathbb{R}^D$0 conditioned on the autoregressive context $a \in \mathbb{R}^D$1 from previous chunks and trains it with the standard flow-matching regression loss

$a \in \mathbb{R}^D$2

Under the linear schedule, the paper states that

$a \in \mathbb{R}^D$3

so the implemented loss becomes

$a \in \mathbb{R}^D$4

The final step maps the denoised continuous output back to discrete token IDs. For each scalar coordinate, Dragon-FM applies finite scalar quantization by selecting the nearest codebook center:

$a \in \mathbb{R}^D$5

This continuous-to-discrete bridge is one of the defining claims of the model. The paper explicitly frames Dragon-FM as demonstrating that continuous autoregressive flow-matching can predict discrete tokens with finite scalar quantizers (Liu et al., 30 Jul 2025). A plausible implication is that the architecture is intended to combine the optimization advantages of continuous denoising with the deployment advantages of discrete codec tokens.

4. Generation procedure and parameter sharing

Generation proceeds chunk by chunk for an utterance of $a \in \mathbb{R}^D$6 chunks (Liu et al., 30 Jul 2025). The inputs are a text prompt, an optional speech prompt for speaker style, the number of chunks, the number of flow steps $a \in \mathbb{R}^D$7, and a sequence of step times $a \in \mathbb{R}^D$8. The encoder produces the initial autoregressive context $a \in \mathbb{R}^D$9, after which the model iterates over chunks.

For chunk $a$0, the autoregressive transformer first computes a chunk-level conditioning vector $a$1 and updates the KV cache. The chunk latent is then initialized as Gaussian noise $a$2. For each flow step, the model computes a flow-conditioning representation $a$3, predicts the velocity $a$4, and updates the latent according to

$a$5

After the final step, the denoised outputs are quantized into 25 discrete codec tokens, which are appended back into the autoregressive state for subsequent chunk generation. The concatenation of all generated chunks forms the final token stream.

The paper states that, in practice, $a$6 or $a$7 suffices for high fidelity. It also states that the AR transformer and Flow transformer share most weights, differing only in whether attention is causal or bidirectional. This suggests that the design seeks to minimize architectural duplication while preserving distinct context geometries for cross-chunk and within-chunk computation.

5. Iteration count, memory profile, and scaling behavior

Dragon-FM analyzes efficiency through the Total Number of Function Evaluations, defined as

$a$8

For a 2 s segment, the paper reports three comparison points (Liu et al., 30 Jul 2025). A pure autoregressive system such as VALL-E requires 25 autoregressive token steps, giving $a$9. A pure flow system such as E2 uses one 2 s block but 32 flow steps, giving $C \subset \mathbb{R}$0. Dragon-FM, with a 2 s chunk and $C \subset \mathbb{R}$1, gives $C \subset \mathbb{R}$2.

Model	Cache/TNFE	RTF
E2	no / 32	low
VALL-E	yes / 25	med
Dragon-FM	yes / 2	high

The paper summarizes this as an iteration-count reduction of approximately $C \subset \mathbb{R}$3–$C \subset \mathbb{R}$4 relative to E2 or VALL-E, and notes that the advantage is amortized over long contexts such as 32 s, where Dragon-FM spans 16 chunks (Liu et al., 30 Jul 2025). It also states that AR steps cache only the last chunk’s keys and values, giving $C \subset \mathbb{R}$5 KV growth per chunk, while flow steps recompute full attention only within a fixed chunk length of 25, described as a negligible relative cost.

The reported memory expression is

$C \subset \mathbb{R}$6

for attention per chunk, with the qualification that chunk size is fixed at 25, so scaling depends only on prompt length. This is significant because the model’s efficiency claim is not only about fewer iterations, but also about preserving the favorable cache structure of decoder-only autoregression.

6. Empirical evaluation on podcast speech synthesis

The experiments are conducted on a dataset of 60k h of English podcasts covering diverse topics and speakers (Liu et al., 30 Jul 2025). The reported evaluation setting is zero-shot: a text prompt and a short same-speaker speech prompt are provided, and the model generates new speech.

Fréchet Audio Distance is used as a distributional quality and diversity indicator, with lower FAD interpreted as better match to the real-data distribution. The paper reports the following configurations.

Configuration	Chunk / rate / FM steps	FAD
Dragon-FM-A1	2 s / 12.5 Hz / 12	2.4
Dragon-FM-A2	2 s / 12.5 Hz / 24	2.2
Dragon-FM-A3	2 s / 12.5 Hz / 6	2.6
Dragon-FM-B1	1 s / 12.5 Hz / 12	2.1
Dragon-FM-B2	1 s / 12.5 Hz / 24	1.9
Dragon-FM-C1	2 s / 20 / 12	2.2
Dragon-FM-C2	2 s / 20 / 24	2.0

The reported interpretation is explicit: increasing flow steps, as in A2 versus A1, improves FAD, and using smaller chunks, as in B2, also improves FAD. The paper also states that Dragon-FM preserves high perceptual quality while achieving substantially faster real-time factors through its lower TNFE (Liu et al., 30 Jul 2025).

Codec reconstruction is evaluated separately by speaker similarity (SIM) and word error rate (WER).

Codec model	Frame rate	SIM / WER
Mel-VocoderC	16 Hz	0.904 / 3.01
CodecA	12.5 Hz	0.925 / 2.65
CodecB (default)	12.5 Hz	0.916 / 2.74

The paper states that, at 12.5 Hz, the neural FSQ codec outperforms a 16 Hz mel-spectrogram vocoder in both SIM and WER. This result is important because the system’s generation speed relies on a compact token rate; the codec reconstruction table provides evidence that the low-rate representation remains compatible with intelligibility and speaker similarity requirements.

7. Limitations, edge cases, and proposed extensions

The authors identify two principal limitations (Liu et al., 30 Jul 2025). First, chunk boundaries can produce boundary artifacts. Suggested mitigations are overlap or chunk smoothing. Second, the chunk length is fixed; the paper notes adaptive chunk sizing as future work. These limitations follow directly from the architectural choice to impose a hard decomposition between long-range autoregression and local denoising.

The paper presents the core principle as “chunk-wise AR for long-range dependency plus few-step flow for local parallel refinement,” and states that it is readily extendable beyond speech synthesis. The listed examples are image generation, using patch-level AR for global layout and flow within patches for texture; video, using clip-level AR for temporal coherence and flow per frame or group of frames; and multimodal LLMs, using block-wise AR for discrete tokens together with flow matching on continuous latent spaces. These extensions are presented as prospective rather than experimentally validated within the reported study.

In that sense, Dragon-FM is best understood as a specific TTS instantiation of a more general factorization strategy: autoregression for coarse sequential dependency, and flow-matching for parallel local refinement. The paper’s claim is that this factorization bridges discrete and continuous modeling while preserving both global coherence and computational efficiency in zero-shot long-form speech generation (Liu et al., 30 Jul 2025).