FLM-Audio: Full-Duplex Audio Chat Model
- FLM-Audio is a native full-duplex spoken dialog model that leverages natural monologues for sentence-level alignment, preserving language continuity.
- It employs a dual ASR/TTS training curriculum that alternates leading and trailing audio-text relations to reduce latency and handle interruptions.
- Evaluation demonstrates improved responsiveness, lower WER, and efficient processing compared to traditional TDM systems with reduced computational costs.
FLM-Audio is a native full-duplex spoken dialog model that is designed to let a chatbot listen and speak simultaneously, rather than alternating these behaviors through time-division multiplexing. Its central proposal is to replace word-level text–audio synchronization with natural monologues—continuous textual token sequences aligned with audio at the sentence level—and to train the system with a dual curriculum that alternates whether the monologue stream leads or trails the audio stream. In the reported implementation, FLM-Audio is a 7B autoregressive model evaluated on audio understanding, audio generation, and full-duplex chatting, where it is described as improving responsiveness, duplexity, and chatting experience [2509.02521].
1. Research context and problem setting
FLM-Audio belongs to the broader class of spoken dialog models that seek full duplexity, meaning that the system should process input and generate output simultaneously. The paper frames this requirement in terms of rapid reactions to fast-changing user input, including interruptions, turn-taking, and overlapping speech. Within this setting, it contrasts two families of systems: Time-Division Multiplexing (TDM), in which listen and speak are interleaved in one long sequence, and native full-duplex systems, in which all channels are merged at each time step [2509.02521].
This positioning differs from much of the contemporaneous audio-language literature. Models such as Audio Flamingo and Audio Flamingo 3 emphasize audio understanding, few-shot learning, long-audio reasoning, and multi-turn dialogue over speech, sound, and music, but they are not primarily organized around the systems problem of simultaneous listening and speaking [2402.01831][2507.08128]. Dataset-centric efforts such as Audio-FLAN target unified audio understanding and generation across speech, music, and general audio, whereas FLM-Audio addresses the stream organization and alignment problem inside native full-duplex spoken interaction [2502.16584].
The paper’s motivation is that TDM expands the effective context length as channels are added, and because Transformer attention is (O(n2)), this hurts latency and scalability. It cites full-duplex delays of up to 2 seconds and practical generation limits around 45 seconds for some TDM systems. Native full-duplex avoids that by not expanding the time axis with channel count; in prior work such as Moshi, this paradigm can reduce latency to around 80 ms [2509.02521].
2. Critique of conventional alignment
The immediate technical target of FLM-Audio is the alignment strategy used in prior native full-duplex models, especially Moshi-style systems. In those models, the monologue stream is synchronized to the exact moment each word is pronounced. Because audio is denser than text, a spoken second may correspond to only 3–4 text tokens but 12.5 audio frames. The result is a text stream broken into word-aligned fragments, with padding markers such as <pad> and <end-of-pad> inserted to force timing consistency [2509.02521].
The paper argues that this word-level alignment has three drawbacks. First, splitting a sentence into isolated fragments weakens the natural autoregressive flow of the language model and can “break the language capability” of large pretrained models. Second, it depends on highly accurate token timestamps, which are difficult to obtain at scale and can introduce cascading supervision errors. Third, it imposes substantial preprocessing cost because fine-grained alignment is required for every token rather than only sentence-level transcript alignment [2509.02521].
This critique is central to FLM-Audio’s design. The paper does not present the problem as a minor efficiency issue; it treats the alignment abstraction itself as the source of degraded language ability and brittle supervision. The model is therefore organized around a different assumption: spoken dialogue should not be forced into token-by-token synchronization between text and audio.
3. Natural monologues
FLM-Audio’s replacement for word-level synchronization is the natural monologue. Instead of tying each text token to a specific spoken moment, the monologue channel carries a continuous text sequence at the sentence or paragraph level, while the speaking channel emits audio concurrently. Alignment is performed only at the sentence level [2509.02521].
Operationally, the text stream remains uninterrupted until the sentence ends; after that point, it emits <wait> tokens until the corresponding audio finishes or until a new sentence or turn begins. The paper presents this as closer to human conversational behavior, where language is not naturally organized as isolated word fragments synchronized to acoustic frames [2509.02521].
Natural monologues can be arranged in two temporal modes. In Lead, the text monologue precedes the audio by about 0–2 tokens, which is described as TTS-like. In Follow, the text monologue trails the audio, which is useful for ASR-like behavior. FLM-Audio does not commit to only one of these relations; the dual-training curriculum uses both [2509.02521].
The significance of this design is linguistic as much as acoustic. The paper’s claim is that contiguous text preserves the pretrained backbone’s language modeling strength, whereas word-level alignment fragments it. This suggests that FLM-Audio treats text continuity as a systems variable rather than merely a formatting choice.
4. Dual training, architecture, and data pipeline
The second major contribution is the dual training paradigm, which alternates between TTS-like and ASR-like temporal relations across training stages. The authors’ claim is that once text is no longer forced into word-level alignment, the model must explicitly learn both regimes: text-first speech generation and audio-first transcription-like understanding [2509.02521].
At each autoregressive step, the backbone input embedding includes text, listen-channel audio tokens, and speak-channel audio tokens, and the backbone transformer produces a hidden state used by a depth transformer to generate audio codes. Audio is discretized at 12.5 frames per second, with 8 audio tokens per frame; the first token in the RQ-Transformer output is semantic, followed by 7 acoustic tokens. FLM-Audio is initialized from the language-model component of Qwen-2.5-VL, and for audio processing it follows the RQ-Transformer architecture used by Moshi and related work, with streaming Mimi encoders and decoders [2509.02521].
The training curriculum has four stages.
| Stage | Purpose | Salient format |
|---|---|---|
| Post-training-1 | listening and speaking initialization | TTS Style / ASR Style |
| Post-training-2 | stronger speech supervision | adds human-annotated ASR datasets |
| Fine-tuning-1 | semi-duplex transition | ASR-Response-TTS |
| Fine-tuning-2 | final full-duplex behavior | Response-TTS with interruptions |
In post-training, the model is trained on about 1 million hours of Chinese and English speech audio from audiobooks, podcasts, TV shows, and vlogs. Chinese transcripts are produced with FunASR, English transcripts with Whisper, and noisy samples are filtered. Post-training-2 adds human-annotated ASR datasets such as ST_CMDS, Aishell3, Magicdata, primewords_md_2018_set1, and Thchs30. Only sentence-level timestamps are required [2509.02521].
The post-training objective is a weighted cross-entropy over speaking-channel semantic tokens, speaking-channel acoustic tokens, monologue tokens, and <wait> tokens, with reported effective weights (\alpha_1 = 1), (\alpha_2 = 0.5), (\beta = 1), and (\gamma = 0.01). The paper notes that this differs from Moshi, which used very large weights and additional tricks such as text-token masking and separate optimizers [2509.02521].
Supervised fine-tuning turns the model into a general-purpose spoken dialog assistant. Instruction data are synthesized from Chinese and English sources including Magpie, Belle, Infinity-Instruct, WizardLM, and Ultrachat; responses are refined with DeepSeek-V3; TTS is synthesized with over 700 human voices filtered by DNSMOS; and noise augmentation uses DNS Challenge, RNNoise, and speech clips. Fine-tuning-1 uses an ASR-Response-TTS format with <asr> and <answer> markers, while Fine-tuning-2 removes explicit ASR supervision and trains a Response-TTS format. A variant of the final stage introduces free user inputs: interruptions are inserted with probability 0.7, reaction delay is set to 0.5 seconds, and speech leakage augmentation mixes the speaking channel back into the listening channel with probability 0.3 [2509.02521].
The reported optimization schedule uses batch size 256 for all stages, with cosine learning-rate decay: Post-training-1 for 3.3 epochs at (2\mathrm{e}{-}4 \to 1\mathrm{e}{-}5), Post-training-2 for 1 epoch at (1\mathrm{e}{-}5 \to 8\mathrm{e}{-}6), Fine-tuning-1 for 2 epochs at (1\mathrm{e}{-}5 \to 8\mathrm{e}{-}6), and Fine-tuning-2 for 6 epochs at (8\mathrm{e}{-}6 \to 7\mathrm{e}{-}6) [2509.02521].
5. Evaluation and reported performance
The paper evaluates FLM-Audio on three capability groups: audio understanding, audio generation, and full-duplex chatting [2509.02521].
For audio understanding, the tasks are ASR on Fleurs-zh and LibriSpeech-clean, plus Speech QA on LlamaQuestions. The reported metrics are WER for ASR and Accuracy for LlamaQuestions. After SFT-1, FLM-Audio reaches 5.4 WER on Fleurs-zh, 3.2 WER on LibriSpeech-clean, and 56.3 on LlamaQuestions. The paper emphasizes that it uses less than 15% of Moshi’s audio data, does not rely on fine-grained timestamps, and still beats Moshi on LibriSpeech-clean by a large margin [2509.02521].
For audio generation, the evaluation uses Seed-TTS-en and Seed-TTS-zh, with WER and speaker similarity (SIM) as metrics. FLM-Audio SFT-2 reports 2.95 WER / 0.543 SIM on Seed-TTS-en and 2.10 WER / 0.601 SIM on Seed-TTS-zh. The paper notes that these similarity scores are lower than those of specialized TTS systems such as Seed-TTS or CosyVoice2, but presents this as acceptable given that FLM-Audio is a duplex dialog model rather than a specialist voice-cloning system [2509.02521].
For full-duplex chatting, the automatic evaluation is an LLM-score obtained by scoring spoken instruction-following responses against references. In that setting, Qwen2.5-Omni scores 6.36, while FLM-Audio SFT full scores 6.58. The human evaluation is a double-blind head-to-head comparison against Qwen2.5-Omni on multi-turn audio conversations, rated by five annotators on helpfulness, naturalness, responsiveness, and robustness [2509.02521].
| Evaluation | FLM-Audio | Comparison |
|---|---|---|
| LLM-score | 6.58 | Qwen2.5-Omni 6.36 |
| Helpfulness | 7.2 | Qwen2.5-Omni 7.4 |
| Naturalness | 8.2 | Qwen2.5-Omni 7.9 |
| Responsiveness | 8.8 | Qwen2.5-Omni 8.1 |
| Robustness | 8.0 | Qwen2.5-Omni 7.7 |
The most direct ablation concerns the transitional semi-duplex stage. A variant without SFT-1 obtains an LLM-score of 4.59, compared with 6.58 for the full model. The paper uses this result to argue that ASR-style supervision in the transitional stage is essential, and more broadly that TTS-style data alone is not sufficient for native full-duplex dialog [2509.02521].
6. Limitations and significance
The paper is explicit about several limitations. FLM-Audio is only about 7B parameters, and the authors state that they did not scale it further because of compute and data limits. The training corpus is about 1 million hours, which is less than Moshi’s 8M+ hours. FLM-Audio is not optimized for perfect speech similarity and does not outperform specialist TTS models on similarity metrics. Its alignment scheme still assumes reliable sentence-level transcript alignment, even though this is much cheaper than word-level alignment. Multilingual expansion beyond English and Chinese is left open [2509.02521].
Within the broader literature, FLM-Audio is significant because it shifts the emphasis from backbone novelty to stream organization, alignment, and curriculum design. This suggests that progress in spoken dialog agents may depend as much on how text and audio are temporally organized as on raw scale. It also complements other directions in audio-language research: dataset unification for audio understanding and generation in Audio-FLAN [2502.16584], long-audio reasoning in the Audio Flamingo line [2503.03983][2507.08128][2604.10905], and evaluation work showing that benchmark gains in audio-language models do not always correspond to robust auditory dependence [2604.24401].
A common misconception is that native full-duplex quality is determined mainly by architectural choice. FLM-Audio argues otherwise. Its core claim is that word-by-word synchronization is itself the wrong abstraction for spoken dialogue, and that natural monologues combined with dual ASR/TTS-style training preserve the pretrained language model’s competence while still enabling responsive, interruptible speech interaction. In that sense, FLM-Audio is best understood not simply as another spoken assistant, but as a specific proposal about how language and audio streams should be aligned inside native full-duplex chatbots [2509.02521].