- The paper introduces an end-to-end audio scene generator that combines a fine-tuned Qwen3-1.7B LLM with per-token conditional flow matching over continuous semantic-acoustic latents for variable-length speech, music, and sound-effect generation.
- The model substantially improves speech intelligibility over Dasheng AudioGen, reducing English WER from 12.15% to 2.79% and Chinese CER to 3.87%, while achieving stronger mixed-scene FAD and improved MusicCaps distribution scores.
- Ablations show that audio-text alignment is essential for speech quality and that the flow-matching DiT width must exceed the latent dimensionality, although low-resource languages, long-form generation, speaker control, and some fidelity metrics remain limitations.
Motivation and positioning
Generating coherent audio scenes that blend intelligible speech, music, and sound effects requires modeling temporal coordination, energy balance, and acoustic consistency across sources — capabilities that neither single-type generators (TTS, TTM, TTA) nor task-unified models provide. Task-unified systems such as UniAudio and UniFlow-Audio generate each audio type independently, while Dasheng AudioGen, the closest predecessor, couples a frozen text encoder with a non-autoregressive diffusion transformer, restricting it to fixed-length, monolingual outputs with poor speech intelligibility. MiDashengLM-Gen addresses these constraints by replacing the generation backbone with a pre-trained LLM combined with per-token conditional flow matching, retaining Dasheng AudioGen's structured multi-view captions and high-dimensional semantic-acoustic latent space. The authors claim this is the first end-to-end trained model for general text-to-audio scene generation; the claim rests on the removal of any separate frozen text encoder, since the LLM jointly encodes text and audio.
Architecture
The system has three components. First, a pre-trained LLM backbone (Qwen3-1.7B, fully fine-tuned) processes structured multi-view captions decomposed into six views — global scene description, transcript, speaker identity/style, sound effects, music content, and recording environment — each gated by a dedicated special token, with inapplicable fields filled by an <|unknown|> token. Second, an audio tokenizer reuses the encoder of a custom MiDashengLM-0.6B model (DashengTokenizer architecture), producing 768-dimensional continuous latents at 25 Hz that preserve both semantic and acoustic information, unlike low-dimensional VAE latents. An MLP adapter groups $k=5$ consecutive frames into 5 Hz tokens (200 ms) projected into the LLM's 2048-dimensional space. Third, a per-token flow-matching DiT (16 layers, width 2048, 8 heads) generates each group of five encoder frames conditioned on the LLM hidden state at the corresponding position, latent history, and a timestep embedding, trained with the standard conditional flow-matching objective. Inference uses a 10-step Euler ODE solver with classifier-free guidance (scale 2.0), and output length is controlled by a learned stop head trained with balanced cross-entropy over hard negatives.
Two design choices distinguish this from prior work. Operating on continuous latents rather than discrete codec tokens avoids quantization artifacts while preserving autoregressive length control. And because flow matching is applied per token rather than over the full sequence, the model retains variable-length generation that full-sequence non-autoregressive decoders cannot support.
Results: speech intelligibility is the headline gain
The most striking result is speech intelligibility. On Seed-TTS English, WER drops from 12.15% (Dasheng AudioGen) to 2.79%, approaching dedicated TTS systems (Qwen3-TTS at 1.24%); Chinese CER improves from above 100% to 3.87%. Multilingual evaluation across nine languages shows competitive error rates in high-resource languages (2.42% English, 3.51% Chinese) but substantially worse performance for under-represented languages (16.45% Japanese, 13.19% Russian), which the authors attribute directly to training data distribution — Japanese and Russian have minimal representation in the TTS corpora. This is a fair caveat: multilingual capability here tracks corpus coverage rather than architectural generalization.
On emotion expressiveness (CV3-Eval), the model achieves the best text-related accuracy across all three emotions (0.98 happy / 0.96 sad / 0.92 angry), exceeding even CosyVoice3-DiffRO-EMO, though CosyVoice3 retains an advantage in text-unrelated settings where emotion must be conveyed purely through prosody.
On single-type benchmarks, results are mixed as expected. On AudioCaps, the model trails TangoFlux (FAD 5.01 vs. 2.26), which the authors attribute to training on mixed scenes rather than pure sound effects. On MusicCaps, it outperforms Dasheng AudioGen on FD (14.58 vs. 18.45) and KL (1.24 vs. 1.37).
Mixed-audio scene generation on MECAT
On MECAT, the primary mixed-audio benchmark, the picture is nuanced. Across all speech-containing mixed categories (S0A, SM0, SMA), MiDashengLM-Gen achieves consistently lower FAD than Dasheng AudioGen (e.g., 0.98 vs. 1.70 on SM0), indicating better multi-source coordination. However, Dasheng AudioGen retains advantages on FD and KL in these categories, suggesting its non-autoregressive decoder better captures fine-grained distributional detail. On music (0M0), the LLM-based model reduces FD by more than 2× (6.82 vs. 14.90); on speech (S00), it achieves better FAD (0.87 vs. 1.76). The authors interpret this tradeoff honestly: the autoregressive LLM backbone improves coordination and spectral quality, while the non-autoregressive DiT holds an edge in isolated distributional fidelity. Notably, MECAT speech-related comparisons are restricted to the English subset for fairness, since the benchmark is multilingual.
Ablations: two structural requirements
The ablation studies establish two prerequisites that arguably constitute the paper's main technical contribution beyond the system itself.
Audio-text alignment is strictly necessary. Removing the alignment pre-training stage (randomly initialized adapter, original Qwen3 weights) degrades AudioCaps FAD from 5.01 to 9.39 and, disproportionately, speech metrics: Seed-TTS English WER rises 4.4× (2.79% → 12.17%), Chinese CER 3.7×, and mean multilingual WER 4.1× (7.68% → 31.73%). The authors conclude that alignment is especially critical for speech, where precise text-to-phonetic mapping depends on a pre-aligned embedding space.
DiT width must strictly exceed latent dimensionality. Under fixed parameter budgets trading off width against depth, convergence occurs only when DiT width exceeds the audio latent dimensionality. For 768-dim latents, widths of 752 and 768 fail (STFT L2 distance 3.89 vs. a reconstruction topline of 1.30), while 784 and 1024 converge. The pattern replicates across 1280-dim latents and across Qwen3-0.6B and 4B backbones, consistent with the theoretical finding of Zheng et al. that noise injection expands the data manifold to full rank, requiring decoder width to match token dimensionality. This is presented as a structural property generalizing across scales, though the evidence comes from single-sample overfit experiments rather than full-scale training runs at every configuration.
Limitations and open questions
The paper concedes several limitations explicitly. Variable-length generation is bounded by the 1–20 second training distribution; coherent generation beyond this range remains untested. Speech intelligibility still trails dedicated TTS by 2.2× on English, with larger gaps in low-resource languages. Speaker control is coarse — there is no voice cloning or explicit speaker identity conditioning. Additionally, the width-convergence rule is validated only via overfitting experiments, and the FD/KL deficit relative to the non-autoregressive baseline on mixed categories is left unexplained mechanistically. Whether the alignment stage requirement persists at larger LLM scales, and whether the width threshold interacts with latent dimensionality in a predictable way, are open questions the paper does not resolve.
Conclusion
MiDashengLM-Gen demonstrates that coupling a pre-trained LLM with per-token conditional flow matching over high-dimensional semantic-acoustic latents yields unified audio scene generation with near-dedicated-TTS speech intelligibility, variable-length multilingual output, and competitive mixed-audio quality. Its two ablation findings — the necessity of explicit audio-text alignment and the strict DiT-width-over-latent-dimensionality convergence condition — provide actionable design constraints for future LLM-conditioned continuous-latent audio generators.