AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation

Abstract: Unifying speech, sound, and music generation in one model is hindered by tradeoffs between fidelity, end-to-end training, in-context conditioning, and variable-length synthesis that no current paradigm fully resolves. To address this challenge, we present AudioCALM, a universal audio generation framework that extends autoregressive (AR) next-token prediction from discrete tokens to continuous audio latents: a thin flow-matching head replaces the softmax to predict rectified-flow velocities at each position, and a block-causal AR-Flow attention pattern produces arbitrary-length output. Joint training of multiple audio generation tasks faces an asymmetric text--audio mismatch: speech transcripts align to specific time spans and demand tight, time-aligned attention, whereas sound and music captions describe only overall semantics and rely on diffuse, holistic attention; mixing the two disproportionately degrades sound and music generation. We address this asymmetry at two levels: a data reformulation strategy that unifies all three tasks under a single description-style conditioning interface, and a novel architecture Asymmetric Mixture-of-Modality-Experts (A-MoME), which adds a dedicated residual expert for speech while sound and music share the backbone, incurring no inference overhead on non-speech inputs. Experimental results demonstrate that AudioCALM matches modality-specific state-of-the-art and outperforms prior unified baselines on speech, sound, and music generation benchmarks.

• The paper introduces Continuous Autoregressive Language Modeling, combining continuous audio latents, a flow-matching head, and block-causal attention for streaming, variable-length generation.
• AudioCALM matches or surpasses specialist systems, achieving WERs of 0.020 and 0.011 for speech, FAD/CLAP scores of 1.95/0.37 for sound, and 2.02/0.36 for music.
• The paper shows that speech creates asymmetric cross-modal interference, which description-based conditioning and speech-specific residual experts reduce without adding inference cost for non-speech audio.

AudioCALM is a universal audio generation framework that generates speech, sound, and music with a single set of weights by extending autoregressive next-token prediction from discrete codec tokens to continuous audio latents (2606.23080). The framework, termed Continuous Autoregressive Language Modeling (CALM), replaces the softmax output layer with a thin flow-matching head that predicts rectified-flow velocities at each latent position, and introduces a block-causal attention pattern (AR-Flow) that supports streaming, arbitrary-length generation without an external duration predictor. The paper's central empirical claim is that a unified model can match modality-specific state-of-the-art systems on all three domains, and the reported numbers substantiate this: lowest WER and MOS among all compared systems on both zero-shot TTS benchmarks, and the best FAD/CLAP on both text-to-sound and text-to-music.

Motivation and the paradigm gap

The authors organize prior work into three paradigms, each with a structural deficiency. Discrete-token autoregression (VALL-E, AudioLM, MusicGen, UniAudio) offers a clean LM interface but suffers a codec information bottleneck that caps fidelity on polyphonic music and complex sound. Cascaded LM-then-diffusion systems (CosyVoice2, MaskGCT) remove that bottleneck but freeze an intermediate semantic code, severing end-to-end optimization. Non-autoregressive flow matching (Audiobox, Stable Audio Open, UniFlow-Audio) achieves strong fidelity but requires externally specified target durations and forfeits autoregressive in-context conditioning. AudioCALM is positioned as satisfying four desiderata—fidelity, end-to-end training, in-context conditioning, and variable-length synthesis—simultaneously. It extends the continuous-latent AR line (GIVT, MAR in vision; DiTAR in speech) from speech-only generation to universal audio, differing from DiTAR in that the LLM itself serves as the denoiser, a modality-asymmetric expert design resolves cross-modal interference absent in single-modality settings, and speech is recast as a special case of audio under a shared conditioning interface.

Continuous autoregression via AR-Flow

The sequence formulation factorizes $p(x \mid c) = \prod_i p(x_i \mid x_{<i}, c)$ over continuous VAE latents at approximately 10.75 Hz. A linear input projection embeds noisy latents together with RoPE position and sinusoidal timestep embeddings; a zero-initialized linear head maps hidden states to velocities in the same latent space, trained with a rectified-flow objective under a logit-normal timestep schedule. Zero-initialization of both the output head and the expert FFN ensures the added components begin as no-ops relative to the pretrained Qwen3-1.7B backbone.

The AR-Flow attention mask is causal across latent blocks and fully bidirectional within the active noisy block, so flow matching can denoise a block jointly while the LM commits the previous block to the KV cache. A notable engineering contribution is the single-pass teacher-forced training scheme: all blocks of an utterance are packed into one extended sequence containing clean and noisy copies, with a mask constructed so that the training attention pattern exactly matches the inference pattern. This yields per-token training cost comparable to a standard LM without train–inference mask mismatch. Variable length is handled by a lightweight binary stop head trained with a linearly ramped target over the last tokens of each clip, giving token-level rather than block-level termination granularity. Two regularizers address exposure-bias drift: per-block clean-prefix noise (scale ramping to $\sigma_\text{clean}=0.1$) applied only to input embeddings, and an exposure-bias perturbation ($\gamma_\text{exp}=0.1$) that perturbs both the prefix and the target.

The speech–audio asymmetry

The paper's most substantive analytical claim is that joint training exhibits an asymmetric text–audio mismatch. Speech transcripts align to specific time spans and demand tight, locally aligned attention, whereas sound and music captions describe only global semantics and rely on diffuse attention. The ablation isolates this cleanly: adding non-speech data to a speech-only run leaves speech metrics essentially unchanged (WER 0.022→0.024, SIM 0.628→0.620), while adding speech to a non-speech-only run inflates sound and music FAD by roughly 35% each—a directional crowd-out in which non-speech pays a substantially larger fidelity tax.

The remedy operates at two levels. At the data level, a description-style conditioning interface recasts all three tasks uniformly: an audio-conditioned MLLM (Gemini 3 Pro) converts each clip plus its bare annotation into a long-form natural-language description grounded in the waveform (speaker timbre, prosody, acoustics for speech; events and scene for sound; instrumentation, tempo, mood for music). For speech, the verbatim transcript is spliced into the description inside dedicated <spoken> delimiters added as single special tokens to the tokenizer, preserving local content alignment within the global description. Short and long variants are sampled with equal probability during training. At the architecture level, Asymmetric Mixture-of-Modality-Experts (A-MoME) keeps self-attention and the main FFN shared while adding one deterministic, speech-only residual FFN per block—no gating network, no overhead on non-speech inputs, and roughly one extra FFN sub-layer versus the three duplicated FFNs of a symmetric modality-expert design.

Main results

The model is trained for 300k steps on A800×8 GPUs on LibriTTS + Emilia (speech), VGGSound/AudioCaps/WavCaps (sound), and FMA/MTG-Jamendo (music), with all audio re-captioned offline and evaluation sets held out. Key numbers:

Benchmark	Metric	AudioCALM	Best specialist
LibriTTS test-clean	WER / MOS	0.020 / 4.02	CosyVoice 3.0: 0.022 / 3.96
SeedTTS-eval (en)	WER / MOS	0.011 / 3.95	Ming-omni: 0.013 / 3.80
AudioCaps (sound)	FAD / CLAP	1.95 / 0.37	TangoFlux: 2.70 / 0.36
Song-Describer (music)	FAD / CLAP	2.02 / 0.36	Stable Audio Open: 2.23 / 0.32

The 28% relative FAD reduction over TangoFlux on sound is the strongest single result. Speaker similarity is the only axis where a specialist (CosyVoice 3.0) remains ahead, a gap the authors attribute to its larger English-only speech corpus and dedicated speaker-fidelity objective. Against unified baselines the contrast is stark: UniAudio trails badly on every modality, UniFlow-Audio loses ≥2.8× FAD on music, and Ming-omni degrades sharply on music (FAD 7.98, CLAP 0.07). AudioCALM is the only system ranking first or second on every reported column, supporting the claim that continuous-latent autoregression with description-style conditioning eliminates the cross-modal interference that has characterized unified audio generation.

Ablations

The cumulative ablation attributes gains in a clear ordering. Replacing a discrete codec head with the continuous flow-matching head is the largest single contributor across all domains (e.g., music FAD 4.80→3.45, speech SIM 0.560→0.620), confirming the codec bottleneck as the dominant fidelity cap. Description-style reframing improves sound and music (FAD 3.30→2.70 and 3.45→2.85) while leaving speech within noise. A-MoME then improves all six metrics beyond a symmetric three-expert MoME (e.g., music FAD 2.40→2.02) with fewer added parameters and no non-speech inference cost—capacity allocated by mismatch structure rather than modality count. Appendix sweeps show monotone gains from 0.6B to 4B backbones, an optimum near CFG $w{=}3$, and a block-size sweet spot at $B{=}1.0$ s, with the non-autoregressive variant (no block decoding) degrading speech WER substantially.

Limitations

The authors concede three constraints. Training covers only English speech and public sound/music corpora, excluding non-English speech, singing voice, and rare audio events. The backbone-scale study stops at 4B parameters, so scaling trends beyond that remain unverified. Long-form generation is not studied in depth; long-horizon coherence and termination behavior are left open. A further reproduction caveat, acknowledged in the paper's LLM-usage declaration, is that the captioner is closed-source; although prompts and cached annotations are released and evaluation uses original benchmark text, exact training reproduction requires comparable multimodal-LLM access. The paper also notes that license-based safeguards are insufficient against determined misuse of zero-shot voice cloning.

Conclusion

AudioCALM demonstrates that continuous autoregressive language modeling—flow-matching heads over continuous latents with block-causal attention—can unify speech, sound, and music generation at specialist-level fidelity, and that the residual cross-modal interference is directional and best resolved by asymmetric, mismatch-aware design rather than uniform capacity allocation. Open questions the paper leaves include scaling beyond 4B parameters, extending coverage to singing voice and non-English speech, and characterizing long-form coherence under the AR-Flow decoding scheme.