Papers
Topics
Authors
Recent
Search
2000 character limit reached

Eureka-Audio: Triggering Audio Intelligence in Compact Language Models

Published 15 Feb 2026 in cs.SD and cs.AI | (2602.13954v1)

Abstract: We present Eureka-Audio, a compact yet high-performance audio LLM that achieves competitive performance against models that are 4 to 18 times larger across a broad range of audio understanding benchmarks. Despite containing only 1.7B parameters, Eureka-Audio demonstrates strong performance on automatic speech recognition (ASR), audio understanding, and dense audio captioning, matching or surpassing multiple 7B to 30B audio and omni-modal baselines. The model adopts a unified end-to-end architecture composed of a lightweight language backbone, a Whisper-based audio encoder, and a sparsely activated Mixture-of-Experts (MoE) adapter that explicitly accounts for audio heterogeneity and alleviates cross-modal optimization conflicts under limited capacity. To further enhance paralinguistic reasoning, we introduce DataFlux, a closed loop audio instruction data synthesis and verification pipeline that constructs high quality, logically consistent supervision from raw audio. Extensive evaluations across ASR, knowledge reasoning, safety, instruction following, and paralinguistic benchmarks, demonstrate that Eureka-Audio achieves an efficient balance between computational cost and performance. These results establish Eureka Audio as a strong and practical baseline for lightweight audio understanding models.

Summary

  • The paper introduces Eureka-Audio, a 1.7B-parameter model that combines a Whisper encoder, sparse MoE audio adapter, and Qwen3 language backbone to separate semantic and paralinguistic audio features efficiently.
  • The paper shows that DataFlux synthetic instruction data raises MMAU performance from 66.93 to 74.67, while the model reaches 269.7 tokens per second—3.7× faster than Qwen3-Omni-30B-A3B.
  • The paper demonstrates strong compact-model performance for audio understanding and English ASR, but larger systems remain better on deep reasoning, knowledge benchmarks, and several Mandarin speech tasks.

Overview

Eureka-Audio is a 1.7B-parameter audio-LLM developed by Baidu and collaborators that targets both semantic audio understanding (transcription, content comprehension) and paralinguistic reasoning (emotion, tone, sound events, music). The central claim of the paper is that a compact dense model, when paired with a heterogeneous-aware adapter and high-quality synthesized instruction data, can match or exceed audio and omni-modal baselines in the 7B–30B range. The headline result is a score of 74.67 on the MMAU benchmark, competitive with models 4–17× larger, alongside the highest decode throughput among compared models at 269.7 tokens/sec — 3.7× faster than Qwen3-Omni-30B-A3B while being 17× smaller.

Architecture

The model follows the standard end-to-end LALM design with three components:

  • Audio encoder: a Whisper-based encoder producing continuous, high-temporal-resolution acoustic features from raw waveforms.
  • Sparse MoE adapter: rather than a dense projector, acoustic tokens are routed via Top-kk sparse softmax gating to lightweight feed-forward experts (SiLU activation), aggregated by routing weights, projected to the LLM embedding dimension, and layer-normalized. A load-balancing auxiliary loss Laux\mathcal{L}_{\text{aux}} over expert importance and token fractions mitigates expert collapse; training uses the standard next-token prediction objective plus this auxiliary term.
  • Language backbone: Qwen3-1.7B-base, processing concatenated audio embeddings and text tokens autoregressively.

The motivation for the MoE adapter is explicit: speech, environmental sounds, and music have distinct statistical structures, and a single shared projection introduces conflicting optimization signals. Under limited capacity, semantic and paralinguistic information compete for representational space; sparse routing is proposed as the mechanism to disentangle these demands without inflating total parameters. Notably, unlike recent omni models that use MoE backbones (e.g., Qwen3-Omni's ~3B active parameters), Eureka-Audio keeps a dense 1.7B backbone and confines sparsity to the adapter, which the authors argue avoids the large total memory footprint that hampers edge deployment.

Pretraining

Pretraining proceeds in two stages. Stage 1 (alignment) trains only the MoE adapter on roughly 100B tokens, with the encoder and backbone frozen, using unimodal modeling (text and discrete semantic audio tokens from the GLM-4-Voice tokenizer), audio–text mapping (ASR/TTS), and interleaved modeling. Stage 2 (joint pretraining) unfreezes all parameters and scales to approximately 1T tokens, adding audio captioning data (220K hours) for high-level semantic supervision. Audio and text are sampled at a fixed 1:1 ratio throughout.

DataFlux

The post-training data pipeline, DataFlux, addresses the observation that existing open-source instruction datasets derive from earlier, weaker model generations and may misalign with current model capabilities. It operates in three steps:

  1. Query–Choice generation: Qwen3-Omni-30B-A3B-Captioner produces dense captions from raw audio; GPT-OSS-120B converts them into structured multiple-choice pairs under a predefined paralinguistic taxonomy, with verification against the taxonomy hierarchy.
  2. Answer generation: two audio LLMs with distinct reasoning profiles (Qwen3-Omni-Thinking and Step-Audio-R1) generate reasoning traces and answers conditioned on the same audio, inducing answer diversity for hard-example mining.
  3. Answer verification: an automated judge (GPT-OSS-120B) evaluates consensus, internal logical consistency, and consistency with the ground-truth caption, retaining reliable samples and discarding conflicting ones.

SFT then runs on approximately 30B tokens with all parameters trainable, dominated by ASR data (60% of audio sampling), with semantic understanding, paralinguistic understanding, and dense captioning each contributing smaller shares.

Evaluation results

ASR. On LibriSpeech test-clean/test-other, Eureka-Audio-Instruct achieves 1.46/3.24 WER, outperforming Qwen2.5-Omni-7B (1.53/3.19 on clean), MiniCPM-o, and several larger models, though trailing Kimi-Audio-7B (1.33/2.57) and Step-Audio-2-mini. On Fleurs-en it scores 5.39 WER, better than most baselines except Step-Audio-2-mini. Mandarin results are weaker relative to the field: AISHELL-2 CER of 3.10 and WenetSpeech meeting/net CERs of 9.14/7.55 trail Step-Audio-2-mini and Qwen3-Omni, indicating that ASR parity with larger systems does not extend uniformly across languages and acoustic conditions.

Audio understanding. The strongest result is MMAU: 74.67 with DataFlux versus 66.93 without — a 7.74-point gain attributable directly to the synthesized data — surpassing Qwen3-Omni-Instruct (74.57), Audio Flamingo 3 (74.77 is marginally higher), and all other baselines. On MMAR, however, Eureka-Audio reaches only 56.20, well below Qwen3-Omni's 67.10, so deep-reasoning superiority does not carry over to that benchmark. Knowledge benchmarks show a clear capacity effect: MMSU 55.63 and OpenBookQA 69.23 fall short of Qwen3-Omni (77.00/92.31). Conversely, IFEval instruction following (53.21) exceeds every baseline including Qwen3-Omni (81.17 aside — actually Qwen3-Omni leads here), and AdvBench safety reaches 99.81, second only to Kimi-Audio's perfect score.

Dense captioning evaluation. The paper proposes an indirect QA-based protocol: generated captions are fed with downstream questions to GPT-OSS-120B, so answer accuracy measures caption faithfulness. Eureka-Audio-Instruct scores 52.96 (MMAU) / 41.70 (MMAR), exceeding Qwen3-Omni-Instruct (48.24/36.90) but below the dedicated 30B Captioner (56.68/46.40).

Limitations and open questions

Several caveats bear directly on the headline claims. First, the DataFlux ablation is reported only on MMAU and MMAR; its contribution to ASR, knowledge, safety, and instruction-following performance is not isolated, so the pipeline's general benefit remains partially unquantified. Second, the advantage over Qwen3-Omni on MMAU (74.67 vs. 74.57) is within noise margins, and the model trails substantially on MMAR and knowledge benchmarks, suggesting the "competitive with 4–18× larger models" claim holds mainly for paralinguistic perception rather than reasoning depth or world knowledge. Third, Mandarin ASR lags the best baselines by meaningful margins. Fourth, the dense-captioning evaluation depends on GPT-OSS-120B as both judge and answerer, introducing potential evaluator bias shared across the pipeline and the metric. Finally, the throughput comparison is conducted on a specific H100 configuration; absolute speedups may not transfer to other hardware. Open questions include whether the MoE adapter's expert specialization actually disentangles semantic from paralinguistic routing (no interpretability analysis is provided) and how DataFlux scales beyond the current taxonomy.

Conclusion

Eureka-Audio demonstrates that careful adapter design and systematically verified synthetic instruction data can deliver strong paralinguistic audio understanding at 1.7B parameters, with state-of-the-art-level MMAU performance and leading decode throughput. The evidence also shows clear boundaries: deep audio reasoning (MMAR), knowledge-intensive QA, and Mandarin ASR still favor larger models. The paper establishes a practical baseline for resource-constrained audio understanding while leaving open whether its data-centric gains extend to unified perception-and-generation settings.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.