- The paper presents a novel diagnostic framework for entropy allocation in LLM-ASR, identifying the encoder-LLM interface as a critical information bottleneck.
- It introduces specific metrics (NSE, PAI, CSAI) and a multi-stage training strategy that balances acoustic grounding with semantic precision for enhanced parameter efficiency.
- Empirical results demonstrate that a 2.3B-parameter model achieves competitive recognition performance and reduced hallucination rates compared to larger baselines.
Entropy Allocation in LLM-based ASR: Diagnostic Perspective and Capability-Aware Optimization
Introduction
The integration of LLMs with automatic speech recognition (ASR) systems has become central to next-generation ASR architectures, leveraging the contextual reasoning and world knowledge embedded in LLMs to resolve semantic ambiguity. Despite strong performance on public benchmarks, contemporary LLM-based ASR models face persistent challenges in balancing recognition accuracy, efficiency, and robustness, particularly under practical constraints on model size, latency, and risk of hallucination. This paper presents a rigorous analysis of LLM-ASR dynamics through the lens of entropy allocation, introducing diagnostic metrics and an optimized multi-stage training strategy to explicitly align uncertainty reduction with functional module capabilities (2604.08003).
Entropy Allocation: Metrics and Problem Diagnosis
The core insight formalizes the encoder-LLM interface as an information-theoretic bottleneck, where the speech encoder maps high-entropy acoustic input into lower-entropy intermediate representations, and the LLM subsequently resolves residual uncertainty for final token prediction. The authors introduce three primary metrics to characterize representation evolution and uncertainty reduction:
- Normalized Spectral Entropy (NSE): Measures the geometric concentration of encoder representations. Low NSE indicates strongly compressed, anisotropic representations; high NSE indicates isotropy and higher residual entropy.
- Phonetic Accessible Information (PAI): Probes the linear accessibility of phonetic information in the encoder state, serving as a proxy for transcription-relevant acoustic content.
- Conditional Semantic Accessible Information (CSAI): Measures how much semantic information is accessible from encoder representations, beyond phonetic structure.
Empirical analysis across model families (FireRedASR, Whisper/Voxtral) reveals two characteristic suboptimal regimes:
- Representation Drift: End-to-end LLM gradients drive the encoder toward semantic priors, lowering acoustic fidelity, as evidenced by decreasing PAI and increasing CSAI during joint training.
- Encoder-Weak Interface: Minimal compression at the encoder (high NSE) passes excessive uncertainty to the LLM, which must then resolve distinctions at the phoneme level—leading to inefficient parameter usage.
Capability-Boundary-Aware Training: Methodological Innovations
To address these diagnostic findings, the authors propose a multi-stage training paradigm explicitly designed to enforce modular capability boundaries and optimize entropy allocation:
- Phoneme-Level CTC Pretraining: Initializes the encoder using phoneme-level Connectionist Temporal Classification (CTC), prioritizing acoustically grounded, language-agnostic representations, reducing the risk of premature semantic coupling.
- Iterative Asynchronous SFT (IA-SFT): Introduced between alignment and joint SFT, IA-SFT asynchronously hot-swaps encoder checkpoints according to Centered Kernel Alignment (CKA) scores, enabling the LLM and adaptor to adapt in curriculum fashion to increasingly refined encoder representations while limiting representation drift.
- Joint SFT: Once encoder-LLM alignment is established, end-to-end joint fine-tuning consolidates cross-modal interactions, preserving the acoustic-semantics division of labor.
This process effectively narrows the speech-text modality gap in early stages, while decoupling functional roles and restricting cross-module contamination during optimization.
Empirical Results and Analysis
With only 2.3B parameters, the proposed system achieves results on par with or better than 8B+ parameter baselines across Mandarin, English, and code-switching benchmarks. Notably, the model is robust on entity-dense and dialect benchmarks, underscoring the preservation of acoustic discriminative capacity via elevated PAI values.
Hallucination Mitigation
The methodology yields substantially reduced hallucination rates across all test scenarios. Ablations demonstrate that the IA-SFT stage is critical; its removal increases the encoder’s susceptibility to LLM-driven semantic shortcuts, directly amplifying hallucination risk. This empirical evidence supports the theoretical claim that hallucination is a function of misallocated entropy reduction and contaminated capability boundaries.
Representation Dynamics
Metric trajectory analysis across training stages reveals that, following IA-SFT, the encoder maintains low NSE and high PAI while suppressing CSAI growth in joint SFT, confirming effective acoustic grounding and semantic containment. Layer-wise CKA analysis further shows that the adaptor performs a clean geometrical mapping to the text embedding space without compensating for unstable, high-entropy representations.
Ablation Studies
Controlled ablations of joint SFT and IA-SFT validate their necessity. The encoder hot-swapping mechanism within IA-SFT, acting as regularization by exposing the LLM to diversified yet stable encoder states, is further proven critical for adaptation and generalization.
Theoretical and Practical Implications
This work situates LLM-ASR within an explicit zero-sum entropy budget, clarifying the operational consequences of different allocation strategies:
- From an information-theoretic perspective, the optimal interface maximally reduces uncertainty in ways aligned with module-specific priors—preserving the division between acoustic feature resolution (encoder) and context-driven semantic modeling (LLM).
- Faulty training that ignores this division results in either overreliance on LLM priors (hallucination) or underutilization of available capacity (inefficient scaling in small models).
This capability-boundary-aware strategy has direct implications for parameter efficiency, deployment cost, and real-world robustness. Robust decoupling facilitates modular system updates and supports easier transfer across LLM backbones and varying computational budgets.
Future Directions
Potential extensions include scaling analysis to Large Audio-LLMs (LALMs), which support general audio understanding tasks and have distinct entropy dynamics, and the integration of reinforcement learning, which could further modulate the entropy allocation across modules via experience-driven policy gradients.
Conclusion
By reframing LLM-based ASR as a problem of principled entropy allocation and inter-module capability decoupling, this work provides a formal diagnostic toolkit and practical multi-stage optimization pipeline. The result is an ASR system that is more robust, parameter-efficient, and resistant to hallucination, with improved deployment viability for multilingual and code-switching scenarios (2604.08003).