LLM can Read Spectrogram: Encoder-free Speech-Language Modeling

Abstract: Recent speech-aware LLMs (Speech-LLMs) rely on a pre-trained speech encoder to convert audio into semantic-rich representations consumable by LLM. In this work, instead, we explore: can an LLM learn to read Mel spectrogram directly without a dedicated speech encoder? We propose Mel-LLM, an encoder-free Speech-LLM that feeds lightly pre-processed Mel spectrogram patches directly into the LLM through a linear projection, allowing the LLM to learn speech-text alignment purely through its own parameters. We conduct extensive experiments on both automatic speech recognition (ASR) and text-to-speech (TTS) tasks. For ASR, we evaluate on the OpenASR leaderboard public sets and production-level scaling experiments, demonstrating that the encoder-free solution achieves competitive performance with only limited degradation compared to encoder-initialized counterparts. We find that when data is limited, initialization from a multimodal checkpoint (Phi-4-MM) is crucial for maintaining performance. We also present ablation studies revealing which LLM layers are less relevant to speech encoding. For TTS, we show preliminary results with a next-token VAE approach. While TTS performance is not yet optimal, these results establish the feasibility of a fully unified encoder-free architecture for autoregressive speech-text modeling.

• The paper demonstrates that a large language model can achieve competitive ASR and TTS performance by directly ingesting Mel spectrograms, bypassing conventional speech encoders.
• It leverages multimodal pretraining and layer adaptation, revealing that only lower LLM layers require adjustment for effective speech feature encoding.
• Results highlight improved computational efficiency and reduced architectural complexity, paving the way for unified speech-text modeling applications.

Encoder-Free Speech-Language Modeling with Direct Mel Spectrogram Input

Introduction

The paper "LLM Can Read Spectrogram: Encoder-Free Speech-Language Modeling" (2606.10231) challenges the canonical Speech-LLM paradigm, which integrates a dedicated pre-trained speech encoder into LLMs by replacing the encoder with direct ingestion of Mel spectrogram patches via linear projection. This approach, termed Mel-LLM, interrogates whether LLMs with sufficient parameter capacity can implicitly learn both speech encoding and cross-modal alignment tasks, potentially enabling unified architectures for both automatic speech recognition (ASR) and text-to-speech (TTS).

Motivation and Contributions

Traditional Speech-LLMs rely on a modular architecture: a heavy speech encoder (e.g., Whisper, Conformer), a modality projector, and an LLM backbone. This strategy incurs substantial computational and architectural overhead and may introduce a representational bottleneck due to limited information flow from the compressed encoder output. Recent advancements in vision-language modeling have demonstrated that encoder-free approaches—directly projecting raw image patches into LLMs—are viable and competitive. Drawing on these insights, this work investigates the analogous design in the audio domain and advances the following contributions:

1. Direct ASR from Mel Spectrogram: Demonstrates that an LLM can learn to process raw speech features, performing ASR without a speech encoder and attaining competitive performance on OpenASR benchmarks.
2. Scalability and Degradation Analysis: Quantifies the performance degradation of encoder-free models vs. encoder-initialized baselines, revealing that the gap is largely mitigated at scale with sufficient training data.
3. Multimodal Pretraining Insights: Identifies that initialization from multimodal checkpoints, specifically Phi-4-MM, is crucial for data-efficient training and for both ASR and TTS convergence.
4. Layerwise Speech Encoding Analysis: Provides ablation evidence that only the lower layers of the LLM are primarily responsible for speech feature encoding.
5. Preliminary Unified TTS Architecture: Introduces a next-token VAE-based decoder coupled to the LLM, yielding a fully unified encoder-free speech-text modeling pipeline.

Methodology

Encoder-Free Mel-LLM Design

The Mel-LLM architecture eliminates all transformer/conformer encoder blocks from the speech branch. Only a simple convolutional submodule for potential downsampling is retained before a linear projection injects (possibly downsampled) Mel spectrogram sequences directly into the LLM's embedding space. The LLM, built atop the Phi-4-MM backbone adaptation, is trained via LoRA with frozen base weights, allowing most model adaptation to occur in the low-rank modules.

For ASR, audio is chunked and linearly projected, concatenated with an optional text prompt embedding, and autoregressively decoded as text by the LLM. For TTS, text input is processed by the LLM to generate autoregressive continuous Mel spectrograms, which are further refined by a post-VAE and postnet module, bypassing the need for discrete tokens or vector quantization.

Training and Optimization

All ASR and TTS experimentation utilizes public data exclusively. ASR is evaluated over approximately 31M utterances, utilizing staged finetuning, downsampling factor ablation, and LoRA layerwise freezing. Phi-4-MM multimodal pretraining is leveraged where specified, and all non-LoRA weights are frozen during adaptation. TTS tasks utilize Libriheavy and apply significant data augmentation and regularization to mitigate exposure bias and increase generation fidelity.

Results and Analysis

ASR: Performance and Scaling Properties

The encoder-free Mel-LLM model initialized from Phi-4-MM achieves a mean WER only marginally worse (7.12% vs 6.97%) than the best encoder-initialized baseline on OpenASR public sets. Random initialization (no multimodal pretraining) increases the gap, highlighting the importance of inductive bias during cross-modal alignment when data is limited. Notably, the relative performance difference diminishes to 3.8% when extensive amounts of internal training data are used, with the Mel-LLM eventually converging near encoder-based models given sufficient scale.

The ablation on downsampling rate finds that aggressive downsampling (12.5Hz tokens) offers substantial speedups (up to 1.57× vs encoder-based) with minimal accuracy trade-off, positioning encoder-free modeling as both efficient and pragmatic for deployment.

Layer freezing experiments further reveal that only the lower ~24 layers in the 32-layer LLM require modality-specific adaptation; upper layers (24–31) can remain fixed from task-agnostic pretraining, consistent with similar claims for vision-LLMs.

TTS: Feasibility and Initialization Dependence

For TTS, Mel-LLM with a VAE decoder trained from scratch fails to generate audible output unless initialized from a multimodal checkpoint. When properly initialized, the system produces intelligible speech with moderate WER (11.03%) and UTMOS perceptual scores typical of direct-spectrogram autoregressive architectures. Dropout regularization is found to be critical for successful AR training; insufficient regularization precipitates catastrophic failure modes due to exposure bias. Ablations on normalization and KL divergence variants offer marginal trade-offs in ASR and perceptual quality.

Contradictory and Notable Claims

• Contradicts the necessity of modular encoders: The work empirically refutes the assumption that a pre-trained speech encoder is essential for high-performing ASR within large language modeling systems, at least once sufficiently strong pretraining and/or scaling are leveraged.
• Layer specialization emerges: The claim that only lower LLM layers require adaptation for speech processing, with upper layers already well-tuned for language modeling, is strongly supported by layerwise freezing results.

Practical and Theoretical Implications

From a deployment perspective, Mel-LLM's architectural simplicity (no large speech encoder) directly translates into computational efficiency, parameter reduction, and potential latency advantages. Furthermore, this model offers a unified backbone for speech and text tasks, facilitating cross-domain and cross-modal research (e.g., direct speech-to-speech translation, multi-modal dialogue) within a single model.

Theoretically, these results suggest that sufficiently LLMs possess latent capacity for modality-specialized processing, provided adequate pretraining or data, raising questions about the optimal organization and reuse of LLM layers for multimodal, encoderless learning.

The results also highlight the importance of robust initialization in data-scarce regimes—mirror findings across other multimodal research efforts—further motivating the curation of high-coverage multimodal pretraining corpora and checkpoints.

Future Directions

Immediate open tasks include joint multi-task training on ASR and TTS for improved speech-only representation learning, extending to unified S2ST, and further optimization of autoregressive spectrogram generation (e.g., by exploring diffusion or representation-forcing objectives). More aggressive scaling, sharing across more modalities, and testing in non-English, low-resource, or real-time settings form promising avenues for further investigation.

Conclusion

This work demonstrates that encoder-free Speech-LLMs, which ingest Mel spectrograms directly via a simple projection, are viable for both ASR and TTS as long as sufficient pretraining and/or scale is present. The paradigm reduces architectural complexity while retaining or approaching state-of-the-art performance. Detailed analysis reveals that LLM lower layers adapt to audio processing tasks, while high layers remain domain-agnostic. These findings point toward simplified, unified modeling pathways for future speech and multimodal research.