Mel-LLM: Encoder-Free Speech Language Model

• Mel-LLM is an encoder-free Speech-LLM that feeds pre-processed Mel spectrogram patches directly into a linear-projected LLM, bypassing a dedicated speech encoder.
• The model unifies ASR and TTS using a causal decoder-only backbone, demonstrating competitive performance with simplified architecture.
• It reduces compute and memory overhead by internalizing acoustic processing in the LLM’s lower layers, streamlining training and inference.

Mel-LLM is an encoder-free Speech-LLM that feeds lightly pre-processed Mel spectrogram patches directly into the LLM through a linear projection, allowing the LLM to learn speech-text alignment purely through its own parameters rather than through a dedicated speech encoder. In the formulation reported in "LLM can Read Spectrogram: Encoder-free Speech-Language Modeling" (Fan et al., 8 Jun 2026), the same causal decoder-only backbone is used for automatic speech recognition (ASR) and text-to-speech (TTS), making Mel-LLM a unified autoregressive speech-text model rather than an encoder-augmented LLM.

1. Concept and motivation

Mel-LLM is motivated by the design overhead of conventional Speech-LLMs, which typically use a large pre-trained speech encoder plus a projector. The paper explicitly contrasts this with encoder-free operation, noting that conventional systems may rely on a speech encoder such as Whisper-large at roughly 600M parameters, whereas Mel-LLM removes the Transformer or Conformer encoder altogether and retains only lightweight convolutional downsampling when needed. The intended effect is to reduce compute and memory overhead, eliminate an external encoder bottleneck, and let the LLM’s own lower Transformer layers learn the role of a speech encoder internally (Fan et al., 8 Jun 2026).

The model is framed as a direct test of whether an LLM can learn to read Mel spectrograms without a dedicated speech encoder. This design is also presented as a way to simplify scaling: instead of separately scaling a speech front end and a LLM, the architecture concentrates adaptation inside the LLM backbone. The paper further argues that this can reduce latency and training cost while avoiding representational mismatch between an external acoustic encoder and the LLM.

A plausible implication is that Mel-LLM recasts speech-language modeling as an internal specialization problem for the decoder stack rather than as a cross-module interface problem. The paper’s ablations support this interpretation by showing that lower layers are more important for acoustic processing than upper layers.

2. Input representation and architectural design

The speech input is an 80-dimensional log-Mel spectrogram with mean-variance normalization. The paper gives the usual signal-processing pipeline in terms of STFT, power, and log-Mel features:

$X(k,n) = \sum_m x[m] w[m-nH] e^{-j 2\pi k m / N}$

$$S(f,n) = |X(f,n)|^2,\qquad M(b,n) = \sum_f H_b(f) S(f,n),\qquad \tilde{M}(b,n) = \log(\epsilon + M(b,n))$$

where $H_b(f)$ is the Mel filter bank response and $\epsilon$ is a numerical-stability constant. These Mel features are chunked in time and projected directly into the model space through a single learned linear mapping,

$h = Wm + b$

with no dedicated speech encoder between the spectrogram and the LLM (Fan et al., 8 Jun 2026).

Optional convolutional downsampling reduces sequence length by a factor $r$ in time. The reported settings are $r=8$ for ASR, corresponding to 12.5 Hz speech tokens, and $r=2$ for TTS. The paper evaluates token rates of 100, 50, 25, 12.5, and 6.25 Hz as part of its quality-speed ablation.

The backbone is Phi-4-MM with hidden size $d_{\text{model}} = 3072$, 32 layers, 24 attention heads, and 8 KV heads. Spectrogram embeddings are concatenated with prompt text embeddings and passed to a causal decoder-only LLM. The paper reports LoRA adaptation on attention and MLP linear layers with rank $r=320$ and $$S(f,n) = |X(f,n)|^2,\qquad M(b,n) = \sum_f H_b(f) S(f,n),\qquad \tilde{M}(b,n) = \log(\epsilon + M(b,n))$$0, while the base LLM weights remain frozen. This makes the trainable part relatively small compared with the backbone and locates adaptation in the multimodal interface and decoder layers rather than in a standalone encoder.

3. Training objectives and optimization

For ASR, Mel-LLM uses standard autoregressive next-token prediction rather than CTC or an auxiliary alignment loss. The paper writes the conditional factorization as

$$S(f,n) = |X(f,n)|^2,\qquad M(b,n) = \sum_f H_b(f) S(f,n),\qquad \tilde{M}(b,n) = \log(\epsilon + M(b,n))$$1

with cross-entropy loss

$$S(f,n) = |X(f,n)|^2,\qquad M(b,n) = \sum_f H_b(f) S(f,n),\qquad \tilde{M}(b,n) = \log(\epsilon + M(b,n))$$2

The reported public-data ASR setup uses approximately 31M utterances, or roughly 64k hours, drawn from LibriSpeech, GigaSpeech, MLS English, SPGISpeech, CommonVoice 15 English, VoxPopuli English, TED-LIUM, AMI, Earnings-22, and FLEURS English. Optimization uses AdamW, peak learning rate $$S(f,n) = |X(f,n)|^2,\qquad M(b,n) = \sum_f H_b(f) S(f,n),\qquad \tilde{M}(b,n) = \log(\epsilon + M(b,n))$$3, linear warmup-decay with 9000 warmup steps, gradient clipping at 1.0, effective batch size 512, DeepSpeed ZeRO-1, and 16 H100 GPUs. The paper states that three data sweeps were sufficient, with no gain beyond that point (Fan et al., 8 Jun 2026).

For TTS, Mel-LLM adds a next-token VAE head that generates continuous Mel frames autoregressively. The LLM hidden state is mapped to Gaussian parameters,

$$S(f,n) = |X(f,n)|^2,\qquad M(b,n) = \sum_f H_b(f) S(f,n),\qquad \tilde{M}(b,n) = \log(\epsilon + M(b,n))$$4

followed by residual MLP and Postnet refinement:

$$S(f,n) = |X(f,n)|^2,\qquad M(b,n) = \sum_f H_b(f) S(f,n),\qquad \tilde{M}(b,n) = \log(\epsilon + M(b,n))$$5

The combined TTS loss is

$$S(f,n) = |X(f,n)|^2,\qquad M(b,n) = \sum_f H_b(f) S(f,n),\qquad \tilde{M}(b,n) = \log(\epsilon + M(b,n))$$6

Here $$S(f,n) = |X(f,n)|^2,\qquad M(b,n) = \sum_f H_b(f) S(f,n),\qquad \tilde{M}(b,n) = \log(\epsilon + M(b,n))$$7 is L1 plus MSE on Mel reconstruction, $$S(f,n) = |X(f,n)|^2,\qquad M(b,n) = \sum_f H_b(f) S(f,n),\qquad \tilde{M}(b,n) = \log(\epsilon + M(b,n))$$8 is the KL term from the VAE objective, $$S(f,n) = |X(f,n)|^2,\qquad M(b,n) = \sum_f H_b(f) S(f,n),\qquad \tilde{M}(b,n) = \log(\epsilon + M(b,n))$$9 is binary cross-entropy for end-of-audio prediction, and $H_b(f)$0 is a spectrogram-temporal-smoothness term following MELLE. The TTS training set is Libriheavy 50k hours English, with dropout 0.5 on the input projection and Mel head, $H_b(f)$1, $H_b(f)$2, and $H_b(f)$3.

4. Automatic speech recognition performance

On OpenASR public test sets, the strongest encoder-based baseline remains better in absolute WER, but the encoder-free design is reported as competitive. The paper gives the following average WERs: 5.61% for a pretrained-encoder plus fine-tuning baseline, 6.97% for a random encoder plus fine-tuning baseline, 7.12% for encoder-free Mel-LLM with Phi-4-MM LoRA initialization, and 7.44% for encoder-free Mel-LLM with random initialization (Fan et al., 8 Jun 2026).

The paper emphasizes that multimodal initialization is crucial when data is limited. Moving from random initialization to Phi-4-MM LoRA initialization improves the encoder-free model from 7.44% to 7.12% average WER. On the public benchmarks, the Phi-4-MM initialized model reports per-set WERs of 12.91 on AMI, 12.99 on Earnings22, 10.95 on Gigaspeech, 1.70 on LibriSpeech clean, 4.83 on LibriSpeech other, 2.28 on SPGISpeech, 3.55 on TED-LIUM, and 7.76 on VoxPopuli.

The scaling experiments are central to the paper’s argument. With limited public data, the encoder-initialized system averages 12.52% WER on in-house tests, whereas the encoder-free system averages 13.93%, a relative gap of 11.3%. With approximately ten times more anonymized in-house English data, the encoder-free system improves to 12.99%, reducing the relative gap to 3.8%. This suggests that the encoder-free formulation is more data-sensitive than encoder-initialized systems, but that the gap narrows substantially with scale.

5. Ablations, efficiency, and text-to-speech

The main efficiency ablation studies token rate. At 12.5 Hz, the model reports 7.12% average WER together with a 1.57× speedup over the encoder-based baseline. Higher token rates improve accuracy but reduce speed: 100 Hz yields 6.58% average WER at 0.33× speed, and 50 Hz yields 6.71% at 0.65× speed. The paper presents 12.5 Hz as the best quality-speed trade-off in its setting (Fan et al., 8 Jun 2026).

Layer-freezing experiments are used to localize where speech encoding occurs. Freezing LoRA on layers 28–31 changes average WER only slightly, from 7.12% to 7.40%, and freezing 24–31 gives 7.43%. Freezing deeper blocks harms performance more: 20–31 gives 7.77%, and 16–31 gives 7.94%. The paper interprets this as evidence that lower layers learn acoustic processing while upper layers already carry higher-level language semantics.

For TTS, the results are explicitly described as preliminary. On zero-shot synthesis on LibriSpeech test-clean, Mel-LLM with Phi-4-MM and no MVN reports WER 11.03 and UTMOS 3.10, while the MVN variant reports WER 14.75 and UTMOS 3.25. A low-dropout configuration fails badly, with WER 85.51 and UTMOS 1.38. Additional KL variants are reported: a fixed KL target gives 12.65 WER and 3.22 UTMOS, and sigma-VAE gives 18.07 WER and 3.29 UTMOS.

The limitations are correspondingly direct. TTS is reported as below state of the art, the ASR study is English-only, and encoder-free performance degrades more than encoder-initialized systems when training data is limited. Future directions named in the paper include joint ASR/TTS training, speech-only pretraining, and improvements to the VAE Mel head.

6. Position within the literature and naming ambiguities

The term “Mel-LLM” has been used in more than one sense in adjacent literature, and the distinction matters. In (Fan et al., 8 Jun 2026), Mel-LLM denotes an encoder-free Speech-LLM that reads Mel spectrograms directly. By contrast, "MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning" describes a PEFT method in which “Mel-LLM” refers to applying MELoRA to LLM fine-tuning, not to speech input modeling (Ren et al., 2024). "Mell: Memory-Efficient LLM Serving via Multi-GPU KV Cache Management" is a serving system whose name is pronounced “Mel-LLM,” but it addresses multi-GPU KV-cache management rather than speech-language modeling (Qianli et al., 12 Jan 2025). The Persian benchmark suite MELAC further states that it is not affiliated with, nor derived from, any “Mel-LLM” or similarly named suite (Farsi et al., 1 Aug 2025).

Within speech-language modeling itself, Mel-LLM occupies a clear position relative to encoder-based Speech-LLMs. The paper presents it as simpler, cheaper, and more unified than architectures that combine a dedicated speech encoder with an LLM, while acknowledging that encoder-based systems still achieve stronger absolute ASR accuracy in the reported experiments. Its distinctive claim is therefore not unconditional superiority, but feasibility: an LLM can, under the reported conditions, learn to interpret normalized Mel spectrogram patches directly and support both ASR and TTS in one autoregressive backbone.

A plausible implication is that Mel-LLM reopens the design space for Speech-LLMs by treating acoustic front-end learning as an emergent function of lower decoder layers. In the paper’s formulation, this is the main scientific significance of the model: it shows that direct Mel-to-LLM interfacing is workable, competitive under scaling or multimodal initialization, and extensible to generation as well as recognition.