Phoneme-First Prediction for LLM-Based Speech Recognition

Abstract: Recent research has explored integrating LLMs with speech encoders to create speech-augmented LLMs capable of contextualized speech recognition. The main challenge lies in aligning the semantic embeddings of LLMs with the acoustic representations of speech encoders. We propose a novel approach that teaches the LLM to first predict phonemes from the speech features before generating the final transcript. By integrating a phoneme prediction step directly into the LLM, the model develops a fine-grained knowledge of pronunciation, reducing acoustic confusion and improving transcription accuracy and explainability. Our method is cheap and simple, as phoneme targets can be automatically derived from existing transcripts. Through comprehensive experiments, we show that intermediate phoneme prediction can improve speech recognition, particularly in low-resource settings, and yields outputs that are acoustically more faithful to the speech.

• The paper introduces a phoneme-first prediction strategy that integrates phoneme prediction into LLMs to bridge acoustic and semantic gaps.
• It leverages joint training with both speech-to-text and phoneme-first prompts to significantly reduce word error rates and improve acoustic discrimination.
• Empirical results on LibriSpeech and TED-LIUM datasets demonstrate enhanced explainability and robustness, especially in low-resource settings.

Phoneme-First Prediction for LLM-Based Speech Recognition: A Technical Analysis

Introduction

The integration of LLMs with speech processing systems is redefining approaches to Automatic Speech Recognition (ASR), particularly via speech-augmented LLM frameworks. The core challenge in this paradigm arises from the incongruity between the semantic-centric latent spaces of LLMs and the pronunciation-based embeddings produced by speech encoders. "Phoneme-First Prediction for LLM-Based Speech Recognition" (2606.10864) introduces a method in which the LLM is trained to first predict phonemic sequences prior to word-level transcription. This essay provides a detailed technical analysis of this approach, highlighting empirical results, architectural innovations, and its implications for the future of ASR and multimodal AI systems.

Methodology and Model Architecture

The proposed speech-augmented LLM architecture is composed of three main components:

1. Frozen Speech Encoder: The authors utilize pre-trained encoders such as Whisper and HuBERT to extract robust acoustic features, leveraging large-scale supervised or self-supervised training paradigms.
2. Projection Layer: An MLP-based adapter projects the speech features into the embedding space of the LLM, facilitating their alignment despite pronounced differences in representation.
3. Backbone LLM with Low-Rank Adaptation: Llama-3.1-8B, quantized and fine-tuned via QLoRA, enables efficient adaptation to speech tasks without full-precision updates.

The pipeline is trained end-to-end using the LLM’s next-token cross-entropy loss, updating both the projection layer and low-rank LLM adaptation weights. The phoneme-first prediction strategy modifies the standard speech-to-text (S2T) prompt to instruct the model to output a sequence of phonemes before generating the orthographic transcription. Phoneme labels are systematically derived through forced alignment or G2P models, obviating the need for manual phonetic annotation.

A key aspect is the joint training scheme, which randomly alternates between S2T and phoneme-first prompts, mitigating over-reliance on either task and exposing the LLM to both direct and intermediate transduction scenarios. During inference, either decoding strategy can be used depending on application requirements.

Experimental Evaluation and Results

Low-Resource Regime: LibriSpeech-100h and TED-LIUM-100h

Experiments on LibriSpeech and TED-LIUM subsets with 100 hours of transcribed speech demonstrate that joint phoneme-first prompting consistently reduces Word Error Rate (WER) compared to baseline S2T fine-tuning. Notable findings include:

• On LibriSpeech dev-clean, WER is reduced from 5.9% (S2T) to 4.2% (joint, PF-S2T decode).
• On TED-LIUM, joint training reduces WER by up to 25% compared to S2T.

These improvements are pronounced for clean domains and are strongly attributed to enhanced acoustic discrimination facilitated by the additional phoneme prediction step.

Scaling to Large Datasets: LibriSpeech-960h

With an order of magnitude more data, the method generalizes well across larger LLM quantization ranks and stronger encoders:

• Using Whisper-large and HuBERT-large encoders, joint S2T/PF-S2T training achieves a WER of 2.0–2.3% on clean development/test sets, closely approaching the performance of dedicated ASR models.
• The HuBERT-based system outperforms the Whisper-based system in this setting, indicating that self-supervised encoders, especially when layer aggregation is trainable, are particularly suitable for integrating into LLM pipelines.

While the fully fine-tuned HuBERT CTC model slightly outperforms the speech-augmented LLM in pure ASR, the difference is marginal, and the LLM-based approach demonstrates higher flexibility and potential for generalized language tasks.

Label Quality and Language Generalization

By leveraging the Spoken Dutch Corpus, experiments measured sensitivity to phoneme label quality:

• Joint training with manually annotated phoneme labels leads to lower WERs and Phoneme Error Rates (PERs) than with automatically generated labels, but automatic labels still confer substantial benefits.
• This demonstrates the approach's scalability to settings where manual phonetic transcriptions are unavailable, provided that a pronunciation lexicon and forced alignment tools are accessible.

Acoustic Faithfulness and Error Analysis

Through direct computation of G2P-mapped PER between predicted and reference transcripts, the paper shows that phoneme-first methods systematically reduce phonemic mismatches—even in instances where WER is only modestly improved. Example analyses reveal that model errors under PF-S2T training are more acoustically plausible, with reduced instances of hallucinated content.

Moreover, the intermediate phoneme prediction step yields inherently more explainable outputs, potentially benefiting downstream interpretability in deployment scenarios.

Theoretical and Practical Implications

The phoneme-first approach challenges the current paradigm of directly mapping acoustic features to text in speech-augmented LLMs. By enforcing an intermediate, linguistically meaningful representation, the method:

• Enhances acoustic modeling: The LLM acquires sensitivity to fine-grained pronunciation, which is typically underrepresented in purely semantic embeddings.
• Mitigates acoustic confusion: Homophones and near-homophones are better disambiguated due to explicit phoneme modeling.
• Improves resource efficiency: The approach demonstrates robust learning even with limited labeled data and noisy phoneme supervision.
• Enables explainability: Transcripts can be traced through their corresponding phonemic representations, supporting use cases such as pronunciation feedback, speech assessment, and error analysis.
• Generalizes across languages and encoders: Results hold across English and Dutch, with both supervised and self-supervised encoders.

Future directions could focus on:

• Exploring more fine-grained subword modeling (e.g., articulatory features).
• Adapting the approach to low-resource or zero-resource languages using cross-lingual G2P or unsupervised phoneme discovery.
• Joint training for other downstream speech tasks such as direct speech translation, diarization, or emotion recognition.
• Integration with speech-augmented LLMs for improved contextual biasing and in-context learning capabilities.

Conclusion

Phoneme-first prediction in LLM-based ASR systems introduces a minimal yet highly effective architectural modification that improves both transcription quality and model interpretability. The method achieves statistically significant reductions in both WER and PER, especially in low-resource regimes and across diverse speech encoders. These results substantiate the claim that requiring LLMs to explicitly model phonemic information bridges the representational gap between acoustic and semantic domains. This work provides a strong foundation for the further integration of linguistically motivated intermediate representations into multimodal, speech-grounded LLMs (2606.10864).