Phonemes vs. Projectors: An Investigation of Speech-Language Interfaces for LLM-based ASR

Abstract: Integrating pretrained speech encoders with LLMs is promising for ASR, but performance and data efficiency depend on the speech-language interface. A common choice is a learned projector that maps encoder features into the LLM embedding space, whereas an alternative is to expose discrete phoneme sequences to the LLM. Using the same encoder and LLM backbones, we compare phoneme-based and vanilla projector-based interfaces in high-resource English and low-resource Tatar. We also propose a BPE-phoneme interface that groups frequent local phoneme patterns while preserving explicit word-boundary cues for phoneme-to-grapheme generation. On LibriSpeech, the phoneme-based interface is competitive with the vanilla projector, and the BPE-phoneme interface yields further gains. On Tatar, the phoneme-based interface substantially outperforms the vanilla projector. We further find that phoneme supervision yields a phoneme-informed hybrid interface that is stronger than the vanilla projector.

• The paper demonstrates that discrete phoneme-based interfaces, particularly with BPE-driven segmentation, outperform continuous projector methods in both high- and low-resource settings.
• Empirical results from LibriSpeech and Tatar show that BPE-phoneme prompts improve WER by leveraging linguistic structure and enhanced LLM contextualization.
• The study highlights the critical role of interface design in ASR pipelines, suggesting that phonetic supervision and hybrid models yield competitive, robust performance.

Speech–Language Interface Design for LLM-based ASR: Phonemes versus Projectors

Introduction

This paper presents a comparative and systematic analysis of continuous (projector-based) versus discrete (phoneme-based) speech–language interfaces within LLM-based ASR architectures. The authors examine data efficiency, recognition accuracy, and the effect of interface type under both high-resource (LibriSpeech) and low-resource (Tatar) conditions, using controlled encoder and LLM backbones. A further contribution is the introduction of the BPE-phoneme interface, which merges frequent local phoneme patterns while preserving word boundaries, seeking to exploit the strengths of both atomic and chunked phonotactics.

Figure 1: Projector-based interfaces use an MLP to align downsampled speech representations with the LLM embedding space, while phoneme-based interfaces employ a CTC-based speech-to-phoneme model followed by phoneme-to-grapheme LLM generation.

Speech–Language Interface Architectures

Continuous interfaces map downsampled encoder states into an LLM's embedding space through a learnable MLP projector. The main challenge arises from the necessity to bridge independently pretrained acoustic and linguistic latent spaces using only limited paired supervision, which can cause robustness issues—especially in low-resource settings.

In contrast, discrete interfaces employ explicit tokenization via an S2P model, producing phoneme sequences that serve as direct prompts to the LLM, which is then trained for phoneme-to-grapheme sequence generation. This decouples pronunciation from orthographic generation, with the linguistic grounding of phonemes providing resilience and cross-lingual generalization. The paper further innovates with a BPE-phoneme variant: BPE is applied to phoneme sequences, segmenting frequent local patterns while explicitly maintaining word boundaries. This improves the quality of the prompt passed to the LLM without simply reducing token sequence length.

Empirical Findings in High-Resource Settings

LibriSpeech evaluations highlight several core results:

• Vanilla projector-based interfaces under fixed, pretrained encoders are consistently weaker than phoneme-based or phoneme-informed hybrid approaches.
• Fine-tuning the speech encoder with phoneme-level supervision prior to projector training yields substantial gains, making the hybrid “phoneme-informed projector” competitive with discrete phoneme-based interfaces.
• The BPE-phoneme interface achieves the best WER (2.78/6.75 on clean/other), slightly outperforming phoneme-informed projectors with the same encoders.
• Scaling the encoder or LLM backbone under the BPE-phoneme interface further reduces WER, supporting the view that LLM capacity is crucial for resolving contextual and lexical ambiguity from phonotactically richer inputs.

Notably, the BPE-phoneme interface does not gain from a shortened LLM input as its prompts actually contain marginally more tokens compared to mono-phoneme baselines, but the boundary-aware segmentation and chunking provide higher-quality intermediate representations.

Findings in Low-Resource Settings

On 20h Tatar (CV-tt), both interfaces operate with the same frozen multilingual Whistle-large encoder. The phoneme-based interface outperforms the projector-based interface by nearly halving WER (from 33.58% to 17.44%) without any Tatar-specific encoder tuning. These results underscore the superior data efficiency and robustness of the linguistically motivated discrete interface in situations where paired supervision is limited or the target language is under-represented in pretraining.

Strong Numerical Claims and Contextualization

The BPE-phoneme interface, combined with scaled encoder and LLM backbones, achieves 1.97%/3.78% WER on LibriSpeech test-clean/other, matching or surpassing recent highly-tuned LLM-ASR baselines that use stronger or larger models and/or more elaborate adaptation protocols. The empirical comparison with Q-Former, SALM, and various LoRA-adapted LLM-ASR paradigms establishes that the phoneme-based discrete interface paradigm yields state-of-the-art competitive performance within the constraints of this architectural family.

Theoretical and Practical Implications

The results formalize a critical observation: the effectiveness of LLM-based ASR frameworks hinges not only on the model capacity of speech encoders and LLMs but, crucially, on the intermediate interface. Phoneme-based (discrete) interfaces provide both practical gains (robustness, data efficiency, cross-lingual transfer) and a theoretically interpretable inductive bias grounded in linguistic structure. The data suggest that the cross-modal alignment achieved by projectors is brittle under limited supervision unless explicitly informed by phonetic structure learned beforehand.

Meanwhile, advanced subword modeling (BPE-phoneme) further maximizes the utility of discrete interfaces, not by prompt-length reduction but by presenting the LLM with segmentation cues aligned to meaningful phonotactics, thus making better use of the generative capacity of LLMs.

Future Directions

Promising future directions include the end-to-end, lexicon-free training of S2P and P2G models (eliminating hand-crafted lexicons), unified architectures that effectively combine discrete and continuous intermediate representations, and extension toward more diverse, challenging, or typologically resource-poor languages. Unifying optimization—possibly via joint training or stochastic marginalization—could further enhance interface robustness, generalization, and transfer.

Conclusion

Discrete phoneme-based interfaces, especially with BPE-driven segmentation, are empirically validated as superior or highly competitive compared to projector-based approaches for LLM-ASR. These findings should inform interface selection in future architectures, especially under constraints of limited data or in multilingual/crosslingual scenarios. Theoretical progress toward lexicon-free training and better end-to-end integration is warranted for further advancement (2604.09332).