Revisiting the Relation Between Language Model Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognition

Abstract: LLM (LM) perplexity (PPL) has historically been used as a proxy for automatic speech recognition (ASR) word error rate (WER), with prior work reporting an approximately linear relation in log-log space. Modern end-to-end ASR systems challenge this assumption because they already contain internal language modeling capacity, are often evaluated without external LLMs, and can now be combined with neural LMs and LLMs through different recognition strategies. This paper revisits the relation between PPL and WER for modern ASR systems. We study whether external LMs still improve current end-to-end ASR systems, whether the PPL-WER relation remains linear in log-log space, how encoder context length affects this relation, and how LLM perplexities fit into the trend observed for standard neural LMs. We further investigate internal language modeling (ILM) in attention-based encoder-decoder systems and show that ILM subtraction changes the observed PPL-WER relation, indicating that the decoder's internal LM must be considered when interpreting the effect of external LM quality.

• The paper demonstrates that the log-log linear correlation between language model perplexity and word error rate persists locally but flattens in high-PPL regimes.
• It shows that restricting encoder context significantly boosts the benefits of external LMs while temperature smoothing decouples PPL changes from WER variations.
• The study finds that applying ILM subtraction in AED systems sharpens the PPL-WER relationship and underscores challenges in tokenization and EOS alignment.

Revisiting the Relation Between LLM Perplexity and ASR Word Error Rate for Modern End-to-End Speech Recognition

Introduction and Context

The empirical correlation between LLM perplexity (PPL) and automatic speech recognition (ASR) word error rate (WER) played a foundational role in shaping recognition system development, especially under the paradigm of hybrid ASR using n-gram LMs. However, with the transition to highly integrated end-to-end (E2E) architectures—particularly Connectionist Temporal Classification (CTC) and attention-based encoder-decoder (AED) models, sometimes augmented with LLMs—the underlying assumptions of this relationship are called into question. This work provides a thorough reevaluation of the PPL-WER relationship on modern E2E ASR, quantifying the residual benefits of external neural LMs, analyzing the log-log linearity of PPL-WER curves, investigating the impact of encoder context and internal LLM (ILM) capacity, and situating different LM classes—including LLMs—within this analysis.

Core Findings: PPL-WER Correlation Across Architectures and LM Types

Log-Log Linearity and Regimes

The canonical log-log linear dependence of WER on PPL, previously documented in hybrid ASR, is demonstrated to persist as a local (regime-wise) property for modern E2E systems. On the LibriSpeech CTC system with Transformer LMs, the relationship displays a higher slope $\alpha$ (sensitivity) in the low-PPL regime and a noticeably flatter one at higher PPLs, i.e., beyond a split point (subword-level PPL $\sim50$).

Figure 1: The log-log relation between subword-level PPL and WER on LibriSpeech CTC demonstrates a strong slope (high sensitivity of WER to PPL) below PPL 50, flattening at higher PPLs.

This pattern is consistent but more pronounced in resource-rich English datasets like LibriSpeech than in the AppTek Spanish CTC system, which presents a much lower slope—particularly above its split point of PPL $\sim450$—due to a reduced ratio of external LM text to supervised acoustic transcriptions. Both n-gram and modern neural LMs (LSTM, Transformer) exhibit monotonically improved WER as PPL drops, but the incremental WER gains from reducing PPL diminish rapidly at high perplexities.

Figure 2: Lower slopes on the AppTek Spanish CTC task reflect reduced impact of external LM quality relative to supervised data scale.

Encoder Context and External LM Utility

Controlling the acoustic context available to the CTC encoder demonstrates that external LMs provide sharply increasing WER improvement as encoder context is restricted. Without access to entire utterances, the encoder's internal language modeling capabilities are impaired, thus increasing the complementarity of external LM priors.

Figure 3: LM gains surge as encoder context shrinks; when context is heavily constrained, the external LM recovers a substantial fraction of recognition performance, though never fully matching full-context baselines.

Temperature Smoothing as a Decoupling Mechanism

Post-hoc temperature adjustment of the LM output proves that PPL can be manipulated independently from WER: tuning the softmax temperature alters PPL dramatically (by nearly a factor of four), while WER remains nearly invariant (within a narrow 0.16% band). Thus, absolute PPL changes driven by calibration/uncertainty are not always predictive of recognition accuracy if the top-hypothesis rankings are unchanged.

Internal LM (ILM) Subtraction in AED Systems

AED decoders act as strong conditional LLMs trained on paired speech-text, thus exhibiting substantial internal language modeling. Applying explicit ILM subtraction during inference increases both the observed PPL-WER correlation and the absolute gain from external LMs. The regime change—where the slope nearly vanishes as external LM PPL exceeds the ILM's own PPL—indicates external LM redundancy in this range.

Figure 4: ILM subtraction in AED recognition steepens the log-log trend in the low-PPL regime and shifts the regime change to align with the ILM's own perplexity.

LLMs in the PPL-WER Framework

Integration of pretrained LLMs (e.g., Qwen2) as external LMs into CTC ASR reveals that naive perplexity comparisons are confounded by vocabulary and EOS conventions. LLMs with large vocabularies exhibit higher (word-level) PPL than conventional Transformer LMs but still reduce WER relative to no-LM baselines. The treatment of EOS tokens significantly alters PPL but only minimally influences WER, underscoring the necessity of standardizing scoring conventions. Full adaptation—i.e., fine-tuning LLMs on task-specific vocabularies and data—closes the WER gap with best-performing LMs.

Figure 5: Despite higher word-level PPL, LLMs trained with EOS alignment and vocabulary adaptation can produce low WERs, matching or exceeding conventional LMs when appropriately fine-tuned.

Theoretical and Practical Implications

The results substantiate that while PPL remains a meaningful proxy for WER, especially for system development and model selection under controlled conditions, its global predictive strength is limited by architecture-specific factors, regime effects, calibration/post-processing, and tokenization dependencies. External LMs provide tangible WER benefits in CTC-based recognition, with especially pronounced gains under constrained encoder context. However, internal LM capacity learned by AED decoders requires careful compensation (ILM subtraction) for external LM improvements to manifest.

For the practical adoption of LLMs in ASR, vocabulary mismatch, EOS convention, and integration methodology (e.g., delayed fusion for mismatched tokenizations) are critical to realizing theoretical perplexity benefits as actual recognition accuracy.

Future Directions

This analytical framework invites further research into architectural design that jointly optimizes internal and external LM capacity in E2E systems, more robust PPL normalization/cross-model comparability (especially in the context of rapidly evolving LLMs and compositional subword frameworks), and strategies for effective ILM estimation/subtraction. Potential exists for adaptive LM fusion conditioned on available acoustic context, and for deeper study of WER sensitivity in domain-mismatched or low-resource settings. Finally, generalization of results to other sequence modeling tasks where LLM priors compete or interact with context encoders would yield broader insights.

Conclusion

This study redefines the practical and theoretical contours of the PPL-WER relationship in modern E2E ASR. Log-log linearity is a robust local property, particularly in strong external LM regimes, but is subject to pronounced flattening in data-rich, ILM-dominated, or post-processed models. External LMs remain essential for CTC and context-restricted models, with variable impact in AED systems subject to ILM subtraction. The interpretability of LLM PPL is highly contingent on consistent tokenization and EOS handling; optimal integration requires substantial adaptation. The overarching conclusion is that PPL, though informative, is neither a universal nor a sufficient statistic for WER under modern ASR architectures, and systematic consideration of model internals and integration strategies is necessary for principled recognition system engineering.