Internal Language Modeling in ASR

• Internal Language Modeling is the implicit sequence prior learned from paired speech and transcript data that captures textual regularities for effective decoding.
• Estimation methods range from zeroed contexts and mini-LSTM surrogates to adaptive techniques that enable precise subtraction during external LM fusion.
• Adaptive ILM training and adaptation lower perplexity and improve rare-word recognition by better balancing acoustic cues with linguistic priors.

Internal language modeling (ILM) in end-to-end automatic speech recognition denotes the implicit prior over output token sequences that is learned inside the recognizer itself. In this usage, the ASR posterior is not treated as a purely acoustic score: attention-based encoder-decoder (AED) decoders, RNN-Transducer (RNN-T) prediction pathways, and even modern CTC encoders absorb textual regularities from paired speech-transcript training data. ILM research therefore studies how this internal prior should be defined, estimated, subtracted or discounted during external language-model fusion, strengthened or adapted with text-only data, and interpreted in relation to perplexity, word error rate, rare-word recognition, and domain mismatch (Meng et al., 2020, Zeineldeen et al., 2021, Yang et al., 6 Jun 2025, Zeineldeen et al., 6 Jul 2026).

1. Definition and architectural scope

In the strongest theoretical form, ILM is the label-sequence distribution induced by marginalizing the ASR posterior over acoustics. For AED, one formulation is

$$P_{\mathrm{ILM}}(w_1^N)=\sum_{T,x_1^T} P_{\mathrm{AED}}(w_1^N\mid x_1^T)\,P(x_1^T),$$

and for CTC the analogous definition is

$$P_{\mathrm{ILM}}(a_1^S)=\sum_X Pr(X)\,P(a_1^S\mid X).$$

Both papers state that this exact marginalization is intractable, which is why practical ILM work is dominated by estimators rather than direct computation (Zeineldeen et al., 2021, Yang et al., 6 Jun 2025).

The architectural location of ILM depends on the recognition model. In RNN-T-style systems, the internal LM is commonly associated with the prediction network plus joint network once encoder contribution is removed. In AED, it is associated with the autoregressive decoder when attention-derived acoustic context is suppressed. In factorized transducers and MHAT-like models, the linguistic component is made more explicit by construction: non-blank token prediction contains a dedicated ILM term, while acoustic and blank pathways are separated more cleanly (Meng et al., 2021, Meng et al., 2023, Guo et al., 2024).

A recurrent misconception is that ILM is only relevant for explicitly autoregressive decoders. Recent CTC work disputes that view. Although classical CTC assumes label-context independence at the output factorization, modern Conformer-based CTC systems can still learn a context-dependent ILM through powerful encoders, and cross-domain results show that context-dependent ILM estimators outperform context-independent priors (Yang et al., 6 Jun 2025). A related masking-based CTC study argues that contextual encoders exhibit source-domain semantic bias even without an autoregressive decoder, and operationalizes that bias as a pseudo internal LM estimated from masked-acoustic perturbations (Das et al., 2023).

2. Bayesian interpretation and decoding with prior compensation

The central ILM argument is Bayesian. If an end-to-end posterior already contains a language prior, then adding an external LM by shallow fusion risks double counting. This motivates decoding rules of the form

$$\hat{\mathbf{Y}}=\arg\max_{\mathbf{Y}}\left[\log P(\mathbf{Y}\mid \mathbf{X};\theta_{\mathrm{E2E}})+\lambda_E \log P(\mathbf{Y};\theta_{\mathrm{LM}})-\lambda_I \log P(\mathbf{Y};\theta_{\mathrm{ILM}})\right],$$

which is the common structure behind internal LLM estimation (ILME) for RNN-T and AED (Meng et al., 2020, Meng et al., 2021).

For AED, the same idea is often written as a shallow-fusion score augmented with ILM compensation,

$$\log p_{\mathrm{AED}}(a_1^S\mid x_1^T)+\lambda \log p_{\mathrm{LM}}(a_1^S)-\gamma \log p_{\mathrm{ILM}}(a_1^S),$$

where $\lambda$ scales the external LM and $\gamma$ scales ILM subtraction. When $\gamma=0$, the rule reduces to ordinary shallow fusion. The 2026 AED study treats this subtraction term as essential for interpreting the relation between external-LM perplexity and ASR error, because otherwise the external LM is fused with a score that already contains a decoder prior (Zeineldeen et al., 6 Jul 2026).

A unified RNN-T analysis gives two decoding-level reasons for the gains from ILM correction. First, prior removal rebalances the label distribution so that the external LM contributes more directly to context modeling. Second, dividing by the ILM boosts non-blank labels against the usually high blank probability, permitting larger external-LM weights without inducing severe deletion behavior (Zhou et al., 2021). That paper verifies the two effects separately: a length reward mainly simulates the blank-suppression component, while a blank-preserving renormalization mainly simulates label rebalancing.

One extension is adaptive rather than fixed prior compensation. ILME-ADA compares scaled internal-LM and external-LM log-likelihoods at each decoding step and chooses the larger one. If the scaled ILM score dominates, the subtraction and addition cancel and decoding falls back to the baseline E2E objective; if the external LM dominates, decoding reduces to standard ILME-style fusion. In Mandarin RNN-T and LAS domain adaptation, this adaptive rule gave substantially better target-domain CER with much smaller degradation on the general domain than either shallow fusion or fixed ILME, especially with n-gram LMs (Ma et al., 2022).

3. Estimation methods across AED, RNN-T, and CTC

Because exact acoustic marginalization is intractable, ILM research is largely a study of estimators.

For AED, the standard estimator family replaces the decoder context vector $c_i$ with a surrogate $\hat c_i$. Early baselines used zero context, but several papers report that this is too crude. One AED study showed that using encoder-bias information is better than masking the input representation: global attention-context averages, global encoder averages, and especially a trained mini-LSTM context generator all outperformed zero-attention and density-ratio baselines, with the mini-LSTM giving the best WER and the lowest ILM perplexity among the valid prior estimators it studied (Zeineldeen et al., 2021). A later LAS paper proposed explicit context-vector learning. OTCL replaces every decoder context vector with a single learned vector, while LSCL predicts a dynamic context vector from the decoder query/state by a lightweight feed-forward network. On BLSTM, Transformer, and Conformer encoders, LSCL achieved the lowest held-out transcript perplexities—$235$, $$P_{\mathrm{ILM}}(a_1^S)=\sum_X Pr(X)\,P(a_1^S\mid X).$$0, and $$P_{\mathrm{ILM}}(a_1^S)=\sum_X Pr(X)\,P(a_1^S\mid X).$$1, respectively—whereas zero-out context gave $$P_{\mathrm{ILM}}(a_1^S)=\sum_X Pr(X)\,P(a_1^S\mid X).$$2, $$P_{\mathrm{ILM}}(a_1^S)=\sum_X Pr(X)\,P(a_1^S\mid X).$$3, and $$P_{\mathrm{ILM}}(a_1^S)=\sum_X Pr(X)\,P(a_1^S\mid X).$$4, indicating that learned context surrogates are far better matched to the frozen decoder than naive zeroing (Liu et al., 2022).

For RNN-T, a common approximation is to remove encoder contribution and reuse the prediction network and joint network as a text-only scorer. Several alternatives then differ in how closely that scorer is tied to the transducer. Density-ratio methods use a standalone source LM trained on transcripts; zero-encoder methods derive the ILM directly from the transducer; low-order density ratio replaces the usual strong source LM by a deliberately weak low-order model, based on the claim that the RNN-T ILM is weak and low-order rather than a full-context neural LM. In LibriSpeech in-domain results, the source-trained DR LM had the lowest transcript perplexity among the compared ILM estimates, but ILME still achieved the best WER and LODR stayed close, supporting the claim that lower perplexity does not imply a better match to the transducer’s internal prior (Zheng et al., 2022). A complementary theoretical paper later generalized the HAT-style decomposition and introduced exact-ILM training via a learned acoustic-independent logit term $$P_{\mathrm{ILM}}(a_1^S)=\sum_X Pr(X)\,P(a_1^S\mid X).$$5, improving history-dependent mini-LSTM ILM estimation (Zhou et al., 2021).

CTC required separate developments. One line of work derives prefix-conditioned next-label posteriors from CTC prefix probabilities and distills them into a small autoregressive LM. In TED-LIUM2 cross-domain evaluation, label-level knowledge distillation with smoothing reduced test WER from $$P_{\mathrm{ILM}}(a_1^S)=\sum_X Pr(X)\,P(a_1^S\mid X).$$6 with shallow fusion to $$P_{\mathrm{ILM}}(a_1^S)=\sum_X Pr(X)\,P(a_1^S\mid X).$$7, more than $$P_{\mathrm{ILM}}(a_1^S)=\sum_X Pr(X)\,P(a_1^S\mid X).$$8 relative, and outperformed context-independent frame priors and unigrams. The same study argues that these gains are evidence that modern CTC models learn a context-dependent ILM despite the classical label-context-independence assumption (Yang et al., 6 Jun 2025). Another line of work estimates a pseudo ILM by iteratively masking equal acoustic partitions, accumulating masked log-posteriors only at timesteps whose posteriors change substantially, and subtracting that pseudo-likelihood during decoding. Across multiple out-of-domain datasets, this masking-based CTC ILME improved WER by up to $$P_{\mathrm{ILM}}(a_1^S)=\sum_X Pr(X)\,P(a_1^S\mid X).$$9 and OOV F1 by up to $$\hat{\mathbf{Y}}=\arg\max_{\mathbf{Y}}\left[\log P(\mathbf{Y}\mid \mathbf{X};\theta_{\mathrm{E2E}})+\lambda_E \log P(\mathbf{Y};\theta_{\mathrm{LM}})-\lambda_I \log P(\mathbf{Y};\theta_{\mathrm{ILM}})\right],$$0 relative to shallow fusion when target-domain text was available; in zero-shot adaptation it still improved WER by up to $$\hat{\mathbf{Y}}=\arg\max_{\mathbf{Y}}\left[\log P(\mathbf{Y}\mid \mathbf{X};\theta_{\mathrm{E2E}})+\lambda_E \log P(\mathbf{Y};\theta_{\mathrm{LM}})-\lambda_I \log P(\mathbf{Y};\theta_{\mathrm{ILM}})\right],$$1 relative (Das et al., 2023).

4. Training, adaptation, and explicit factorization of the internal LM

A major shift in the literature is from estimating ILM at inference time to training or adapting the internal LM directly.

Internal LM training (ILMT) adds an auxiliary ILM loss to ordinary end-to-end training, but applies that loss only to the components used later as the internal LM estimate. For RNN-T this means the prediction and joint networks; for AED it means the decoder. The resulting objective is

$$\hat{\mathbf{Y}}=\arg\max_{\mathbf{Y}}\left[\log P(\mathbf{Y}\mid \mathbf{X};\theta_{\mathrm{E2E}})+\lambda_E \log P(\mathbf{Y};\theta_{\mathrm{LM}})-\lambda_I \log P(\mathbf{Y};\theta_{\mathrm{ILM}})\right],$$2

In 30K-hour RNN-T and AED experiments, ILMT reduced RNN-T ILM perplexity from $$\hat{\mathbf{Y}}=\arg\max_{\mathbf{Y}}\left[\log P(\mathbf{Y}\mid \mathbf{X};\theta_{\mathrm{E2E}})+\lambda_E \log P(\mathbf{Y};\theta_{\mathrm{LM}})-\lambda_I \log P(\mathbf{Y};\theta_{\mathrm{ILM}})\right],$$3 to $$\hat{\mathbf{Y}}=\arg\max_{\mathbf{Y}}\left[\log P(\mathbf{Y}\mid \mathbf{X};\theta_{\mathrm{E2E}})+\lambda_E \log P(\mathbf{Y};\theta_{\mathrm{LM}})-\lambda_I \log P(\mathbf{Y};\theta_{\mathrm{ILM}})\right],$$4 and AED ILM perplexity from $$\hat{\mathbf{Y}}=\arg\max_{\mathbf{Y}}\left[\log P(\mathbf{Y}\mid \mathbf{X};\theta_{\mathrm{E2E}})+\lambda_E \log P(\mathbf{Y};\theta_{\mathrm{LM}})-\lambda_I \log P(\mathbf{Y};\theta_{\mathrm{ILM}})\right],$$5 to $$\hat{\mathbf{Y}}=\arg\max_{\mathbf{Y}}\left[\log P(\mathbf{Y}\mid \mathbf{X};\theta_{\mathrm{E2E}})+\lambda_E \log P(\mathbf{Y};\theta_{\mathrm{LM}})-\lambda_I \log P(\mathbf{Y};\theta_{\mathrm{ILM}})\right],$$6, and ILMT combined with ILME-based inference achieved up to $$\hat{\mathbf{Y}}=\arg\max_{\mathbf{Y}}\left[\log P(\mathbf{Y}\mid \mathbf{X};\theta_{\mathrm{E2E}})+\lambda_E \log P(\mathbf{Y};\theta_{\mathrm{LM}})-\lambda_I \log P(\mathbf{Y};\theta_{\mathrm{ILM}})\right],$$7 relative WER reduction from standard E2E training with shallow fusion on out-of-domain LibriSpeech (Meng et al., 2021).

Internal LLM adaptation (ILMA) goes further by fine-tuning the internal LM with text-only data so that no external LM is needed at inference. For transformer transducers, the key design choice was to update only the non-blank output rows of the joiner, because text-only data provide no blank supervision and modifying components that affect blank probability can harm alignment. In 30K-hour transformer-transducer experiments, ILMA achieved up to $$\hat{\mathbf{Y}}=\arg\max_{\mathbf{Y}}\left[\log P(\mathbf{Y}\mid \mathbf{X};\theta_{\mathrm{E2E}})+\lambda_E \log P(\mathbf{Y};\theta_{\mathrm{LM}})-\lambda_I \log P(\mathbf{Y};\theta_{\mathrm{ILM}})\right],$$8 relative WER reduction from the unadapted baseline, and the most effective setting updated only the non-blank joiner output layer (Meng et al., 2021).

JEIT extends this logic to joint training with large-scale unpaired text. It optimizes the normal E2E loss on paired speech and an ILM cross-entropy loss on unpaired text in one stage, so that the paired-data objective regularizes the strengthened internal LM and obviates the separate Kullback-Leibler regularization required by ILMA-style adaptation. On HAT and MHAT, JEIT improved rare-word recognition, and with 100B unpaired sentences JEIT/CJJT improved rare-word recognition accuracy by up to $$\hat{\mathbf{Y}}=\arg\max_{\mathbf{Y}}\left[\log P(\mathbf{Y}\mid \mathbf{X};\theta_{\mathrm{E2E}})+\lambda_E \log P(\mathbf{Y};\theta_{\mathrm{LM}})-\lambda_I \log P(\mathbf{Y};\theta_{\mathrm{ILM}})\right],$$9 over a model trained without unpaired text (Meng et al., 2023).

Factorized transducer work treats ILM as an explicit branch rather than only an estimated hidden prior. In the factorized model, non-blank prediction is decomposed into an acoustic log-probability $$\log p_{\mathrm{AED}}(a_1^S\mid x_1^T)+\lambda \log p_{\mathrm{LM}}(a_1^S)-\gamma \log p_{\mathrm{ILM}}(a_1^S),$$0 and an internal LM log-probability $$\log p_{\mathrm{AED}}(a_1^S\mid x_1^T)+\lambda \log p_{\mathrm{LM}}(a_1^S)-\gamma \log p_{\mathrm{ILM}}(a_1^S),$$1, while blank probability is modeled separately. That paper argues that explicitness alone is insufficient: the ILM branch must be pretrained on text-only data, decoded with a score that uses $$\log p_{\mathrm{AED}}(a_1^S\mid x_1^T)+\lambda \log p_{\mathrm{LM}}(a_1^S)-\gamma \log p_{\mathrm{ILM}}(a_1^S),$$2 both inside and outside the non-blank softmax, and ideally optimized with an ILM-fusion-aware MWER objective. On LibriSpeech, the proposed decoding rule delivered a $$\log p_{\mathrm{AED}}(a_1^S\mid x_1^T)+\lambda \log p_{\mathrm{LM}}(a_1^S)-\gamma \log p_{\mathrm{ILM}}(a_1^S),$$3 relative improvement over standard factorized-transducer decoding, and the resulting system exceeded a strong RNN-T plus external-LM shallow-fusion baseline by $$\log p_{\mathrm{AED}}(a_1^S\mid x_1^T)+\lambda \log p_{\mathrm{LM}}(a_1^S)-\gamma \log p_{\mathrm{ILM}}(a_1^S),$$4 relative on general sets and by $$\log p_{\mathrm{AED}}(a_1^S\mid x_1^T)+\lambda \log p_{\mathrm{LM}}(a_1^S)-\gamma \log p_{\mathrm{ILM}}(a_1^S),$$5 on rare-word WER without any external LM at inference (Guo et al., 2024).

5. Empirical phenomena: perplexity, saturation, rare words, and sequence training

One of the most developed empirical findings is that ILM changes how external-LM perplexity maps to ASR error. In a 2026 study on Conformer AED for LibriSpeech dev-other, the relation between subword-level external-LM perplexity and WER remained piecewise linear in log-log space, but ILM subtraction substantially steepened the low-perplexity slope: the fitted low-PPL slope changed from $$\log p_{\mathrm{AED}}(a_1^S\mid x_1^T)+\lambda \log p_{\mathrm{LM}}(a_1^S)-\gamma \log p_{\mathrm{ILM}}(a_1^S),$$6 without ILM subtraction to $$\log p_{\mathrm{AED}}(a_1^S\mid x_1^T)+\lambda \log p_{\mathrm{LM}}(a_1^S)-\gamma \log p_{\mathrm{ILM}}(a_1^S),$$7 with ILM subtraction, while the high-PPL slopes stayed near zero at $$\log p_{\mathrm{AED}}(a_1^S\mid x_1^T)+\lambda \log p_{\mathrm{LM}}(a_1^S)-\gamma \log p_{\mathrm{ILM}}(a_1^S),$$8 and $$\log p_{\mathrm{AED}}(a_1^S\mid x_1^T)+\lambda \log p_{\mathrm{LM}}(a_1^S)-\gamma \log p_{\mathrm{ILM}}(a_1^S),$$9. The same paper reported the AED’s internal LM perplexity as $\lambda$0, very close to the empirical break point around PPL $\lambda$1. Its interpretation is that once the external LM becomes weaker than, or comparable to, the decoder’s own internal LM, WER saturates and differences among weaker external LMs matter little (Zeineldeen et al., 6 Jul 2026).

The same work contrasts AED with CTC. On LibriSpeech dev-other, the fitted CTC slope dropped from $\lambda$2 below PPL $\lambda$3 to $\lambda$4 above PPL $\lambda$5, so the high-perplexity regime still benefited from external-LM quality. AED, with a strong decoder prior, showed much stronger saturation. The paper also showed that weakening CTC encoder context made external LM gains much larger: with full context, the best Transformer LM improved average dev-other/test-other WER from $\lambda$6 to $\lambda$7, whereas with only $\lambda$8 s of context, LM-assisted decoding reduced WER from $\lambda$9 to $\gamma$0. This suggests that internal sequence-modeling strength, whether decoder-side or encoder-side, directly controls how much external LM quality can be observed in WER (Zeineldeen et al., 6 Jul 2026).

Another important empirical relation is between ILM subtraction and sequence discriminative training. For neural transducers, a theoretical derivation shows that the global optimum of MMI training with integrated LM has the form of the empirical posterior divided by an LM prior, up to renormalization and exponentiation, which is structurally similar to ILM subtraction. Empirically, on LibriSpeech full-context RNN-T, CE training plus shallow fusion gave test-other WER $\gamma$1; CE plus ILM subtraction reduced it to $\gamma$2; but MMI or MBR fine-tuning with shallow fusion already reached about $\gamma$3, leaving only negligible additional gain for ILM subtraction. That paper further showed that sequence discriminative training had little effect on the commonly used zero-encoder ILM estimate after blank renormalization, but it did reshape both encoder and prediction/joint behavior, including blank suppression (Yang et al., 2023).

Rare-word and hallucination behavior provide a separate diagnostic. AdaptLMD studies RNN-T from the perspective that the prediction network acts as an overconfident internal LM which can override acoustics, especially for rare or out-of-domain words. It estimates an implicit acoustic model and an ILM by masking one side or the other before the joint network, then discounts the ILM adaptively based on recent token rarity and the KL divergence between ILM and IAM distributions. On conversational code-mixed Hindi-English ASR, this reduced overall WER by up to $\gamma$4 and rare-word PER by up to $\gamma$5 relative, supporting the claim that internal language priors can directly cause acoustically inconsistent hallucinations (Unni et al., 2022).

6. Limitations, ambiguity of the acronym, and adjacent usages

Across the ASR literature, ILM remains an estimated object rather than an exactly recovered one. The common derivations rely on approximate decompositions of end-to-end posteriors, such as removing encoder contribution in RNN-T or zeroing attention context in AED, and several papers explicitly note that these are not exact probabilistic factorizations (Meng et al., 2020, Meng et al., 2021). AED work likewise reports that estimator quality varies strongly with architectural details, that zero-context methods can be badly mismatched to Transformer and Conformer decoders, and that some studies establish the phenomenon more clearly than the estimator design space itself (Liu et al., 2022, Zeineldeen et al., 6 Jul 2026).

Estimator quality also conditions downstream conclusions. ILME-ADA shows that adaptive fusion is only as good as the ILM estimate it compares against; in LAS, zeroing the context vector was markedly weaker than AvgH, LSCL, or OTCL under a general-domain-degradation constraint (Ma et al., 2022). For CTC, one paper reports that ILM perplexity is not predictive of ASR performance, so checkpoint selection must still rely on development-set WER rather than ILM PPL alone (Yang et al., 6 Jun 2025). These results suggest that ILM research is less about identifying a single canonical estimator than about matching the estimator to the model family and the decoding objective.

The acronym itself is overloaded. In a distinct literature on language evolution and emergent communication, “ILM” denotes the Iterated Learning Model or Iterated Learning Method rather than internal language modeling. There it refers to tutor-pupil transmission through a bottleneck, used to study compositionality, language contact, and neural sender-receiver systems (Bullock et al., 2024, Perkins, 2021). That usage is conceptually separate from ASR ILM, even though both concern the interaction between learned linguistic structure and transmission constraints.

Taken together, the ASR papers suggest a stable interpretation. ILM is the model-internal sequence prior that modern end-to-end recognizers learn from transcripts; external-LM fusion is therefore a prior-replacement problem rather than simple score addition; and the practical value of perplexity, LM fusion, or domain adaptation depends on how accurately that internal prior is estimated, discounted, trained, or factorized (Zeineldeen et al., 6 Jul 2026).