Inverse Language Modeling (ILM)

• Inverse Language Modeling (ILM) is a set of techniques that explicitly manages the latent language prior, enabling both bias correction in ASR and input recovery in LLMs.
• In end-to-end ASR, ILM methods subtract or adapt the internal language model to rebalance label distributions and improve decoding performance.
• In large language model contexts, ILM involves prompt inversion and gradient-based reconstruction strategies to recover or optimize the inputs that lead to desired outputs.

Searching arXiv for recent and foundational papers on inverse/internal language modeling relevant to the requested encyclopedia entry. arXiv search query: "inverse language modeling internal LLM estimation RNN-T AED LLM inversion" Inverse Language Modeling (ILM) denotes a set of techniques that make the latent language prior of a sequence model explicit and then either compensate, adapt, discount, or invert it. In end-to-end automatic speech recognition (ASR), the term usually refers to the internal LLM implicitly learned by attention-based encoder–decoder (AED) and RNN-Transducer (RNN-T) systems, together with decoding rules that subtract or control that prior when integrating external LLMs or mitigating hallucinations (Zeineldeen et al., 2021, Zhou et al., 2021, Unni et al., 2022). In more recent LLM work, the same phrase has also been used literally: given a frozen LLM and a desired output, recover prompts or input tokens that could have produced that output, or train the model so that output-conditioned gradients reconstruct the input (Denamganaï et al., 11 Feb 2026, Gabrielli et al., 2 Oct 2025). A plausible implication is that ILM has become an umbrella label for methods that treat the language prior of a sequence model as an explicit object rather than an opaque by-product of training.

1. Terminological scope and conceptual split

In the ASR literature represented here, ILM primarily abbreviates internal LLM or internal LLM estimation. AED work frames the decoder-side prior as $P_{\text{prior}}(W)$, which must be estimated and subtracted when an external LM is fused by Bayes-style density-ratio scoring (Liu et al., 2022). RNN-T work uses the same idea in transducer form: the prediction network behaves as an implicit LM, and ILM correction removes its sequence prior so that an external LM can replace or rebalance it (Zhou et al., 2021). Closely related papers use terms such as ILM correction, ILM subtraction, density ratio, HAT, ILMT, and ILMA (Zhou et al., 2021, Meng et al., 2021).

In the LLM literature represented here, LLM Inversion and Inverse Language Modeling refer to a backward problem. One line formulates inversion as prompt optimization for a frozen autoregressive model,

$$\mathbf{x}^*=\arg\min_{\mathbf{x}\in\mathcal{V}^N} d\bigl(f_\theta(\mathbf{x}),\mathbf{y}^*\bigr),$$

with the objective of finding a prompt that causes the model to generate a target output (Denamganaï et al., 11 Feb 2026). Another line augments ordinary next-token training with an inverse objective that reconstructs input tokens from gradients of the forward loss, thereby coupling robustness and grounding (Gabrielli et al., 2 Oct 2025). These uses are not identical, but both treat the forward LM as invertible, at least approximately.

A plausible synthesis is that the ASR usage is chiefly about removing or reshaping an internal prior, whereas the LLM usage is chiefly about recovering inputs from outputs or output-conditioned signals. The shared mathematical theme is explicit control of the latent sequence prior.

2. Bayesian and architectural foundations in end-to-end ASR

For AED models, the central Bayesian observation is that the posterior can be interpreted as

$P(W\mid X)\propto P(X\mid W)\,P_{\text{prior}}(W),$

where $P_{\text{prior}}(W)$ is the internal language prior learned from training transcripts (Liu et al., 2022). If an external LM $P_{\text{ext}}(W)$ is added naively, the internal prior is counted twice. This leads to the standard ILM-corrected decoding rule

$$\widehat{\mathbf{W}}=\arg\max_{\mathbf{W}}\left[\log P(\mathbf{W}\mid\mathbf{X};\theta^{AED})-\lambda_{\text{ILM}}\log P(\mathbf{W};\theta^{AED})+\lambda_{\text{LM}}\log P(\mathbf{W};\theta^{\text{ext}}_{\text{LM}})\right],$$

in which the estimated internal LM score is subtracted before the external LM is added (Liu et al., 2022).

For AED, an exact internal LM can be defined as a marginal over acoustics,

$$P_{\text{ILM}}(w_1^N)=\sum_{T,x_1^T}P_{\text{AED}}(w_1^N\mid x_1^T)P(x_1^T),$$

but this quantity is intractable (Zeineldeen et al., 2021). Practical ILM estimation therefore replaces the attention context $c_i$ by a surrogate $\hat c_i$, such as zero context, corpus averages, or a learned context derived only from label history (Zeineldeen et al., 2021).

For RNN-T, the internal LM emerges from the separation of the transcription network, prediction network, and joint network. The full model defines

$$rnnt(y_{t,u}\mid y_{<u},x_t)=\text{softmax}\{J(h_t\oplus g_u)\},$$

with $$\mathbf{x}^*=\arg\min_{\mathbf{x}\in\mathcal{V}^N} d\bigl(f_\theta(\mathbf{x}),\mathbf{y}^*\bigr),$$0 and $$\mathbf{x}^*=\arg\min_{\mathbf{x}\in\mathcal{V}^N} d\bigl(f_\theta(\mathbf{x}),\mathbf{y}^*\bigr),$$1 (Unni et al., 2022). Because the prediction network is autoregressive over past non-blank tokens, it functions as an implicit LM. Several papers operationalize this by removing the encoder contribution. One transducer formulation writes the internal LM as the prediction network plus joint network with the encoder term zeroed,

$$\mathbf{x}^*=\arg\min_{\mathbf{x}\in\mathcal{V}^N} d\bigl(f_\theta(\mathbf{x}),\mathbf{y}^*\bigr),$$2

where only non-blank outputs are retained (Meng et al., 2021). Another exposes both an implicit acoustic model and an implicit LLM via the same joint network,

$$\mathbf{x}^*=\arg\min_{\mathbf{x}\in\mathcal{V}^N} d\bigl(f_\theta(\mathbf{x}),\mathbf{y}^*\bigr),$$3

so that acoustics-only and language-only distributions live in the same output space (Unni et al., 2022).

3. Estimation, subtraction, and adaptation of the internal LM

The basic score-level pattern in ASR is stable across AED and RNN-T: start with the end-to-end posterior, add an external LM, and subtract an estimate of the internal LM. In RNN-T form, the decoding objective becomes

$$\mathbf{x}^*=\arg\min_{\mathbf{x}\in\mathcal{V}^N} d\bigl(f_\theta(\mathbf{x}),\mathbf{y}^*\bigr),$$4

which the literature interprets as removing the sequence prior implicitly carried by the transducer (Zhou et al., 2021). The same paper gives a decoding interpretation for two reasons ILM correction helps: rebalancing the label distribution and boosting label probabilities against the blank (Zhou et al., 2021).

For AED, multiple estimators replace the acoustic context by a surrogate context. Zero-attention is the simplest approximation, but encoder-aware surrogates perform better. On Switchboard and LibriSpeech, the trained-context mini-LSTM method produces the strongest results among the compared AED estimators, improving over shallow fusion, density ratio, and average-context variants; on LibriSpeech test-other it reduces WER from 7.59 with shallow fusion to 6.53, and on TED-LIUM V2 test it reduces WER from 19.3 with shallow fusion to 16.9 (Zeineldeen et al., 2021). The same line of work also studies suppression of the internal LM by decreasing decoder capacity, limiting label context, or training the AED with an external LM already present in the criterion (Zeineldeen et al., 2021).

For LAS-style AED models, One-Time Context Learning (OTCL) and Label-Synchronous Context Learning (LSCL) estimate the ILM by replacing the decoder context vector with either a single learned vector or a decoder-state-conditioned feed-forward mapping. These learned ILMs are trained on transcripts to minimize perplexity, and the resulting estimated prior is subtracted during external LM fusion (Liu et al., 2022). LSCL achieves the lowest perplexity in the reported comparisons and outperforms shallow fusion and two previously proposed ILM estimation approaches on several datasets, including cross-domain English, Mandarin medical, and Japanese settings (Liu et al., 2022).

For RNN-T, the same score-level correction appears in several forms. A common framework compares density-ratio correction, zeroed-encoder ILM, averaged-context ILM, and mini-LSTM ILM in a unified transducer setting (Zhou et al., 2021). In that study, mini-LSTM ILM with exact-ILM training provides the best overall performance, reaching 4.6 WER on LibriSpeech test-other and 14.0 WER on TED-LIUM Release 2 test (Zhou et al., 2021). The paper’s theoretical contribution is a generalized HAT-style factorization in which the internal LM becomes exact if a learned label-history function $$\mathbf{x}^*=\arg\min_{\mathbf{x}\in\mathcal{V}^N} d\bigl(f_\theta(\mathbf{x}),\mathbf{y}^*\bigr),$$5 satisfies the additive decomposition assumed by the proof (Zhou et al., 2021).

A different strategy is Internal LLM Adaptation (ILMA). Instead of subtracting the internal LM at decoding time, ILMA explicitly fine-tunes the internal LM with text-only data while freezing the encoder, and regularizes the adapted ILM toward the original via a Kullback–Leibler term (Meng et al., 2021). The method is most effective when only the last linear layer of the joint network is updated. On a 30K-hour Transformer Transducer, ILMT plus ILMA achieves 8.34 WER on Command Control and 3.95 WER on Voice Search, corresponding to up to 34.9% relative WER reduction from the unadapted baseline (Meng et al., 2021). A plausible interpretation is that ILMA shifts the emphasis from subtracting the internal LM to improving it directly.

4. Local and structured control of the internal LM

Global subtraction is not the only way to control the internal LM. One line of work treats internal LM influence as a time-varying quantity that should be reduced only when it conflicts with acoustics. In streaming RNN-T, this is the motivation for Adaptive LM Discounting (AdaptLMD) (Unni et al., 2022). The method has two training modifications: random masking of the prediction-network output with probability $$\mathbf{x}^*=\arg\min_{\mathbf{x}\in\mathcal{V}^N} d\bigl(f_\theta(\mathbf{x}),\mathbf{y}^*\bigr),$$6, and auxiliary ILM and IAM losses. The full objective is

$$\mathbf{x}^*=\arg\min_{\mathbf{x}\in\mathcal{V}^N} d\bigl(f_\theta(\mathbf{x}),\mathbf{y}^*\bigr),$$7

with typical hyperparameters $$\mathbf{x}^*=\arg\min_{\mathbf{x}\in\mathcal{V}^N} d\bigl(f_\theta(\mathbf{x}),\mathbf{y}^*\bigr),$$8 and $$\mathbf{x}^*=\arg\min_{\mathbf{x}\in\mathcal{V}^N} d\bigl(f_\theta(\mathbf{x}),\mathbf{y}^*\bigr),$$9 (Unni et al., 2022).

At decoding time, AdaptLMD discounts the internal LM only when two signals are jointly large: the local rarity of recently predicted tokens and the KL divergence between ILM and IAM. Rarity is defined through a rolling ILM score, and the discount term is

$P(W\mid X)\propto P(X\mid W)\,P_{\text{prior}}(W),$0

The final score for a non-blank token becomes

$P(W\mid X)\propto P(X\mid W)\,P_{\text{prior}}(W),$1

This yields up to 4% relative reductions in overall WER and 14% relative reductions in rare word PER on a conversational, code-mixed Hindi-English ASR task, while a static, utterance-wide discount factor hurts WER (Unni et al., 2022). The paper explicitly characterizes the method as a soft, local inverse-LM approach.

Another line uses architectural factorization to make the internal LM explicit rather than estimated. In the factorized transducer, the non-blank distribution is decomposed into an acoustic component $P(W\mid X)\propto P(X\mid W)\,P_{\text{prior}}(W),$2 and a standalone internal LM $P(W\mid X)\propto P(X\mid W)\,P_{\text{prior}}(W),$3, while blank is modeled separately (Guo et al., 2024). The reported non-blank distribution is

$P(W\mid X)\propto P(X\mid W)\,P_{\text{prior}}(W),$4

and the final posterior multiplies non-blank probabilities by $P(W\mid X)\propto P(X\mid W)\,P_{\text{prior}}(W),$5 (Guo et al., 2024). Because $P(W\mid X)\propto P(X\mid W)\,P_{\text{prior}}(W),$6 is explicit, it can be pre-trained on text-only data, frozen during transducer training, and fused back in decoding with separate weights $P(W\mid X)\propto P(X\mid W)\,P_{\text{prior}}(W),$7 and $P(W\mid X)\propto P(X\mid W)\,P_{\text{prior}}(W),$8:

$P(W\mid X)\propto P(X\mid W)\,P_{\text{prior}}(W),$9

With a well-trained ILM and this decoding rule, the model reports a 17% relative improvement over the standard decoding method on LibriSpeech, and compared to a strong RNN-T baseline enhanced with external LM fusion, the proposed model yields a 5.5% relative improvement on general-sets and an 8.9% WER reduction for rare words (Guo et al., 2024). The same paper adds an ILM-fusion-aware MWER objective to align sequence-discriminative training with the ILM-aware decoding rule (Guo et al., 2024).

5. Literal inversion of LLMs

Outside ASR, inverse language modeling is formulated as an explicit inverse problem for frozen LMs. In LLM Inversion through End-to-End Differentiation, the model is reinterpreted as a differentiable mapping from sequences of distributions over tokens to sequences of distributions over tokens,

$P_{\text{prior}}(W)$0

with soft embeddings $P_{\text{prior}}(W)$1 and Gumbel-Softmax relaxations providing differentiability through prompt tokens and next-token generation (Denamganaï et al., 11 Feb 2026). Prompt inversion is then written as continuous optimization over prompt logits $P_{\text{prior}}(W)$2, augmented with per-position learnable temperatures and Teacher Forcing. The reported system can reliably and efficiently optimize prompts of lengths $P_{\text{prior}}(W)$3 and $P_{\text{prior}}(W)$4 for targets of length $P_{\text{prior}}(W)$5, for several white-box LMs, including SmolLM2-135M and SmolLM3-3B-Base (Denamganaï et al., 11 Feb 2026). Evaluation is based on the token-level Longest Common Subsequence ratio between the target and the LM’s greedy output from the recovered prompt, not on recovering a unique “ground-truth” prompt (Denamganaï et al., 11 Feb 2026).

A different inversion paradigm appears in Inverse Language Modeling towards Robust and Grounded LLMs, where the inverse signal is not a recovered prompt but the gradient of the forward loss with respect to input embeddings (Gabrielli et al., 2 Oct 2025). The general backward-prediction rule is

$P_{\text{prior}}(W)$6

and training combines the usual forward cross-entropy with a backward reconstruction term,

$P_{\text{prior}}(W)$7

with $P_{\text{prior}}(W)$8 in the experiments (Gabrielli et al., 2 Oct 2025). The paper studies Identity, BERT-like, and Inv-First variants, and reports that the practical Identity (grad. direction) variant reduces GCG success rate from 95.9% in the baseline to 82.8%, while keeping forward perplexity close to baseline at 5.08 versus 4.83 (Gabrielli et al., 2 Oct 2025). The BERT-like (grad. value) variant yields 0.8% GCG success rate but is worse at inversion and forward perplexity (Gabrielli et al., 2 Oct 2025). Inversion quality is also reported through token-level and sentence-level metrics; Identity (grad. direction) obtains the best token-level F1 at approximately 21.2% and the highest or tied-highest semantic similarity at approximately 0.30 (Gabrielli et al., 2 Oct 2025).

These two LLM-oriented uses differ from ASR ILM correction. Here the problem is not removal of a prior from a discriminative posterior, but recovery of compatible inputs, prompts, or gradient-encoded triggers. The commonality lies in making the backward structure of the model operational.

6. Limitations, ambiguities, and open directions

The most immediate limitation is terminological. In some papers, ILM means internal LLM; in others, it means inverse language modeling in the literal sense of recovering inputs from outputs (Liu et al., 2022, Denamganaï et al., 11 Feb 2026). For technical reading, disambiguation by task is necessary.

Within ASR, the exact internal LM is usually intractable. AED work states this directly through the marginal-over-acoustics definition, and therefore relies on surrogate contexts $P_{\text{prior}}(W)$9 (Zeineldeen et al., 2021). Encoder-zeroing is also only an approximation; for Transformer and Conformer encoders in LAS, zero-out context gives extremely high perplexity compared with learned context estimators, indicating a poor internal-LM approximation in those architectures (Liu et al., 2022). RNN-T ILMA likewise depends on prior internal LM training; without ILMT, text-only adaptation can degrade WER because the adapted LM no longer cooperates properly with the encoder (Meng et al., 2021).

Local control methods introduce a different limitation profile. AdaptLMD is architecture-agnostic and uses no external text, LM, or additional parameters, but its reported evaluation is on 628 hours of proprietary code-mixed Hindi-English call center speech, and its adaptive rule depends on tuned hyperparameters $P_{\text{ext}}(W)$0 and $P_{\text{ext}}(W)$1 (Unni et al., 2022). The factorized transducer provides an explicit ILM, but its gains depend on careful tuning of the decoding weights $P_{\text{ext}}(W)$2 and $P_{\text{ext}}(W)$3, and the reported results are on LibriSpeech (Guo et al., 2024).

Literal LM inversion has stronger access assumptions. The differentiable inversion framework requires white-box access to the model and the computation graph, and its reported scale reaches models up to 3B parameters, prompt length 80, and target length 20 (Denamganaï et al., 11 Feb 2026). The robustness-oriented inverse training framework is demonstrated on a small decoder-only transformer trained on TinyStories, and explicitly notes that scale to 1B–70B models remains open (Gabrielli et al., 2 Oct 2025). Neither line provides a formal invertibility guarantee (Denamganaï et al., 11 Feb 2026, Gabrielli et al., 2 Oct 2025).

Several open directions are already stated in the cited work. For ASR, proposed directions include combining local discounting with global density-ratio or external LM fusion, using ILM–IAM divergence for OOD detection or confidence estimation, extending learned-context ILM estimation to RNN-T, and combining ILMA with ILM-based shallow fusion (Unni et al., 2022, Liu et al., 2022, Meng et al., 2021). For general LMs, open problems include black-box inversion, inversion under stochastic decoding policies, advanced gradient estimators beyond Gumbel-Softmax, scaling to larger models and longer contexts, and applying inverse-gradient training during instruction tuning or safety fine-tuning (Denamganaï et al., 11 Feb 2026, Gabrielli et al., 2 Oct 2025).

Taken together, the literature suggests two mature but converging viewpoints. One treats inverse language modeling as debiasing and controlling the internal prior of end-to-end ASR systems. The other treats it as recovering or reconstructing causes from generated effects in general-purpose LMs. A plausible implication is that future work will increasingly connect these viewpoints through explicit latent-variable decompositions, local confidence diagnostics, and training objectives that make the backward semantics of sequence models as accessible as their forward predictions.