Lexical Surprisal: Theory & Applications

• Lexical surprisal is an information-theoretic metric that measures the processing cost of a word using its negative log probability given preceding context.
• It bridges probabilistic language modeling with human behavioral and neural data, effectively predicting reading times and ERP responses such as the N400 and P600.
• Recent advances decompose surprisal into shallow and deep components, enhancing its application in multilingual, neurocognitive, and morphologically complex language studies.

Lexical surprisal is a foundational information-theoretic construct in computational psycholinguistics and cognitive neuroscience, quantifying the processing cost incurred by comprehenders when encountering a word in context. Defined formally as the negative logarithm of a word’s conditional probability given its preceding linguistic context, lexical surprisal underpins a diverse range of modeling efforts aiming to align probabilistic LLMs with human behavioral and neural data. Across traditions, lexical surprisal provides a rigorously specified bridge linking statistical language modeling, psycholinguistic theories of prediction, incremental comprehension, cognitive workload, and brain electrophysiology.

1. Formalization of Lexical Surprisal

Let $c$ denote the preceding context (typically a sequence of words) and $w$ the next word. Lexical surprisal $S(w\,|\,c)$ is defined by:

$S(w\,|\,c) = -\log P(w\,|\,c)$

where $P(w\,|\,c)$ is the context-sensitive probability yielded by a probabilistic LLM, such as a neural autoregressive LM or an $n$-gram model. This definition is agnostic to the granularity of the linguistic units (word, subword, morpheme), modulo the need to aggregate sub-token surprisal via the chain rule when applicable (Oh et al., 2024, Nair et al., 2023). In information-theoretic terms, surprisal directly corresponds to an instantaneous information content and is measured in bits if the logarithm is base 2, or in nats for the natural logarithm.

Surprisal can be interpreted as the Kullback–Leibler divergence between a comprehender’s prior and the posterior over words immediately after observing $w$: $D_{KL}[p_\infty \Vert p_0] = S(w\,|\,c)$ (Li et al., 2024). The expectation of surprisal across all possible words (under $P(w\,|\,c)$) recovers the Shannon entropy: $H(c) = -\sum_w P(w\,|\,c) \log P(w\,|\,c)$ (Černý et al., 8 Jan 2026, Wilcox et al., 2023).

2. Surprisal as a Predictor of Human Processing Cost

Decades of empirical research have established that lexical surprisal is a strong, linear (in bits) predictor of incremental human language processing difficulty as measured by reading times, self-paced reading, eye movement latencies, and event-related brain potentials (ERPs) (Wilcox et al., 2023, Oh et al., 2022, Schijndel et al., 2018).

Key quantitative findings include:

Language	Surprisal effect (ms/bit)	Source
English	3.2	(Wilcox et al., 2023)
Dutch, Russian	3.8	(Wilcox et al., 2023)
Finnish, Turkish	3.0–3.1	(Wilcox et al., 2023)

Linear mixed-effects models consistently show that including wordwise surprisal as a regressor significantly improves fit to human reading data, both in monolingual and multilingual contexts (Wilcox et al., 2023, Schijndel et al., 2018). The linking function is empirically best approximated as linear across most of the surprisal range.

Surprisal also robustly predicts ERP signatures. The N400 amplitude, in particular, is best modeled as proportional to wordwise shallow surprisal under recent dual-stage decompositions (see below) (Li et al., 2024, Madhyastha et al., 2023).

3. Cognitive and Neurocomputational Decomposition

Recent advances have split total lexical surprisal into two theoretically distinct components:

• Shallow surprisal ($A$): The cost to move from prior to a shallow, plausibility-driven posterior over words after partial processing. Quantitatively, $A = D_{KL}[p_\lambda \Vert p_0]$, where $p_\lambda$ is a resource-rational, shallow inference distribution.
• Deep surprisal ($B$): The incremental cost incurred in revising from the shallow to the fully detailed, veridical posterior: $$B = D_{KL}[p_\infty \Vert p_0] - D_{KL}[p_\lambda \Vert p_0]$$.

The total surprisal decomposes additively as $S(w\,|\,c) = A + B$, and these components map directly onto early and late ERP signals: the N400 scales with $A$, while the P600 reflects $B$ (Li et al., 2024). This model formalizes the neurocognitive processes underlying “good-enough” interpretive heuristics and elaborative repair.

4. Methodological Considerations in Estimating Surprisal

Surprisal estimation depends crucially on both the choice of LLM and the definition of linguistic units:

• Tokenization granularity: Using subword units (e.g., Byte-Pair Encoding, UnigramLM) or morphological decomposition modifies how $P(w\,|\,c)$ is assembled. Empirical work shows that vocabularies of size $|V| \approx 8000$ yield the best predictive power for naturalistic reading times, balancing encoding of length, frequency, and compositional effects (Oh et al., 2024). For syntactic paradigms such as garden-path effects, coarser tokenizations (≥32k) can improve the sensitivity of surprisal to structural reanalysis (Oh et al., 2024).
• Model capacity and training scale: The fit between LM surprisal and human data is non-monotonic in both model capacity and training corpus size. Surprisal from LMs peaks in cognitive plausibility after approximately $2 \times 10^9$ training tokens (for contemporary architectures), while additional training drives model predictions away from observed human processing, especially for open-class/novel words (Oh et al., 2023, Oh et al., 2022). Surprisal from large-scale, over-trained models may underpredict human reading times on named entities or rare items—a divergence attributed to memorization and overly precise statistics on frequent sequences.
• Predictive distributions: Transformer-based LMs (GPT-2, BERT, etc.) generally yield more humanlike surprisal estimates than lower-order $n$-gram models, but performance depends on the specific context window and modality. In multimodal settings (audio-visual comprehension), human predictors may weight local context more heavily, favoring lower-order $n$-gram surprisal (Madhyastha et al., 2023).
• Cloze vs. LM-based surprisal: Surprisal derived from LMs outperforms human cloze-probability-based surprisal in predicting processing cost, due to finer resolution, ability to distinguish closely related alternatives, and reliable estimates for rare continuations (Nair et al., 14 Jan 2026).

5. Extensions: Similarity-Adjusted and Morphologically-Aware Surprisal

Standard surprisal treats all vocabulary items as maximally distinct. Extensions include:

• Similarity-adjusted surprisal: Incorporates graded semantic or form similarity via a kernel $z(w,w')$:

$$S_\mathrm{sim}(w_t\,|\,\mathrm{context}) = -\log\left(\sum_{w'} z(w_t, w') P(w'|\mathrm{context})\right)$$

This generalizes standard surprisal and information value, capturing processing cost for words that are semantically or orthographically close to high-probability alternatives. Contexts rich in graded competition (e.g., discourse-rich reading) benefit most from this extension (Meister et al., 2024).

• Morphologically-aware surprisal: For typologically rich languages, word-level surprisal as the sum of morpheme-level surprisal—derived from morphological segmentation—better approximates compositional processing difficulty. For morphologically complex words, this approach provides more cognitively plausible estimates than subword-based or orthographic surprisal (Nair et al., 2023).

6. Surprisal in Neurocognitive and Multilingual Contexts

Surprisal theory generalizes across languages, modalities, and measurement domains:

• Crosslinguistic validations spanning 11 languages confirm that the linear surprisal–processing cost relationship is robust across typologically diverse families (Wilcox et al., 2023).
• Shallow/deep surprisal mapping onto ERP components offers a unified account reconciling information-theoretic modeling with neurobiological signatures such as the N400 and P600 (Li et al., 2024).
• In multimodal language comprehension (audio-visual speech), predictive reliance on local context—hence the best-fit model order—shifts with the presence of non-verbal scaffolding (Madhyastha et al., 2023).

7. Limitations and Theoretical Challenges

While lexical surprisal robustly predicts many graded processing effects, it underestimates the magnitude of reanalysis costs in phenomena such as garden-path ambiguities. Even after independently parameterizing syntactic and lexical predictability, the predicted slowdowns remain an order of magnitude below observed human effects, suggesting the necessity of additional cognitive mechanisms (e.g., explicit reanalysis cost, dual-stage update architectures) not captured by single-update surprisal models (Arehalli et al., 2022). Caution is warranted when interpreting raw surprisal from large, memorization-prone LMs as fully representative of human incremental processing (Oh et al., 2022, Oh et al., 2023).

---

In summary, lexical surprisal furnishes an indispensable, precisely defined metric for quantifying the information-processing demands of incremental language comprehension. Continuing advancements in its theoretical decomposition, efficient estimation, crosslinguistic generalization, and integration with neural and behavioral data solidify its status as a central construct in the computational and cognitive sciences of language.