Modeling semantic association in self-paced reading with language model embeddings

Abstract: Semantic association between a word and its context has been identified as an important component of reading comprehension, even when word predictability is accounted for. Recent research has highlighted the potential of LLM ( LM) embeddings to quantify semantic association. Yet, embedding-based semantic association have been operationalized in a myriad of ways. In this study, we use embeddings from LMs to estimate semantic association on a corpus of joint electroencephalography (EEG) and self-paced reading of natural, Dutch texts. Semantic association is calculated in ten different implementations that vary the embedding model and context lengths. The effects of semantic association across the different implementations on the N400 and self-paced reading times are examined using Bayesian hierarchical models and Bayes factor. The results show that the choice of embedding model can alter the estimated effect of semantic association on both the N400 and self-paced reading times. Furthermore, the results demonstrate a promising potential of sentence embeddings for capturing semantic association, as only implementations relying on sentence embeddings indicate reliable results of semantic association beyond word predictability on both neural and behavioral measures. Together, these findings highlight the importance of methodological choices in quantifying semantic association.

• The paper demonstrates that contextualized, sentence-level embeddings yield more reliable semantic association estimates for predicting both N400 and reading times compared to uncontextualized models.
• Methodologically, it employs Bayesian hierarchical regression to isolate semantic association effects while controlling for word predictability in naturalistic Dutch texts.
• Findings indicate that aspects like embedding choice and context window size critically influence the interpretation of neural and behavioral measures in language comprehension.

Modeling Semantic Association in Self-Paced Reading with LLM Embeddings

Introduction and Theoretical Motivation

Semantic association, distinct from word predictability, modulates both behavioral and neural indices of linguistic processing. Prior cognitive neuroscience work has identified the N400 ERP component and self-paced reading times (SPRs) as sensitive to context-driven predictability and semantic fit [kutasThirtyYearsCounting2011; frankERPResponseAmount2015; federmeierRoseAnyOther1999]. However, the degree to which LM-based embeddings can consistently and robustly quantify semantic association effects beyond predictability, particularly in naturalistic, self-paced reading paradigms with ecologically valid Dutch texts, remains unresolved. The present study directly addresses this by operationalizing and comparing a comprehensive set of embedding-based implementations of semantic association, probing their explanatory strength for the N400 and reading time data, and systematically assessing the influence of embedding type and contextual window on resulting estimates.

Methodology

The study analyzes a joint EEG and SPR corpus (TiNT [ostergaardCorpusJointEEG2025]) comprising 56 participants reading medium-length (>600 words) Dutch texts in a naturalistic setting. EEG preprocessing and N400 extraction follow established standards, targeting centroparietal electrode mean amplitudes within 300–500ms post-stimulus.

Definition and Implementation of Semantic Association

Semantic association is operationalized as the cosine similarity between a critical word's embedding and a context embedding. Two major embedding paradigms are contrasted:

• Uncontextualized word embeddings (word2vec/wikipedia2vec-nl): Context vectors constructed by averaging embeddings for all words (WE, All), only content words (CWE, All), or via local/content word windows and exponentially weighted averages.
• Contextualized sentence embeddings (e5-large-trm-nl): Context vectors taken as sentence/document-level representations, without post-hoc averaging.

The study varies the context window granularity (all preceding words, preceding sentence, windowed contexts of 1–2 words, and exponentionally weighted averages) to probe local/global semantic integration.

Figure 1: Pearson's correlation coefficients between implementations of semantic association, log-probability of words, and Zipf word frequency for all content words in the corpus.

Figure 1 demonstrates that semantic association estimates derived from the same embedding type are highly correlated, whereas those across embedding paradigms are only weakly related, indicating that embedding choice is non-trivial and potentially targets substantially different semantic properties in text.

Statistical Modeling

Bayesian hierarchical regression models are constructed separately for each semantic association implementation for both dependent variables (N400, SPR times), always covarying for log-probability (estimated via GPT ensemble for Dutch). Bayes factors (using the Savage–Dickey method) are reported for the semantic association coefficient, enabling a graded quantification of evidence for/against an association effect after controlling for predictability.

Results

Figure 2: Regression coefficients and 95\% credible intervals for semantic association $\beta_2$ across all implementations for both N400 and reading times.

Regression models yield the following primary findings:

• Embedding model critically modulates effect direction and magnitude: Sentence embedding (SE) models yield positive semantic association effects on N400, aligning with the expectation that reduced semantic fit amplifies N400 negativity. Word embedding (WE) models, in contrast, estimate slightly negative or null effects.
• Effect on reading times is weak throughout: Only the SE, All implementation yields a non-negligible positive effect for reading time (greater association predicts slower reading), but this finding is numerically modest.

  Figure 3: Bayes factor (BF01) for the effect of semantic association across models. $BF01 > 1$ supports the null; $BF01 < 1$ supports the alternative.

Bayes factors indicate that across almost all implementations—especially those based on uncontextualized word embeddings—there is substantial evidence for the null hypothesis, i.e., no independent effect of semantic association on reading times or N400 once predictability is controlled. Only SE-based models provide anecdotal (i.e., non-decisive but suggestive) evidence for a non-null effect.

Crucially, these findings empirically contradict prior literature using similar word embedding-based approaches, which have frequently reported positive semantic association effects on N400 amplitude [broderickElectrophysiologicalCorrelatesSemantic2018; frankWordPredictabilitySemantic2017], demonstrating that results are heavily implementation-dependent.

Qualitative Analysis of Embedding Behavior

Figure 4: Semantic association (via various implementations), log-probability, and Zipf frequency for selected words in two sentence pairs. SE-based models are sensitive to thematic coherence (e.g., "draak", "Nomadisme") where WE-based models are not.

Qualitative inspection reveals SE-derived measures better track higher-level semantic or thematic relevance than WE-based measures. This is evident in cases where only SE implementations correctly reflect strong context association for repeated or topical words, a property not systematically captured by local word averages.

Discussion

The findings illustrate several key technical and theoretical points:

• Embedding choice is paramount: Uncontextualized (word2vec) embeddings exhibit severely attenuated and even reversed N400 association effects in the context of longer, naturalistic texts. Contextualized, sentence-level embeddings (e.g., e5-large-trm-nl) provide more reliable semantic association estimates, which retain some predictive value for both N400 and reading times.
• Context window effects are localized to SE models: For word embedding models, expanding the context window does not enhance semantic association estimates or predictive power, suggesting severe information dilution and lack of sensitivity to thematic structure when using naive averaging schemes. SE-based models, in contrast, are more sensitive to windowing, presumably reflecting their supervision on sentence- and document-level tasks.
• Consistency with prior work: The results call into question the robustness of earlier findings supporting an independent effect of semantic association indexed via uncontextualized embeddings, especially in corpora with more complex context structures, languages with less resource-rich embeddings, or less contrived/synthetic materials.
• Operationalization matters: The paper underscores that seemingly minor methodological choices—embedding architecture, context window, representation aggregation—propagate to qualitative differences in experimental conclusions about the role of semantic association in processing.

  Figure 5: Average semantic association for various target-context categories (validated on classic materials), demonstrating enhanced discrimination by SE-based models.

Implications and Future Directions

The results have both immediate and long-term consequences for the computational-cognitive modeling of language processing:

• Practical modeling guidance: For naturalistic SPR and EEG corpora, researchers should prefer contextualized, sentence- or document-level embeddings for quantifying semantic association, rather than relying on uncontextualized, static word-level representations.
• Theoretical ramifications: The findings raise the possibility that many previously reported effects attributed to "semantic association" using WE-based metrics may have been artifacts of context window size, language, text genre, or model limitations. More robust, context-sensitive embeddings can reveal associations missed by naive averaging.
• Implications for neural modeling: Given the dependency of N400 effects on embedding implementation, computational-level models of ERP signals that incorporate semantic similarity must be closely aligned with the architecture and training objectives of the underlying lexical or sentence representations.
• Resource development needs: Since only a narrow set of embedding models were fully explored, future work should systematically examine a broader landscape of contextualized and hybrid models, especially for languages beyond English where embedding quality may be variable.

Conclusion

This paper provides a rigorous, multilevel evaluation of embedding-based semantic association metrics for predicting neural and behavioral measures in self-paced reading. The findings converge on the strong claim that embedding selection and context formalization non-trivially determine the ability to detect semantic association effects beyond word predictability. Sentence embeddings, in particular, offer a more promising avenue for capturing thematic and context-driven semantic relationships in naturalistic materials. These results should inform both the design of future cognitive- and neural-level linguistic studies and the development of AI systems modeling human-like semantic processing.

Citation: "Modeling semantic association in self-paced reading with LLM embeddings" (2606.07066)