The Effect of Surprisal on Reading Times in Information Seeking and Repeated Reading

Abstract: The effect of surprisal on processing difficulty has been a central topic of investigation in psycholinguistics. Here, we use eyetracking data to examine three language processing regimes that are common in daily life but have not been addressed with respect to this question: information seeking, repeated processing, and the combination of the two. Using standard regime-agnostic surprisal estimates we find that the prediction of surprisal theory regarding the presence of a linear effect of surprisal on processing times, extends to these regimes. However, when using surprisal estimates from regime-specific contexts that match the contexts and tasks given to humans, we find that in information seeking, such estimates do not improve the predictive power of processing times compared to standard surprisals. Further, regime-specific contexts yield near zero surprisal estimates with no predictive power for processing times in repeated reading. These findings point to misalignments of task and memory representations between humans and current LLMs, and question the extent to which such models can be used for estimating cognitively relevant quantities. We further discuss theoretical challenges posed by these results.

• The paper demonstrates that standard surprisal estimates maintain a robust linear link with reading times, though with diminished accuracy in non-ordinary reading contexts.
• The study reveals that regime-specific surprisal estimates do not enhance predictions for first reading in information seeking or repeated reading.
• These findings challenge the broad applicability of surprisal theory and urge refinements in language models to better capture human reading processes.

The Effect of Surprisal on Reading Times in Information Seeking and Repeated Reading

The paper explores the extension of surprisal theory to various reading regimes beyond ordinary reading: information seeking, repeated reading, and a combination of the two. Utilizing eyetracking data, the study examines whether the linear relationship predicted by surprisal theory, which posits a direct link between word surprisal and processing difficulty, holds under these different regimes.

Key Findings

1. Standard Surprisal Estimates:
   • The analysis reveals that regime-agnostic surprisal estimates show a consistent linear relationship across all examined reading regimes: information seeking, repeated reading, and their combination.
   • The linear surprisal effects are robust but exhibit reduced predictive power compared to ordinary reading, indicating potential context-dependent variations in surprisal impact.
2. Regime-Specific Surprisal Estimates:
   • In first reading information seeking, regime-specific context does not improve the predictive power over standard surprisal estimates.
   • For repeated reading, both ordinary and information seeking surprisals align poorly with human reading times. The nearly zero surprisals and lack of predictive power suggest that current models inadequately represent repeated exposure memory effects.

These findings imply a misalignment between human cognitive processing during these regimes and how LLMs estimate surprisal. This raises questions about the psycholinguistic relevance of current models, particularly regarding their predictive power in non-ordinary reading contexts.

Implications

The implications of these results are twofold. Practically, they highlight the limitations of current LLMs in replicating human reading comprehension in varied contexts. Theoretically, they challenge the general applicability of surprisal as a singular measure of cognitive processing difficulty, especially in complex language processing scenarios.

Speculation on Future Developments

Future research may focus on refining LLM architectures and training methodologies to address the observed discrepancies in memory representation and task alignment. Exploration into deeper integration of task-specific cues and advanced memory mechanisms within models could potentially yield a better approximation of human reading processes.

Furthermore, a re-evaluation of surprisal theory, taking into account additional cognitive factors beyond processing difficulty, may provide a more comprehensive framework for understanding human language comprehension across diverse contexts.

In conclusion, while surprisal continues to be a valuable concept within psycholinguistics, its current operationalization in LLMs requires careful consideration. Addressing these challenges could significantly enhance model fidelity and enrich theoretical insights into human language processing.