Incremental Comprehension of Garden-Path Sentences by Large Language Models: Semantic Interpretation, Syntactic Re-Analysis, and Attention

Abstract: When reading temporarily ambiguous garden-path sentences, misinterpretations sometimes linger past the point of disambiguation. This phenomenon has traditionally been studied in psycholinguistic experiments using online measures such as reading times and offline measures such as comprehension questions. Here, we investigate the processing of garden-path sentences and the fate of lingering misinterpretations using four LLMs: GPT-2, LLaMA-2, Flan-T5, and RoBERTa. The overall goal is to evaluate whether humans and LLMs are aligned in their processing of garden-path sentences and in the lingering misinterpretations past the point of disambiguation, especially when extra-syntactic information (e.g., a comma delimiting a clause boundary) is present to guide processing. We address this goal using 24 garden-path sentences that have optional transitive and reflexive verbs leading to temporary ambiguities. For each sentence, there are a pair of comprehension questions corresponding to the misinterpretation and the correct interpretation. In three experiments, we (1) measure the dynamic semantic interpretations of LLMs using the question-answering task; (2) track whether these models shift their implicit parse tree at the point of disambiguation (or by the end of the sentence); and (3) visualize the model components that attend to disambiguating information when processing the question probes. These experiments show promising alignment between humans and LLMs in the processing of garden-path sentences, especially when extra-syntactic information is available to guide processing.

• The paper demonstrates that LLMs initially adopt misinterpretations and later correct their understanding using disambiguating cues.
• The methodology tracks semantic probabilities and incremental parse tree adjustments across 24 garden-path sentences to mirror human comprehension.
• The study shows that attention mechanisms, especially in RoBERTa and LLaMA-2, become sensitive to commas, guiding effective syntactic re-analysis.

Incremental Comprehension of Garden-Path Sentences by LLMs: Semantic Interpretation, Syntactic Re-Analysis, and Attention

This paper investigates the processing of garden-path sentences by LLMs to understand their capability in parsing temporarily ambiguous sentences and resolving lingering misinterpretations. The primary aim is to discern whether LLMs align with human processing of garden-path sentences, particularly in leveraging extra-syntactic cues such as commas to guide interpretation.

The authors evaluate four LLMs—GPT-2, LLaMA-2, Flan-T5, and RoBERTa—against three main research questions: (1) Do LLMs initially adopt the misinterpretation and shift to the correct interpretation upon encountering the disambiguating information? (2) Is this shift reflected in the syntactic parse trees constructed by the LLMs? (3) Are the attentional mechanisms of transformer-based LLMs sensitive to disambiguating cues?

Methodology

The study employs 24 garden-path sentences that create temporary ambiguities via optional transitive and reflexive verbs. For each sentence, a pair of comprehension questions corresponds to the misinterpretation and the correct interpretation. The sentences are processed in chunks, and the LLMs' semantic interpretations, parse tree adjustments, and attention mechanisms are analyzed.

Experiments

1. Dynamic Semantic Interpretations: The semantic probabilities of "yes" and "no" answers to comprehension questions after each chunk are tracked to measure the models' endorsement of misinterpretations and subsequent corrections.
2. Syntactic Re-Analysis: Parse trees are extracted incrementally to inspect whether LLMs reanalyze the syntactic structure from misinterpretation to the correct parse tree at the point of disambiguation.
3. Attention Mechanisms: The study examines the attention weights to determine the model components focusing on disambiguating information.

Results

Surprisal Baseline

Surprisal values indicate that all models show modest increases at the point of disambiguation when there is no comma, consistent with temporary ambiguity. GPT-2 and RoBERTa showed increased sensitivity to disambiguating information when a comma was present.

Semantic Interpretations

The models demonstrated varying degrees of alignment with human-like misinterpretations and corrections. Specifically:

• LLaMA-2 showed a strong initial endorsement of the misinterpretation, which decreased significantly post-disambiguation in the presence of a comma.
• RoBERTa and GPT-2 initially favor the misinterpretation but show a trend toward the correct interpretation with the extra-syntactic comma.

Parse Tree Adjustments

The parse trees extracted reveal that:

• RoBERTa-large achieves human-like performance in reanalyzing and correcting parse trees at and post-disambiguation, especially when commas are used.
• GPT-2 showed moderate success, with increased shifts upon including commas.

Attention Weights

Exploratory analysis of attention weights shows that specific attention heads in LLaMA-2 and RoBERTa-large are sensitive to the point of disambiguation and the correct syntactic reanalysis. Comma presence enhances this sensitivity, demonstrating the model's ability to correct misinterpretations.

Implications and Future Directions

This research contributes to the understanding of LLMs as potential models of human sentence parsing. The experiments highlight that while current LLMs like LLaMA-2 and RoBERTa can align with human-like comprehension patterns, especially with extra-syntactic cues, there are areas for improvement. Future research should expand the dataset of garden-path sentences, include a broader range of LLMs, and investigate emergent abilities in larger models. Additionally, examining a more extensive array of syntactic ambiguities could provide a more comprehensive understanding of these models' cognitive alignment.

In conclusion, the study underscores that LLMs exhibit promising capabilities in processing complex linguistic ambiguities similarly to humans, with certain models demonstrating notable sensitivity to syntactic cues. These results bolster the potential of LLMs not only as engineering feats but also as robust scientific models of human sentence processing.