- The paper proposes to assess LLMs' ability to handle Chinese topic constructions and their sensitivity to island constraints using acceptability judgment tasks.
- It adapts Tian et al. (2024)'s experimental design to compare responses across both gapped and gapless topical structures in multiple state-of-the-art models.
- The study anticipates that larger models like GPT-4, Claude, and Gemini will mirror human judgments, providing insights for future linguistic and cross-linguistic research.
This research proposal outlines a plan to evaluate the grammatical knowledge of LLMs concerning Chinese topic constructions, particularly their sensitivity to island constraints. The work is directly inspired by the experimental methodology and findings of Tian et al. (2024), who investigated human judgments on these structures.
Chinese is a topic-prominent language featuring sentence-initial topics followed by comments. These can be "gapped" (topic linked to a gap in the comment, similar to English topicalization) or "gapless" (topic linked semantically/pragmatically without an overt gap, a characteristic feature). A key debate in linguistics revolves around whether these topics are base-generated in their initial position or derived by syntactic movement from within the comment. Movement theories predict sensitivity to island constraints (syntactic structures from which elements cannot typically be extracted), while base-generation theories might predict immunity, especially for gapless topics.
The proposal reviews Tian et al. (2024), who conducted acceptability judgment experiments using a factorial design (crossing Topicalization presence/absence with Island structure presence/absence). Their findings indicated that both gapped and gapless topics exhibit sensitivity to island constraints in Mandarin, challenging previous claims (e.g., Huang et al., 2009) that gapless topics were immune. Tian et al.'s results support a uniform movement analysis for Chinese topic constructions and suggest that universal island constraints apply.
Building on this, the proposal aims to test whether state-of-the-art LLMs (specifically mentioning GPT-4, Claude Sonnet 3.7, Gemini 2.5, and LLaMA 3.2) replicate these findings. The proposed methodology involves two main tasks:
- Acceptability Judgment Tasks: Presenting LLMs with the same types of controlled sentence stimuli used by Tian et al. (minimal pairs varying topic and island presence) and prompting them to rate sentence acceptability or grammaticality. This directly assesses if LLMs show the same sensitivity patterns (i.e., rating island-violating topicalizations lower) as humans.
- Sentence Continuation Tasks: Prompting LLMs to complete sentence fragments that involve topic structures, particularly in contexts where an island constraint might be relevant. Analyzing the completions can reveal whether the models implicitly respect these constraints by avoiding ungrammatical structures, restructuring the sentence, or using strategies like resumptive pronouns.
The anticipated results suggest that larger, more advanced LLMs (GPT-4, Claude, Gemini) are likely to show greater sensitivity to these syntactic constraints, potentially mirroring human judgments more closely than smaller models like LLaMA. It is expected that these models might avoid generating island violations in continuation tasks. However, challenges are acknowledged, including prompt sensitivity, the difficulty of distinguishing syntactic from semantic errors in model outputs, variability in responses, and the inherent non-symbolic nature of LLM processing.
Finally, the proposal outlines future research directions, such as extending the evaluation to other languages (e.g., Japanese, Korean), investigating different syntactic phenomena (e.g., wh-islands), analyzing the impact of model architecture and training data, refining prompting techniques, and potentially using LLMs as tools for linguistic exploration ("synthetic informants") or psycholinguistic modeling. The paper aims to contribute to understanding both the linguistic capabilities of LLMs and the nature of syntactic constraints in human language.