Causal Drawbridges: Characterizing Gradient Blocking of Syntactic Islands in Transformer LMs

Abstract: We show how causal interventions in Transformer models provide insights into English syntax by focusing on a long-standing challenge for syntactic theory: syntactic islands. Extraction from coordinated verb phrases is often degraded, yet acceptability varies gradiently with lexical content (e.g., "I know what he hates art and loves" vs. "I know what he looked down and saw"). We show that modern Transformer LLMs replicate human judgments across this gradient. Using causal interventions that isolate functionally relevant subspaces in Transformer blocks, attention modules, and MLPs, we demonstrate that extraction from coordination islands engages the same filler-gap mechanisms as canonical wh-dependencies, but that these mechanisms are selectively blocked to varying degrees. By projecting a large corpus of unrelated text onto these causally identified subspaces, we derive a novel linguistic hypothesis: the conjunction "and" is represented differently in extractable versus non-extractable constructions, corresponding to expressions encoding relational dependencies versus purely conjunctive uses. These results illustrate how mechanistic interpretability can inform syntax, generating new hypotheses about linguistic representation and processing.

• The paper demonstrates that Transformer LMs robustly capture gradient syntactic island phenomena via precise causal interventions.
• It employs Distributed Alignment Search to localize and manipulate subspaces controlling filler-gap dependencies in coordination structures.
• The study reveals strong correlations between learned subspace positions and human acceptability ratings, supporting a graded context-sensitive linguistic framework.

Causal Drawbridges: Characterizing Gradient Blocking of Syntactic Islands in Transformer LMs

Introduction

The paper "Causal Drawbridges: Characterizing Gradient Blocking of Syntactic Islands in Transformer LMs" (2604.13950) presents a mechanistic investigation into how transformer LMs process the complex, gradient phenomenon of syntactic islands, specifically focusing on coordination islands. It leverages recent advances in mechanistic interpretability, employing causal interventions to localize and quantify the internal mechanisms that underlie gradient acceptability in extraction from coordination structures. The study both quantifies model-human alignment in judgments and demonstrates that LMs recapitulate known filler-gap mechanisms, with gradient blocking controlled by specific learned subspaces. Additionally, it proposes a novel linguistic hypothesis regarding the representation of conjunctions in extractable vs. non-extractable contexts.

Agreement Between LMs and Human Judgments of Syntactic Islands

The paper establishes that transformer LMs exhibit strong quantitative agreement with human gradient acceptability judgments on extraction from coordinated verb phrases. Using minimal pairs from a validated human-rated dataset, model wh-licensing effects (measuring representation of long-distance dependencies) are shown to correlate robustly with human Likert acceptability ratings (Pearson $r$ 0.54–0.80 across diverse LM families and scales). Notably, the gradient of LM responses matches human gradience with respect to lexical content—certain conjuncts allow extraction with little degradation while others yield strong blocking.

Figure 1: LM judgments of gradiently acceptable conjuncts correlate with human judgments, depend on blocking of filler-gap mechanism, and are explained by the identification of causally relevant subspaces.

Figure 2: Correlation between LM mean licensing interaction and human acceptability judgments across embedded wh-questions with verb-phrase conjuncts.

This evidence refutes any purely categorical analysis of the model’s syntactic knowledge and supports the view that LMs represent islandhood as a graded, content-sensitive phenomenon, paralleling contemporary views in psycholinguistics.

Uncovering Mechanisms: Causal Interventions and Filler-Gap Generalization

By employing Distributed Alignment Search (DAS), a supervised causal interpretability technique, the authors identify model-internal subspaces that mediate the filler-gap dependency within canonical wh-extraction sentences. Interventions on these subspaces robustly alter LM predictions in held-out minimal pairs, demonstrating high causal efficacy.

Figure 3: Odds for learned embedded wh-question interventions evaluated on a held-out test set. High causal efficacy indicates recovery of the filler-gap mechanism.

Projection of these filler-gap interventions onto coordination island stimuli reveals that the mechanisms underlying classic filler-gap dependency generalize to 'extractable' coordinations but are only partially active (blocked) for non-extractable configurations. The degree to which these interventions affect the model’s decisions is proportional to the observed gradient of acceptability both in the model and humans, suggesting a shared pathway for these syntactic processes.

Discovery of Causal Drawbridges: Mechanisms of Gradient Blocking

To probe the cause of gradient blocking, the study learns new DAS interventions discriminating highly extractable from unextractable coordination islands, uncovering subspaces that act as "causal drawbridges"—internal mechanisms that modulate the application of the filler-gap dependency.

Figure 4: Odds for learned interventions at each position of the conjuncts and layer of every LM. Strong causal efficacy suggests DAS robustly discovers the blocking mechanism.

Figure 5: Absolute correlation between each conjunct's average position along the learned subspaces and human judgments. Strong correlation confirms that subspace position faithfully encodes gradient acceptability.

Analysis reveals that the gradient position within these learned subspaces is strongly correlated with human acceptability judgments, providing evidence that the mechanisms responsible for gradient blocking are both highly localized and causally explanatory. The drawbridge effect is tightly linked to the syntactic and lexical properties of the verbs in the coordinated structure, with evidence also suggesting that LMs incrementally track semantic cues as soon as they become available.

Linguistic Implications: Representation of Conjunction

By projecting a large sample of unrelated text from the Gutenberg Corpus onto the discovered blocking subspaces, the authors find that the representations of the word "and" diverge systematically between extractable and non-extractable usages. In extractable contexts, "and" codes for relational and purposive event sequences (e.g., "crept up and stole"), while in blocked cases it is used as a logical conjunction for noun or event lists ("meat and bread", "books and papers"). This supports a graded, context-dependent analysis of coordination type, with direct ramifications for theories of the syntax/semantics interface. Notably, these findings echo classic but debated linguistic accounts (e.g., Ross's analysis) while recasting them in a gradient/probabilistic framework.

Theoretical and Practical Implications

From a theoretical perspective, this work strengthens the argument that LMs, trained on purely distributional data, acquire representations and mechanisms closely paralleling those hypothesized for human syntax—including sensitivity to gradience, context, and lexical idiosyncrasy. The ability to localize and causally manipulate these mechanisms offers a template for further reverse-engineering of complex grammatical phenomena. The precise correlation between learned subspace positions and human acceptability ratings invites further experimental psycholinguistic work testing analogous representations and mechanisms in human subjects.

Practically, the methodology showcased—using DAS to discover causally relevant subspaces—can be applied to a broad range of model behaviors, advancing interpretability research and enabling more targeted interventions in syntactic, semantic, or pragmatic reasoning in LMs.

Conclusion

This paper provides compelling evidence that transformer LMs encode gradient acceptability for syntactic islands in a manner highly homologous to human judgments, mediated by mechanisms localized via causal analysis. The discovery of "causal drawbridges" responsible for modulating filler-gap dependencies bridges mechanistic interpretability and linguistic theory, and the methodology paves the way for more comprehensive reverse-engineering of contextual grammar in large LMs. Future directions should include cross-linguistic extension, testing the effect in state-of-the-art models, and human experiments probing the cognitive reality of the hypothesized drawbridge subspaces.