Grammatical Structure Bias Overview

• Grammatical structure bias is a systematic preference for certain structural configurations in language models, even when alternatives are equally grammatical.
• It emerges from neural architectures, data distributions, and crosslinguistic influences, and is measured using surprisal metrics, minimal pairs, and statistical tests.
• Understanding this bias is key to improving grammar induction techniques, refining language processing models, and enhancing crosslinguistic insights.

Grammatical structure bias refers to systematic, model-internal or language-internal tendencies to prefer specific structural configurations over others, even in the absence of categorical (ungrammatical/grammatical) distinctions. It encompasses a range of phenomena, from architecture-driven inductive preferences in neural networks, to persistent asymmetries driven by data distributions, typological patterns, or crosslinguistic transfer effects. This article synthesizes empirical definitions, mathematical characterizations, methodologies, and theoretical implications from recent research on arXiv, spanning language modeling, grammar induction, word embedding spaces, and neural architectures.

1. Definitions and Core Phenomena

Grammatical structure bias arises when a learner, model, or system exhibits systematic preferences for certain grammatical analyses or constructions over equally well-formed alternatives. In the context of neural LMs, this includes interpretive biases (e.g., attachment ambiguities), architecture-induced generalization (e.g., constituency vs. dependency), and crosslingual "bleed-over" (e.g., English-like structures favored in other languages) (Davis et al., 2020, Papadimitriou et al., 2022, Lepori et al., 2020).

Key operationalizations:

• Structural generalization vs. linear generalization: A structural bias refers to rules referencing hierarchical relations in a parse tree (e.g., c-command, clause boundaries) rather than mere linear order (Warstadt et al., 2020).
• Interpretation bias: Preference for one parse over another in structural ambiguity (e.g., low vs. high attachment in relative clauses) (Davis et al., 2020).
• Data-induced production bias: Acquisition of model biases that reflect production distributions in corpora, but not necessarily human interpretive biases (Davis et al., 2020).
• Redundancy in grammatical cues: The tendency to maintain multiple, overlapping signals for core roles (subject, object), leading to structural bias in how subjecthood is marked and learned (Mahowald et al., 2022).
• Structural optimization ambiguity and simplicity bias: In grammar induction, the likelihood objective creates ambiguity in optimizing over structurally-equivalent grammars and biases models toward low rule diversity (simplicity) (Park et al., 2024).

2. Methodological Approaches and Metrics

Modern research evaluates grammatical structure bias using both behavioral/linguistic paradigms and controlled neural experiments:

1. Minimal Pair Paradigms. Departing from grammaticality contrasts, models are tested on pairs that are both grammatical but structurally distinct, such as RC attachment or dative alternation (Davis et al., 2020, Michaelov et al., 2023).
2. Probability and Surprisal-based Metrics.
   • For LMs, the probability assigned to a sentence (or fragment) is decomposed via chain rule:

   $P(s) = \prod_{t=1}^{T} P(w_t | w_{<t})$

   and per-word surprisal:

   $\text{surprisal}(w_i) = -\log P(w_i | w_1…w_{i-1})$

• The difference in surprisal at disambiguating points quantifies attachment or syntactic preference (Davis et al., 2020).

1. Statistical and Information-Theoretic Measures.

   • Attachment preference scores ($$\Delta = \text{surprisal}_{\text{HIGH}} - \text{surprisal}_{\text{LOW}}$$) and mutual information for redundancy of cues:

   $$R = I(C; \text{Role}) = \sum_{c, r} p(c, r) \log \Bigg( \frac{p(c, r)}{p(c)p(r)} \Bigg)$$

   (Mahowald et al., 2022).

2. Grammar Induction Formulations.

   • Negative log-likelihood over all parses:

   $L(s; G) = -\log\sum_{\tau \in T(s)} p(\tau | G)$

• Sentence-wise parse-focusing to restrict sum to parses generated by multiple unsupervised parsers, mitigating both structural optimization ambiguity (SOA) and structural simplicity bias (SSB) (Park et al., 2024).

1. Embedding and Association Tests.

   • Word Embedding Association Test (WEAT) and projections for disentangling grammatical gender (Sabbaghi et al., 2022, McCurdy et al., 2020).
2. Psycholinguistic Structural Priming.
   • Probability of choosing a structure in a target after a structurally-congruent or incongruent prime, quantifying the extent of abstract structural transfer, both monolingually and cross-lingually (Michaelov et al., 2023, Jumelet, 2024).

3. Empirical Manifestations Across Languages and Models

The empirical literature demonstrates grammatical structure bias across a range of settings:

Bias Type	Manifestation Example	Supporting Papers
Interpretive	Universal LOW RC-attachment in RNN LMs (regardless of training language)	(Davis et al., 2020)
Data-induced	Lack of Spanish HIGH-attachment bias in RNN LMs, despite human preference	(Davis et al., 2020)
Redundancy	88.9% of English and 86.7% of Russian SVO triads recoverable by semantics alone	(Mahowald et al., 2022)
Model architecture	Constituency Tree-LSTMs outgeneralize Dependency Tree-LSTMs on subject-verb agreement, especially when lexical cues are neutralized	(Lepori et al., 2020)
Cross-lingual	Multilingual BERT assigns higher likelihood to English-like SVO and explicit-pronoun forms in other languages	(Papadimitriou et al., 2022)
Grammar Induction	PCFGs trained on likelihood alone underutilize rule diversity; parse-focusing increases S-F1 and rule-diversity	(Park et al., 2024)
Priming	Transformer LMs exhibit strong monolingual and crosslingual structural priming across alternations (dative, genitive, voice)	(Michaelov et al., 2023, Jumelet, 2024)

Quantitative highlights:

• In RC attachment, English RNN LMs: mean $\Delta = 0.77$, $p < 10^{-5}$ (robust LOW bias); Spanish RNN LMs: mean $\Delta = 0.21$, $p < 10^{-5}$ (nonhuman LOW bias) (Davis et al., 2020).
• Redundancy: for word-order cues alone, mutual information $I(\text{Cue}; \text{Role})$ is near zero; with case marking, rises to ~98% (Mahowald et al., 2022).
• Parse-focused grammar induction: S-F1 increases from $\text{surprisal}(w_i) = -\log P(w_i | w_1…w_{i-1})$0 (baseline) to $\text{surprisal}(w_i) = -\log P(w_i | w_1…w_{i-1})$1 (parse-focused), with average rule diversity reaching $\text{surprisal}(w_i) = -\log P(w_i | w_1…w_{i-1})$285–90% of gold parses (Park et al., 2024).

4. Theoretical Analysis and Typological Implications

Research articulates multiple drivers and implications of grammatical structure bias:

• Production vs. Interpretation Split. LMs learn preferences aligned with production frequency (e.g., recency bias in English RC attachment), not always with human interpretation or comprehension (Davis et al., 2020).
• Redundancy and Robustness. Grammatical cues often overlap with world knowledge (e.g., animacy), but play a critical role in rare or ambiguous contexts; redundancy enables error correction (Mahowald et al., 2022).
• Typological Alignment and Cognitive Constraints. RNNs and LSTMs generalize more robustly to typologically frequent word orders under length extrapolation (SOV/SVO), paralleling the distribution in natural language (El-Naggar et al., 14 Oct 2025).
• Model Architecture Effects. Constituency structure in recursive models yields more robust inductive bias than dependency-based alternatives under adversarial or lexically ambiguous conditions (Lepori et al., 2020).
• Crosslingual Transfer. Multilingual pretraining introduces structural "accent," favoring high-resource (often