Collocational bootstrapping: A hypothesis about the learning of subject-verb agreement in humans and neural networks

Abstract: In what ways might statistical signals in linguistic input assist with the acquisition of syntax? Here we hypothesize a mechanism called collocational bootstrapping, in which regularities in word co-occurrence patterns can provide cues to syntactic dependencies. We investigate whether this mechanism can support the acquisition of English subject-verb agreement. First, we simulate language acquisition by training neural networks on synthetic datasets that vary in how predictable their subject-verb pairings are. We find that there is a range of variability levels at which these statistical learners robustly learn subject-verb agreement. We then analyze the variability of subject-verb pairings in child-directed language, and we find that the variability in such data falls within the range that supported robust generalization in our computational simulations. Taken together, these results suggest that collocational bootstrapping is a viable learning strategy for the type of input that children receive.

• The paper demonstrates that intermediate Zipfian subject-verb pairing distributions (α≈1.4) maximize syntactic generalization in neural models.
• It employs controlled neural simulations with 2-layer GPT-2 style transformers and synthetic grammars to isolate statistical effects.
• Empirical analysis reveals that child-directed speech naturally aligns with the optimal statistical regime for subject-verb agreement acquisition.

Collocational Bootstrapping as a Mechanism for Syntactic Acquisition

Hypothesis and Motivation

This paper introduces the collocational bootstrapping hypothesis, positing that statistical regularities in word co-occurrence serve as pivotal cues for syntactic dependency acquisition, focusing on subject-verb agreement in English. The approach is motivated by limitations in non-syntactic bootstrapping proposals (prosody, semantics) and the demonstrated ability of neural models to infer syntax-sensitive phenomena solely from raw linguistic data. The central tension addressed is between predictability and variability: highly predictable input enables memorization but stifles generalization, whereas highly variable input supports abstraction but obscures statistical cues underlying syntactic dependencies.

Controlled Neural Network Simulation

To isolate statistical factors and rigorously test the hypothesis, neural LLMs (2-layer GPT-2 style transformers) are trained on synthetic grammars. Subject-verb pairings are sampled from truncated Zipfian distributions, where the parameter $\alpha$ modulates the degree of predictability versus variability. Lower $\alpha$ yields uniform (high-variance) distributions, while higher $\alpha$ produces concentrated (low-variance) distributions.

Figure 1: Noun probability distributions across $\alpha$ values; lower $\alpha$ produce flatter distributions, higher $\alpha$ yield heavy concentration on select nouns.

The experimental setup enforces ambiguity between competing syntactic rules (Agree-Subject vs. Agree-Recent) during training, ensuring that only collocational statistics—without explicit disambiguating examples—can drive differentiation. Models are evaluated using targeted minimal pairs, systematically varying both lexical familiarity (SEEN/UNSEEN) and number attraction (MATCH/MISMATCH), probing generalization beyond training distributions.

Quantitative Findings

A decisive outcome is that model accuracy peaks robustly in all generalization conditions (including unseen subject-verb pairs with syntactic attractors) when the training data’s subject-verb pairing statistics follow a Zipfian distribution with intermediate $\alpha \approx 1.4$. Accuracy collapses for both low ($\alpha \rightarrow 0$) and high ($\alpha \rightarrow \infty$) extremes, supporting the claim that only certain statistical configurations robustly enable collocational bootstrapping.

Figure 2: Model accuracy vs. Zipfian parameter $\alpha$; optimal generalization at $\alpha$0, diminished accuracy outside this range.

Training and validation loss trajectories further confirm absence of overfitting regardless of $\alpha$1, demonstrating that the observed generalization is not an artifact of memorization.

Figure 3: Training and validation loss curves at three $\alpha$2 values illustrating similar learning dynamics across conditions.

Corpus Analysis: Empirical Validation

To assess real-world viability, the authors analyze subject-verb pairing statistics in child-directed speech (CHILDES, 0–96 months), extracting all relevant dependencies via dependency parsing. For each verb, the empirical distribution of subject frequencies is computed and fit to a Zipfian law across verbs; the best-fit $\alpha$3 is found to be 1.43, closely matching the model-optimal regime.

Figure 4: Empirical distribution of subject-verb pairings in CHILDES, alongside Zipfian fit ($\alpha$4).

Further stratification by child age shows a decreasing trend in $\alpha$5, with values ranging from $\alpha$6 (youngest) to $\alpha$7 (oldest), yet all fall within or near the optimal generalization range.

Figure 5: Fitted Zipf parameter $\alpha$8 as a function of child age, with reference lines for corpus-wide and model-optimal values.

Speaker role distributions in the data confirm that input is dominated by primary caregivers and adults interacting with children.

Figure 6: Distribution of utterances by speaker role in the English subset of CHILDES (ages 0–96 months).

Theoretical Implications and Directions

The strong numerical result—child-directed data nearly precisely matches the statistical regime enabling robust collocational bootstrapping in computational models—suggests that statistical properties of natural language are well-tuned for effective syntactic acquisition via data-driven mechanisms. While the synthetic setups differ qualitatively from natural input and the models lack multimodal information accessible to children, the evidence establishes the statistical sufficiency of collocational cues for learning subject-verb dependencies.

The authors note conceptual intersections with semantic bootstrapping, given correlations between semantic relatedness and co-occurrence statistics, but stress that collocational bootstrapping targets syntactic dependencies, not category acquisition. They advocate extending the framework to other agreement phenomena (adjective-noun gender agreement, anaphor agreement) and investigating the interplay with indirect evidence and learning failures documented in child acquisition research.

Practical Applications

These findings have direct implications for the architecture and dataset design of artificial learners: manipulating statistical distributional properties in synthetic corpora can systematically induce or inhibit syntactic generalization. For developmental linguists, the analysis offers a quantitative link between structured linguistic input and successful acquisition, potentially informing curriculum design for language interventions and further corpus studies.

Conclusion

Collocational bootstrapping is quantitatively verified as a mechanism for subject-verb agreement acquisition in statistical learners, with optimal performance arising at intermediate levels of subject-verb variability. Empirical analysis shows that natural child-directed language embodies precisely this statistical structure. Theoretical and practical implications extend to the design of data-driven acquisition models, and the hypothesis invites future investigation across syntactic phenomena and input modalities.