Collocational Bootstrapping

• Collocational Bootstrapping is a linguistic hypothesis that uses word co-occurrence patterns to infer syntactic and semantic relations, exemplified by subject-verb agreement and noun-compound interpretation.
• It employs bootstrapping strategies that capitalize on structured regularities in language data, combining synthetic experiments and real-world child-directed input to reveal underlying dependencies.
• This approach underpins various NLP applications such as relation extraction and topic modeling by iteratively refining lexical associations and filtering noise with statistical measures.

Searching arXiv for the cited papers and closely related work. Collocational bootstrapping is a hypothesis and a family of bootstrapping strategies in which regularities in word co-occurrence patterns are used to infer linguistic structure or to expand linguistic resources. In its most specific recent formulation, collocational bootstrapping is a proposal about how learners can use ordinary word co-occurrence statistics to infer abstract syntactic dependencies, with subject-verb agreement as the central test case (Hobbs et al., 19 May 2026). In a broader NLP sense, related work uses lexical associations, paraphrasing patterns, or iteratively lexicalized multiword units to acquire relation extractors, semantic interpretations of noun compounds, or improved topic-model inputs (Eyal et al., 2021, Kim et al., 2019, Poon et al., 2020). Across these settings, the common premise is that collocational structure can provide indirect evidence for relations that are not overtly labeled in the input.

1. Conceptual scope and definition

The paper "Collocational bootstrapping: A hypothesis about the learning of subject-verb agreement in humans and neural networks" defines collocational bootstrapping as a mechanism by which regularities in word co-occurrence patterns can provide cues to syntactic dependencies (Hobbs et al., 19 May 2026). Its core claim is that words standing in a syntactic dependency often co-occur in systematically nonrandom ways, so a learner may first infer likely word-word dependencies from collocational regularities and then abstract away from particular lexical items to the underlying syntactic rule.

A crucial distinction in that formulation is that collocational bootstrapping is not just “distributional learning” in the sense of grouping words into categories; it is specifically a proposal for learning relations between words, such as subject-verb dependencies, from co-occurrence structure (Hobbs et al., 19 May 2026). The canonical illustrative contrast is that in a sentence such as the dog on the couch barks, the learner could infer that the dependency is more likely between dog and barks than between couch and barks because dog co-occurs with barks much more often than couch does.

Related papers broaden the operational interpretation of the term. Kim and Nakov treat noun compound interpretation as an alternating process in which noun compounds and paraphrasing patterns are jointly extracted from the web, making the paper “a concrete instance of what one could call collocational bootstrapping” (Kim et al., 2019). Gupta and Manning’s syntactic-search method is explicitly described as bootstrapping in the same problem space, but with a shift from lexical collocations and hand-written textual patterns to dependency-graph similarity (Eyal et al., 2021). Chen et al. do not use the phrase “collocational bootstrapping,” but their iterative token concatenation procedure for discovering collocations before hierarchical latent tree analysis has a clear discovery-and-reuse dynamic in which collocations found in one round become atomic units for later rounds (Poon et al., 2020). Park et al., finally, contribute a cooccurrence-based criterion for specificity that is directly relevant to ranking and filtering candidate collocations, even though their method is not itself iterative bootstrapping (Stewart, 2014).

This suggests a useful generalization: collocational bootstrapping is best understood as a family of methods in which collocational regularities serve as indirect evidence for hidden structure. The structure may be syntactic dependency, semantic relation, extractable relation instance, or multiword lexicalization, but the shared mechanism is the use of nonrandom lexical association as a guide.

2. Subject-verb agreement as the canonical modern formulation

The most explicit theoretical development of collocational bootstrapping concerns English subject-verb agreement (Hobbs et al., 19 May 2026). The paper asks whether co-occurrence statistics can help a learner discover that the correct rule is Agree-Subject rather than the plausible but incorrect Agree-Recent. The test case is informative because many ordinary English sentences do not distinguish these rules; they diverge only when an intervening noun appears, as in the dogs in the park bark, where the correct controller is the subject dogs, not the more recent noun park.

The study makes the problem sharper by constructing training data that are fully ambiguous between competing agreement rules: all nouns in a sentence have the same number, so the training strings never directly reveal whether the learner should track the subject, the most recent noun, or some other position (Hobbs et al., 19 May 2026). The synthetic corpora contain 12,000 unique grammatical sentences built from four templates:

• Det N V
• Det N PP V
• PP Det N V
• PP Det N PP V

Each sentence has one subject noun phrase, one intransitive present-tense verb, and optionally a prepositional phrase before and/or after the subject. The vocabulary contains 40 noun stems and 40 verb stems, each with singular and plural forms, yielding 80 noun tokens and 80 verb tokens total. The determiner is always the; the prepositions are by and near (Hobbs et al., 19 May 2026).

What varies across datasets is the distribution of possible subjects for each verb, parameterized by a truncated Zipfian distribution:

$f(r) = \frac{K}{r^\alpha}$

with a more specific subject-sampling rule:

$$\frac{1}{((j-i \bmod 40)+1)^\alpha} \quad \text{if } j-i \bmod 40 < 30,$$

and $0$ otherwise (Hobbs et al., 19 May 2026). For each verb, 30 nouns can appear with it in training, while 10 nouns were withheld entirely and later used to test generalization to unseen subject-verb pairs. This creates a direct tradeoff between variability and predictability: at $\alpha = 0$, the distribution is uniform; as $\alpha \to \infty$, each verb appears with only one subject noun.

The models are 2-layer decoder-only Transformer LLMs in the GPT-2 style, trained from scratch using nanoGPT/PyTorch, each with 4 attention heads per layer, embedding size 256, and about 1.6M parameters (Hobbs et al., 19 May 2026). For each $\alpha$ from 0.0 to 3.0 in steps of 0.1, plus $\alpha \to \infty$, the authors trained 10 random seeds on newly generated datasets, using an 80/10/10 train/validation/test split. Training used AdamW with learning rate 0.0006, batch size 32, 300 batches per epoch for 4 epochs, for 1,200 total iterations; the best validation-checkpoint model was selected (Hobbs et al., 19 May 2026).

Evaluation uses targeted minimal pairs. A model is counted correct if it assigns higher log probability to the grammatical sentence than to the ungrammatical one, giving the effective accuracy criterion:

$$\text{Accuracy} = \frac{\#\{\text{pairs where } \log p(\text{grammatical}) > \log p(\text{ungrammatical})\}}{\text{total number of pairs}}$$

(Hobbs et al., 19 May 2026). Four test conditions are defined: SEEN, MATCH; UNSEEN, MATCH; SEEN, MISMATCH; and UNSEEN, MISMATCH. The critical condition is UNSEEN, MISMATCH, because success there requires both structural generalization and resistance to distractor nouns (Hobbs et al., 19 May 2026).

The principal result is that agreement performance as a function of $\alpha$ is non-monotonic (Hobbs et al., 19 May 2026). At low $\alpha$, subject-verb pairings are highly variable and nearly uniform, so models struggle in mismatch conditions. At high $$\frac{1}{((j-i \bmod 40)+1)^\alpha} \quad \text{if } j-i \bmod 40 < 30,$$0, lexical associations become highly predictive, and mismatch performance improves for seen pairings, but generalization to unseen pairings collapses. Between these extremes lies a robust intermediate regime, with a peak around $$\frac{1}{((j-i \bmod 40)+1)^\alpha} \quad \text{if } j-i \bmod 40 < 30,$$1, and Figure 1 highlighting $$\frac{1}{((j-i \bmod 40)+1)^\alpha} \quad \text{if } j-i \bmod 40 < 30,$$2 as the optimum (Hobbs et al., 19 May 2026). At that level, the models achieve near-perfect performance even in UNSEEN, MISMATCH. The paper’s central claim is therefore not merely that co-occurrence matters, but that successful acquisition depends on a specific tradeoff between predictability and variability.

3. Compatibility with child-directed input

The second major component of the subject-verb agreement work asks whether the relevant statistical regime occurs in children’s input (Hobbs et al., 19 May 2026). The authors analyze English child-directed language in CHILDES, extracting all adult-to-child utterances for target children aged 0–96 months and obtaining 4,739,189 utterances after filtering. Using the spaCy dependency parser, they extract all noun-subject/verb pairs with dependency label nsubj, using lemmas for both subject and verb, for a total of 2,802,071 subject-verb pairs (Hobbs et al., 19 May 2026).

To quantify collocational variability, the analysis focuses on the 100 most frequent verbs in the corpus, each occurring at least 2,396 times. For each verb, subjects are ranked by frequency and converted to proportions; these per-rank proportions are then averaged across verbs. The empirical rank-frequency curve is defined as

$$\frac{1}{((j-i \bmod 40)+1)^\alpha} \quad \text{if } j-i \bmod 40 < 30,$$3

and fitted to the Zipfian prediction

$$\frac{1}{((j-i \bmod 40)+1)^\alpha} \quad \text{if } j-i \bmod 40 < 30,$$4

by searching over $$\frac{1}{((j-i \bmod 40)+1)^\alpha} \quad \text{if } j-i \bmod 40 < 30,$$5 in increments of 0.01 and minimizing mean squared error (Hobbs et al., 19 May 2026).

The fitted result is $$\frac{1}{((j-i \bmod 40)+1)^\alpha} \quad \text{if } j-i \bmod 40 < 30,$$6, which is almost identical to the synthetic-data optimum of about 1.4 (Hobbs et al., 19 May 2026). Age-stratified analyses show that fitted $$\frac{1}{((j-i \bmod 40)+1)^\alpha} \quad \text{if } j-i \bmod 40 < 30,$$7 decreases with child age, ranging from 1.46 for speech to 0–12 month-olds to 1.23–1.25 for the oldest groups, with intermediate ages often clustered around 1.37–1.44 (Hobbs et al., 19 May 2026).

The paper is explicit about the interpretive limit of this result. The CHILDES analysis does not prove that children actually use collocational bootstrapping; it shows that their input appears compatible with it (Hobbs et al., 19 May 2026). This is theoretically important because it converts the simulation from a purely in-principle demonstration into a plausibility argument grounded in child-directed English. A plausible implication is that collocational bootstrapping is best viewed as a viable learning strategy whose feasibility depends jointly on the learner and the input distribution, rather than as a claim that collocational cues are necessary or sufficient for syntax acquisition.

The same paper situates collocational bootstrapping alongside other acquisition proposals. It is explicitly contrasted with distributional bootstrapping, which uses context distributions to infer syntactic categories, and it is described as potentially related to semantic bootstrapping, because semantic relatedness and distributional co-occurrence often overlap (Hobbs et al., 19 May 2026). The authors do not claim exclusivity: bootstrapping mechanisms are “not mutually exclusive,” and children may also use prosodic, semantic, and other cues (Hobbs et al., 19 May 2026).

4. Bootstrapping lexical-semantic resources from collocations and paraphrases

In lexical semantics, collocational bootstrapping appears as an alternating extraction process in which lexicalized patterns and lexicalized constructions recursively support one another. Kim and Nakov’s noun-compound study is the clearest example (Kim et al., 2019). Their target is fine-grained semantic interpretation of noun compounds such as orange juice or chocolate bar by acquiring paraphrases that make the latent relation explicit, for example “squeezed from” or “made of.”

The framework has two interdependent object types: noun compounds and paraphrasing patterns (Kim et al., 2019). The initial seeds are a small set of noun compounds instantiating Levi’s Make$$\frac{1}{((j-i \bmod 40)+1)^\alpha} \quad \text{if } j-i \bmod 40 < 30,$$8 relation and a small set of associated paraphrasing patterns. The seed set contains all 20 Make$$\frac{1}{((j-i \bmod 40)+1)^\alpha} \quad \text{if } j-i \bmod 40 < 30,$$9 examples from Levi’s appendix, and the seed pattern inventory contains 18 patterns, including “be composed of, be comprised of, be inhabited by, be lived in by, be made from, be made of, be made up of, be manufactured from, be printed on, consist of, contain, have, house, include, involve, look like, resemble, taste like” (Kim et al., 2019).

The alternation proceeds in two directions. Given patterns, the system extracts noun compounds from the web; given noun compounds, it extracts new paraphrasing patterns from the web (Kim et al., 2019). For noun-compound extraction, the generalized query templates are:

$0$0

$0$1

$0$2

For pattern extraction, the generalized query is

$0$3

with $0$4 and * standing for 1–6 wildcard tokens (Kim et al., 2019). The web is used as a corpus because explicit paraphrases are sparse.

The paper defines three bootstrapping variants: Loose bootstrapping, Strict bootstrapping, and NC-only strict bootstrapping (Kim et al., 2019). The central empirical conclusion is that fixing one constituent noun yields stronger lexical-semantic constraints. The paper states that having one compound noun fixed “yields both a higher number of semantically interpreted NCs and improved accuracy due to stronger semantic restrictions” (Kim et al., 2019).

The aggregate results make that point concrete. Across all iterations, Loose bootstrapping, $0$5 extracts 1,662 NCs at 61.67% accuracy and 12 patterns at 65.83% accuracy; Strict bootstrapping, $0$6 extracts 16,090 NCs at 68.27% accuracy and 16 patterns at 78.98% accuracy; NC-only strict bootstrapping, $0$7 extracts 100,550 NCs at 70.43% accuracy (Kim et al., 2019). The paper explicitly states that fixing one of the two nouns yields significantly higher accuracy by a $0$8 test for both NC and NC-pattern pair extraction compared to loose bootstrapping (Kim et al., 2019).

This work exemplifies collocational bootstrapping in a strong lexical sense. The learned knowledge is not merely that two words co-occur, but that noun compounds and paraphrastic predicates stand in reciprocal lexical association. A plausible implication is that partially fixed collocational frames can act as semantic constraints in a way analogous to how subject-verb collocations act as syntactic cues in agreement learning.

5. Relation extraction and the shift from collocation to syntax-guided expansion

Low-resource relation extraction offers a contrasting but closely related bootstrapping paradigm (Eyal et al., 2021). Gupta and Manning propose “bootstrapping relation extractors using syntactic search by example,” starting from a very small number of positive seed sentences—3 examples per relation in the main experiments. The user lightly annotates each sentence by marking the two relation arguments and, when applicable, a trigger word. A system based on SPIKE converts this markup into a dependency-graph query and retrieves syntactically similar sentences from a parsed corpus, here Wikipedia (Eyal et al., 2021).

The conceptual parallel to classical collocational bootstrapping is explicit. The method has seed examples, an expansion mechanism that harvests more candidate positives from a large corpus, and a downstream extractor trained on the harvested data (Eyal et al., 2021). The difference is representational: classical collocational or pattern-based bootstrapping often expands through lexical co-occurrence and surface patterns, whereas this method expands through dependency-graph similarity and entity-type constraints.

Retrieved search matches are treated as positive training instances; negatives are sampled from type-compatible sentences in the same domain and filtered to avoid syntactic patterns attested by the positives (Eyal et al., 2021). The resulting weakly supervised dataset trains a binary relation classifier based on the Entity Markers architecture with RoBERTa rather than BERT. The paper defines a relation instance as

$0$9

where $\alpha = 0$0 and $\alpha = 0$1 (Eyal et al., 2021).

The experiments on TACRED and DocRED show that the method is a substantive bootstrapping mechanism rather than a brittle rule system. Average test F1 over all relations for Syntactic Search (3 queries) is 0.443 on TACRED and 0.266 on DocRED; Search + Generation reaches 0.491 on TACRED and 0.277 on DocRED (Eyal et al., 2021). By contrast, Pattern Based RE (3 queries) reaches 0.128 on TACRED, while manual annotation baselines range from 0.097 for Annotated 5+50 on TACRED to 0.569 for Annotated all (Eyal et al., 2021).

The paper is especially relevant to collocational bootstrapping because it preserves the seed-and-expand logic while replacing collocation as the primary generalization engine with syntax. It therefore functions as a modern descendant of classic bootstrapping methods in information extraction, but with explicit syntactic constraints substituted for lexical pattern induction (Eyal et al., 2021). This suggests that collocational bootstrapping and syntax-guided bootstrapping are not mutually exclusive alternatives; rather, they occupy adjacent points on a spectrum of weak-supervision strategies.

6. Iterative collocation discovery, specificity, and representational consequences

Two additional lines of work illuminate how collocational information can be operationalized outside explicit syntax acquisition or extraction. Chen et al. propose an iterative preprocessing procedure for hierarchical latent tree analysis (HLTA) in which selected collocations are replaced with single tokens before topic modeling (Poon et al., 2020). The algorithm Preprocess(\mathcal{D}, r, P) repeatedly computes TF-IDF, selects the top-$\alpha = 0$2 tokens, concatenates consecutive in-vocabulary pairs, rescoring and rewriting the corpus across rounds (Poon et al., 2020). The maximum length of collocations that can be considered is stated to be $\alpha = 0$3.

The method is not presented as collocational bootstrapping, but it has an unmistakable bootstrapping structure: words define candidate pairs, selected pairs become atomic units, and those units become inputs for later concatenations (Poon et al., 2020). On the downstream task, the proposed preprocessing improves topic coherence for HLTA on NIPS, AAN, and JRC. For example, on NIPS, HLTA base = $\alpha = 0$4, HLTA NSP = $\alpha = 0$5, and HLTA r1 = $\alpha = 0$6 (Poon et al., 2020). The representational motivation is that standard HLTA can place a token such as network in only one branch, whereas phrase tokens such as bayesian-network or social-network can occupy different branches.

Park et al. address a different but complementary problem: how to estimate the specificity of a term from the distribution of its cooccurrence relations (Stewart, 2014). Their claim is that specific terms have fewer, stronger, and more predictable relations, yielding lower entropy over their relational distribution than general terms. Using a cooccurrence matrix $\alpha = 0$7, they define a relation probability for target term $\alpha = 0$8,

$\alpha = 0$9

and the corresponding entropy,

$\alpha \to \infty$0

(Stewart, 2014). They also consider covariance- and mutual-information-based relational distributions. The method is proposed for extracting collocations, keyphrases, multi-word terms, and index terms, and it is explicitly motivated by the limitation that TF-IDF can mistake general idiomatic terms for specific terms (Stewart, 2014).

This line of work does not itself perform iterative bootstrapping, but it provides a ranking and filtering principle that is directly useful for collocational pipelines. The paper reports that cooccurrence-based ranking can elevate expressions such as kernel method, online learning, and convex optimization problem, while suppressing generic expressions such as other hand or show how (Stewart, 2014). A plausible implication is that collocational bootstrapping systems require not only a mechanism for harvesting candidates but also a principled way to distinguish domain-bearing lexical associations from broad or formulaic phraseology.

7. Limitations, controversies, and theoretical boundaries

The recent subject-verb agreement work is careful not to claim too much (Hobbs et al., 19 May 2026). Its strongest claim is that collocational bootstrapping is a viable learning strategy and likely one useful contributor among several, not that collocational cues are necessary or sufficient for all syntax acquisition. Several caveats are emphasized: the synthetic corpora are highly idealized; they overrepresent PP-rich structures relative to natural input; they use a tiny vocabulary; they consider only present-tense verbs with overt number marking; and the models are text-only, lacking prosody and visual context (Hobbs et al., 19 May 2026). The paper also notes a substantive empirical tension: children and LLMs trained on child-directed data still make agreement-attraction errors, so favorable collocational statistics alone cannot explain the whole developmental picture.

Lexical-semantic and information-extraction variants face related but distinct limitations. Kim and Nakov’s method is relation-specific, focused on Levi’s Make$\alpha \to \infty$1, depends on English relative-clause paraphrases, and is vulnerable to noisy snippets and heuristic shallow parsing (Kim et al., 2019). Gupta and Manning’s method is less tied to lexical collocations but more dependent on infrastructure: it assumes access to a dependency-parsed corpus, a parse-tree query/index engine, and by-example graph-query compilation (Eyal et al., 2021). It is also “not iterative in the classic bootstrapping sense,” because the main pipeline is a single expansion step followed by supervised training rather than repeated extractor-induced pattern promotion (Eyal et al., 2021).

The preprocessing line of Chen et al. has its own boundary conditions. The proposed method uses adjacency, vocabulary filtering, top-$\alpha \to \infty$2 TF-IDF selection, and iterative corpus rewriting rather than direct significance tests such as PMI or likelihood ratio (Poon et al., 2020). It improves HLTA on three of four datasets but is essentially neutral on Reuters, where HLTA base = $\alpha \to \infty$3 and HLTA r1 = $\alpha \to \infty$4 (Poon et al., 2020). Park et al., similarly, warn that “the assumption that cooccurrence implies a relationship is not always correct,” that their graph is dense and unpruned, and that the method is $\alpha \to \infty$5 in vocabulary size (Stewart, 2014).

A broader misconception is that collocational bootstrapping is simply another name for frequency-based distributional learning. The evidence surveyed here points to a narrower and more technical characterization. The defining property is not the use of counts per se, but the use of structured regularities in co-occurrence to infer hidden relational organization. In syntax learning, that organization is the dependency controlling agreement (Hobbs et al., 19 May 2026). In noun-compound interpretation, it is the paraphrastic relation linking constituents (Kim et al., 2019). In low-resource relation extraction, the classic collocational logic survives even where syntax becomes the organizing principle of retrieval (Eyal et al., 2021). In topic modeling and feature selection, collocational information reshapes the representational vocabulary or filters candidate lexical units (Poon et al., 2020, Stewart, 2014).

Taken together, these results support a constrained but substantive view of collocational bootstrapping. It is not a substitute for syntax, semantics, or explicit supervision in general. Rather, it is a mechanism by which nonrandom lexical association can provide indirect evidence for abstract relations, sometimes revealing them, sometimes constraining them, and sometimes supplying the lexicalized units on which later modeling depends.