Language Acquisition Device in Large Language Models

Abstract: LLMs remain substantially less data-efficient than humans. Pre-pretraining (PPT) on synthetic languages has been proposed to close this gap, with prior work emphasizing highly expressive formal languages such as $k$-Shuffle Dyck. Inspired by the Language Acquisition Device (LAD) hypothesis, which posits that innate constraints preemptively restrict the learner's hypothesis space to natural-language-like structure, we propose LAD-inspired PPT: pre-pretraining on MP-STRUCT, a formal language whose strings encode hierarchical composition, feature-based dependencies, and long-distance displacement via MERGE, AGREE, and MOVE. A brief 500-step PPT with MP-STRUCT matches strong formal-language baselines in token efficiency while additionally imparting a human-like resistance to structurally implausible languages (e.g., REVERSE). Analyzing simplified variants, we find that MP-STRUCT CORE outperforms $k$-Shuffle Dyck despite not being definable in C-RASP (a formal bound on transformer expressivity), challenging the prior hypothesis that effective PPT languages must be both hierarchically expressive and circuit-theoretically learnable. We show that functional landmarks, which reduce dependency resolution ambiguity, are a key driver, suggesting that effective PPT design depends not only on expressivity but also on the accessibility of dependency resolution.

• The paper introduces MP-STRUCT, a LAD-inspired synthetic generator that improves token efficiency by encoding innate grammatical structures via MERGE, AGREE, and MOVE operations.
• Experiments with Pythia-1B demonstrate MP-STRUCT achieving MRS scores of 15.3 and 16.2, yielding 29%-31% efficiency gains compared to formal baselines.
• Ablation studies reveal that removing any component (MERGE, AGREE, MOVE) undermines performance, underscoring the significance of explicit dependency organization.

Formal Summary of "Language Acquisition Device in LLMs" (2605.16758)

Motivation and Conceptual Framework

The paper addresses the persistent gap in data efficiency between human linguistic acquisition and LLMs, despite LLMs exhibiting general linguistic competence. Current LLM training regimes require substantially more data, attributed to an overly permissive hypothesis space. Prior research on pre-pretraining (PPT) has focused on synthetic formal languages with high expressivity, such as k-Shuffle Dyck, assuming that inducing hierarchical structural biases is sufficient for improved token efficiency. This paper challenges and complements this assumption by leveraging the Language Acquisition Device (LAD) hypothesis from generative grammar, positing that innate structural constraints (Universal Grammar, UG) restrict hypothesis space to natural-language-like structures.

MP-STRUCT is proposed as a synthetic sequence generator inspired by the Minimalist Program, operationalizing key linguistic operations: hierarchical composition (MERGE), feature-based dependencies (AGREE), and long-distance displacement (MOVE). The central conjecture is that innate structural biases, explicitly encoded via MP-STRUCT, result in greater token efficiency and more human-like inductive biases than formal structural expressivity alone.

Methodology: LAD-inspired Pre-Pretraining

The MP-STRUCT generator produces serialized structural representations devoid of lexical content, thus forcing models to acquire structural and dependency representations independently of surface co-occurrence. The data generation algorithm comprises:

• MERGE: Ensures recursive hierarchical composition, constructing phrase structure bottom-up.
• AGREE: Encodes feature-based dependencies (e.g., subject-verb agreement).
• MOVE: Implements long-distance dependencies by copying elements to higher positions and leaving traces.
• Linearization: Traverses the parse tree and outputs sequences containing brackets, nonterminal labels, features, and traces, excluding all lexical tokens.

Ablation studies further isolate the contributions of each linguistic operation through removal of MERGE, AGREE, and MOVE components.

Experimental Design and Baseline Comparisons

Experiments utilized Pythia-1B, with a fixed PPT budget (500 steps), followed by standard C4 pretraining. Baselines include training from scratch, unstructured random PPT, context-free 1-Dyck, and context-sensitive k-Shuffle Dyck. Measures of learning efficiency include Marginal Rate of Substitution (MRS) and Efficiency Gain, quantifying the reduction in required training steps post-PPT.

Grammatical generalization is evaluated via BLIMP minimal pair benchmarks; robustness to semantic perturbation is assessed using Jabberwocky transformations.

Numerical Results and Empirical Claims

MP-STRUCT, and its idealized MP-STRUCT CORE variant, match or slightly outperform k-Shuffle Dyck in learning efficiency (MP-STRUCT: MRS 15.3, Efficiency Gain 29%; MP-STRUCT CORE: MRS 16.2, Efficiency Gain 31%). Importantly, MP-STRUCT CORE yields these efficiency scores despite not being definable in C-RASP, formally challenging the hypothesis that C-RASP-definability is necessary for effective PPT languages.

Ablation confirms that hierarchical phrase structure (MERGE) and functional dependencies (AGREE, MOVE) act synergistically; removal of any component negatively impacts performance.

Structurally, MP-STRUCT imparts stronger resistance to synthetically reversed input sequences relative to k-Shuffle Dyck, indicating an asymmetric inductive bias reminiscent of human incremental processing.

Analysis of Inductive Bias and Structural Robustness

MP-STRUCT outperforms baselines on Jabberwocky meaning attenuation, maintaining loss robustness when semantic cues are suppressed, supporting reliance on structural regularities. On "impossible" languages (SHUFFLE, REVERSE, HOP), MP-STRUCT demonstrates notable selectivity, notably resisting REVERSE transformations, consistent with the LAD-inspired directional asymmetry.

Theoretical analysis demonstrates that the organization and accessibility of dependency information—specifically, the presence of functional landmarks (explicit structural markers adjacent to dependency endpoints)—play a pivotal role in efficiency. Dependency identification ambiguity is reduced in MP-STRUCT CORE, enabling attention-based models to resolve dependencies more efficiently, independent of circuit-theoretic expressivity.

Implications and Future Directions

The findings underscore that the formal expressivity of synthetic PPT languages, as measured by the Chomsky hierarchy and C-RASP-definability, is not alone predictive of sample-efficient linguistic acquisition. Explicit structural organization, via landmark tokens that reduce dependency retrieval ambiguity, constitutes a complementary—and potentially more critical—factor.

Practically, these results motivate PPT design optimizing both structural expressivity and accessibility of dependency cues. Theoretically, the results call for a refinement of model inductive bias frameworks, incorporating dependency organization as an independent axis beyond traditional circuit complexity.

Scaling and generalization of these findings remain open: verifying linear scalability to larger model architectures and multilingual settings is necessary. Operationalizing dependency ambiguity quantitatively for diverse linguistic typologies also represents a key methodological requirement.

Conclusion

The paper proposes and empirically validates LAD-inspired PPT using MP-STRUCT, demonstrating comparative token efficiency and novel human-like inductive biases relative to prior formal structure baselines. The results formally challenge the sufficiency of Circuit-RASP expressivity and highlight dependency organization as a critical factor in efficient LLM training. These insights contribute to both practical pretraining optimization and theoretical understanding of inductive biases in artificial language learners, with future directions aimed at scaling, cross-lingual generalization, and formal characterization of dependency ambiguity.