MP-Struct Core: Formal Pre-Pretraining Language

• MP-Struct Core is a formal language defined by a fixed CP–TP–vP structure, a single recursive bracket type, and four dependency types to isolate structural organization.
• It employs explicit functional landmarks (H_CP, H_TP, H_VP) and bracketed dependencies to reduce identification ambiguity and streamline dependency resolution.
• Empirical results using a Pythia-1B Transformer show that MP-Struct Core offers improved token efficiency and transfer progress over k-Shuffle Dyck despite its non-C-RASP definability.

MP-Struct Core is a pure formal language introduced as a deliberately stripped-down, fully formal version of the MP-Struct generator for LAD-inspired pre-pretraining of LLMs. It preserves the Minimalist core of MERGE, AGREE, and MOVE, while abstracting away lexical content and most feature detail, so that the analysis can isolate how structural information is organized rather than how much structure is expressible. In the source formulation, its central design choice is a fixed CP–TP–vP skeleton with explicit functional landmarks, one structural bracket type, and four dependency types, yielding a language that is formally highly structured yet engineered to test whether dependency organization and accessibility matter for token-efficient transfer to natural-language pretraining (Mita et al., 16 May 2026).

1. Formal definition and symbolic inventory

MP-Struct Core is introduced as an abstract, fixed-topology version of full MP-Struct. Its strings are built from a small inventory of symbolic primitives rather than lexical items. The structural inventory contains one type of hierarchy bracket, [ ], and a family of typed dependency brackets, (k ... )k. Its functional heads are abstract head tokens $H_{CP}$, $H_{TP}$, and $H_{VP}$, corresponding to complementizer-like, T-like, and vP-/V-like heads. Its dependency types are an abstract agreement feature $\mathrm{AGR} \in \{\mathrm{AGR}_A,\mathrm{AGR}_B\}$, a selectional dependency $\mathrm{SEL}$, and a movement dependency $\mathrm{MOVE}$. A binary feature $wh \in \{+,-\}$ controls whether movement is present, and trace tokens $t$ mark displacement sites (Mita et al., 16 May 2026).

The language consists of token sequences $S$ built by repeatedly generating one clause skeleton and concatenating until a target length $L$ is reached. In the experimental setup described for the abstraction study, $H_{TP}$0. The fixed structural spine is: $H_{TP}$1 This spine is not merely expository. It constrains where each dependency type can occur: agreement is encoded in the TP region, movement is anchored in CP, and selection is anchored in vP. The paper characterizes the resulting system as retaining a single recursive structural bracket type, four dependency types implemented as disjoint kinds of bracket arcs, a fixed topological arrangement of those dependencies, and explicit head tokens adjacent to dependency sites (Mita et al., 16 May 2026).

The full MP-Struct generator is procedural rather than a traditional grammar. It builds structural strings by composing MERGE, AGREE, and MOVE and then linearizing the resulting tree in preorder. In the full system, lexical items $H_{TP}$2 are sampled, but lexical items are removed from the output so that the observed sequences are purely structural: $H_{TP}$3

$H_{TP}$4

$H_{TP}$5

MP-Struct Core abstracts this richer generator into a formal language whose tokens are structural markers, dependency markers, head landmarks, and traces rather than category-bearing lexical material (Mita et al., 16 May 2026).

2. Generative procedure and relation to full MP-Struct

The core generator proceeds in four stages. First, it constructs a vP domain. It samples $H_{TP}$6 and $H_{TP}$7, introduces the head token $H_{TP}$8, and builds a vP containing an internal subject slot, an object, and a selection dependency. The paper describes this as a structure of the form

$H_{TP}$9

followed by an internal shuffle of vP-internal elements to vary surface order while preserving linkage (Mita et al., 16 May 2026).

Second, it adds functional structure and agreement. The selected agreement feature is assigned to $H_{VP}$0, a T head with an agreement feature is created, and TP is formed as: $H_{VP}$1 If $H_{VP}$2, the subject occupies Spec-TP; if $H_{VP}$3, Spec-TP is left empty because the subject will move higher. The paper’s point is that subject–head agreement is encoded and marked by the functional landmark $H_{VP}$4 (Mita et al., 16 May 2026).

Third, it applies movement. If $H_{VP}$5, the subject is copied to Spec-CP, a trace is left in its lower position, and C bears the movement licensor: $H_{VP}$6

$H_{VP}$7

If $H_{VP}$8, the CP layer is present but movement is absent: $H_{VP}$9 Fourth, the tree is linearized in preorder, outputting structural brackets, dependency brackets, head tokens, and traces, and clause skeletons are concatenated until the length target is reached (Mita et al., 16 May 2026).

The relation between full MP-Struct and MP-Struct Core is therefore asymmetric. Core retains the fixed hierarchical skeleton, the three core dependency types, and the functional landmarks, but omits lexical categories, concrete feature values such as Num:sg/pl and +EPP, and the richer category labels visible in the full output. The paper explicitly states that MP-Struct Core captures only a single recursive structural bracket type, four dependency types, fixed topological arrangement of those dependencies in a CP–TP–vP spine, and explicit head tokens adjacent to dependency sites (Mita et al., 16 May 2026).

A full MP-Struct example is given as: $\mathrm{SEL}$9

An abstract MP-Struct Core example is: $\mathrm{MOVE}$0

These examples illustrate the transition from feature-rich but still lexical-less structural strings to an abstract dependency language whose entire burden is structural organization (Mita et al., 16 May 2026).

3. Expressivity, calibration, and non-definability in C-RASP

The paper calibrates MP-Struct Core against formal-language baselines using two measures: hierarchical structure and dependency type inventory. In that calibration,

$\mathrm{AGR} \in \{\mathrm{AGR}_A,\mathrm{AGR}_B\}$0

so that MP-Struct Core implements “1-Dyck + 4-Shuffle-Dyck.” This means that, at the level of raw dependency inventory, it matches a suitably calibrated shuffle-Dyck language while differing in how dependencies are arranged and cued (Mita et al., 16 May 2026).

This calibration is not meant to assert equivalence of organization. The paper emphasizes that generic $\mathrm{AGR} \in \{\mathrm{AGR}_A,\mathrm{AGR}_B\}$1-Shuffle Dyck encodes dependencies purely through bracket matching, with all bracket tokens of the same type indistinguishable and no head landmarks. MP-Struct Core uses the same cardinality of dependency types but restricts them to a fixed CP–TP–vP scaffolding, places distinct head tokens near the dependencies, and limits where each dependency can occur. It is therefore presented as highly expressive but topologically constrained (Mita et al., 16 May 2026).

A central theoretical claim is that MP-Struct Core is not definable in C-RASP. The paper’s argument is that the generator enforces a strict adjacency constraint: functional head tokens such as $\mathrm{AGR} \in \{\mathrm{AGR}_A,\mathrm{AGR}_B\}$2 are systematically placed immediately before their associated dependency brackets, and expressing that constraint requires a predicate that jointly references both the head position and the dependency position. Because C-RASP is described as a restriction of FO(M) permitting only single-index predicates per quantifier, it cannot express such a two-position constraint. The immediate significance in the paper is that MP-Struct Core falls outside the class proposed by the prior “Expressivity Hypothesis,” according to which effective pre-pretraining languages should be hierarchically structured and C-RASP-definable (Mita et al., 16 May 2026).

The paper uses this result polemically but narrowly. It does not claim that circuit-theoretic considerations are irrelevant. Rather, it argues that C-RASP-definability is neither necessary nor sufficient for effective pre-pretraining, because MP-Struct Core, despite being non-C-RASP-definable, still matches or slightly exceeds the token efficiency of the previously strongest formal-language baseline (Mita et al., 16 May 2026).

4. Role in pre-pretraining and empirical performance

MP-Struct Core is used in the paper’s abstraction study rather than as the sole language in the full experimental program. Its role is to remove lexical and morphological detail while preserving one structural bracket type, four dependency types, fixed topology, and functional landmarks. This lets the experiments ask whether organization and landmarking matter beyond raw expressivity (Mita et al., 16 May 2026).

The training setup is held constant across pre-pretraining conditions. The model is a Pythia-1B Transformer, approximately 1B parameters. Pre-pretraining runs for 500 optimization steps on synthetic data with standard autoregressive next-token prediction. The paper reports a learning rate of $\mathrm{AGR} \in \{\mathrm{AGR}_A,\mathrm{AGR}_B\}$3, cosine schedule with 1000 warmup steps, AdamW, batch size 16 with gradient accumulation to 32, and maximum length 1024. After pre-pretraining, weights are transferred to pretraining on C4 for 25,000 steps (Mita et al., 16 May 2026).

The main comparison relevant to MP-Struct Core is against $\mathrm{AGR} \in \{\mathrm{AGR}_A,\mathrm{AGR}_B\}$4-Shuffle Dyck. The reported results are:

PPT language	BLiMP	MRS	Efficiency Gain
MP-Struct Core	0.764	$\mathrm{AGR} \in \{\mathrm{AGR}_A,\mathrm{AGR}_B\}$5	$\mathrm{AGR} \in \{\mathrm{AGR}_A,\mathrm{AGR}_B\}$6
$\mathrm{AGR} \in \{\mathrm{AGR}_A,\mathrm{AGR}_B\}$7-Shuffle Dyck	0.764	$\mathrm{AGR} \in \{\mathrm{AGR}_A,\mathrm{AGR}_B\}$8	$\mathrm{AGR} \in \{\mathrm{AGR}_A,\mathrm{AGR}_B\}$9

These figures support two claims made explicitly in the paper. First, MP-Struct Core matches the formal-language baseline on BLiMP while slightly exceeding it on MRS and Efficiency Gain. Second, this occurs despite MP-Struct Core being non-C-RASP-definable. The paper therefore frames MP-Struct Core as evidence against the view that effective pre-pretraining languages must simultaneously maximize hierarchical expressivity and remain within a particular circuit-theoretic expressivity class (Mita et al., 16 May 2026).

The paper also reports that all abstract conditions improve over Non-PPT, that MP-Struct Core consistently achieves lower C4 validation loss at 25,000 pretraining steps than Generic $\mathrm{SEL}$0-SD, and that the MRS and Efficiency Gain improvements indicate that each MP-Struct-Core pre-pretraining step buys more natural-language pretraining progress than a $\mathrm{SEL}$1-Shuffle-Dyck step (Mita et al., 16 May 2026).

5. Functional landmarks and dependency resolution

The paper gives special explanatory weight to “functional landmarks.” In the full MP-Struct setting, these are functional categories such as T, C, and v. In MP-Struct Core, they are abstracted to $\mathrm{SEL}$2, $\mathrm{SEL}$3, and $\mathrm{SEL}$4. The stated role of a functional landmark is to signal the presence and type of nearby dependencies and sharply narrow the set of candidate antecedents (Mita et al., 16 May 2026).

This is tied to what the paper calls “dependency identification ambiguity.” That notion is defined as the extent to which a dependency endpoint such as )1 provides sufficient cues to uniquely identify its corresponding start point (1. When cues are limited, multiple candidate antecedents remain plausible; when structural markers are present, the candidate set is sharply constrained (Mita et al., 16 May 2026).

The contrast is made concrete through the paper’s examples. A Generic $\mathrm{SEL}$5-SD sequence is: $\mathrm{MOVE}$1 An MP-Struct Core sequence is: $\mathrm{MOVE}$2 In the first case, dependencies are randomly interleaved and unlabeled except for the bracket index. In the second, movement is locally associated with $\mathrm{SEL}$6, agreement with $\mathrm{SEL}$7, and selection with $\mathrm{SEL}$8. The paper’s interpretation is that a transformer can focus attention around the landmark token to resolve the appropriate dependency instead of searching globally among all openers of the same type (Mita et al., 16 May 2026).

This interpretive claim is central but remains partly inferential. The paper states that functional landmarks reduce retrieval ambiguity and that reduced dependency identification ambiguity may contribute to more efficient learning. A plausible implication is that the empirical advantage of MP-Struct Core arises less from raw formal expressivity than from making dependency resolution more accessible to a decoder-only transformer trained by next-token prediction (Mita et al., 16 May 2026).

6. Theoretical significance, misconceptions, and open questions

Within the paper’s broader LAD framing, MP-Struct Core functions as an operational hypothesis-space restriction. It restricts synthetic pre-pretraining data to structures with a fixed, head-driven CP–TP–vP architecture, feature-driven dependencies, and directional movement patterns. The paper argues that this restriction induces two outcomes: improved token efficiency in later natural-language pretraining and more human-like resistance to structurally implausible systems such as REVERSE (Mita et al., 16 May 2026).

One misconception directly addressed by the results is that a pre-pretraining language should be judged primarily by how much structure it can express. MP-Struct Core is deliberately stripped down. It removes lexical content and most feature detail, yet still outperforms a stronger-seeming formal benchmark in token efficiency. The paper’s interpretation is that effective design depends not only on expressivity but also on the accessibility of dependency resolution. Another misconception challenged by the paper is that C-RASP-definability is a necessary condition for transfer-effective synthetic languages; MP-Struct Core is presented as a counterexample (Mita et al., 16 May 2026).

The paper is explicit about its limits. All experiments use Pythia-1B and a standard decoder-only transformer, so persistence of the effect at 7B or 70B is unknown. MP-Struct Core abstracts away lexical content, morphology, islands, control, binding, and other natural-language phenomena. “Dependency identification ambiguity” is proposed as an explanatory notion, but not formally quantified across arbitrary corpora. The paper also notes a confound between adding head landmarks and increasing vocabulary size, and it leaves open the exact relationship between logical definability and learned representations in finite-capacity transformers (Mita et al., 16 May 2026).

Taken together, MP-Struct Core occupies a specific position in current research on synthetic pre-pretraining languages. It is neither a general grammar of natural language nor merely another shuffle-Dyck variant. It is a tightly controlled formal language whose importance lies in the claim that fixed topology, explicit landmarks, and organized dependency placement can improve transfer efficiency even when the language lies outside a previously favored expressivity class. This suggests that, for pre-pretraining design, the organization of structure may be as consequential as the amount of structure available (Mita et al., 16 May 2026).