MP-STRUCT: Synthetic Language for Pre-Pretraining

• MP-STRUCT is a synthetic language that incorporates Merge, Agree, and Move to encode hierarchical syntactic structure for pre-pretraining language models.
• It utilizes a derivational system with explicit functional landmarks to reduce ambiguity in dependency retrieval for transformer models.
• Empirical results indicate that MP-STRUCT CORE outperforms baseline systems on token efficiency and dependency resolution metrics.

Searching arXiv for papers on MP-STRUCT and closely related synthetic-language pre-pretraining work. arxiv_search(query="MP-STRUCT Language Acquisition Device in LLMs", max_results=5) arxiv_search(query="MP-STRUCT synthetic language pre-pretraining LLMs", max_results=10) arxiv_search(query="pre-pretraining synthetic languages k-Shuffle Dyck LLMs", max_results=10) MP-STRUCT is a synthetic sequence generator for pre-pretraining LLMs that operationalizes three Minimalist Program operations—Merge, Agree, and Move—into token sequences encoding hierarchical composition, feature-based dependencies, and long-distance displacement. It was introduced as a LAD-inspired pre-pretraining language intended to constrain the learner’s hypothesis space toward natural-language-like structure with head-driven organization, directional displacement, and explicit functional landmarks that reduce dependency retrieval ambiguity for attention-based models (Mita et al., 16 May 2026).

1. Conceptual definition and design objective

MP-STRUCT is defined at two levels: a derivational system over syntactic objects supporting MERGE, AGREE, and MOVE, and a serializing grammar that linearizes those derivations into strings with explicit structural tokens, features, and movement traces. The central design goal is LAD-inspired pre-pretraining: the language is meant to encode head-driven hierarchy, structure-dependent agreement, and directional movement in a form that is accessible to transformer architectures (Mita et al., 16 May 2026).

A syntactic object has the form

$$X = (\mathrm{label}(X), \mathrm{fs}(X), \mathrm{children}(X)),$$

where $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$, $\mathrm{fs}$ is a feature structure over a finite feature inventory, and $\mathrm{children}$ is an ordered list of syntactic objects. The feature inventory is

$\Phi = \{Num, uNum, wh, EPP\}.$

Interpretable number features are $iNum \in \{sg, pl\}$; uninterpretable or valued features include $uNum \in \{sg, pl\}$, $wh \in \{+, -\}$, and $EPP \in \{+, -\}$ (Mita et al., 16 May 2026).

The language is expressly structured around functional heads and clause-spine organization. This means that MP-STRUCT does not merely encode nested bracketing. It encodes who licenses which dependency, where movement lands, and how agreement is valued. The paper argues that these landmarks make dependency identification more accessible than in generic formal languages that encode hierarchy but provide fewer retrieval cues. A plausible implication is that the representational burden is shifted away from purely latent induction and toward explicit structural indexing.

2. Derivational system: Merge, Agree, and Move

MERGE is binary composition. Given syntactic objects $X$ and $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$0, $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$1 yields a new object $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$2 whose label is determined by projection and whose children are $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$3, subject to head-selection constraints. In the transitive frame used in the generator, $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$4 selects $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$5, $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$6 selects $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$7, $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$8 selects $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$9, and $\mathrm{fs}$0 selects $\mathrm{fs}$1 (Mita et al., 16 May 2026).

AGREE values uninterpretable features on a head from interpretable features on a goal under locality and c-command. For agreement features $\mathrm{fs}$2, here $\mathrm{fs}$3, the compact formulation is

$\mathrm{fs}$4

with $\mathrm{fs}$5. The concrete instantiation used is subject–T agreement:

$\mathrm{fs}$6

MOVE copies a constituent to a higher position when licensed by a functional head and leaves a trace at the original position. Subject raising is licensed by $\mathrm{fs}$7 and wh-movement by $\mathrm{fs}$8. Formally, MOVE creates a copy $\mathrm{fs}$9 in the specifier of the licensor’s projection and replaces the original occurrence with $\mathrm{children}$0, together with a dependency link $\mathrm{children}$1. Movement must target the edge of the next phase head and is monotonic and leftward in the serialization adopted here (Mita et al., 16 May 2026).

The well-formedness conditions are correspondingly explicit. Selection requires compatible complements; AGREE requires local valuation of $\mathrm{children}$2 by the subject $\mathrm{children}$3; MOVE requires licensed landing sites, traces, and a single filler-gap dependency per clause. These constraints are not incidental. They determine which strings are generated at all, and they are part of the intended inductive bias.

3. Serialization, traces, and functional landmarks

MP-STRUCT linearizes derived trees in pre-order and emits structural brackets, category labels, functional heads, and their features, while stripping lexical content in the reported setup. The alphabet includes brackets $\mathrm{children}$4, dependency markers, category labels such as $\mathrm{children}$5, $\mathrm{children}$6, $\mathrm{children}$7, $\mathrm{children}$8, $\mathrm{children}$9, and feature renderings such as $\Phi = \{Num, uNum, wh, EPP\}.$0, $\Phi = \{Num, uNum, wh, EPP\}.$1, $\Phi = \{Num, uNum, wh, EPP\}.$2, $\Phi = \{Num, uNum, wh, EPP\}.$3, $\Phi = \{Num, uNum, wh, EPP\}.$4, $\Phi = \{Num, uNum, wh, EPP\}.$5, and $\Phi = \{Num, uNum, wh, EPP\}.$6. The start symbol is $\Phi = \{Num, uNum, wh, EPP\}.$7 (Mita et al., 16 May 2026).

A canonical derivation proceeds as follows. First, $\Phi = \{Num, uNum, wh, EPP\}.$8 and $\Phi = \{Num, uNum, wh, EPP\}.$9 are built by merging $iNum \in \{sg, pl\}$0 and $iNum \in \{sg, pl\}$1. Then $iNum \in \{sg, pl\}$2 is formed by merging $iNum \in \{sg, pl\}$3 with $iNum \in \{sg, pl\}$4, and $iNum \in \{sg, pl\}$5 by combining the subject with $iNum \in \{sg, pl\}$6. Next, $iNum \in \{sg, pl\}$7 is merged, AGREE values $iNum \in \{sg, pl\}$8, and the subject moves to Spec-TP, leaving $iNum \in \{sg, pl\}$9. Optionally, if $uNum \in \{sg, pl\}$0, a $uNum \in \{sg, pl\}$1 is copied to Spec-CP and a trace is left in its original position (Mita et al., 16 May 2026).

A representative lexical-stripped string is: $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$19

The paper also gives invalid configurations. An agreement violation assigns $uNum \in \{sg, pl\}$2 to $uNum \in \{sg, pl\}$3 while the subject bears $uNum \in \{sg, pl\}$4. A movement violation fills Spec-TP without leaving $uNum \in \{sg, pl\}$5 in the lower position (Mita et al., 16 May 2026).

Functional landmarks are a defining feature. Full MP-STRUCT uses explicit functional heads such as $uNum \in \{sg, pl\}$6, $uNum \in \{sg, pl\}$7, and $uNum \in \{sg, pl\}$8 with features and consistent placement adjacent to dependency sites. In the CORE variant, these are replaced by abstract tokens $uNum \in \{sg, pl\}$9, $wh \in \{+, -\}$0, and $wh \in \{+, -\}$1. Their stated roles are localization and directionality: they identify the domain and licensor responsible for each dependency and encode the clause spine $wh \in \{+, -\}$2. The paper’s interpretation is that such landmarks reduce the ambiguity of antecedent retrieval for attention-based models.

4. Pre-pretraining protocol and empirical results

The reported corpus uses $wh \in \{+, -\}$3 sequences with maximum sequence length $wh \in \{+, -\}$4, lexical terminals stripped, $wh \in \{+, -\}$5 on $wh \in \{+, -\}$6 set to True, $wh \in \{+, -\}$7, $wh \in \{+, -\}$8, number prior $wh \in \{+, -\}$9, and agreement-match ratio $EPP \in \{+, -\}$0. Tokenization uses the Pythia-1B tokenizer on emitted structural tokens. The base model is Pythia-1B with context length $EPP \in \{+, -\}$1 and AdamW optimization with $EPP \in \{+, -\}$2, $EPP \in \{+, -\}$3, $EPP \in \{+, -\}$4, learning rate $EPP \in \{+, -\}$5 with cosine schedule and warmup, minimum learning rate $EPP \in \{+, -\}$6, weight decay $EPP \in \{+, -\}$7, gradient clipping $EPP \in \{+, -\}$8, mixed precision bf16, batch size $EPP \in \{+, -\}$9, gradient accumulation $X$0, effective batch $X$1, and a pre-pretraining budget of $X$2 steps (Mita et al., 16 May 2026).

The downstream evaluation uses token-efficiency metrics per Hu et al., BLiMP minimal pairs, Jabberwocky sensitivity $X$3, and structural selectivity $X$4 on impossible languages such as SHUFFLE, REVERSE, and HOP. The main quantitative comparisons are:

Variant	MRS / Efficiency Gain	BLiMP
MP-STRUCT	$X$5 / $X$6	$X$7
$X$8-Shuffle Dyck	$X$9 / $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$00	$$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$01
MP-STRUCT CORE	$$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$02 / $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$03	$$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$04

These results support three distinct claims. First, MP-STRUCT matches the strong formal-language baseline $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$05-Shuffle Dyck in token efficiency. Second, MP-STRUCT CORE surpasses it on MRS and Efficiency Gain. Third, BLiMP performance is comparable to the non-PPT baseline for MP-STRUCT and significant versus Non-PPT for MP-STRUCT CORE and $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$06-Shuffle Dyck (Mita et al., 16 May 2026).

The paper also reports that removing any single operation—Merge, Agree, or Move—degrades final loss relative to full MP-STRUCT. Jabberwocky sensitivity is lower than for $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$07-Shuffle Dyck, suggesting stronger reliance on structural cues when semantics are attenuated. On impossible languages, MP-STRUCT shows positive structural selectivity in REVERSE, whereas $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$08-Shuffle Dyck yields $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$09 (Mita et al., 16 May 2026).

5. MP-STRUCT CORE and the C-RASP question

MP-STRUCT CORE retains the clause spine, agreement, movement, and fixed transitive selection while stripping lexical content and replacing explicit heads with abstract landmarks. Its abstract generation procedure samples $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$10 and agreement values, builds the $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$11 domain, introduces a $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$12 domain with $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$13 and agreement marking, optionally copies a subject into Spec-CP with $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$14 as movement licensor, and linearizes the resulting structure in pre-order (Mita et al., 16 May 2026).

The central theoretical claim attached to CORE is that it is not definable in C-RASP. The reason given is that the generator imposes joint adjacency constraints such as immediate placement of $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$15 before the bracket that starts the relevant Agree/Move domain, and such head–dependency adjacency requires predicates referencing two positions simultaneously. The paper contrasts this with prior hypotheses that effective synthetic pre-pretraining languages must be both hierarchically expressive and circuit-theoretically learnable (Mita et al., 16 May 2026).

This matters because CORE nonetheless outperforms $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$16-Shuffle Dyck. The paper’s stated implication is that effective PPT design depends not only on expressivity but also on the accessibility of dependency resolution. In this account, functional landmarks are not superficial annotation; they are an architectural affordance that changes how easily a transformer can retrieve the relevant dependency endpoints.

6. Theoretical framing, limits, and significance

The work situates MP-STRUCT within the Language Acquisition Device hypothesis. In that framing, natural-language learning is aided by innate constraints that restrict the learner’s hypothesis space toward head-driven hierarchy, structure-dependent agreement, and directional displacement. MP-STRUCT is presented as a practical instantiation of such constraints for pre-pretraining, without lexical semantics and within a brief $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$17-step budget (Mita et al., 16 May 2026).

The paper is explicit about its limitations. Validation is reported on Pythia-1B; scaling to larger models or alternate architectures remains open. BLiMP is English-only, so transfer to typologically diverse languages is unresolved. The notion of dependency identification ambiguity is argued to be useful, but it does not yet have a corpus-independent formal metric. Functional landmarks may also change vocabulary statistics and entropy, which the paper identifies as a potential confound. Future directions listed include controlling for unigram entropy, extending the generator to multi-clause and embedding structures, directly probing attention heads for landmark-driven retrieval, and varying wh/EPP rates as part of curriculum design (Mita et al., 16 May 2026).

Within the literature on synthetic languages for pre-pretraining, MP-STRUCT is notable for combining three properties in one formalism: explicit hierarchical serialization, feature-based dependencies, and displacement with traces. The empirical finding that MP-STRUCT CORE can outperform $$\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$$18-Shuffle Dyck despite being non-C-RASP-definable suggests that the decisive factor is not formal expressivity alone. The paper’s own conclusion is more specific: accessibility of dependency resolution is a key determinant of effective PPT language design (Mita et al., 16 May 2026).