---
title: 'MP-STRUCT: Synthetic Language for Pre-Pretraining'
url: https://www.emergentmind.com/topics/mp-struct
type: topic
---

# MP-STRUCT: Synthetic Language for Pre-Pretraining

Searching arXiv for recent papers on MP-STRUCT and closely related synthetic-language pre-pretraining work.
arxiv_search(query="MP-STRUCT Language Acquisition Device in Large Language Models", max_results=5)
arxiv_search(query="MP-STRUCT synthetic language pre-pretraining large language models", max_results=10)
arxiv_search(query="pre-pretraining synthetic languages k-Shuffle Dyck large language models", max_results=10)
MP-STRUCT is a synthetic sequence generator for pre-pretraining large language models that operationalizes three Minimalist Program operations—Merge, Agree, and Move—into token sequences encoding hierarchical composition, feature-based dependencies, and long-distance displacement. It was introduced as a LAD-inspired pre-pretraining language intended to constrain the learner’s hypothesis space toward natural-language-like structure with head-driven organization, directional displacement, and explicit functional landmarks that reduce dependency retrieval ambiguity for attention-based models [2605.16758].

## 1. Conceptual definition and design objective

MP-STRUCT is defined at two levels: a derivational system over syntactic objects supporting MERGE, AGREE, and MOVE, and a serializing grammar that linearizes those derivations into strings with explicit structural tokens, features, and movement traces. The central design goal is LAD-inspired pre-pretraining: the language is meant to encode head-driven hierarchy, structure-dependent agreement, and directional movement in a form that is accessible to transformer architectures [2605.16758].

A syntactic object has the form
$$
X = (\mathrm{label}(X), \mathrm{fs}(X), \mathrm{children}(X)),
$$
where $\mathrm{label} \in \{CP, C, TP, T, vP, V′, V, DP, D, N, TR\}$, $\mathrm{fs}$ is a feature structure over a finite feature inventory, and $\mathrm{children}$ is an ordered list of syntactic objects. The feature inventory is
$$
\Phi = \{Num, uNum, wh, EPP\}.
$$
Interpretable number features are $iNum \in \{sg, pl\}$; uninterpretable or valued features include $uNum \in \{sg, pl\}$, $wh \in \{+, -\}$, and $EPP \in \{+, -\}$ [2605.16758].

The language is expressly structured around functional heads and clause-spine organization. This means that MP-STRUCT does not merely encode nested bracketing. It encodes who licenses which dependency, where movement lands, and how agreement is valued. The paper argues that these landmarks make dependency identification more accessible than in generic formal languages that encode hierarchy but provide fewer retrieval cues. A plausible implication is that the representational burden is shifted away from purely latent induction and toward explicit structural indexing.

## 2. Derivational system: Merge, Agree, and Move

MERGE is binary composition. Given syntactic objects $X$ and $Y$, $\mathrm{MERGE}(X,Y)$ yields a new object $Z$ whose label is determined by projection and whose children are $[X,Y]$, subject to head-selection constraints. In the transitive frame used in the generator, $V$ selects $DP_{\mathrm{object}}$, $v$ selects $V′$, $T$ selects $vP$, and $C$ selects $TP$ [2605.16758].

AGREE values uninterpretable features on a head from interpretable features on a goal under locality and c-command. For agreement features $\Phi_{\mathrm{agr}} \subseteq \Phi$, here $\{Num\}$, the compact formulation is
$$
\forall f \in \Phi_{\mathrm{agr}}:\ \mathrm{val}(H, uf) = \mathrm{val}(G, if)
$$
with $G \in D(H)$. The concrete instantiation used is subject–T agreement:
$$
\mathrm{AGREE}(T[uNum], DP_{\mathrm{subj}}[iNum]) \Rightarrow \mathrm{val}(T, uNum) := \mathrm{val}(DP_{\mathrm{subj}}, iNum).
$$

MOVE copies a constituent to a higher position when licensed by a functional head and leaves a trace at the original position. Subject raising is licensed by $T[EPP=+]$ and wh-movement by $C[wh=+]$. Formally, MOVE creates a copy $X′$ in the specifier of the licensor’s projection and replaces the original occurrence with $TR[X]$, together with a dependency link $\delta(X′,TR[X])$. Movement must target the edge of the next phase head and is monotonic and leftward in the serialization adopted here [2605.16758].

The well-formedness conditions are correspondingly explicit. Selection requires compatible complements; AGREE requires local valuation of $T[uNum]$ by the subject $DP[iNum]$; MOVE requires licensed landing sites, traces, and a single filler-gap dependency per clause. These constraints are not incidental. They determine which strings are generated at all, and they are part of the intended inductive bias.

## 3. Serialization, traces, and functional landmarks

MP-STRUCT linearizes derived trees in pre-order and emits structural brackets, category labels, functional heads, and their features, while stripping lexical content in the reported setup. The alphabet includes brackets $[\,]$, dependency markers, category labels such as $CP$, $TP$, $VP$, $DP$, $TR$, and feature renderings such as $Num:sg$, $Num:pl$, $uNum:sg$, $uNum:pl$, $+EPP$, $wh:+$, and $wh:-$. The start symbol is $S \to CP$ [2605.16758].

A canonical derivation proceeds as follows. First, $DP_{\mathrm{subj}}$ and $DP_{\mathrm{obj}}$ are built by merging $D$ and $N$. Then $V′$ is formed by merging $V$ with $DP_{\mathrm{obj}}$, and $vP$ by combining the subject with $V′$. Next, $T[+EPP,uNum:?]$ is merged, AGREE values $uNum$, and the subject moves to Spec-TP, leaving $TR[DP]$. Optionally, if $wh=+$, a $DP$ is copied to Spec-CP and a trace is left in its original position [2605.16758].

A representative lexical-stripped string is:
```text
[ CP [ C ] [ [ TP [ [ DP[Num:pl] [ D ] [ N ] ] ] [ T(+EPP,uNum:pl) ] [ [ VP V [ [ DP[Num:pl] [ D ] [ N ] ] ] [ TR[DP] ] ] ] ] ] ]
```

The paper also gives invalid configurations. An agreement violation assigns $uNum:sg$ to $T$ while the subject bears $Num:pl$. A movement violation fills Spec-TP without leaving $TR[DP]$ in the lower position [2605.16758].

Functional landmarks are a defining feature. Full MP-STRUCT uses explicit functional heads such as $C$, $T$, and $v$ with features and consistent placement adjacent to dependency sites. In the CORE variant, these are replaced by abstract tokens $H_{CP}$, $H_{TP}$, and $H_{VP}$. Their stated roles are localization and directionality: they identify the domain and licensor responsible for each dependency and encode the clause spine $CP \to TP \to vP$. The paper’s interpretation is that such landmarks reduce the ambiguity of antecedent retrieval for attention-based models.

## 4. Pre-pretraining protocol and empirical results

The reported corpus uses $n = 100{,}000$ sequences with maximum sequence length $1{,}024$, lexical terminals stripped, $EPP$ on $T$ set to True, $P(C[+wh]) = 0.2$, $P(DP[-wh]) = 0.2$, number prior $P(sg)=0.5$, and agreement-match ratio $1.0$. Tokenization uses the Pythia-1B tokenizer on emitted structural tokens. The base model is Pythia-1B with context length $1{,}024$ and AdamW optimization with $\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=10^{-6}$, learning rate $5\times10^{-4}$ with cosine schedule and warmup, minimum learning rate $5\times10^{-5}$, weight decay $0.1$, gradient clipping $1.0$, mixed precision bf16, batch size $16$, gradient accumulation $2$, effective batch $32$, and a pre-pretraining budget of $500$ steps [2605.16758].

The downstream evaluation uses token-efficiency metrics per Hu et al., BLiMP minimal pairs, Jabberwocky sensitivity $\Delta_{\mathrm{sens}}$, and structural selectivity $\Delta_{\mathrm{sel}}$ on impossible languages such as SHUFFLE, REVERSE, and HOP. The main quantitative comparisons are:

| Variant | MRS / Efficiency Gain | BLiMP |
|---|---:|---:|
| MP-STRUCT | $15.3 \pm 2.86$ / $0.29 \pm 0.057$ | $0.755$ |
| $k$-Shuffle Dyck | $15.6 \pm 3.31$ / $0.29 \pm 0.068$ | $0.764$ |
| MP-STRUCT CORE | $16.2 \pm 2.97$ / $0.31 \pm 0.055$ | $0.764$ |

These results support three distinct claims. First, MP-STRUCT matches the strong formal-language baseline $k$-Shuffle Dyck in token efficiency. Second, MP-STRUCT CORE surpasses it on MRS and Efficiency Gain. Third, BLiMP performance is comparable to the non-PPT baseline for MP-STRUCT and significant versus Non-PPT for MP-STRUCT CORE and $k$-Shuffle Dyck [2605.16758].

The paper also reports that removing any single operation—Merge, Agree, or Move—degrades final loss relative to full MP-STRUCT. Jabberwocky sensitivity is lower than for $k$-Shuffle Dyck, suggesting stronger reliance on structural cues when semantics are attenuated. On impossible languages, MP-STRUCT shows positive structural selectivity in REVERSE, whereas $k$-Shuffle Dyck yields $\Delta_{\mathrm{sel}} \approx 0$ [2605.16758].

## 5. MP-STRUCT CORE and the C-RASP question

MP-STRUCT CORE retains the clause spine, agreement, movement, and fixed transitive selection while stripping lexical content and replacing explicit heads with abstract landmarks. Its abstract generation procedure samples $wh$ and agreement values, builds the $VP$ domain, introduces a $TP$ domain with $H_{TP}$ and agreement marking, optionally copies a subject into Spec-CP with $H_{CP}$ as movement licensor, and linearizes the resulting structure in pre-order [2605.16758].

The central theoretical claim attached to CORE is that it is not definable in C-RASP. The reason given is that the generator imposes joint adjacency constraints such as immediate placement of $H_{TP}$ before the bracket that starts the relevant Agree/Move domain, and such head–dependency adjacency requires predicates referencing two positions simultaneously. The paper contrasts this with prior hypotheses that effective synthetic pre-pretraining languages must be both hierarchically expressive and circuit-theoretically learnable [2605.16758].

This matters because CORE nonetheless outperforms $k$-Shuffle Dyck. The paper’s stated implication is that effective PPT design depends not only on expressivity but also on the accessibility of dependency resolution. In this account, functional landmarks are not superficial annotation; they are an architectural affordance that changes how easily a transformer can retrieve the relevant dependency endpoints.

## 6. Theoretical framing, limits, and significance

The work situates MP-STRUCT within the Language Acquisition Device hypothesis. In that framing, natural-language learning is aided by innate constraints that restrict the learner’s hypothesis space toward head-driven hierarchy, structure-dependent agreement, and directional displacement. MP-STRUCT is presented as a practical instantiation of such constraints for pre-pretraining, without lexical semantics and within a brief $500$-step budget [2605.16758].

The paper is explicit about its limitations. Validation is reported on Pythia-1B; scaling to larger models or alternate architectures remains open. BLiMP is English-only, so transfer to typologically diverse languages is unresolved. The notion of dependency identification ambiguity is argued to be useful, but it does not yet have a corpus-independent formal metric. Functional landmarks may also change vocabulary statistics and entropy, which the paper identifies as a potential confound. Future directions listed include controlling for unigram entropy, extending the generator to multi-clause and embedding structures, directly probing attention heads for landmark-driven retrieval, and varying wh/EPP rates as part of curriculum design [2605.16758].

Within the literature on synthetic languages for pre-pretraining, MP-STRUCT is notable for combining three properties in one formalism: explicit hierarchical serialization, feature-based dependencies, and displacement with traces. The empirical finding that MP-STRUCT CORE can outperform $k$-Shuffle Dyck despite being non-C-RASP-definable suggests that the decisive factor is not formal expressivity alone. The paper’s own conclusion is more specific: accessibility of dependency resolution is a key determinant of effective PPT language design [2605.16758].

Source: https://www.emergentmind.com/topics/mp-struct