Structure-Inducing Language Models

• SiLMs are language models that induce latent hierarchical structures using self-supervised objectives, enabling implicit syntax and structural biases.
• They employ diverse architectures like StructFormer, UDGN, and GPST, which differ in parsing mechanisms, induced structures, and computational efficiency.
• Evaluations reveal trade-offs between grammatical accuracy and efficiency, emphasizing the need for robust and consistent structure induction methods.

Structure-inducing LLMs (SiLMs) are a family of language-modeling approaches in which structural regularities are induced as part of learning or inference rather than supplied as gold supervision. In the narrow syntactic sense, a SiLM is a neural model trained with a self-supervised language-modeling objective that, when processing an input sequence $x$, induces an unobserved hierarchical representation $t_x$ such as a tree or bracket structure without ever seeing gold parses during training (Arps et al., 11 Aug 2025). In adjacent literatures, the label also appears in work on structure-inducing pre-training over graphs, causal intervention objectives that carve out symbolic subspaces, post-hoc steering of internal states, and domain-specific “structure language modeling” for proteins and structured retrieval (McDermott et al., 2021, Huang et al., 2022, Belitsky et al., 11 Jul 2025, Lu et al., 2024). Across these variants, the central question is how structural bias can be made operative inside a LLM’s representations, dynamics, or decoding process.

1. Formal definitions and conceptual scope

The canonical syntactic definition treats a SiLM as any model that satisfies two conditions: it is trained with a self-supervised language-modeling objective, and it induces a latent hierarchical representation as a byproduct of processing an input (Arps et al., 11 Aug 2025). For an autoregressive model, the objective is

$$\mathcal{L}(\theta) = - \sum_{t=1}^n \log p_\theta(x_t \mid x_{<t}) \; .$$

For masked-language-model variants, the same cross-entropy loss is applied to a subset of tokens masked out at random, typically $15\%$ of positions. In this formulation, structure is latent: the model posits a discrete $z$ and defines

$p_{\theta}(x) = \sum_{z} p_{\theta}(x,z) \; ,$

with test-time decoding sometimes taking $\hat z=\arg\max_z p_\theta(z\mid x)$ (Arps et al., 11 Aug 2025).

This syntactic notion contrasts with a standard transformer or recurrent LM that performs sequential prediction without explicitly committing to a hierarchical parse. A related line of work generalizes the structural objective from sentence-internal syntax to the geometry of the latent space itself. In Structure-Inducing Pre-Training (SIPT), the pre-training graph $G_{PT}$ is treated as a first-class input, and an encoder $f_\theta$ is optimized with

$$\mathcal L_{PT}(\theta) = (1-\lambda)\,M(\theta) + \lambda\,SI(\theta) \; ,$$

where $t_x$0 is a conventional intra-sample objective such as MLM and $t_x$1 explicitly regularizes pairwise geometry so that graph edges are recoverable by distance in the learned embedding space (McDermott et al., 2021). That framework introduces the distinctions between explicit and implicit structural constraints, and between shallow and deep constraints. This broader perspective suggests that SiLM research is not restricted to syntax induction: it also studies how structure can be encoded in neighborhood geometry, token subspaces, discourse plans, or cache states.

2. Core syntactic architectures

The most developed SiLM literature concerns unsupervised syntactic induction from self-supervised language modeling. A comparative study of three representative architectures—StructFormer, UDGN, and GPST—shows that they differ primarily in where the parser is placed, what structure is induced, and how that structure constrains token prediction (Arps et al., 11 Aug 2025).

Architecture	Induced structure	Main mechanism
StructFormer	Soft dependency-style head matrix $t_x$2	Transformer encoder split into front block, unsupervised CNN parser, and back block with gated attention
UDGN	Soft dependency adjacency matrix $t_x$3	BiLSTM parser plus multi-layer Dependency Graph Network trained with masked LM
GPST	Binary constituency tree plus actions	Inside-outside autoencoder with autoregressive shift-reduce transformer

StructFormer uses a shallow unsupervised CNN parser to predict adjacent-pair distance scores $t_x$4 and token heights $t_x$5, from which a full head matrix $t_x$6 is constructed. Attention in the back block is then multiplied by learned gating parameters $t_x$7, so parsing and language modeling are coupled through attention control (Arps et al., 11 Aug 2025). A detailed analysis of StructFormer shows that the parser network predicts syntactic distances and heights, and that an unlabeled binary constituency tree can be recovered at test time by recursively splitting at the largest learned distance $t_x$8 (Momen, 2024).

UDGN replaces the CNN parser with a bidirectional LSTM parser that predicts a soft dependency adjacency matrix $t_x$9, then feeds that structure into a gated multi-head attention stack called the Dependency Graph Network. Its scoring mechanism is identical in spirit to StructFormer’s gated-attention masked LM, but the parser is LSTM-based rather than convolutional (Arps et al., 11 Aug 2025).

GPST differs more substantially. It combines a bidirectional inside-outside autoencoder, which induces a binary constituency tree using a pruned chart parser heuristic, with an autoregressive shift-reduce transformer that generates both parse actions and next tokens. Its total loss is

$$\mathcal{L}(\theta) = - \sum_{t=1}^n \log p_\theta(x_t \mid x_{<t}) \; .$$0

Because GPST jointly optimizes constituency induction, action prediction, and next-token prediction, it occupies a middle ground between grammar induction and generative language modeling (Arps et al., 11 Aug 2025).

3. Evaluation regimes and empirical behavior

Recent comparative evaluation has emphasized that SiLM assessment must go beyond a single parsing score. On English, the main setup uses $$\mathcal{L}(\theta) = - \sum_{t=1}^n \log p_\theta(x_t \mid x_{<t}) \; .$$1M tokens from the cleaned BabyLM corpus with a held-out $$\mathcal{L}(\theta) = - \sum_{t=1}^n \log p_\theta(x_t \mid x_{<t}) \; .$$2 test split, while on formal bracketing expressions it uses four Dyck languages—Dyck-1, Dyck-2, Dyck-64, and Dyck-u—with training splits of approximately $$\mathcal{L}(\theta) = - \sum_{t=1}^n \log p_\theta(x_t \mid x_{<t}) \; .$$3M tokens and generalization tests up to twice the training length, $$\mathcal{L}(\theta) = - \sum_{t=1}^n \log p_\theta(x_t \mid x_{<t}) \; .$$4 tokens (Arps et al., 11 Aug 2025). The evaluation suite spans induced-structure metrics such as UAS and constituent $$\mathcal{L}(\theta) = - \sum_{t=1}^n \log p_\theta(x_t \mid x_{<t}) \; .$$5, grammaticality judgments on BLiMP and Dyck minimal pairs, language-modeling loss, and training efficiency.

A central empirical finding is that no single SiLM dominates across all evaluation metrics. GPST performs most consistently across evaluation settings and outperforms the other models on long-distance dependencies in bracketing expressions (Arps et al., 11 Aug 2025). On Dyck minimal pairs, GPST consistently exceeds $$\mathcal{L}(\theta) = - \sum_{t=1}^n \log p_\theta(x_t \mid x_{<t}) \; .$$6 accuracy even for long-distance swaps, UDGN ranges around $$\mathcal{L}(\theta) = - \sum_{t=1}^n \log p_\theta(x_t \mid x_{<t}) \; .$$7, and StructFormer barely outperforms the transformer baseline at approximately $$\mathcal{L}(\theta) = - \sum_{t=1}^n \log p_\theta(x_t \mid x_{<t}) \; .$$8–$$\mathcal{L}(\theta) = - \sum_{t=1}^n \log p_\theta(x_t \mid x_{<t}) \; .$$9. On English BLiMP, the reported overall accuracies are approximately $15\%$0 for a standard RoBERTa-style transformer, $15\%$1 for GPST, $15\%$2 for StructFormer, and $15\%$3 for UDGN. These numbers make clear that inducing explicit structure is not equivalent to uniformly improving all forms of grammatical generalization.

The induced structures themselves can be unstable. Different random seeds yield significant variation in the induced parses of StructFormer and UDGN, with UAS variation of approximately $15\%$4–$15\%$5 points on English; GPST’s constituency $15\%$6 also varies across runs. On formal languages, only GPST remains consistent across runs, whereas StructFormer’s parser often collapses to near-uniform head distributions, described as “self-loops,” which yield almost random $15\%$7 (Arps et al., 11 Aug 2025). Training dynamics also show a separation between structure formation and token prediction: the first $15\%$8K steps produce large jumps in UAS or $15\%$9, but beyond $z$0K steps induced structure changes by less than $z$1 points even as perplexity continues to decline slowly.

Efficiency comparisons further differentiate the architectures. Per $z$2 steps on an A100 GPU, UDGN takes approximately $z$3 minutes on English, StructFormer approximately $z$4 minutes, and GPST approximately $z$5 minutes (Arps et al., 11 Aug 2025). A complementary thesis-scale study of StructFormer reports that, on Penn Treebank, StructFormer reduces adapted MLM-PPL from approximately $z$6 to $z$7 and raises UF1 from approximately $z$8 to $z$9; on the BabyLM benchmark, a StructFormer variant with parser inserted at $p_{\theta}(x) = \sum_{z} p_{\theta}(x,z) \; ,$0 reaches an aggregate score of approximately $p_{\theta}(x) = \sum_{z} p_{\theta}(x,z) \; ,$1, compared with approximately $p_{\theta}(x) = \sum_{z} p_{\theta}(x,z) \; ,$2 for the transformer baseline (Momen, 2024). Taken together, these results support a nuanced view: SiLMs can induce nontrivial syntax, but structure quality, downstream utility, and computational cost do not align perfectly.

4. Alternative induction mechanisms and post-hoc control

Not all SiLM work modifies the architecture. One line injects inductive bias through synthetic pre-training data while keeping the backbone fixed. Using GPT-2-small with no architectural modifications, pre-training on synthetic corpora that encode recursive Dyck structure, crossing non-context-free dependencies, or Zipfian vocabularies yields a controlled testbed for structural bias. Across English, Japanese, and Basque, the downstream perplexity ranking is reported as $p_{\theta}(x) = \sum_{z} p_{\theta}(x,z) \; ,$3, with English values of approximately $p_{\theta}(x) = \sum_{z} p_{\theta}(x,z) \; ,$4 for cross, $p_{\theta}(x) = \sum_{z} p_{\theta}(x,z) \; ,$5 for nest, $p_{\theta}(x) = \sum_{z} p_{\theta}(x,z) \; ,$6 for rep, and $p_{\theta}(x) = \sum_{z} p_{\theta}(x,z) \; ,$7 for rand (Papadimitriou et al., 2023). Mixing just $p_{\theta}(x) = \sum_{z} p_{\theta}(x,z) \; ,$8 cross into nest yields an approximately $p_{\theta}(x) = \sum_{z} p_{\theta}(x,z) \; ,$9-point PPL improvement, and $\hat z=\arg\max_z p_\theta(z\mid x)$0 cross yields approximately $\hat z=\arg\max_z p_\theta(z\mid x)$1 points, supporting the claim that non-context-free relationships form a strong inductive bias.

A second route induces structure through causal intervention training. Type-level Interchange Intervention Training (TIIT) defines each character as a typed variable in a causal model and reserves a contiguous hidden-state slice for character representations at a selected transformer layer. Training combines standard MLE with an interchange intervention objective, typically with $\hat z=\arg\max_z p_\theta(z\mid x)$2, so that swapping hidden slices aligned to same-type variables produces the correct counterfactual output (Huang et al., 2022). Across a suite of character-level tasks, Subword+IIT improves OOV accuracy by $\hat z=\arg\max_z p_\theta(z\mid x)$3 percentage points on average over the base subword model and yields human-interpretable clusters in PCA space, with vowels, consonants, and digits forming distinct regions.

A third route is inference-time structure induction through internal-state perturbation. Cache steering is a one-shot intervention on the key–value cache of an autoregressive transformer. After a prompt forward pass, the model perturbs only the final prompt token’s cache entries at each layer:

$\hat z=\arg\max_z p_\theta(z\mid x)$4

The steering tensors are extracted from contrastive prompt pairs with and without explicit chain-of-thought traces, using mean-of-differences in the last-token key and value tensors (Belitsky et al., 11 Jul 2025). On reasoning benchmarks, the method improves both reasoning style and task accuracy, while adding negligible per-token cost, measured at $\hat z=\arg\max_z p_\theta(z\mid x)$5 ms/token on an H100, versus a $\hat z=\arg\max_z p_\theta(z\mid x)$6–$\hat z=\arg\max_z p_\theta(z\mid x)$7 slowdown for continuous activation steering. It also nearly triples chain-of-thought length on small models—for example, SmolLM2 increases from $\hat z=\arg\max_z p_\theta(z\mid x)$8 to $\hat z=\arg\max_z p_\theta(z\mid x)$9 tokens—and achieves intended rhetorical style transfer in up to $G_{PT}$0 of outputs.

The term SiLM has also been used in emergent-communication experiments, where LLM agents repeatedly learn and use artificial lexica in referential games. In that setting, SiLMs are defined as LLM agents whose implicit biases reshape initially unstructured holistic languages into more regular, compositional systems. Across $G_{PT}$1 random initializations and $G_{PT}$2 generation chains, TopSim increases linearly per round with $G_{PT}$3, $G_{PT}$4, $G_{PT}$5; N-gram diversity decreases with $G_{PT}$6, $G_{PT}$7, $G_{PT}$8; and communicative success asymptotes at approximately $G_{PT}$9, compared with chance at $f_\theta$0 (Kouwenhoven et al., 2024). At the same time, iterated learning can produce non-humanlike degenerate vocabularies, indicating that induced structure and communicative adequacy do not guarantee humanlike language organization.

5. Cross-domain extensions and applications

SiLM ideas have been extended well beyond sentence-level syntax. In long-form text generation, Structural Alignment aligns an instruction-tuned LLM policy with human-like discourse structures by integrating surface-level and graph-level reward models into PPO. The graph-level variant parses generated text with an RST parser, extracts discourse motifs, scores their human distinctiveness, and uses the resulting $f_\theta$1 as reward (Kim et al., 4 Apr 2025). On GOVREPORT, Base+SAG improves ROUGE from $f_\theta$2 to $f_\theta$3 on ROUGE-1/2/L, outperforming both the base model and an off-the-shelf RLHF reward model. On essay generation, graph-level alignment yields the largest gains, while two-step combinations add only marginal extra improvement.

Another extension uses explicit structured representations as LLM inputs, but recasts them in a language-model-friendly form. SR-LLM studies training-free integration via SR-to-Natural Language Description (SR-NLD) and training-dependent integration via supervised fine-tuning on mixed text and SR-augmented examples (Zhang et al., 20 Feb 2025). On PAWS with Llama3.1-8B, the training-free baseline without structure scores $f_\theta$4 F1, raw structured representation injection drops to $f_\theta$5, and SR-NLD improves to $f_\theta$6. In the training-dependent setting, a $f_\theta$7 text $f_\theta$8 $f_\theta$9 SR mixture gives the strongest result, improving from $$\mathcal L_{PT}(\theta) = (1-\lambda)\,M(\theta) + \lambda\,SI(\theta) \; ,$$0 without SR to $$\mathcal L_{PT}(\theta) = (1-\lambda)\,M(\theta) + \lambda\,SI(\theta) \; ,$$1 with SR-NLD. This result directly addresses a common failure mode of earlier zero-shot structure injection: code-style graph linearizations can be mismatched to LLM pre-training distributions.

Prompt-only structuralization provides a related but more general route. Unified Text Structuralization wraps input text between a prefix instruction and a suffix instruction and relies on an instruction-tuned LLM to output structured records without any additional fine-tuning (Ni et al., 2023). On Wikibio with GPT-3 $$\mathcal L_{PT}(\theta) = (1-\lambda)\,M(\theta) + \lambda\,SI(\theta) \; ,$$2B, the instruction-only setting reports Key-F1 $$\mathcal L_{PT}(\theta) = (1-\lambda)\,M(\theta) + \lambda\,SI(\theta) \; ,$$3, Value-F1 $$\mathcal L_{PT}(\theta) = (1-\lambda)\,M(\theta) + \lambda\,SI(\theta) \; ,$$4, and Structure-Degree $$\mathcal L_{PT}(\theta) = (1-\lambda)\,M(\theta) + \lambda\,SI(\theta) \; ,$$5; on ACE2005, text-davinci-003 with instruction prompting reaches $$\mathcal L_{PT}(\theta) = (1-\lambda)\,M(\theta) + \lambda\,SI(\theta) \; ,$$6 NER-F1, $$\mathcal L_{PT}(\theta) = (1-\lambda)\,M(\theta) + \lambda\,SI(\theta) \; ,$$7 EE-F1, and $$\mathcal L_{PT}(\theta) = (1-\lambda)\,M(\theta) + \lambda\,SI(\theta) \; ,$$8 RE-F1 in zero-shot mode. The approach is explicitly framed as a unified interface over NER, RE, EE, and other structuralization tasks.

Protein modeling has produced a distinct “structure language modeling” literature. One framework first encodes a 3D backbone conformation $$\mathcal L_{PT}(\theta) = (1-\lambda)\,M(\theta) + \lambda\,SI(\theta) \; ,$$9 into discrete structure tokens $t_x$00 with a discrete VAE, then learns a conditional LM $t_x$01 from amino-acid sequence $t_x$02 to structure-token sequence (Lu et al., 2024). Its ESMDiff variant is a BERT-like model fine-tuned with a masked-diffusion objective. Across multiple settings, these SLMs run $t_x$03–$t_x$04 faster than SE(3)-equivariant diffusion methods when generating ensembles of $t_x$05, and on BPTI ESMDiff (DDPM) achieves JS-PwD $t_x$06 and clash-free validity $t_x$07. A related method, the Structure-Aligned Protein LLM, aligns residue embeddings from a protein LM with frozen protein-graph-neural-network embeddings and adds a structural-token prediction task. Applied to ESM2, it raises contact $t_x$08 from $t_x$09 to $t_x$10, a $t_x$11 increase, while also improving fold classification and secondary-structure accuracy (Chen et al., 22 May 2025).

Structure-aware encoding has also been applied to embedding models. Struc-EMB studies sequential concatenation and parallel KV-caching as in-process methods for encoding text together with structural neighbors such as citations or hyperlinks (Liu et al., 9 Oct 2025). On zero-shot tasks with Qwen3-4B, Struc-Emb-Seq improves MuSiQue nDCG@10 from $t_x$12 for text-only embeddings to $t_x$13, and HotpotQA nDCG@10 from $t_x$14 to $t_x$15. The paper characterizes a trade-off: sequential concatenation is stronger under noisy, moderate-length context, whereas parallel caching scales better to long, high-signal context but is more susceptible to distractors.

6. Debates, misconceptions, and research directions

A recurrent misconception is that “structure induction” names a single method or a single target formalism. The literature instead uses the term for several partly overlapping enterprises: latent syntactic tree induction, graph-shaped latent geometry, discourse-level RL alignment, character-level causal modularization, inference-time reasoning control through cache perturbation, and domain-specific tokenization of protein structure (Arps et al., 11 Aug 2025, McDermott et al., 2021, Kim et al., 4 Apr 2025, Huang et al., 2022, Belitsky et al., 11 Jul 2025, Lu et al., 2024). Current usage is therefore not uniform. What unifies these lines is not one architecture but a family of attempts to make structure available to the model as an inductive bias, a latent variable, or a controllable internal state.

A second misconception is that better induced structure should automatically imply better downstream language behavior. The evidence is mixed. In syntax induction, no model dominates all evaluation metrics, and explicit structure recovery can remain weak even when grammaticality performance is competitive (Arps et al., 11 Aug 2025). In discourse alignment, gains are strongest in genres with well-defined discourse organization, while the method remains limited by an RST parser capped at approximately $t_x$16-token segments and by a genre scope restricted to formal essays and government reports (Kim et al., 4 Apr 2025). In emergent communication, increased structure accompanies improved learnability but can also produce degeneracy, underspecification, and excessive message length (Kouwenhoven et al., 2024). In protein modeling, efficiency gains are clear, but the representation target is structural biology rather than linguistic syntax (Lu et al., 2024).

Several limitations recur across the literature. Seed sensitivity and parse inconsistency remain serious problems for syntactic SiLMs, especially StructFormer and UDGN (Arps et al., 11 Aug 2025). Prompt-based structuralization can hallucinate spurious key-value pairs on very long texts, and its strongest models are often closed-source or expensive to run (Ni et al., 2023). Cache steering has only been validated up to Llama-3.1-8B and still relies on a fixed target token for intervention (Belitsky et al., 11 Jul 2025). SR-LLM identifies a mismatch between structured-code formats and LLM pre-training distributions, suggesting that representational compatibility is itself a bottleneck (Zhang et al., 20 Feb 2025).

The main research directions are correspondingly diverse. Syntactic work emphasizes improving the consistency and robustness of induced parses, extending evaluation to typologically diverse languages and more complex formal grammars, and designing more efficient scalable architectures that preserve hierarchical inductive bias (Arps et al., 11 Aug 2025). Intervention-based work points toward adaptive or learned steering strengths, multi-token or dynamic cache manipulations, and broader non-reasoning behaviors (Belitsky et al., 11 Jul 2025). Discourse alignment motivates richer discourse theories and long-context parsing beyond fixed windowing (Kim et al., 4 Apr 2025). Structure-aligned prompting suggests end-to-end SR-to-language mapping and dynamic structure retrieval (Zhang et al., 20 Feb 2025). More generally, the field continues to investigate which forms of structure are genuinely causal for generalization, which are merely correlational artifacts of training data or decoding, and how far structural bias can be scaled without sacrificing efficiency or robustness.