---
title: 'StructFormer: Structure-Aware Pre-Training'
url: https://www.emergentmind.com/topics/structformer
type: topic
---

# StructFormer: Structure-Aware Pre-Training

Searching arXiv for the specified paper and closely related StructFormer variants to ground the article in current literature.
StructFormer is a document-structure-aware pre-training method for transformer language models that uses explicit document landmarks to control sparse attention during masked language modeling. In the formulation introduced for scientific documents, StructFormer is implemented as a Longformer-based BERT pre-training approach in which tokens belonging to titles, section headings, and subheadings are marked as global tokens, while the remaining tokens use local windowed attention [2411.16618]. The resulting masked attention pattern preserves the near-linear scaling of sparse attention, exposes the model to document structure already during pre-training rather than only at fine-tuning time, and is evaluated primarily on long-form scientific text derived from arXiv [2411.16618]. The term “StructFormer” has also been used for distinct models in unsupervised syntax induction [2012.00857] and robotic rearrangement [2110.10189], but in contemporary language-model pre-training usage it most commonly denotes the structure-aware Longformer variant introduced for document understanding [2411.16618].

## 1. Definition and problem setting

StructFormer addresses a limitation of standard transformer language models on long documents. Full self-attention has $O(n^2)$ time and memory complexity per layer, which constrains practical sequence lengths and makes long-document pre-training expensive [2411.16618]. Sparse attention patterns such as local and global attention reduce this burden, yielding complexity $O(nw + ng)$, where $w$ is the local window size and $g$ is the number of global tokens [2411.16618]. Prior work had established that sparse attention can retain strong theoretical expressivity, but the practical effect of using global attention during pre-training remained comparatively underexplored [2411.16618].

The central design choice in StructFormer is to tie global attention to document structure itself. Rather than introducing special task tokens or activating global attention only at fine-tuning time, the model derives a global attention mask from LaTeX-parsed arXiv documents and marks heading-level tokens as globally connected [2411.16618]. This creates a pre-training regime in which the model learns to route information through titles and headings as salient document anchors. The intended benefit is not merely efficiency, but improved abstraction over long contexts and better performance on document understanding tasks [2411.16618].

This suggests that StructFormer should be understood not as a new transformer family in the broadest sense, but as a specific pre-training protocol and masking policy layered onto Longformer. Its novelty lies in the source of the sparse pattern: document structure, extracted from the corpus rather than learned post hoc or specified only by downstream tasks [2411.16618].

## 2. Attention mechanism and structural masking

StructFormer uses AllenAI’s `longformer-base-4096` architecture and retains Longformer’s sparse attention scheme, but supplies a structure-derived global attention mask during pre-training [2411.16618]. Tokens corresponding to titles, section headers, and subheadings are assigned global attention, and all other tokens remain local [2411.16618]. The model therefore combines short-range contextualization within paragraphs with long-range routing through headings.

The attention mechanism is expressed analytically through masked scaled dot-product attention:
$$
A = \mathrm{softmax}\!\left(\frac{QK^{\top} + M}{\sqrt{d_k}}\right)V,
$$
where $Q, K, V \in \mathbb{R}^{n \times d_k}$ and the additive mask $M \in \mathbb{R}^{n \times n}$ determines which edges are permitted [2411.16618]. The mask is defined so that $M_{ij}=0$ for allowed attention edges and $M_{ij}=-\infty$ for disallowed ones, implemented in practice as a large negative constant [2411.16618].

Longformer’s sparse pattern is decomposed into a local window and a global mask. For local attention, token $i$ may attend only to positions $j$ satisfying $|i-j| \le w$; in the reported experiments $w=256$ [2411.16618]. For global attention, a binary vector $g \in \{0,1\}^n$ indicates which tokens are global. If $g_i=1$, then token $i$ may attend to all tokens and all tokens may attend to $i$ [2411.16618]. For local tokens, long-range edges are blocked except toward global tokens or nearby neighbors [2411.16618].

Although the source document structure is hierarchical, StructFormer does not explicitly encode a full tree-structured graph over sections and subsections [2411.16618]. Instead, it uses an implicit approximation: all heading-level tokens are made global, and the hierarchy is reflected indirectly through the document’s linearized order and the special connectivity of headings [2411.16618]. A plausible implication is that StructFormer trades representational granularity for simplicity and compatibility with existing sparse-attention implementations.

## 3. Corpus construction and pre-training regime

The pre-training corpus is built from 1,129,787 arXiv LaTeX documents spanning 2000–2018 [2411.16618]. Comments, figures, tables, and equations are removed, while title, abstract, sections, subsections, subsubsections, and paragraphs are extracted using TexSoup [2411.16618]. Each token is stored together with metadata such as bold, italic, and underline, plus a flag indicating whether it belongs to a title or header [2411.16618]. Custom PyTorch data loaders then produce token sequences and the corresponding global attention masks [2411.16618].

The extracted corpus exhibits substantial variation in document length and structure. Reported statistics include token counts ranging from 2 to 4,553,287 with mean 15,266 and standard deviation 31,993; header counts ranging from 1 to 498 with mean 14 and standard deviation 9; and tokens per header ranging from 1 to 40,592 with mean 106 and standard deviation 204 [2411.16618]. For pre-training, documents are filtered to the range 2,000–12,000 tokens, and 100,000 documents are used [2411.16618].

Two pre-training conditions are compared under the same masked language modeling objective. In the structure-aware setting, header tokens are marked global; in the vanilla Longformer baseline, all global-mask entries are set to zero, yielding pure local attention [2411.16618]. Both models are trained for 9,000 runs and each model is reported to train for approximately 16 hours [2411.16618]. On a held-out arXiv test set, the structure-aware model reaches 2.2136 bits-per-character, compared with 2.3051 for the default pre-training condition [2411.16618].

The improvement in bits-per-character is interpreted as evidence that explicit document landmarks reduce uncertainty in masked language modeling on scientific articles [2411.16618]. Because the baseline shares the same architecture and corpus but disables global tokens during pre-training, the result isolates the contribution of the structure-derived masking policy more cleanly than comparisons against unrelated long-context baselines would.

## 4. Attention-pattern analysis

StructFormer is not evaluated only through end-task scores; the paper also examines how pre-training with structure-aware masking changes attention behavior [2411.16618]. The analysis uses an unseen 2023 arXiv dataset containing section headings and several subsequent sentences, with manually annotated keywords judged critical for section-level context [2411.16618]. Average attention scores are then computed between heading tokens and these keywords across layers for the structure-aware model and the vanilla Longformer baseline [2411.16618].

The reported result is that StructFormer exhibits more than 20% higher attention between keywords and header tokens than the vanilla model [2411.16618]. The vanilla Longformer, by contrast, distributes attention more uniformly within the local window and displays a recency bias [2411.16618]. Last-layer visualizations further show a heading such as “Introduction” attending strongly to relevant nearby keywords [2411.16618].

These observations support the claim that the model learns a structural prior rather than merely inheriting a hard masking rule. Since both models share the same sparse-attention backbone, the divergence in attention allocation

Source: https://www.emergentmind.com/topics/structformer