StructFormer: Structure-Aware Pre-Training

• StructFormer is a document-structure-aware pre-training method that marks titles and headings as global tokens for sparse attention.
• It combines local windowed attention with structure-derived global connections to reduce computational cost and improve long-context abstraction.
• Evaluations on arXiv scientific texts show enhanced attention to key document landmarks and improved bits-per-character metrics.

Searching arXiv for the specified paper and closely related StructFormer variants to ground the article in current literature. StructFormer is a document-structure-aware pre-training method for transformer LLMs that uses explicit document landmarks to control sparse attention during masked language modeling. In the formulation introduced for scientific documents, StructFormer is implemented as a Longformer-based BERT pre-training approach in which tokens belonging to titles, section headings, and subheadings are marked as global tokens, while the remaining tokens use local windowed attention (Ponkshe et al., 2024). The resulting masked attention pattern preserves the near-linear scaling of sparse attention, exposes the model to document structure already during pre-training rather than only at fine-tuning time, and is evaluated primarily on long-form scientific text derived from arXiv (Ponkshe et al., 2024). The term “StructFormer” has also been used for distinct models in unsupervised syntax induction (Shen et al., 2020) and robotic rearrangement (Liu et al., 2021), but in contemporary language-model pre-training usage it most commonly denotes the structure-aware Longformer variant introduced for document understanding (Ponkshe et al., 2024).

1. Definition and problem setting

StructFormer addresses a limitation of standard transformer LLMs on long documents. Full self-attention has $O(n^2)$ time and memory complexity per layer, which constrains practical sequence lengths and makes long-document pre-training expensive (Ponkshe et al., 2024). Sparse attention patterns such as local and global attention reduce this burden, yielding complexity $O(nw + ng)$, where $w$ is the local window size and $g$ is the number of global tokens (Ponkshe et al., 2024). Prior work had established that sparse attention can retain strong theoretical expressivity, but the practical effect of using global attention during pre-training remained comparatively underexplored (Ponkshe et al., 2024).

The central design choice in StructFormer is to tie global attention to document structure itself. Rather than introducing special task tokens or activating global attention only at fine-tuning time, the model derives a global attention mask from LaTeX-parsed arXiv documents and marks heading-level tokens as globally connected (Ponkshe et al., 2024). This creates a pre-training regime in which the model learns to route information through titles and headings as salient document anchors. The intended benefit is not merely efficiency, but improved abstraction over long contexts and better performance on document understanding tasks (Ponkshe et al., 2024).

This suggests that StructFormer should be understood not as a new transformer family in the broadest sense, but as a specific pre-training protocol and masking policy layered onto Longformer. Its novelty lies in the source of the sparse pattern: document structure, extracted from the corpus rather than learned post hoc or specified only by downstream tasks (Ponkshe et al., 2024).

2. Attention mechanism and structural masking

StructFormer uses AllenAI’s longformer-base-4096 architecture and retains Longformer’s sparse attention scheme, but supplies a structure-derived global attention mask during pre-training (Ponkshe et al., 2024). Tokens corresponding to titles, section headers, and subheadings are assigned global attention, and all other tokens remain local (Ponkshe et al., 2024). The model therefore combines short-range contextualization within paragraphs with long-range routing through headings.

The attention mechanism is expressed analytically through masked scaled dot-product attention:

$$A = \mathrm{softmax}\!\left(\frac{QK^{\top} + M}{\sqrt{d_k}}\right)V,$$

where $Q, K, V \in \mathbb{R}^{n \times d_k}$ and the additive mask $M \in \mathbb{R}^{n \times n}$ determines which edges are permitted (Ponkshe et al., 2024). The mask is defined so that $M_{ij}=0$ for allowed attention edges and $M_{ij}=-\infty$ for disallowed ones, implemented in practice as a large negative constant (Ponkshe et al., 2024).

Longformer’s sparse pattern is decomposed into a local window and a global mask. For local attention, token $i$ may attend only to positions $O(nw + ng)$0 satisfying $O(nw + ng)$1; in the reported experiments $O(nw + ng)$2 (Ponkshe et al., 2024). For global attention, a binary vector $O(nw + ng)$3 indicates which tokens are global. If $O(nw + ng)$4, then token $O(nw + ng)$5 may attend to all tokens and all tokens may attend to $O(nw + ng)$6 (Ponkshe et al., 2024). For local tokens, long-range edges are blocked except toward global tokens or nearby neighbors (Ponkshe et al., 2024).

Although the source document structure is hierarchical, StructFormer does not explicitly encode a full tree-structured graph over sections and subsections (Ponkshe et al., 2024). Instead, it uses an implicit approximation: all heading-level tokens are made global, and the hierarchy is reflected indirectly through the document’s linearized order and the special connectivity of headings (Ponkshe et al., 2024). A plausible implication is that StructFormer trades representational granularity for simplicity and compatibility with existing sparse-attention implementations.

3. Corpus construction and pre-training regime

The pre-training corpus is built from 1,129,787 arXiv LaTeX documents spanning 2000–2018 (Ponkshe et al., 2024). Comments, figures, tables, and equations are removed, while title, abstract, sections, subsections, subsubsections, and paragraphs are extracted using TexSoup (Ponkshe et al., 2024). Each token is stored together with metadata such as bold, italic, and underline, plus a flag indicating whether it belongs to a title or header (Ponkshe et al., 2024). Custom PyTorch data loaders then produce token sequences and the corresponding global attention masks (Ponkshe et al., 2024).

The extracted corpus exhibits substantial variation in document length and structure. Reported statistics include token counts ranging from 2 to 4,553,287 with mean 15,266 and standard deviation 31,993; header counts ranging from 1 to 498 with mean 14 and standard deviation 9; and tokens per header ranging from 1 to 40,592 with mean 106 and standard deviation 204 (Ponkshe et al., 2024). For pre-training, documents are filtered to the range 2,000–12,000 tokens, and 100,000 documents are used (Ponkshe et al., 2024).

Two pre-training conditions are compared under the same masked language modeling objective. In the structure-aware setting, header tokens are marked global; in the vanilla Longformer baseline, all global-mask entries are set to zero, yielding pure local attention (Ponkshe et al., 2024). Both models are trained for 9,000 runs and each model is reported to train for approximately 16 hours (Ponkshe et al., 2024). On a held-out arXiv test set, the structure-aware model reaches 2.2136 bits-per-character, compared with 2.3051 for the default pre-training condition (Ponkshe et al., 2024).

The improvement in bits-per-character is interpreted as evidence that explicit document landmarks reduce uncertainty in masked language modeling on scientific articles (Ponkshe et al., 2024). Because the baseline shares the same architecture and corpus but disables global tokens during pre-training, the result isolates the contribution of the structure-derived masking policy more cleanly than comparisons against unrelated long-context baselines would.

4. Attention-pattern analysis

StructFormer is not evaluated only through end-task scores; the paper also examines how pre-training with structure-aware masking changes attention behavior (Ponkshe et al., 2024). The analysis uses an unseen 2023 arXiv dataset containing section headings and several subsequent sentences, with manually annotated keywords judged critical for section-level context (Ponkshe et al., 2024). Average attention scores are then computed between heading tokens and these keywords across layers for the structure-aware model and the vanilla Longformer baseline (Ponkshe et al., 2024).

The reported result is that StructFormer exhibits more than 20% higher attention between keywords and header tokens than the vanilla model (Ponkshe et al., 2024). The vanilla Longformer, by contrast, distributes attention more uniformly within the local window and displays a recency bias (Ponkshe et al., 2024). Last-layer visualizations further show a heading such as “Introduction” attending strongly to relevant nearby keywords (Ponkshe et al., 2024).

These observations support the claim that the model learns a structural prior rather than merely inheriting a hard masking rule. Since both models share the same sparse-attention backbone, the divergence in attention allocation