StructFormer: Structure-Aware Pre-training

• StructFormer is a BERT-style language model that uses document structure by assigning global attention to titles and headers during pre-training.
• The method modifies Longformer’s sparse attention, allowing structural tokens to act as context hubs and improve salient cluster detection in document understanding tasks.
• Empirical results show improved MLM loss, selective attention patterns, and competitive general NLP performance over the vanilla Longformer model.

Searching arXiv for the target paper and closely related work on Longformer / sparse attention / document understanding. I’ll look up the target paper and a few directly related arXiv papers to ground the article’s references. StructFormer is a BERT-style LLM that uses document structure as an explicit inductive bias during pre-training by assigning global attention to structural tokens such as titles and section headings. Implemented on top of Longformer’s sparse attention, it asks a specific empirical question: whether document-structure-based masked attention during pre-training measurably changes attention patterns and improves document-understanding tasks without harming general NLP performance. The model is trained on arXiv-derived scientific text whose structure is recoverable from LaTeX, so the distinction between header tokens and paragraph tokens can be encoded directly into the attention mask (Ponkshe et al., 2024).

1. Definition and design objective

StructFormer is not a new transformer architecture from scratch. It is the Longformer encoder, specifically AllenAI’s longformer-base-4096, pre-trained with global attention turned on for structural tokens rather than only using local attention during pre-training. In the formulation studied here, the decisive modification is that titles, section headers, subsections, and related header tokens act as global attention anchors already in pre-training, whereas a vanilla Longformer baseline uses the same architecture and the same text but sets the global mask to zero everywhere (Ponkshe et al., 2024).

The motivating problem is the mismatch between long, hierarchically organized documents and dense self-attention. Standard BERT-style encoders are structure-agnostic and apply dense attention to all token pairs, but longer inputs make computation increasingly expensive and typically confine such models to short passages. Longformer mitigates this by mixing local and global attention. StructFormer specializes that idea by making the choice of global tokens structurally meaningful rather than task-specific or deferred to fine-tuning. This suggests that the model treats document organization itself as a pre-training signal rather than as metadata external to the LLM.

A central point of the paper is causal isolation of the pre-training change. The structure-aware corpus and the text-only counterpart contain the same textual content and the same token sequences; the difference lies in whether header-derived tokens are granted global visibility. Consequently, reported differences between StructFormer and the vanilla Longformer baseline are attributed to structure-aware masked attention during pre-training rather than to different corpora or model capacity (Ponkshe et al., 2024).

2. Structure-based masked attention

StructFormer inherits Longformer’s sparse pattern: every token attends within a fixed local window, while a small subset of tokens is allowed global attention. The paper instantiates this by defining a binary global mask $g_i \in \{0,1\}$ over the token sequence $X=(x_1,\dots,x_n)$, where header tokens receive $g_i=1$ and all other tokens receive $g_i=0$. The resulting attention mask is

$$M_{ij} = \begin{cases} 1, & \text{if } g_i = 1 \text{ or } g_j = 1 \ 1, & \text{if } |i-j| \le w \ 0, & \text{otherwise} \end{cases}$$

with local window size $w=256$ (Ponkshe et al., 2024).

Attention scores are then computed in the usual transformer form, except that disallowed positions are masked with a large negative constant. In effect, the connectivity graph becomes block-sparse: a banded local pattern plus full rows and columns at header positions. Conceptually, section headings function as context hubs, and the title acts as a document-level hub. The paper does not introduce a new per-layer masking scheme; rather, it uses the standard Longformer mask type across layers and leaves head- and layer-specific weighting to learning (Ponkshe et al., 2024).

The structural information is extracted from arXiv LaTeX. The parser records title, abstract, and recursively nested sub-levels, with each word tagged as either a title/header word or a paragraph word. This label is preserved through tokenization as (token, is_header). During pre-training, tokens derived from headers receive global attention; paragraph tokens remain local. The preprocessing also preserves bold, italic, and underline flags, but these cues are not used in the core attention mechanism in the reported experiments (Ponkshe et al., 2024).

3. Corpus construction and pre-training protocol

The data pipeline begins from 1,129,787 LaTeX documents from arXiv spanning 2000–2018. Comments, figures, tables, and equations are removed, and the remaining text is organized as a nested structure containing title, abstract, sections, subsections, and paragraphs. From this source, two aligned corpora are constructed: a structure-aware corpus, in which header tokens are global during pre-training, and a text-only counterpart, in which all tokens are local. The paper emphasizes that both corpora contain the same text, so the attention mask is the only experimental variable (Ponkshe et al., 2024).

Pre-training uses only Masked Language Modeling. If $\mathcal{M}$ denotes the masked positions, the objective is the standard MLM loss

$$\mathcal{L}_{\text{MLM}} = - \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \log p_\theta(x_i \mid X_{\text{masked}}).$$

The difference is that, for StructFormer, $p_\theta$ is computed under the structure-based sparse mask rather than an all-local Longformer mask (Ponkshe et al., 2024).

The reported configuration is concise enough to isolate the effect of structure-aware pre-training.

Setting	Value
Base model	longformer-base-4096
Context length	4096 tokens
Local window	256
Training subset	100,000 documents
Length filter	2,000–12,000 tokens
Objective	MLM only
Pre-training runs	9,000
Runtime per pre-train	~16 hours

The held-out pre-training metric is bits-per-character. Structure-aware pre-training yields a test BPC of 2.2136, compared with 2.3051 for default pre-training. The paper characterizes this as a slight improvement despite the noisiness of the corpus, including LaTeX artifacts and abrupt transitions (Ponkshe et al., 2024).

4. Empirical effects on attention and document understanding

The evaluation combines three layers of evidence: pre-training metrics, direct analysis of learned attention patterns, and downstream task performance. The attention analysis uses a custom dataset built from unseen 2023 arXiv papers, where section headings and subsequent sentences are paired with manually annotated keywords. On this dataset, StructFormer shows more than 20% higher attention between header tokens and their associated keywords than vanilla Longformer. The reported qualitative contrast is also specific: StructFormer’s attention is heterogeneous and semantically selective, whereas the vanilla model exhibits a stronger recency bias and more uniform local attention (Ponkshe et al., 2024).

This change in connectivity is reflected downstream most clearly on SciREX, a document-level scientific information extraction benchmark. The model is evaluated in the end-to-end predicted input setting, replacing the BERT component in the original SciREX pipeline.

SciREX task	StructFormer F1	Comparison
Salient Clusters	0.3419	0.3182 vanilla Longformer; 0.3070 SciREX Base
Binary Relations	0.0890	0.0740 vanilla Longformer; 0.0960 SciREX Base
4-ary Relations	0.0037	0.0025 vanilla Longformer; 0.0080 SciREX Base

The most substantive gain is on salient clusters, where StructFormer improves both over vanilla Longformer and over the original SciREX baseline. On binary and 4-ary relations, absolute performance remains weak for all systems, and StructFormer is best viewed as competitive rather than dominant. The paper interprets the salient-cluster improvement as evidence that structure-aware pre-training improves the ability to identify salient entities and clusters, which it treats as a core document-understanding capability (Ponkshe et al., 2024).

The attention analysis supplies a mechanistic interpretation of these results. The learned pattern is not merely “more global”; it preferentially links section headings to semantically important local tokens. A plausible implication is that the model has internalized section-topic association as part of its representation geometry, which then benefits tasks requiring document-level aggregation rather than purely sentence-local prediction.

5. General NLP performance and theoretical placement

A common concern is that aggressive specialization to structured scientific documents could reduce general-purpose transfer. The GLUE results reported in the paper do not support that concern. StructFormer is generally competitive with BERT-base and often slightly better than vanilla Longformer. It improves over vanilla Longformer on MRPC accuracy and F1, QNLI, SST-2, and MNLI, while trailing BERT-base on CoLA and RTE. The paper’s summary claim is narrower and more precise: there is no evidence that structure-aware pre-training hurts general NLU (Ponkshe et al., 2024).

The theoretical context is sparse-attention expressivity. The paper situates StructFormer relative to work arguing that sparse attention mechanisms such as Longformer-, BigBird-, and Sparse Transformer-style patterns can be Turing-complete, but it does not claim a new proof for StructFormer’s particular mask. Its argument is empirical rather than formal: even if sparse patterns are in principle expressive enough, the placement of global connections matters in practice. Structurally motivated global tokens yield better MLM scores, stronger header–keyword connectivity, and better SciREX salient-cluster performance than an otherwise identical all-local pre-training regime (Ponkshe et al., 2024).

Within long-context modeling more broadly, StructFormer’s novelty is therefore not simply “using sparse attention.” Its novelty is using document structure to determine global attention anchors during pre-training itself. Longformer’s architecture is retained unchanged, but the pre-training signal now includes a hierarchy-sensitive routing prior. This suggests a design principle for long-document encoders: efficiency and inductive bias should be co-designed, and the sparse pattern should reflect the latent organization of the corpus rather than computational convenience alone.

6. Limitations, future directions, and nomenclature

The paper describes the study as preliminary in several respects. The attention analysis is coarse rather than exhaustive; it does not provide a systematic layer-wise or head-wise causal account. The corpus is exclusively scientific arXiv LaTeX, so transfer to books, legal documents, technical manuals, or web documents is unresolved. The method also relies on clean structural markup: headings are easy to recover from LaTeX, whereas noisy PDFs or HTML would require a robust structure-extraction stage not addressed here. Scale is another open variable: only base-sized models and about 100,000 filtered pre-training documents are used, so the interaction with larger models, larger corpora, and longer contexts remains unknown. Finally, the attention mask distinguishes only header/title material from paragraph text; richer cues such as lists, figures, cross-references, and layout are left unused (Ponkshe et al., 2024).

The future directions named in the paper follow directly from these limits. They include deeper attention analysis, pre-training on broader structured corpora, extension to XML/HTML/EPUB or PDF-derived structure, addition of richer structural and layout signals, potential integration with vision models for document layout understanding, and scaling studies over model size and context length. Taken together, these directions frame StructFormer less as a completed document model than as a controlled empirical demonstration that pre-training can absorb organizational structure, not only lexical and sentence-level regularities.

The name “StructFormer” also requires disambiguation. In arXiv usage it is not unique to this document-structure-aware LLM. Unrelated models with the same or closely similar name include a structure-inducing masked LLM for unsupervised constituency and dependency induction (Shen et al., 2020), a transformer for language-guided robotic rearrangement from point clouds and structured commands (Liu et al., 2021), and a behavior-modeling transformer based on structured tokenization for player-event sequences (Smirnov et al., 2024). In the present sense, however, StructFormer specifically denotes the Longformer-based encoder pre-trained with document-structure-derived global attention over scientific text (Ponkshe et al., 2024).