DocPolarBERT: Polar Positional Document Encoder

• DocPolarBERT is a layout-aware BERT-style encoder that replaces absolute 2D positional embeddings with a polar coordinate bias to model relative token positions.
• It leverages relative distance and angle between text blocks to better capture document structure in forms, invoices, and tables.
• Pre-trained on diverse datasets with a 1-LOP auxiliary objective, DocPolarBERT achieves state-of-the-art average F1 scores on document named-entity recognition benchmarks.

Searching arXiv for DocPolarBERT and closely related document-understanding models to ground the article in current papers. DocPolarBERT is a layout-aware, BERT-style encoder for document understanding that replaces absolute 2D Cartesian positional embeddings with a relative polar coordinate bias in self-attention. It is designed for document tasks in which semantic structure depends heavily on 2D layout, including forms, invoices, and tables, and it uses relative distance and angle between text blocks rather than absolute page coordinates. The model is introduced as a BERT-like document encoder without any absolute positional embeddings, and it is reported to achieve state-of-the-art or better results on five named-entity recognition benchmarks despite pre-training on approximately 1.8 million documents, more than six times fewer than the 11 million documents in IIT-CDIP (Uthayasooriyar et al., 11 Jul 2025).

1. Position within layout-aware document modeling

Document understanding differs from ordinary sequence modeling because semantic organization is distributed across the page rather than only along a 1D token order. The motivating examples given for DocPolarBERT include headers versus cells, nearby fields, and column structure, all of which depend on horizontal and vertical alignment, proximity of labels and values, and table row or column regularities (Uthayasooriyar et al., 11 Jul 2025). Standard text-only BERT is described as inadequate for such settings because it treats the input as a 1D sequence and uses a 1D absolute positional embedding depending on token index; linear OCR order may not reflect true reading order or semantic grouping in multi-column forms or tables (Uthayasooriyar et al., 11 Jul 2025).

The model is situated against several prior lines of research in layout-aware document representation. LayoutLM, LayoutLMv2, and LayoutLMv3 inject layout through absolute 2D embeddings derived from discretized bounding-box coordinates and, in later variants, incorporate relative Cartesian geometry and visual features (Xu et al., 2019, Xu et al., 2020, Huang et al., 2022). BROS is described as a text-plus-layout model that encodes spatial relationships while avoiding discretization and lookup tables, but still in Cartesian form (Yang et al., 2022). LiLT is another text-plus-layout baseline used in the experimental comparison (Wang et al., 2022). Multi-modal systems such as DocFormer and Wukong-Reader are characterized as text-layout-image encoders that may obtain higher performance at greater computational cost; DocPolarBERT is explicitly positioned as a text-plus-layout model without a vision backbone (Uthayasooriyar et al., 11 Jul 2025).

The central argument made for DocPolarBERT is that many document relations are better expressed as direction and distance than as independent $x$ and $y$ magnitudes. The paper therefore treats relative structure as primary and removes all absolute 2D positional embeddings, using bounding boxes only to compute relative polar biases in attention (Uthayasooriyar et al., 11 Jul 2025).

2. Architectural definition

DocPolarBERT is a BERT-like Transformer encoder with 12 Transformer encoder layers, 12 attention heads per layer, hidden size 768, and approximately 125 million parameters, which the paper states is the same size as RoBERTa-base and comparable to ReLayout base (Uthayasooriyar et al., 11 Jul 2025). Aside from the modified attention mechanism, the remaining stack follows standard BERT or LayoutLM conventions.

The input consists of OCR-derived tokens

$x = (x_1,\dots,x_n)$

and bounding boxes

$b = (b_1,\dots,b_n),$

where $b_i$ is the box of token $x_i$ (Uthayasooriyar et al., 11 Jul 2025). OCR supplies a linearization of tokens, and that 1D ordering is preserved. For ablations, the paper considers an embedding construction that combines token, 1D positional, and 2D positional components:

$$E(x_i) \triangleq E_{\text{voc}(x_i)} + E_{\text{1D-pos}(i)} + E_{\text{2D-pos}(b_i)},$$

where $E_{\text{2D-pos}(b_i)}$ is the sum of four learned embeddings representing discretized absolute box coordinates, following LayoutLM-style practice (Uthayasooriyar et al., 11 Jul 2025). In the final DocPolarBERT model, however, the absolute 2D component is removed.

The defining modification appears in self-attention. The paper first recalls standard scaled dot-product attention:

$$w \triangleq V \,\mathrm{softmax}\!\left(\frac{Qk}{\sqrt{d}}\right),$$

and then frames DocPolarBERT as a 2D geometric generalization of relative attention (Uthayasooriyar et al., 11 Jul 2025). For each query token, the attention logits are supplemented with learned biases determined by relative distance and relative angle to every context token. This makes the model “layout-aware” through attention itself rather than through additive absolute coordinate embeddings.

A plausible implication is that DocPolarBERT shifts the inductive bias from page-template memorization toward pairwise structural reasoning. The paper’s own phrasing is stronger in empirical than theoretical terms: it states that carefully designed attention can compensate for reduced pre-training data and that absolute embeddings can hurt downstream performance (Uthayasooriyar et al., 11 Jul 2025).

3. Relative polar coordinate encoding

The geometric core of DocPolarBERT is defined over token-box centers. For token $i$ with bounding box $y$0, the model defines its center in Cartesian coordinates as

$y$1

and for tokens $y$2 and $y$3 it defines the relative distance and angle as

$y$4

and

$y$5

with $y$6 when $y$7 (Uthayasooriyar et al., 11 Jul 2025). The origin of the polar system is always the center of the query token’s box, and the polar axis is aligned with the first Cartesian dimension.

Because the model does not use continuous $y$8 and $y$9 directly, it discretizes both quantities. Distance is capped at $x = (x_1,\dots,x_n)$0 and the interval $x = (x_1,\dots,x_n)$1 is uniformly divided into 4 bins, each mapped to a learned embedding by $x = (x_1,\dots,x_n)$2 (Uthayasooriyar et al., 11 Jul 2025). Angle is discretized into 8 bins over the interval $x = (x_1,\dots,x_n)$3 and mapped by $x = (x_1,\dots,x_n)$4 (Uthayasooriyar et al., 11 Jul 2025). For a fixed query token $x = (x_1,\dots,x_n)$5, the corresponding embedding matrices are

$x = (x_1,\dots,x_n)$6

collected into $x = (x_1,\dots,x_n)$7 (Uthayasooriyar et al., 11 Jul 2025).

The resulting polar-biased attention is given as

$x = (x_1,\dots,x_n)$8

equivalently described as adding a distance-dependent bias and an angle-dependent bias to each attention logit (Uthayasooriyar et al., 11 Jul 2025). Here, $x = (x_1,\dots,x_n)$9 is the standard content term, while $b = (b_1,\dots,b_n),$0 and $b = (b_1,\dots,b_n),$1 encode relative geometry.

The conceptual rationale offered for polar rather than Cartesian representation is twofold. First, angle and distance are explicitly disentangled, which the authors argue better matches document semantics such as “above,” “below,” “left,” “right,” and “near” versus “far” (Uthayasooriyar et al., 11 Jul 2025). Second, the representation is less directly tied to absolute page coordinates and therefore less sensitive to variations in scanner resolution, margins, and templates. This suggests a form of robustness to layout-preserving transformations, although the paper supports that point primarily through downstream accuracy rather than direct invariance tests (Uthayasooriyar et al., 11 Jul 2025).

4. Pre-training corpus and objectives

DocPolarBERT is pre-trained on publicly annotated document sources rather than IIT-CDIP. The reason given is reproducibility: the public IIT-CDIP release lacks layout annotations, and prior OCR outputs are not publicly shared (Uthayasooriyar et al., 11 Jul 2025). The pre-training corpus instead combines DOCILE, described as containing more than 900k invoices with text and layout annotations, and OCR-IDL, described as containing approximately 26 million documents from the Industry Document Library, including letters, reports, memos, and news articles (Uthayasooriyar et al., 11 Jul 2025). From these sources, the paper samples 1.8 million documents total, evenly split into approximately 0.9 million DOCILE and 0.9 million OCR-IDL documents (Uthayasooriyar et al., 11 Jul 2025).

Two pre-training objectives are used. The first is masked language modeling, with 30% of tokens masked and cross-entropy loss over the masked positions (Uthayasooriyar et al., 11 Jul 2025). The second is 1D Local Order Prediction (1-LOP), described as an auxiliary objective from Jiang et al. (2025) in which 30% of tokens have their 1D positions masked and the model predicts their original index or order in the linearized OCR sequence (Uthayasooriyar et al., 11 Jul 2025). The paper states that the final DocPolarBERT variant uses both MLM and 1-LOP and that this combination yields the best downstream results.

Training is initialized from RoBERTa-base weights, with batch size 2048 documents, 20 epochs, a learning rate of $b = (b_1,\dots,b_n),$2, cosine schedule with 5% warmup, and NVIDIA Tesla V100 32GB GPUs (Uthayasooriyar et al., 11 Jul 2025). These details are presented as part of a fully reproducible pre-training pipeline based on publicly available datasets with layout annotations.

A plausible implication is that DocPolarBERT’s empirical efficiency claim rests on the interaction of architectural bias and pre-training design rather than architecture alone. The paper explicitly reports gains from adding OCR-IDL to DOCILE and from adding 1-LOP, indicating that relative polar encoding does not eliminate the importance of corpus diversity and auxiliary supervision (Uthayasooriyar et al., 11 Jul 2025).

5. Empirical evaluation on document NER

The downstream evaluation is restricted to document named-entity recognition at token level and spans five datasets with word-level boxes: FUNSD, SROIE, CORD v2, PAYSLIPS, and DOCILE* (Uthayasooriyar et al., 11 Jul 2025). The paper gives the following dataset descriptions: FUNSD contains 199 forms with a 149/50 train/test split; SROIE contains 973 receipts with 626 train and 347 test; CORD v2 contains 1000 receipts with 800 train, 100 validation, and 100 test; PAYSLIPS contains 611 pay statements with 485 train and 126 test and has approximately 95% of tokens labeled as “O”; and DOCILE* uses the validation split as test because official test labels are not public (Uthayasooriyar et al., 11 Jul 2025).

All models are fine-tuned for 30 epochs with batch size 16 and learning rate $b = (b_1,\dots,b_n),$3, except DOCILE*, which uses $b = (b_1,\dots,b_n),$4 (Uthayasooriyar et al., 11 Jul 2025). Reported scores are average F1 over 10 runs with different seeds, except DOCILE*, where 4 runs are used due to cost (Uthayasooriyar et al., 11 Jul 2025). The comparisons focus on text-plus-layout models without vision features: LayoutLM-base, BROS-base, LiLT-base, and DocPolarBERT variants (Uthayasooriyar et al., 11 Jul 2025).

The principal quantitative results are summarized below.

Model / Pre-training	AVG F1
LayoutLM-base (IIT-CDIP)	77.87
LayoutLM-base (DOCILE)	74.65
BROS-base	78.21
LiLT-base	78.60
DocPolarBERT (DOCILE)	75.61
DocPolarBERT (DOCILE+OCR-IDL)	80.95
DocPolarBERT (+1-LOP)	81.30

The full paper reports that DocPolarBERT (+1-LOP) reaches F1 scores of 77.10 on FUNSD, 97.48 on SROIE, 97.46 on CORD, 74.43 on PAYSLIPS, and 59.99 on DOCILE* (Uthayasooriyar et al., 11 Jul 2025). According to the reported comparison, this gives the highest average F1 among text-plus-layout models. On SROIE and CORD, the reported scores surpass all listed text-plus-layout baselines. On PAYSLIPS, the model shows a particularly large margin over LayoutLM-base, BROS-base, and LiLT-base, which the paper interprets as support for effective modeling of long-range table dependencies (Uthayasooriyar et al., 11 Jul 2025).

The main weakness in the benchmark table is FUNSD, where DocPolarBERT (+1-LOP) is below BROS and below LayoutMask in a non-directly comparable setting (Uthayasooriyar et al., 11 Jul 2025). The authors hypothesize that FUNSD’s origin from Truth Tobacco Industry Docs, a source related to IIT-CDIP and RVL-CDIP, gives models pre-trained on IIT-CDIP a distributional advantage (Uthayasooriyar et al., 11 Jul 2025). That explanation is presented as a hypothesis rather than a demonstrated causal account.

6. Ablations, efficiency, and limitations

The central ablation concerns absolute 2D positional embeddings. The paper compares a variant “with 2D-Pos (absolute)” against the final model “without 2D-Pos (ours)” and reports average F1 of 79.89 versus 81.29, respectively (Uthayasooriyar et al., 11 Jul 2025). The gains from removing absolute 2D embeddings are reported as especially strong on PAYSLIPS and positive on DOCILE*, leading the authors to conclude that absolute 2D embeddings are not only unnecessary but can deteriorate performance when combined with polar relative encoding (Uthayasooriyar et al., 11 Jul 2025).

The paper also interprets the comparison between pre-training settings as evidence for the importance of corpus diversity and the 1-LOP objective. DocPolarBERT pre-trained on DOCILE only attains 75.61 average F1; adding OCR-IDL raises this to 80.95; adding 1-LOP further raises it to 81.30 (Uthayasooriyar et al., 11 Jul 2025). This supports the specific claim that more diverse pre-training data matters even when the overall corpus is substantially smaller than IIT-CDIP.

In efficiency terms, DocPolarBERT is reported to have 125 million parameters, smaller than LayoutLMv2-base at 200 million and LayoutMask-base at 182 million, while matching RoBERTa-base and ReLayout base in size (Uthayasooriyar et al., 11 Jul 2025). The polar attention overhead is described as extra embedding lookups plus linear projections via $b = (b_1,\dots,b_n),$5 and simple elementwise operations, characterized as negligible relative to standard attention and feed-forward cost (Uthayasooriyar et al., 11 Jul 2025). Because the model uses no image backbone, the paper further states that inference is lighter and faster than in multi-modal systems such as LayoutLMv2 or LayoutLMv3 (Xu et al., 2020, Huang et al., 2022, Uthayasooriyar et al., 11 Jul 2025).

The limitations are explicitly stated. DocPolarBERT depends on OCR and bounding boxes; OCR quality strongly affects performance, and noisy boxes may degrade polar encoding (Uthayasooriyar et al., 11 Jul 2025). The evaluation covers only named-entity recognition and does not include classification, relation extraction, table structure recognition, or question answering (Uthayasooriyar et al., 11 Jul 2025). The pre-training scale remains smaller than IIT-CDIP, even though that choice is motivated by reproducibility. Finally, the model is restricted to text plus layout and may be outperformed by large text-layout-image systems on heavily visual tasks (Uthayasooriyar et al., 11 Jul 2025).

7. Practical use and research significance

Operationally, the model requires tokenized text, bounding boxes for each token, and a consistent page coordinate system (Uthayasooriyar et al., 11 Jul 2025). The described preprocessing procedure computes box centers, then for each token pair computes

$b = (b_1,\dots,b_n),$6

clips distance to $b = (b_1,\dots,b_n),$7, bins distance into 4 intervals and angle into 8 intervals, maps these bins to embeddings, and injects them into attention through the $b = (b_1,\dots,b_n),$8 and $b = (b_1,\dots,b_n),$9 matrices (Uthayasooriyar et al., 11 Jul 2025). For named-entity recognition, fine-tuning proceeds by feeding token sequences and bounding boxes into the encoder, taking the final hidden state $b_i$0 for each token, adding a classification head mapping $b_i$1 to label logits, and training with cross-entropy over labeled tokens (Uthayasooriyar et al., 11 Jul 2025).

The code and models are stated to be public at https://github.com/buthaya/docpolarbert, and the pre-training data sources, DOCILE and OCR-IDL, are public as well (Uthayasooriyar et al., 11 Jul 2025). This reproducibility emphasis is a defining part of the project’s significance. In the paper’s framing, DocPolarBERT is not only an architectural proposal but also a demonstration that a carefully specified attention bias can compensate for reduced pre-training scale in document understanding (Uthayasooriyar et al., 11 Jul 2025).

Within the broader trajectory of document AI, DocPolarBERT represents a shift away from absolute page-coordinate encoding toward explicitly relational geometry. Relative attention itself is not new in NLP, nor is layout awareness in document models, but the specific use of distance-and-angle biases in polar coordinates is presented as the novel contribution (Uthayasooriyar et al., 11 Jul 2025). This suggests a broader methodological point: for structured documents, the geometry most relevant to semantics may be the geometry of relations rather than the geometry of absolute placement.