- The paper introduces POTATR, a 29M-parameter non-autoregressive model that jointly detects tables, captions, footers, structures, bounding boxes, and parent-child relationships as a spatial graph.
- POTATR achieves a GriTSCon score of 0.9636 and TEDS of 0.9565 with direct text, outperforming evaluated VLMs and MLLMs while processing 52,031 pages per hour at about $47 per million pages.
- The paper shows that modular OCR integration improves results without retraining, while a lightweight continuation classifier raises multi-page table extraction from 0.671 to 0.827 GriTSCon, though cross-domain generalization remains untested.
POTATR (Page-Object Table Transformer) is a 29M-parameter image-to-graph model that extends the Table Transformer (TATR) (Hadas et al., 2022, Andronikos et al., 2023) from cropped-table structure recognition to contextually aware, page-level table extraction (TE). Rather than autoregressive HTML/markdown generation as in vision-LLMs (VLMs), POTATR performs parallel-decoded spatial graph prediction: it jointly detects tables, their structures, captions, and footers on a page, predicts directed parent-child edges among them, and grounds every element with a bounding box. On PubTables-v2 Single Pages (Smock et al., 11 Dec 2025), it achieves a GriTSCon of 0.964, exceeding all evaluated models including frontier MLLMs, while running over 130× faster and roughly 300× cheaper.
Motivation and positioning
The dominant paradigm for contextual TE is autoregressive text generation with VLMs and MLLMs (Nassar et al., 14 Mar 2025, Bai et al., 19 Feb 2025, Team et al., 14 Feb 2025). The authors identify three structural drawbacks: inherently sequential decoding, no structural validity guarantees, and weak spatial grounding, plus deployment costs that are prohibitive at scale (Poznanski et al., 25 Feb 2025). Task-specific models are orders of magnitude less expensive than general-purpose generative models (Angelelli et al., 2024). POTATR's central claim is that a small, non-autoregressive, spatially grounded model, composed with a separate text recognizer, can outperform frontier MLLMs on page-level TE at a fraction of the cost.
Architecture and training
POTATR retains the original DETR backbone of TATR (ResNet-18, transformer encoder-decoder) and makes three extensions. It expands the class set from 6 to 16 by adding table caption and table footer classes plus rotated counterparts of all eight base classes, handling rotated tables directly in page context. It doubles the object queries from 125 to 250. It adds a relation head—modeled after Relationformer (Forsgren et al., 2022) but simplified to a three-layer MLP without a relation token—predicting binary directed parent-child edges, where the table object is always the parent of its structures, caption, and footer. This learned relation prediction replaces TATR's bounding-box-overlap grouping heuristic and departs from ETT (Jiang et al., 2024), which uses symmetric same-table adjacency, trains from scratch, and omits captions, footers, and rotation handling.
The relation-token omission is a hypothesis: the authors reason it is unnecessary with full self-attention, and the ablation below supports it, though no direct ablation of the token is reported. Training proceeds in two stages on PubTables-v2 Single Pages (468k pages, 548k tables): 100 epochs on the standard distribution, then 35 epochs augmented with table-free document pages and repeats of the 10k largest tables to address the long tail. Weights are initialized from TATR-v1.1-Pub; the relation head, expanded class head, and additional queries are randomly initialized. Training uses 8 T4 GPUs at an effective batch size of 16, with a relation loss weight of 0.05.
Pre-trained initialization matters
In a controlled 10-epoch comparison against Relationformer and EGTR (Kim et al., 2024), both parameter-matched, POTATR dominates on every metric: AP 0.698 versus 0.549 (EGTR) and 0.484 (Relationformer), and edge F1 0.746 versus 0.707 and 0.339. The edge-F1 result is notable because the relation head is an entirely novel component; the benefit of TATR's pre-trained weights transfers to the new task, not just detection. This validates extending a pre-trained TSR model rather than adopting an image-to-graph architecture trained from scratch, though the comparison is limited to 10 epochs and one dataset.
Evaluated on PubTables-v2 Single Pages against seven specialized document VLMs and three frontier MLLMs (all zero-shot), POTATR with direct text (DT) extraction achieves the best scores across the board:
| Model |
GriTS_Top |
GriTS_Con |
TEDS |
| Gemini 3.1 Pro (best MLLM) |
0.9500 |
0.9418 |
0.9250 |
| dots.ocr (best small VLM) |
0.9241 |
0.8991 |
0.9168 |
| POTATR + docTR |
0.9601 |
0.8799 |
0.9500 |
| POTATR + DT |
0.9665 |
0.9636 |
0.9565 |
The error decomposition is the paper's most informative analysis. The gap from POTATR + docTR (0.880) to POTATR + DT (0.964) is 0.084—more than twice the residual error with perfect text—so text recognition error dominates, and structure recognition is near its ceiling. The OCR progression (EasyOCR 0.782 → PaddleOCR 0.876 → docTR 0.880 → DT 0.964) shows the system improves automatically with better OCR without retraining. A caveat: the best frontier MLLM outperforms POTATR when paired with traditional OCR, so the headline result depends on text-source quality; the DT setting presumes digitally-native input.
Caption extraction, a byproduct of detection plus text assignment, reaches character-level LCS F1 of 0.978 with direct text, matching Gemini 3.1 Pro (0.977). The authors observe that MLLMs show higher precision but lower recall, occasionally omitting captions, whereas detection guarantees systematic coverage.
Cost and throughput
On a single A10G GPU, POTATR processes 52,031 pages/hour at $47 per million pages—over 130× the throughput of the fastest frontier MLLM and 28× that of the fastest local VLM, at roughly 300× and 28× lower cost respectively. Because decoding is parallel with a fixed 250 queries, inference time is constant at ~21 ms regardless of table size, whereas autoregressive VLM latency grows from 1.8 s to 12 s as cells per page increase. The throughput advantage therefore widens on table-heavy documents. One nuance from the appendix: frontier MLLMs show slightly increasing accuracy with cell count, which the authors speculate reflects more uniform page layouts offsetting longer generations; POTATR instead peaks on the most typical tables in its training distribution.
Modularity and multi-page composition
POTATR's separation of structure detection from text recognition makes errors decomposable and components independently upgradable, in contrast to VLMs where structure and text errors are entangled in a single generation process. This modularity is demonstrated on the hardest out-of-distribution case: multi-page tables in PubTables-v2 Full Documents. Processing page-by-page, POTATR achieves $\textrm{GriTS}_\textrm{Con}$ of 0.671 versus 0.931 for Gemini 3.1 Pro with full-document context. Composing POTATR with an 86M-parameter ViT-B-16 continuation classifier and vertical merging—no retraining required—raises this to 0.827, a 47.4% error reduction that closes 60% of the gap to Gemini 3.1 Pro and 73% to GPT-5.4, at roughly two orders of magnitude lower cost. Continuation prediction itself is near-optimal (0.995 recall), so the residual gap is attributable to page-level extraction on multi-page content POTATR never saw in training, and to within-page table-part merging, which the pipeline does not address.
Limitations
The paper is explicit that POTATR is trained and evaluated exclusively on English scientific articles from PubMed; generalization to other document types and languages is untested, with the argument that TATR has shown cross-domain generalization suggesting adaptation is feasible. Additional unstated assumptions bear on the headline numbers: the best result requires direct text extraction (digitally-native input), and the small-scale architecture comparison is limited to 10 epochs. Open questions include whether the simplified relation head (no relation token) remains optimal under Deformable-DETR-style attention, and how to extend the merging pipeline to table parts split within a single page.
Conclusion
POTATR demonstrates that a 29M-parameter, non-autoregressive image-to-graph model, initialized from pre-trained TSR weights and composed with external text recognition, outperforms frontier MLLMs on page-level table extraction (GriTS_Con 0.964, caption F1 0.979) at constant per-page inference cost. Its spatially grounded, modular output composes with OCR models and lightweight continuation classifiers to handle scanned documents and multi-page tables. The results make the case that small specialized detectors remain a competitive deployment option for document understanding at scale, with the main open problems being cross-domain generalization and out-of-distribution multi-page content.