DocSplit: Document Packet Splitting Benchmark

• DocSplit is a benchmark dataset and evaluation approach for splitting multi-page document packets into structured sub-documents via boundary detection, classification, and ordering.
• It introduces five benchmark regimes that progressively challenge models with shuffled, interleaved, and randomized page orders, emphasizing separate metrics for clustering and sequencing.
• The framework integrates multimodal techniques using OCR-extracted text, layout, and image features with BIO tagging and a CRF layer to enhance document packet segmentation.

Searching arXiv for the specified DocSplit paper and closely related supporting work. {"query":"(Islam et al., 17 Feb 2026) DocSplit A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splitting", "max_results": 5} {"query":"(Islam et al., 26 Feb 2026) IDP Accelerator Agentic Document Intelligence from Extraction to Compliance Validation DocSplit", "max_results": 5} DocSplit is a benchmark dataset and evaluation approach for document packet recognition and splitting, a task that arises when heterogeneous, multi-page document packets contain multiple documents stitched together. It formalizes document packet splitting as boundary detection, document-type classification, and page sequence reconstruction, and introduces five datasets of increasing difficulty together with evaluation metrics that separate clustering quality from ordering quality (Islam et al., 17 Feb 2026). A related systems paper presents DocSplit within the IDP Accelerator framework as a multimodal page-level sequence-labeling component using BIO tagging for agentic document intelligence workflows (Islam et al., 26 Feb 2026).

1. Problem setting and scope

In many enterprise pipelines, including legal e-discovery, loan or insurance claim processing, healthcare records, and mortgage closings, scanned or digitally collected document packets bundle multiple distinct forms, contracts, invoices, medical notes, and related materials into a single stream of pages (Islam et al., 17 Feb 2026). Errors in manual assembly, including page shuffling, duplication, and missing separators, force human specialists to segment and classify pages before downstream extraction or retrieval can proceed. DocSplit is explicitly motivated by the observation that existing benchmarks and models excel at single-page classification or layout analysis, but do not address the multi-page grouping, classification, and re-ordering problem inherent in real document packets.

The benchmark is framed as the first comprehensive public benchmark for document packet splitting, with goals that include formalizing the task, supplying five datasets of increasing difficulty, proposing continuous diagnostic evaluation metrics, and measuring state-of-the-art multimodal LLMs on these benchmarks (Islam et al., 17 Feb 2026). This scope is broader than page categorization alone: the target output is a structured decomposition of an entire packet into logically distinct sub-documents with correct types and correct internal page order.

A related formulation in IDP Accelerator treats DocSplit as a multimodal classifier using BIO tagging to segment complex document packets (Islam et al., 26 Feb 2026). That formulation places DocSplit within an end-to-end document intelligence stack that also includes extraction, agentic analytics, and rule validation, but DocSplit itself remains the segmentation and packet-structuring component.

2. Formal task decomposition

DocSplit defines an input packet as a sequence of $n$ pages,

$P=\{p_1,p_2,\dots,p_n\}.$

The objective is to split $P$ into $m$ logical sub-documents,

$\mathcal D=\{d_1,d_2,\dots,d_m\},$

where each $d_i\subset P$ is a contiguous, after re-ordering, subset of pages belonging to the same document (Islam et al., 17 Feb 2026).

The task is decomposed into three modules. Boundary detection is defined by

$B(P):P\to\{0,1\}^n,$

where $B(P)_j=1$ if page $p_j$ is marked as the start of a new sub-document. Document classification is defined by

$$C(d_i):d_i\to\{\text{type}_1,\text{type}_2,\dots\},$$

which assigns one of $P=\{p_1,p_2,\dots,p_n\}.$0 pre-defined document types to the $P=\{p_1,p_2,\dots,p_n\}.$1th sub-document. Page ordering seeks, for $P=\{p_1,p_2,\dots,p_n\}.$2, a permutation

$P=\{p_1,p_2,\dots,p_n\}.$3

that recovers the original page order.

This decomposition is central to the benchmark’s design. Boundary detection governs segmentation; classification governs semantic typing; and page ordering governs sequence reconstruction. The benchmark explicitly targets settings in which these components interact under realistic packet corruption, including out-of-order pages, interleaved documents, and packets lacking clear demarcations (Islam et al., 17 Feb 2026).

3. Benchmark construction and packet regimes

DocSplit comprises five distinct benchmarks, and train, validation, and test splits are provided for all of them (Islam et al., 17 Feb 2026).

Benchmark	Packet structure	Scale/details
Mono-Seq	Single-type documents concatenated in order	Packet size 5–20 pages; 1 type per packet; no shuffling; ~63 K large and ~91 K small packets
Mono-Rand	Same as Mono-Seq but pages fully shuffled	Tests ordering recovery
Poly-Seq	Multi-type, sequential	Heterogeneous concatenation; ~288 K large and ~152 K small
Poly-Int	Multi-type, interleaved	Round-robin mixing of pages across documents
Poly-Rand	Multi-type, fully randomized	Most difficult; no structural or sequential cues remain

The corpus covers domains spanning legal, financial, healthcare, transportation proofs, compliance forms, resumes, letters, budgets, questionnaires, scientific publications, and more (Islam et al., 17 Feb 2026). Layouts range from text-only letters and memos, to structured forms with tables and key–value pairs, to scanned images with signatures, and mixed-modality documents. The benchmark explicitly tests out-of-order pages, interleaved pages from multiple documents, homogeneous packets lacking type transitions, packets without clear separators, scanned quality issues, and duplicate or missing pages.

A related dataset description in IDP Accelerator presents DocSplit as 500 synthesized multi-document packets containing 13 distinct document types and a total of 7 330 pages (Islam et al., 26 Feb 2026). The listed types include invoices, contracts, letters, purchase orders, medical records, technical reports, receipts, forms, memos, NDAs, policies, resumes, and statements. This related formulation focuses on page-level sequence labeling over packet sequences rather than the five-benchmark regime emphasized in the benchmark paper.

4. Annotation schema and multimodal sequence labeling

Within IDP Accelerator, DocSplit is annotated with page-level BIO tagging to identify boundaries and assign a document type to each page (Islam et al., 26 Feb 2026). The labels are $P=\{p_1,p_2,\dots,p_n\}.$4 for the beginning of a document of type TYPE, $P=\{p_1,p_2,\dots,p_n\}.$5 for pages inside the same TYPE, and $P=\{p_1,p_2,\dots,p_n\}.$6 for pages outside any document, such as blank covers or irrelevant pages. For 13 types, the full tag set is $P=\{p_1,p_2,\dots,p_n\}.$7, yielding $P=\{p_1,p_2,\dots,p_n\}.$8 tags.

The annotation process is described as follows. Human annotators use a web-based labeling tool to load each packet, assign a tag to every page according to BIO rules, and produce a page-wise sequence of BIO tags plus the packet ID. A second annotator reviews 20% of the data for quality control, with inter-annotator Cohen’s $P=\{p_1,p_2,\dots,p_n\}.$9 (Islam et al., 26 Feb 2026).

The model architecture is a multimodal page-level sequence labeling model in which pages are processed independently but tags are decoded jointly in sequence. The per-page inputs are OCR-extracted text tokens $P$0, layout features $P$1, and a raster image $P$2. The encoder components are a BERT-style Transformer for text, an MLP or small GCN over bounding boxes for layout, and a ResNet or ViT for vision: $P$3 Fusion is defined as concat-then-project: $P$4 where FusionLayer is linear + GELU + dropout. The sequence labeling head applies softmax over 27 tags,

$P$5

with an optional linear-chain CRF to enforce valid BIO transitions. The loss is cross-entropy plus $P$6-regularization: $P$7

The reported training protocol uses an 80% train / 10% validation / 10% test split by packet, AdamW, learning rate $P$8 with linear warmup over 10% of steps and then linear decay, weight decay $P$9, batch size 16 pages, 10 epochs with early stopping if validation BIO F1 does not improve for 2 epochs, dropout $m$0, and gradient clipping at 1.0 (Islam et al., 26 Feb 2026).

5. Evaluation methodology

DocSplit’s benchmark paper introduces packet-level metrics that are intended to avoid the all-or-nothing behavior of classical exact-match scoring (Islam et al., 17 Feb 2026). Boundary detection is evaluated with precision, recall, and F1 over predicted versus true boundaries. Document-type classification uses standard accuracy,

$m$1

Ordering quality is measured with Kendall’s Tau for each multi-page sub-document: $m$2 where $m$3 is the number of concordant pairs and $m$4 is the number of discordant pairs. The benchmark averages these values over documents with $m$5: $m$6

Clustering quality is defined from the ground-truth partition $m$7 and predicted partition $m$8 using Rand Index and V-measure: $m$9 where $\mathcal D=\{d_1,d_2,\dots,d_m\},$0 counts page pairs in the same cluster in both partitions, $\mathcal D=\{d_1,d_2,\dots,d_m\},$1 counts page pairs in different clusters in both partitions, $\mathcal D=\{d_1,d_2,\dots,d_m\},$2 is homogeneity, and $\mathcal D=\{d_1,d_2,\dots,d_m\},$3 is completeness. With weighting $\mathcal D=\{d_1,d_2,\dots,d_m\},$4,

$\mathcal D=\{d_1,d_2,\dots,d_m\},$5

The combined packet score balances clustering and ordering: $\mathcal D=\{d_1,d_2,\dots,d_m\},$6 The default setting is

$\mathcal D=\{d_1,d_2,\dots,d_m\},$7

with $\mathcal D=\{d_1,d_2,\dots,d_m\},$8 (Islam et al., 17 Feb 2026). The benchmark states that these metrics provide continuous scores in $\mathcal D=\{d_1,d_2,\dots,d_m\},$9, separate boundary versus ordering errors, and assign partial credit for near-correct groupings or orderings.

IDP Accelerator reports a different but related evaluation layer: page-level accuracy, macro-averaged precision/recall/F$d_i\subset P$0 over BIO classes, section-grouping accuracy

$d_i\subset P$1

and strict sequence accuracy, defined as 1 when the predicted sequence of segments with page orders exactly matches the ground truth and 0 otherwise, averaged over packets (Islam et al., 26 Feb 2026). The two formulations emphasize different granularities: packet-level continuous diagnostics in the benchmark paper, and page- and packet-level sequence labeling quality in the systems paper.

6. Empirical results, tooling, and future work

The benchmark paper evaluates multimodal LLMs in zero-shot prompting with temperature 0.0, top-p 0.1, and top-k 5, including Claude Sonnet 4.5, Claude Haiku 4.5, DeepSeek V3.1, Gemma 3 27B, and Qwen 3 VL 235B (Islam et al., 17 Feb 2026). Packet scores $d_i\subset P$2 range from approximately 0.80, reported for Gemma on the easiest Mono-Seq setting, to approximately 0.95, reported for Qwen on Poly-Seq. Ordering performance remains very high, with $d_i\subset P$3, once pages are correctly clustered, whereas clustering $d_i\subset P$4 shows large variance from 0.56 to 0.90 and is identified as the primary bottleneck. The paper further reports that classical metrics such as Page+Split+Order often under-report partial success, sometimes yielding 0% even when clustering is near-perfect. Strong models such as Qwen degrade only approximately 3% when moving from Mono-Seq to Mono-Rand, while weaker models collapse under Poly-Rand. Reported failure modes include over- and under-segmentation, misclassification of document types leading to incorrect group assignments, and boundary ambiguity under shuffled or interleaved pages when types repeat.

In IDP Accelerator, the reported DocSplit test-set results are page-level accuracy 98.7%, macro F$d_i\subset P$5 97.9%, section-grouping accuracy 97.4%, and strict sequence accuracy 95.8% (Islam et al., 26 Feb 2026). Single-modality ablations report text-only accuracy 94.2% with strict accuracy 88.5%, and image-only accuracy 90.7% with strict accuracy 82.3%. Latency is reported as 2.5 s/packet for a baseline OCR-only splitting pipeline and 0.5 s/packet for multimodal DocSplit, approximately an 80% reduction. Error analysis attributes approximately 60% of errors to visually similar templates, 25% to very short documents of 1–2 pages lacking clear headers, and 15% to borderline cases where human annotators disagreed. Ablation results indicate that removing layout features reduces macro F$d_i\subset P$6 by 2.3 percentage points, removing image features reduces macro F$d_i\subset P$7 by 3.1 percentage points, and removing CRF smoothing reduces strict accuracy by 1.1 percentage points. The same paper reports a healthcare deployment with 98% classification accuracy on live packets, 80% latency reduction, and 77% cost saving versus legacy, and a community-management use case with 95% doc-type accuracy on 48 million documents, with processing reduced from hours to seconds.

DocSplit is released at https://huggingface.co/datasets/amazon/doc_split, with the license described as publicly available for research and full terms on the HuggingFace page, typically CC-BY-4.0 or similar (Islam et al., 17 Feb 2026). The per-packet annotation format is described with JSON per subdocument, using one-based original ordinals:

$d_i\subset P$8

The evaluation directory structure is given as:

$d_i\subset P$9

An AWS-hosted “IDP Accelerator Test Studio” supports no-code evaluation, cost attribution, and result dashboards, and custom datasets can be uploaded in this structure for instant benchmarking (Islam et al., 17 Feb 2026). Future directions named for DocSplit include extending to very long packets of 100+ pages, evaluating a broader variety of multimodal LLM and VLM architectures including fine-tuning strategies, incorporating visual and layout cues such as tables, signatures, and stamps directly in multimodal evaluations, exploring alternative metrics such as Variation of Information and Levenshtein distance on page sequences, investigating domain-specific models and priors for boundary accuracy, and applying DocSplit to downstream information extraction, question answering over multi-document packets, and retrieval augmentation (Islam et al., 17 Feb 2026).