Aryn Partitioner: Segmenting Unstructured Documents

• The Aryn Partitioner is a document segmentation tool that uses advanced models like Deformable DETR and Table Transformer to extract text, tables, and images from unstructured sources.
• It creates hierarchical DocSets that serialize documents as semantic trees, enabling scalable, distributed processing for diverse analytics workflows.
• Empirical benchmarks show that the partitioner achieves higher mAP and mAR than commercial APIs, enhancing precision and explainability in downstream analyses.

The Aryn Partitioner is a document segmentation component in the Aryn unstructured analytics system, responsible for converting raw PDFs and document images into structured, machine-usable representations termed “DocSets.” This function is foundational for large-scale semantic analytics over unstructured documents, providing the structured tree abstractions required by downstream declarative and distributed processing engines such as Sycamore and enabling high-accuracy, explainable analytics workflows. Architecturally, the Aryn Partitioner incorporates advanced vision models—specifically Deformable DETR and Table Transformer models—combined with OCR, in order to robustly segment the heterogeneity of document content (text, tables, images) encountered in diverse real-world corpora.

1. Role and Integration in the Aryn Analytics System

The Aryn Partitioner operates as the starting point for document ingestion within the Aryn system. Its output, the DocSet, is a distributed collection of semantic trees representing each document’s structural and logical layout. Integration with Aryn’s Sycamore engine and Luna query planner is tightly coordinated:

• Input: Accepts raw, unstructured document data—PDFs and images—without assumptions about document type or domain.
• Processing: Applies document layout analysis, bounding box detection, table extraction, and region-specific OCR.
• Output: Emits a serialization of the document as a tree of nodes (sections, paragraphs, tables, images, etc.) with type metadata, geometric information, and text content. This JSON-compatible structured representation is ready for distributed ETL and analytic workflows within Sycamore.
• Downstream coupling: DocSets can be programmatically transformed, filtered, and queried at a fine granularity by functional operators and LLM-powered enrichments in Sycamore; Luna can then synthesize and execute complex queries over these processed collections.

2. Model Architecture and Extraction Pipeline

The core of the Aryn Partitioner’s technical approach is a Deformable DETR (“DEtection TRansformer”) object detector, fine-tuned on the DocLayNet dataset:

• Deformable DETR [DocLayNet]: This architecture uses multi-scale attention to detect and segment regions corresponding to logical entities (text boxes, tables, images) in document layouts. Its deformable attention modules enable efficient identification of variable-sized, non-axis-aligned regions.
• Table Transformer: For regions predicted as tables, a dedicated model further extracts structural information (row and column demarcations) and associates text via OCR.
• OCR Integration: Image regions with embedded text are passed through OCR models, whose outputs are inserted into the semantic tree with position and type annotations.
• Postprocessing: All detected elements are serialized into a format that preserves ordering, hierarchy, bounding box coordinates, and semantic labels.

This processing achieves efficient segmentation while maximizing recall for diverse layout features in heterogeneous, naturally-occurring documents.

3. DocSet Abstraction and Representation

The output of the Aryn Partitioner is the DocSet, which generalizes tabular paradigms (such as DataFrames) to accommodate hierarchical, unstructured content:

• Tree Structure: Each document is encoded as a tree, with interior nodes representing logical containers (e.g., section, subsection, table) and leaf nodes representing atomic elements (e.g., paragraph, cell, image), each adorned with fine-grained content and type metadata.
• Metadata: Nodes encapsulate bounding boxes, region type, order, and extracted content. For tables, cells are further associated with coordinate triples (row, column, span).
• Distributed Processing: DocSets are explicitly designed for parallel, fault-tolerant execution. The Sycamore engine leverages Ray for scalable dataflow across large corpora.

This hierarchical abstraction enables subsequent analytic and enrichment workflows to operate seamlessly across text, tabular, and image content.

4. Empirical Performance and Comparative Evaluation

Benchmarking demonstrates the Aryn Partitioner’s segmentation fidelity:

System/Model	mAP	mAR
Aryn Partitioner (Deformable DETR)	0.602	0.743
Cloud Vendor API	0.344	0.466

• Mean Average Precision (mAP): The partitioner more accurately detects relevant elements than typical commercial APIs.
• Mean Average Recall (mAR): Higher recall indicates fewer missed content regions.
• Significance: Improved segmentation directly enhances downstream analytics, as information critical to queries is less likely to be missed or misidentified.

These empirical results are supported by real-world analytics tasks: in NTSB reports, the end-to-end system yields 72% accuracy on complex analytic queries, outperforming retrieval-augmented generation (RAG) approaches that rely on less structured representations.

5. Processing Pipeline and Implementation Details

A typical Sycamore pipeline using the Aryn Partitioner proceeds as follows:

schema = { "us_state": "string", "probable_cause": "string", "weather_related": "bool" } ds = context.read.binary("/path/to/ntsb_data") \ .partition(ArynPartitioner()) \ .extract_properties(OpenAIPropertyExtractor("gpt-4o", schema=schema)) \ .explode() \ .embed(OpenAIEmbedder("text-embedding-3-small"))

• partition(ArynPartitioner()): Converts input data into a DocSet.
• extract_properties(...) and explode(): Downstream enrichment and flattening for analytics.
• embed(...): Vectorizes elements for semantic search and machine learning tasks.

This pipeline highlights the modularity, composability, and transparency enabled by the Partitioner’s structured output.

6. Explainability and Trust

Explainability is a foundational principle in Aryn’s design, including at the partitioning stage:

• Semantic Transparency: All partitioning decisions, detected elements, and their hierarchy are auditable in the DocSet representation.
• Plan Traceability: Query execution via Luna exposes the lineage of all operations applied to the partitioned data, including the exact logic for element filtering, property extraction, and aggregation.
• User Modification: Analysts may interactively adjust or refine plans if automated choices in region labeling or element extraction do not align with their intent.

This architectural transparency supports rigorous auditing and debuggability, essential for trusted use in domains such as finance, law, and safety analysis.

7. Implications and Applications

The Aryn Partitioner’s methodology generalizes beyond NTSB reports to any domain with challenging, mixed-content unstructured documents. Accurate partitioning into semantic trees enables robust ETL transformations and advanced analytics—capabilities critical for regulatory analysis, scientific literature mining, and enterprise knowledge discovery workflows. The documented performance gains in both segmentation fidelity and downstream analytics underscore the value of integrating purpose-built vision models with document analytics pipelines. The combination of distributed, hierarchical processing with explainable query plans defines a new standard for unstructured data analytics at scale.