DocIE Shared Task Benchmark

• DocIE Shared Task is a benchmark that tests systems' ability to perform joint entity and relation extraction from long documents using schema-constrained, zero-shot methods.
• It employs synthetic demonstrations and a two-call inference strategy to address challenges like coreference resolution, long-range relations, and output validity.
• Evaluation relies on precision, recall, and F1 scores, exposing critical bottlenecks in relation extraction and overall document-level reasoning.

Searching arXiv for the specified DocIE-related papers and benchmark context. The DocIE shared task is a benchmark for long-form information extraction at the document level in a zero-shot, schema-constrained setting. At test time, a system receives only the raw text of a long document and a schema consisting of strings naming the entity types and relation types to be extracted; the training and development material resembles the test set only structurally, while schemata and domains differ, so systems must generalize to unseen entity and relation inventories and new domains without task-specific fine-tuning (Popovič et al., 8 Jul 2025). The task therefore combines zero-shot schema transfer with document-level phenomena such as coreference, repeated mentions, and relations spanning multiple sentences.

1. Task definition and scope

The DocIE shared task evaluates joint entity and relation extraction over long documents rather than sentence-bounded IE. Its core requirement is schema-constrained extraction: systems must produce entities and relations using only the type strings supplied at inference time, even though those types may not have appeared in any analogous annotated training resource available to the system. The zero-shot constraint is explicit: no fine-tuning on the shared task data is used in the reported submission, and no manually annotated demonstrations are used (Popovič et al., 8 Jul 2025).

The task is document-level in a strict sense. Systems must handle repeated mentions and coreference across the full document, not merely within local sentence windows. In the reported pipeline, this is represented with entity IDs, so that mentions referring to the same entity share an ID in the annotated output. This enables consistent span aggregation and type assignment across the document, and it makes the relation layer depend on entity consolidation rather than on isolated mention pairs (Popovič et al., 8 Jul 2025).

A plausible implication is that the shared task is designed to test transfer under simultaneous pressure from three sources: unseen schemas, long-context generation, and cross-sentence reasoning. The benchmark structure also discourages methods that rely on domain-specific memorization, because the provided training and development material differs from the test set in both domain and schema.

2. Evaluation protocol and matching regimes

The evaluation protocol reports precision, recall, and F1 for several subtasks on the DocIE test set: entity identification, entity classification, and relation extraction in general and strict modes. The paper states that it follows the official shared-task scripts and does not re-specify the matching rules; strict mode applies a tighter matching criterion than general mode (Popovič et al., 8 Jul 2025).

The standard metrics are:

$P = \frac{TP}{TP + FP}$

$R = \frac{TP}{TP + FN}$

$F1 = \frac{2PR}{P + R}$

Within this framework, entity identification measures whether entity spans are detected, entity classification further requires correct typing, and relation extraction evaluates triple prediction under the benchmark’s two matching regimes. Because matching is performed by official scripts rather than restated by the submission paper, the benchmark separates system design from scorer implementation and centralizes evaluation semantics in the task infrastructure (Popovič et al., 8 Jul 2025).

The presence of both general and strict relation modes is significant. It indicates that relation extraction is assessed not only as a loose document-level linking problem but also under a tighter criterion that penalizes directionality, boundary, or typing inconsistencies more strongly. This is consistent with the paper’s emphasis on subject–object inversion, entity ID mismatches, and long-range reasoning failures as major error sources.

3. Synthetic demonstrations and automatic annotation

One reported approach to the shared task, "DocIE@XLLM25" (Popovič et al., 8 Jul 2025), centers on a fully automatic pipeline that builds a synthetic demonstration database using a reasoning-optimized LLM, retrieves a single most relevant synthetic demonstration at inference time, and performs document-level joint extraction with an in-context prompt tailored to long texts. The method explicitly avoids manual annotation in order to preserve the zero-shot premise.

Synthetic data generation begins with Wikipedia Vital Articles Level 4, approximately 10k articles. The system uses abstracts and truncates each abstract by adding sentences iteratively until the cumulative text length reaches at least 100 words, omitting the remainder. An initial length study up to 300 words showed that error rates are highly correlated with length, which motivated the shorter truncation regime (Popovič et al., 8 Jul 2025).

Zero-shot annotation uses DeepSeek-R1-Distill-Qwen2.5-32B with temperature 0 and a prompt that enforces machine-parseable JSON, a unified schema section listing entity_types and relation_types, full NER with span-level mention detection and coreference resolution through entity IDs, and relation verification hooks in which each triple is accompanied by a natural-language description and entity ID references. Echoing the input with inline tags is used for robust validation (Popovič et al., 8 Jul 2025).

The post-processing stage combines rule-based and LLM-based verification. Entity mention spans must occur in the source text, and relation arguments must reference valid entity IDs. Triple directionality is checked with a dedicated verification prompt that classifies a triple-description alignment as subject-to-object, inverted, both if symmetric, or none. If any inconsistency is detected for a given relation type within a document, all triples of that type are discarded for that document to favor precision over recall. Annotations are also discarded if entity types echo the entity names or if all entities share the same type. When initial verification fails, the zero-shot prompt is run once more with temperature 0.2 to rescue borderline cases (Popovič et al., 8 Jul 2025).

The resulting synthetic dataset has an initial zero-shot annotation yield of 5114 documents, corresponding to a 52.89% success rate on the first pass, and a final post-processed size of 5010 documents. It contains approximately 59k entities across 3466 entity types and approximately 30k relation triples across 7103 relation types. Two documents overlapping with the DocIE test set are removed to avoid contamination (Popovič et al., 8 Jul 2025).

4. Retrieval-based in-context learning for long documents

At inference time, the reported system retrieves a single synthetic demonstration most similar to the query document using dense sentence embeddings from sentence-transformers all-MiniLM-L6-v2. Similarity is computed with cosine similarity between embedding vectors $x$ and $y$:

$\cos(\theta) = \frac{x \cdot y}{\|x\|\,\|y\|}$

The same truncation strategy used during synthetic dataset construction, namely truncation to at least 100 words, is applied to query text to improve comparability (Popovič et al., 8 Jul 2025).

The selected demonstration includes JSON-formatted text_with_spans, entities with IDs and types, relation_types, and relations consisting of triples plus natural-language descriptions. This demonstration is injected as a single-shot example of correct annotation. The inference prompt then presents the query document as an annotation to be completed, with entity_types and relation_types copied from the demonstration. The model is instructed not to introduce new types, to reuse entity IDs where possible, and to complete coreference and missing triples (Popovič et al., 8 Jul 2025).

A central engineering device is a two-call strategy for long documents. Call 1 uses only the first paragraph of the query document. Call 2 supplies the entire document, prefilled with the annotations produced for the beginning paragraph. The paper describes this “bootstrap then complete” pattern as drastically reducing formatting failures for long documents. No sliding window beyond the first-paragraph bootstrap is used (Popovič et al., 8 Jul 2025).

This setup makes the shared task an instructive case of schema transfer through demonstration retrieval rather than parameter adaptation. A plausible implication is that performance depends not only on the base LLM’s extraction capacity but also on schema alignment between the retrieved demonstration and the target document. The paper explicitly notes that poor demonstration relevance can lead either to overfitting to the demonstration schema or to failure to capture needed types and relations (Popovič et al., 8 Jul 2025).

5. Reported performance, bottlenecks, and limitations

Using the official shared-task scripts, the reported results on the DocIE test set are shown below (Popovič et al., 8 Jul 2025).

Subtask	Precision	Recall / F1
Entity Identification	52.36%	23.94% / 32.86%
Entity Classification	25.79%	11.80% / 16.19%
Relation Extraction (General)	5.03%	2.44% / 3.29%
Relation Extraction (Strict)	4.61%	2.23% / 3.01%

The system ranks fourth in the shared task. Only 63.91% of outputs are valid, defined as parseable JSON that passes basic format checks, and even with the two-call strategy 36.09% of outputs remain invalid (Popovič et al., 8 Jul 2025). This makes output validity a first-order issue rather than a peripheral implementation detail.

The error analysis identifies several recurring failure modes. The most common are unparseable JSON and ID mismatches between triples and entities, with document length strongly correlated with such failures. Relation directionality remains problematic: the synthetic-data verification prompt reduces subject–object inversion during data construction, but long-document inference remains error-prone and strict-mode relation scores stay low. Coreference and boundary errors also persist, because the model may miss long-range coreferences or choose inconsistent boundaries, which degrades entity identification and entity classification. Long-range relations that require multi-sentence reasoning are often missed, contributing to low recall, especially in strict mode (Popovič et al., 8 Jul 2025).

The paper’s ablations and sensitivity discussion is deliberately narrow. It uses a single retrieved demonstration; more shots are not explored. The two-call prompt is identified as the main intervention for reducing failures on long documents, but no separate numeric ablation is reported. Retrieval is performed with dense embeddings and cosine similarity; BM25 and other methods are not evaluated. The conservative triple-type-level discarding strategy improves consistency but may limit relation coverage (Popovič et al., 8 Jul 2025).

These findings support a restrained interpretation. Fully synthetic demonstrations can be generated at scale and used operationally for zero-shot in-context extraction, but document-level joint extraction remains difficult for state-of-the-art LLMs in this setting, particularly for relation extraction and robust structured output generation (Popovič et al., 8 Jul 2025).

6. Relation to broader document IE benchmarks

The DocIE shared task occupies a different design space from other document IE benchmarks. A notable comparison point is "DocILE 2023 Teaser: Document Information Localization and Extraction" (Šimsa et al., 2023), a shared task and benchmark jointly hosted as a CLEF 2023 lab and an ICDAR 2023 competition. DocILE focuses on business documents and defines two tasks: Key Information Localization and Extraction (KILE), which requires detection of semantically important values of predefined field types together with bounding boxes, and Line Item Recognition (LIR), which requires detection of fields and grouping them into line items (Šimsa et al., 2023).

This contrast is technically consequential. DocILE operates on pre-processed PDFs with OCR word-level tokens and bounding boxes, emphasizes layout diversity, and evaluates KILE with Average Precision under a Pseudo-Character Centers correctness rule rather than IoU-based true-positive matching. LIR uses micro F1 over line item fields with a maximum-matching procedure between predicted and ground-truth line items (Šimsa et al., 2023). By contrast, the DocIE shared task as described in the long-document extraction submission operates on raw text plus a schema of entity and relation type names, and evaluates entity identification, entity classification, and relation extraction in general and strict modes (Popovič et al., 8 Jul 2025).

A common misconception is that document IE benchmarks are interchangeable because they all involve extracting structured information from documents. The benchmark designs summarized here show otherwise. DocILE is centered on localization, OCR-linked geometry, and line-item grouping in semi-structured business documents, whereas DocIE emphasizes zero-shot schema-constrained document-level entity and relation extraction from long text with coreference and cross-sentence relations. This suggests that progress on one benchmark family does not directly imply progress on the other, even though both belong to the broader document information extraction landscape (Šimsa et al., 2023).

7. Reproducibility, released resources, and research directions

The reported DocIE submission releases its full pipeline and synthetic dataset at https://github.com/nicpopovic/docie-xllm25. The repository includes prompts, verification scripts, and evaluation routines. The released model stack consists of DeepSeek-R1-Distill-Qwen2.5-32B for generation and inference, using temperature 0 except for a single synthetic re-try at temperature 0.2, and sentence-transformers all-MiniLM-L6-v2 for retrieval (Popovič et al., 8 Jul 2025).

The reproduction procedure is explicit. First, build the synthetic demonstration database from Wikipedia Vital Articles abstracts using the zero-shot annotation prompt, then run verification and post-processing. Second, for DocIE test inference, remove overlapping documents from the demonstration database, retrieve a single most similar demonstration using all-MiniLM-L6-v2 and cosine similarity, and run the two-call inference prompts. Third, evaluate with the shared-task scripts to obtain precision, recall, and F1 for entity identification, entity classification, and relation extraction in general and strict modes (Popovič et al., 8 Jul 2025).

The paper also enumerates recommendations for future work. These include better long-context handling and structured decoding to reduce unparseable outputs; improved retrieval, including hybrid dense–sparse or schema-aware selection, together with multi-shot demonstrations; schema alignment strategies that adapt demonstration types to the target schema without violating zero-shot constraints; more robust post-processing and hybrid symbolic–neural checks such as enforcing argument roles and symmetric or antisymmetric relation constraints; and explicit modeling of cross-sentence reasoning and coreference beyond span echoing, possibly via iterative extraction and verification loops (Popovič et al., 8 Jul 2025).

Taken together, these directions frame the DocIE shared task as a benchmark for data-centric and prompt-centric zero-shot IE research rather than for conventional fine-tuned supervised extraction. The released synthetic resources demonstrate that large demonstration databases can be built without manual annotation, while the reported validity and relation-extraction numbers show that long-context structured decoding, schema alignment, and document-level reasoning remain unresolved technical bottlenecks (Popovič et al., 8 Jul 2025).