---
title: Key–Value Extraction
url: https://www.emergentmind.com/topics/key-value-extraction
type: topic
---

# Key–Value Extraction

Key–value extraction (KVE) is the task of identifying key–value pairs within structured or semi-structured documents, transforming visually and semantically linked entities into explicit, machine-readable tuples. In the context of document analysis, a key is an informational label or field name (e.g., “Invoice Date”), while a value represents the associated content (e.g., “2024-06-10”). The problem arises in both born-digital and scanned document images, and spans invoices, forms, receipts, historical records, and more. Modern KVE systems operate across visual, textual, and layout dimensions, employing deep multimodal models, sophisticated linking strategies, and large-scale annotated datasets to achieve robust extraction under arbitrary layouts.

## 1. Formal Task Definition and Problem Formulation

Given a document \(D = \{e_1, e_2, \ldots, e_N\}\) where each entity \(e_i\) has textual content and a spatial bounding box, the objective is to predict a set of pairs \(P = \{(k_j,v_j)\}_{j=1}^{M}\) such that each key \(k_j\) and value \(v_j\) are entities in \(D\), with \(k_j\) functioning as a semantic label for \(v_j\) [2405.00505]. The definition extends to allowing unkeyed values (values without explicit keys) and unvalued keys (keys without a value). KVE is evaluated by matching predicted and ground-truth pairs according to text similarity (e.g., normalized edit distance threshold) and spatial overlap (e.g., Intersection-over-Union threshold).

Several architectural and methodological paradigms have been developed:

- **Pixel-wise segmentation**: Models treat keys/values as categories in a segmentation mask over the document image [2010.05322].
- **Entity-linking via multimodal encoders**: Systems utilize Transformer-based architectures with visual, textual, and spatial cues to encode entities and learn explicit key–value relations [2304.07957, 2307.10551].
- **Pointer–pointer or graph approaches**: Extraction formulated as typed directed links between entity spans, allowing joint decoding of entities and relations [2307.10551].
- **Instruction-tuned multimodal large language models**: Generative models receive the image and a prompt and output structured key–value lists, leveraging content-aware tokenization and region detection [2507.09531, 2405.00505].
- **Segmentation-free handwritten document approaches**: Joint models learn to extract and transcribe key–value fields without explicit intermediate segmentation steps [2304.13606, 2304.13530].

## 2. Datasets and Annotation Protocols

Robust KVE requires large, diverse, and accurately annotated datasets tailored for both predefined and open-schema key–value scenarios.

### Summary Table: Major KVE Benchmarks

| Dataset    | Modality   | Size    | Annotation      | Layout/Schema Diversity       |
|------------|------------|---------|-----------------|------------------------------|
| FUNSD      | Scanned forms | 199 pg | Entity spans, key–value relations | Limited, four classes (“header”, “question”, “answer”, “other”); revised for consistency [2010.05322] |
| KVP10k     | Business docs | 10,707 pg | Rich key–value links, 17 classes | Extreme: invoices, forms, statements, diverse and arbitrary keys [2405.00505] |
| CLEX       | Forms (complex) | 5,860 pg | 1,162 unique semantic labels, key–value links | High: hundreds of categories, zero-/few-shot splits [2307.10551] |
| SIMARA     | Handwritten, archival | 5,393 pg | 7 fixed metadata fields (page-level) | Historic, noisy, no spatial localization [2304.13606] |

KVP10k specifically bridges KIE (predefined keys) and open or emergent key–value pair extraction. Its annotation schema encompasses bounding boxes, semantic class labels, explicit key–value links, unkeyed values, and section blocks. In contrast, SIMARA and related datasets from [2304.13530] focus on segmentation-free, sequence-labeled key–value extractions, particularly from handwritten and historical documents.

## 3. Model Architectures and Extraction Paradigms

### 3.1 Segmentation-based Approaches

Segmentation models (e.g., U-Net with channel-invariant deformable convolutions) directly predict per-pixel class assignments ({key, value, other, background}) from document images (and optionally a text mask). These architectures focus on spatial cues, and post-hoc nearest-neighbor linking assigns values to detected keys [2010.05322].

### 3.2 Transformer-based, Entity-centric, and Pointer Formulations

Entity-centric methods encode detected text spans with visual, textual, and spatial embeddings (using LayoutLM, LayoutXLM, or similar), then explicitly predict key–value associations:

- **KVPFormer** implements a QA-style pipeline: keys are selected as “questions,” and a transformer decoder predicts their values via a two-stage coarse-to-fine ranking, enhanced by spatial compatibility attention [2304.07957]. The shallow encoder performs key detection; the decoder matches each candidate key to value spans using self- and cross-attention with spatial bias terms \(b_{ij}=FFN(r_{ij})\).
- **PPN (Parallel Pointer-based Network)** casts KVE as dense, parallel link prediction between token pairs. Typed links (11 kinds: e.g., QuestionHead→ValueHead) express all relevant entity and relation types. Link logits are scored jointly via a large (N×N) score matrix and optimized with circle-loss to balance sparse positives and dense negatives. All candidate labels are decoded in parallel in a single model pass [2307.10551].

Generative LLM-based approaches (e.g., Mistral-7B in KVP10k) receive OCR-extracted lines and bounding boxes, and are trained to output a JSON-like serialization of all discovered key–value pairs. This enables handling of dynamic keys/values and supports unkeyed/unvalued extraction [2405.00505].

### 3.3 Layout-aware, Content-aware, and Multimodal Large Language Models

Instruction-tuned MLLMs such as VDInstruct [2507.09531] use explicit spatial ROI detection and content-aware tokenization to selectively encode only relevant regions of the document image. Key features include:

- **Content-aware tokenization**: Region proposals (from Faster R-CNN) define ROIs; each receives a minimal, complexity-adaptive set of spatial and semantic tokens. Typical token reduction is ∼3.6× over patch-based tiling.
- **Dual vision encoder**: Separate spatial and semantic channels, with pooled text-region, vision-region, and global tokens.
- **Fusion with LLM**: Tokens from the vision backbone are projected and concatenated with user instructions, then passed to a Vicuna (or similar) decoder for generative key–value output.
- **Three-stage curriculum**: Layout pretraining (ROI detection), feature learning (VQA-style parsing), and final instruction tuning on large, annotated KIE tasks. This results in SOTA zero-shot performance and strong efficiency.

## 4. Benchmarking, Metrics, and Error Analysis

Historically, KVE models have been evaluated with precision, recall, and F1 at the key–value pair level, with “match” criteria defined by both text similarity and spatial overlap [2405.00505]. For more complex documents, additional grouping structure, e.g., tables or line-items, must be preserved.

### Advanced Evaluation: KIEval Metric

KIEval [2503.05488] introduces application-centric, hierarchical metrics:

- **Entity-Level F1**: Standard matching of (key, value) pairs.
- **Group-Level F1**: Evaluates correct grouping of multiple extracted entities (e.g., line-item association).
- **Aligned Correction-Cost Variant**: Directly measures the minimal number of human corrections (substitutions, additions, deletions) required to reach ground truth, aligning model evaluation with practical RPA settings.

KIEval exposes shortcomings of entity-F1, especially for group-structured information extraction. On datasets with explicit grouping (e.g., CORD), entity-F1 may overstate model utility—group-F1 is significantly lower and tightly correlates to true correction cost.

## 5. Empirical Results, Generalization, and Methodological Insights

Recent models achieve strong gains over previous SOTA:

- **KVPFormer**: On FUNSD, 82.23% entity-linking F1; on XFUND (multi-language), 94.42% average F1—surpassing prior state of the art by 7.2 and 13.2 F1 points, respectively [2304.07957].
- **PPN**: On complex, zero-shot CLEX splits, 74.0% F1, outperforming Donut and QA by large margins. Few-shot adaptation produces F1 >83% with just 1 annotated sample per novel layout [2307.10551].
- **VDInstruct**: Achieves zero-shot F1 of 57.2% (vs. DocOwl 1.5 at 51.7%), overall F1 71.4% with only ~485 image tokens per page, and is robust to unseen document types [2507.09531].
- **KVP10k Mistral-7B baseline**: Combined (text+loc) F1 scores by type: regular KVP 0.611, unkeyed 0.584, unvalued 0.588, exposing challenges in variable layout and linkage [2405.00505].
- **SIMARA (handwritten, segmentation-free)**: Micro-F1 ≈95% on most metadata fields; lower for rare categories (“arrangement” field F1=66.3%) [2304.13606]. Integrated architectures outperform classic two-stage systems especially in the presence of rich spatial context [2304.13530].

Ablation studies consistently show the importance of:

- Multimodal fusion (text, vision, layout)
- Explicit spatial modeling (ROI detection, 2D positional embeddings, spatial attention bias)
- Joint, end-to-end optimization of entity and linking objectives

Limiting factors include small training sets, lack of heavy augmentation for rare layout scenarios, absence of fine-grained spatial supervision (for historical and handwritten records), and sparse tagging degrading key–value-specific model performance [2010.05322, 2304.13530].

## 6. Open Challenges and Future Directions

Key areas for development and research include:

- **Generalization across schemas**: Enabling models to extract novel key–value types without retraining; supporting schema-free information extraction, as in KVP10k.
- **Multimodal, instruction-following models**: Further exploration of instruction-tuned MLLMs with content-adaptive image tokenization and explicit cross-region attention [2507.09531].
- **Joint entity extraction and relation linking**: Avoiding error propagation by fully integrating detection and linking, building on pointer networks and graph-based architectures [2307.10551].
- **Balanced evaluation**: Adoption of KIEval for industrial/document-centric pipelines—reporting both entity- and group-level F1, and including correction-cost metrics for deployment decisions [2503.05488].
- **Handwritten and historical documents**: Expanding segmentation-free, end-to-end learning with robust architectures and targeted semi-/self-supervised pretraining [2304.13606, 2304.13530].
- **Layout robustness**: Aggressive synthetic augmentation, graph attention, and hybrid losses to improve resilience to visually or structurally diverse business documents [2405.00505].
- **Active learning and rare entity discovery**: Optimizing annotation efforts for rare key–value patterns and ambiguous document structures.
- **Downstream integration**: Evaluating extraction models within full RPA pipelines (for example, form-to-database ingestion), with group-aware and correction-aware metrics.

Overall, key–value extraction—through the synergy of advanced multimodal encoders, adaptive generation strategies, open and richly annotated datasets, and application-centric evaluation—has made substantial strides but continues to present evolving challenges in recognition, linking, generalization, and deployment.

Source: https://www.emergentmind.com/topics/key-value-extraction