---
title: Image-Decoupled Table Parsing (IDTP)
url: https://www.emergentmind.com/topics/image-decoupled-table-parsing-idtp
type: topic
---

# Image-Decoupled Table Parsing (IDTP)

Image-Decoupled Table Parsing (IDTP) refers to a class of methodologies that structurally and algorithmically "decouple" the visual parsing of table cell content and embedded non-textual elements (such as images or figures) from the extraction and modeling of table structure. The fundamental objective is to prevent non-textual visual cues from interfering with the recognition of cell boundaries, structural tokens, and textual cell contents. Recent implementations, notably in TableFormer and MonkeyOCR v1.5, have realized the IDTP paradigm with marked quantitative and qualitative advances, particularly in robustly handling complex tables with embedded images, diverse layouts, and variable content modalities [2203.01017] [2511.10390].

## 1. Motivation and Problem Definition

Tables in document images may contain text, figures, or photographs intermingled within structured rows and columns. Traditional table parsing architectures process the entire table region as a single image, so the features from embedded non-textual regions can interfere with both cell detection and table-structure decoding. The result is degraded boundary detection, spurious or missing cells, dropped images, and impaired HTML structure recovery. IDTP addresses this by explicitly decoupling image/figure content from the sequence modeling and structure inference pipeline. In MonkeyOCR v1.5, this is achieved through detection and masking of embedded figures prior to structure decoding, ensuring that HTML generation focuses solely on the structural/textual properties, with deferred re-integration of extracted images for accurate restoration [2511.10390].

## 2. Core Architectural Paradigms

Both TableFormer and MonkeyOCR v1.5 instantiate IDTP by introducing clear architectural separation between modules handling visual (cell and/or image) localization and those responsible for structure prediction.

- **TableFormer** couples a ResNet-18 based CNN encoder for image feature extraction with two downstream decoders: a transformer-based structure decoder (that emits tokenized HTML tags) and an object-detection decoder aligned with HTML `<td>` tokens. The structure decoder is strictly responsible for emitting table structure, while the cell detector regresses bounding boxes and "empty vs. non-empty" labels for each cell, based on hidden states at `<td>` token positions [2203.01017].

- **MonkeyOCR v1.5** incorporates an explicit IDTP module as a data preparation step. YOLOv10 is used to localize embedded image regions within a table crop. Detected regions are masked out (zeroed), producing a masked image $I^m$ input to a Vision Transformer (ViT)-based Vision–Language Model for table-HTML decoding. The HTML decoder's output includes explicit placeholder tokens (`<img id=j>`) for image cells. Original images are reinserted in post-processing, guaranteeing fidelity and perfect image retention [2511.10390].

The high-level workflow for IDTP in MonkeyOCR v1.5 can be summarized as:

| Step                      | Component                     | Functionality                                                                                                           |
|---------------------------|-------------------------------|------------------------------------------------------------------------------------------------------------------------|
| Figure Localization       | YOLOv10                       | Detect bounding boxes of embedded images within the table crop                                                          |
| Image Decoupling & Input  | Masking / Mapping             | Mask detected image cells in $I^o$ to produce $I^m$; record crop-to-ID mappings                                         |
| Structure Decoding        | Vision–Language Model (ViT+Tr) | Decode $I^m$ into HTML with placeholder tokens for images                                                               |
| Image Reintegration       | Post-processing               | Replace each `<img id=j>` token with the exact original image patch per recorded mapping                                |

## 3. Mathematical Formulation and Loss Functions

IDTP frameworks are trained to jointly optimize structural and localization objectives, with additional auxiliary losses in IDTP-aware systems.

- **TableFormer** introduces a multi-task loss with cross-entropy over HTML tags and a composite box regression loss:
  $$
  \ell = \lambda\,\ell_{s} + (1-\lambda)\,\ell_{\text{box}}
  $$
  where $\ell_{s}$ is the cross-entropy loss for structure tokens, and
  $$
  \ell_{\text{box}} = \lambda_{\text{iou}} \cdot \ell_{\text{iou}} + \lambda_{l1} \cdot \|b_{\text{pred}}-b_{\text{gt}}\|_1
  $$
  combines generalized-IoU and L1 box regression. An explicit Tree-Edit-Distance Score (TEDS) metric is used:
  $$
  \text{TEDS} = 1 - \frac{\text{EditDist}(T_{\text{pred}},T_{\text{gt}})}{\max(|T_{\text{pred}}|, |T_{\text{gt}}|)}
  $$
  measuring structural similarity at the HTML tag tree level [2203.01017].

- **MonkeyOCR IDTP** adds an auxiliary cross-entropy loss focused on correct placement of image placeholder tokens and, optionally, an alignment loss to render the predicted HTML and compare structural masks:
  $$
  L_{\text{total}} = L_{\text{struct}}(\hat{y},y^*) + \lambda\,L_{\text{img}}(\hat{y},y^*) + \mu\,L_{\text{align}}(I^o,\hat{y})
  $$
  Here, $L_{\text{img}}$ penalizes misplacement of `<img id=j>` tokens, thereby reinforcing correct alignment of image objects within the structure. $L_{\text{align}}$ optionally penalizes discrepancies between rendered structure and the true table boundary map [2511.10390].

Visual-consistency reinforcement learning is further used to optimize global structure+image consistency by maximizing expected reward under a separately trained reward model, leveraging a generalized RPO loop to close the perception-structure gap.

## 4. Data Integration and Preprocessing

Successful operationalization of IDTP depends on the availability of paired structural (HTML) and cell-level or image-level location supervision. TableFormer leverages programmatic PDF sources and datasets (PubTabNet, FinTabNet), where ground-truth (text, cell-box) pairs obviate the need for OCR and provide precise targets for the bounding box decoder. In cases like TableBank with incomplete bounding box annotations, missing bboxes are inferred or the table is pruned if reconstruction is ambiguous, ensuring strict one-to-one correspondence between `<td>` tokens and targets [2203.01017]. This approach enables training the object-detection decoder exclusively on supervised, non-OCR targets.

MonkeyOCR v1.5 extends the paradigm to complex, real-world cases by integrating automatic figure localization (YOLOv10) and placeholder-based mapping of image content to structure. The masking and mapping is purely a data-preparation step; no structural decoder ever receives the original images as inputs, enforcing architectural decoupling [2511.10390].

## 5. Experimental Results and Quantitative Impact

Both TableFormer and MonkeyOCR v1.5 demonstrate substantial quantitative advances attributable to IDTP, specifically in structural fidelity and end-to-end content+structure accuracy.

TableFormer reports:

| Dataset             | EDD (LSTM) TEDS | TableFormer TEDS |
|---------------------|-----------------|------------------|
| PubTabNet Simple    | 91.1%           | 98.5%            |
| PubTabNet Complex   | 88.7%           | 95.0%            |
| FinTabNet Simple    | 88.4%           | 97.5%            |
| FinTabNet Complex   | 92.1%           | 96.0%            |

Further improvements in mean average precision (mAP) for bounding box detection and end-to-end TEDS including cell text are also reported. Notably, integrating TableFormer’s explicit Cell BBox Decoder into EDD yields a 3–4 point mAP improvement, demonstrating the benefit of decoupled object detection for cell localization [2203.01017].

MonkeyOCR v1.5, equipped with IDTP, achieves:

| Benchmark                 | MinerU2.5 TEDS | PaddleOCR-VL TEDS | MonkeyOCR v1.5 TEDS |
|---------------------------|----------------|-------------------|---------------------|
| PubTabNet                 | 89.1%          | 85.2%             | 90.7%               |
| OCRFlux-Complex (figures) | 81.7%          | 81.7%             | 90.9%               |

On OCRFlux-Complex, which contains many embedded figures and heterogeneous cell types, IDTP yields a +9.2 point improvement over baselines. Only MonkeyOCR and PaddleOCR-VL claim full support for embedded-image restoration, with qualitative results showing perfect image re-insertion [2511.10390].

## 6. Integration in Document Understanding Pipelines

IDTP is typically embedded as a modular subroutine within broader document parsing and intelligence pipelines. In MonkeyOCR v1.5, IDTP operates as an intermediate between region detection and vision-language decoding. The workflow is:

1. Layout analysis produces table regions.
2. IDTP detects image cells, masks them, and runs the masked image through the VLM for HTML table structure decoding (including image placeholders).
3. Extracted image patches are re-inserted into the HTML according to the output IDs.
4. Downstream, a Type-Guided Table Merging module merges partial tables split across pages or columns by analyzing header and row continuation.

This modularity ensures IDTP can be adapted or extended for other document intelligence systems capable of region-level detection and structured sequence modeling [2511.10390].

## 7. Limitations and Future Directions

IDTP's reliance on accurate figure localization and precise mapping between image regions and HTML placeholders introduces sensitivity to detection errors—missed or mislocalized images may not be restorable. The architecture presupposes datasets with sufficient annotation and clear mapping between structure and content. A plausible implication is that further gains could be realized by developing end-to-end networks that internally learn both structure-image associations and are robust to incomplete or noisy supervision.

The success of decoupling relies on a modular approach and loss functions that penalize both structure and content misalignment. As the complexity and modality diversity of real-world tables grows (e.g., with formulas, charts, non-rectangular layouts), research into joint vision-language methods and improved reinforcement learning strategies (such as visual consistency reward modeling) may further advance the state-of-the-art.

Source: https://www.emergentmind.com/topics/image-decoupled-table-parsing-idtp