---
title: ICDAR 2025 DIMT Challenge
url: https://www.emergentmind.com/topics/icdar-2025-dimt-challenge
type: topic
---

# ICDAR 2025 DIMT Challenge

The ICDAR 2025 DIMT Challenge, held as part of the 19th International Conference on Document Analysis and Recognition (ICDAR 2025), targets the task of end-to-end document image machine translation (DIMT) for documents with complex and diverse layouts. Huawei Translation Service Center (HW-TSC) presented a unified end-to-end solution that leverages a large vision-language model (LVLM), addressing both OCR-based and OCR-free DIMT tasks through a multi-task learning and perceptual chain-of-thought framework. The system architecture, training methodology, inference process, datasets, and results collectively represent an advanced approach to robust document image translation in real-world settings [2504.17315].

## 1. Unified System Architecture

The DIMT25@ICDAR2025 system is built around a single LVLM architecture, supporting both OCR-based and OCR-free translation pipelines. Both branches share the same multimodal backbone and diverge only at the text acquisition stage:

- **OCR-based path**: The input image undergoes OCR detection and recognition, producing extracted text and detected layout regions, which are then processed by a text-guided encoder and the LVLM decoder to generate translations.
- **OCR-free path**: The input image is processed directly by a visual encoder, which fuses vision features with query tokens via cross-modal attention. The LVLM decoder subsequently generates internal text representations and the final translation.

Both modalities employ the InternVL2.5-MPO (multi-purpose optimizer) transformer as their shared encoder–decoder core. Training occurs under a unified multi-task learning plus perceptual chain-of-thought (PCOT) regime, enabling the system to seamlessly fall back on direct image-to-translation recognition when OCR fails.

## 2. LVLM Base Model and Architectural Details

The underlying backbone is the InternVL2.5-MPO model [Wang et al. 2024], consisting of:

- **Vision Transformer encoder $E_v$**: Maps image patches to a sequence of embeddings $\mathbf{V} = E_v(\mathrm{Img}) \in \mathbb{R}^{N\times d}$.
- **Causal Transformer decoder $D_t$**: At each step $t$, attends to its previous outputs and the vision embeddings $\mathbf{V}$.

Decoder layers contain:
- **Self-Attention (SA)** operating on text tokens:
  $$
  \mathrm{SA}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V
  $$
  with $Q, K, V$ linearly projected from the previous layer’s output.
- **Cross-Modal Attention (CMA)** fusing vision and text modalities:
  $$
  \mathrm{CMA}(Q_t, K_v, V_v) = \mathrm{softmax}\left(\frac{Q_t K_v^\top}{\sqrt{d}}\right)V_v
  $$
  where $Q_t$ derives from text, $K_v$ and $V_v$ from vision outputs.

Modifications are limited to re-initializing the output linear layer for the translation vocabulary and extending positional encodings to support sequences up to 8,192 tokens.

## 3. Multi-Task and Hierarchical Learning Framework

The system adopts a multi-task learning approach integrating three principal objectives:

- **OCR/text recognition**: Sequence modeling for text extracted from the image.
- **Layout understanding**: Region classification and segmentation for document structure.
- **Machine Translation (MT)**: Generation of target language output.

Each task is supervised with a cross-entropy loss, and the overall objective is a weighted sum:
$$
\mathcal{L} = \lambda_1\,\mathcal{L}_{OCR} + \lambda_2\,\mathcal{L}_{layout} + \lambda_3\,\mathcal{L}_{MT}
$$
with $\lambda_i$ tuned from $\{0.5, 1.0, 2.0\}$.

A perceptual chain-of-thought (PCOT) protocol enforces a two-stage inference process:
- **Stage I (Perception)**: Intermediate token sequences $h^{(1)}$ are generated for recognized text under the guidance of layout and OCR modules.
- **Stage II (Translation Reasoning)**: The decoder, conditioned on $h^{(1)}$ and visual context, produces the final translated sequence $h^{(2)}$.

A consistency regularizer ensures smooth transition between stages:
$$
\mathcal{L}_{cons} = \sum_{t=1}^{T} \| h_t^{(2)} - f(h_t^{(1)}) \|_2^2
$$
where $f$ is a shallow linear carry-over projector. The total loss is:
$$
\mathcal{L}_{total} = \mathcal{L} + \alpha\,\mathcal{L}_{cons}
$$

## 4. Inference, Decoding, and Post-Processing

Inference leverages minimum Bayes-risk (MBR) decoding. For each input, the model samples a candidate set $\mathcal{C}$ containing one deterministic (beam search) and ten stochastic outputs (temperature $t=0.7$, nucleus $p=0.95$). The final output $\hat{y}$ is:
$$
\hat{y} = \arg\max_{y\in\mathcal{C}} \sum_{y'\in\mathcal{C}} \mathrm{Sim}_{BLEU}(y, y')
$$
which minimizes the risk under BLEU similarity.

Post-processing steps include:
- Truncation of repeated special symbols to a maximum length of 10.
- Removal of overly complex table transcriptions.
- For Chinese output, application of Jieba tokenization and collapse of redundant spaces.

## 5. Dataset Composition and Training Procedure

The system is trained on the official DIMT25 datasets:

| Track | Dataset      | Train   | Valid/Test |
|-------|-------------|---------|------------|
| 1     | DIMT-WebDoc | 300 K   | 1 K / 1 K  |
| 2     | DIMT-arXiv  | 124 K   | 1 K / 1 K  |

Training images are preprocessed to $224 \times 224$ resolution and normalized per ViT standards; no synthetic augmentations are introduced. Hardware and hyperparameters:

- Models: InternVL2.5-MPO at 1B and 8B parameter scales
- Fine-tuning: 5 epochs on 8 × A100 GPUs using DeepSpeed ZeRO-3 offload
- Effective batch: 128
- Learning rate: $4 \times 10^{-5}$ with one-epoch linear warmup and cosine decay
- Maximum sequence: 8,192 tokens

## 6. Experimental Results and Analysis

Performance is reported via BLEU scores:

| Model                            | Test-OCR | Test-MT  |
|-----------------------------------|----------|----------|
| InternVL2.5-1B SFT               | –        | 67.21    |
| + MTL-PCOT SFT                   | 94.63    | 62.16    |
|   + MBR                          | 96.50    | 64.08    |
|   + Post-processing              | 97.00    | 66.16    |
| InternVL2.5-8B SFT               | –        | 72.74    |
| + MTL-PCOT SFT                   | 94.89    | 65.32    |
|   + MBR                          | 97.16    | 68.26    |
|   + Post-processing              | 97.66    | 70.48    |

Multi-task learning with PCOT improves over single-task fine-tuning by 3–4 BLEU points; MBR and post-processing add a further 2–5 BLEU. The 8B-parameter model consistently outperforms 1B. Error analysis indicates that symbol over-translation and table cell merging are primary failure modes, addressed by truncation and skipping of complex tables, respectively.

A representative case demonstrates that SFT-only models produce excessive repetition for lines (e.g., “——————”), while the final MTL-PCOT solution correctly truncates, yielding accurate paragraph alignment.

## 7. Advances, Limitations, and Prospects

The system demonstrates that a single LVLM backbone, when trained under a unified multi-task and perceptual hierarchical regime and decoded with MBR and targeted post-processing, can handle both OCR-based and OCR-free machine translation tasks for documents with complex layouts.

Identified limitations include incomplete handling of intricate table structures and room for better cross-modal alignment. Future enhancements are directed toward integrating learned table-structure parsing, employing contrastive layout pre-training, and extending chain-of-thought modules with explicit saliency modeling to address challenges in highly cluttered document images [2504.17315].

Source: https://www.emergentmind.com/topics/icdar-2025-dimt-challenge