Practical In-Image Multilingual Machine Translation

Updated 10 July 2026

Practical IIMMT is a task that translates embedded text in images while preserving the visual context, layout, and typography across multiple languages.
It employs diverse methodologies including modular pipelines, unified multimodal models, and end-to-end image-to-image systems to address detection, recognition, and rendering challenges.
Evaluation frameworks now combine linguistic metrics like BLEU and COMET with visual quality measures such as Structure-BLEU and FID to ensure translation and layout fidelity.

Practical In-Image Multilingual Machine Translation (IIMMT) denotes the translation of text embedded within images from one language to another while outputting images rather than text alone, under multilingual and real-world conditions. In the recent literature, the task is defined not only by translation accuracy, but also by preservation of visual context, layout, rendering style, background integrity, and spatial placement of the translated text. The field has moved from synthetic, visually simple, bilingual settings toward real-world captured one-line text images with complex background, various fonts, diverse text positions, and multilingual translation directions, as well as toward multi-scenario evaluation spanning documents, scenes, web images, and presentation-like layouts (Tian et al., 5 Sep 2025, Lyu et al., 23 Jun 2026, Li et al., 25 Mar 2026).

1. Definition, scope, and relation to adjacent tasks

In-image machine translation aims to translate images containing text from one language to another. In its more practical form, IIMMT requires the system to preserve the original visual appearance, layout, and style while rewriting the embedded text into a target language. This distinguishes it from image-to-text translation systems that output only translated text, and from OCR→MT pipelines that do not inherently solve rendering fidelity (Lyu et al., 23 Jun 2026, Lyu et al., 11 Mar 2026).

A useful boundary must also be drawn between IIMMT and end-to-end text-image machine translation (TIMT). TIMT, as formulated in MMTIT-Bench, directly translates textual content in images across languages and is positioned as crucial for real-world multilingual scene understanding, but its central benchmark outputs bilingual translations to English and Chinese rather than translated images (Li et al., 25 Mar 2026). By contrast, practical IIMMT explicitly requires an image output in which translated text is rendered back into the original regions, typically with preservation of background and typography (Lyu et al., 23 Jun 2026).

A plausible precursor lies in multilingual image description, where sequence generation models condition on image features and source-language descriptions to generate target-language descriptions. Mansimov, Stern, Chen, Firat, Uszkoreit, and Jain’s multilingual image description work established that visual features and cross-lingual textual features can be complementary, but the task remains description generation rather than embedded-text replacement inside the original image (Elliott et al., 2015).

2. Benchmark resources and the shift toward realism

Practical IIMMT research has been shaped by a sequence of datasets and benchmarks that progressively expose synthetic-to-real gaps. Earlier work emphasized controlled rendering and synthetic images; more recent resources foreground real backgrounds, multilingual coverage, and scenario diversity.

Resource	Core characteristics	Role
PRIM (Tian et al., 5 Sep 2025)	First annotated real-world dataset for IIMMT; real-world captured one-line text images; diverse backgrounds, fonts, text positions; 5 translation directions	Practical end-to-end IIMMT training and evaluation
IIMT30k (Tian et al., 21 May 2025)	Subtitle text with real-world background; images from Multi30k; multiple fonts; complex scenario IIMMT	Benchmarking realistic background and font adaptation
MMTIT-Bench (Li et al., 25 Mar 2026)	1,400 images; 14 non-English and non-Chinese languages; menus, posters, books, products, attractions, documents, receipts, web and street scenes	Human-verified multilingual and multi-scenario TIMT evaluation
IMTBench (Lyu et al., 11 Mar 2026)	2,500 image translation samples; four practical scenarios; nine languages	Cross-modal collaborative evaluation of practical IIMT

PRIM is presented as the first annotated real-world dataset for IIMMT. Its images are captured from real-world scenarios, each image contains single-line text with diverse backgrounds, fonts, and text positions, and it supports five translation directions: English to German, French, Czech, Russian, and Romanian. Each translation direction consists of 340 images, and each source image has two reference images translated by Google and GPT4 (Tian et al., 5 Sep 2025).

IIMT30k was introduced to study subtitle text with real-world background. It uses images from Multi30k, resized to $512 \times 512$ , with German and English captions rendered onto the images using multiple fonts. Its purpose is explicitly to benchmark IIMMT systems under realistic, visually complex conditions and to test font adaptation (Tian et al., 21 May 2025).

MMTIT-Bench extends the evaluation problem in a different direction. It is a human-verified multilingual and multi-scenario benchmark with 1,400 images spanning 14 non-English and non-Chinese languages, with scenario diversity including menus, posters, books, products, attractions, documents, receipts, web and street scenes. Each image includes parallel bilingual translations to English and Chinese, and the construction uses a model-assisted, human-refined pipeline with mutual cross-checks and language-expert reviews (Li et al., 25 Mar 2026).

IMTBench further formalizes practical evaluation. It contains 2,500 image translation samples covering Documents, Web Pages, Natural Scenes, and Presentation Slides, and supports nine languages spanning Latin, Cyrillic, CJK, and Arabic scripts. Its design explicitly targets realistic complexity and cross-modal faithfulness rather than single-modality evaluation alone (Lyu et al., 11 Mar 2026).

Taken together, these resources document a field-level transition: synthetic black-font-on-white-background settings are no longer considered sufficient proxies for deployment conditions. This suggests that “practical” IIMMT now implies multilinguality, real backgrounds, complex layout, and evaluation beyond text-only metrics.

3. System paradigms and architectural decomposition

The literature now contains at least three broad paradigms: modular pipelines, unified multimodal models, and end-to-end image-to-image models. Comparative evaluation on images with text explicitly organizes systems in exactly these terms: modular pipelines that separate text detection, recognition, and translation; multi-modal LLMs capable of processing image and text jointly; and an end-to-end model, Translatotron-V, that directly generates translated images (Puchol et al., 28 May 2026).

Modular pipelines remain important because they allow specialized components to be swapped independently. One representative implementation combines a custom U-Net for text detection, Tesseract OCR for text recognition, and a from-scratch Seq2Seq Transformer for multilingual NMT over English, French, German, Italian, and Russian. The pipeline is explicitly summarized as:

1	[Input Image] → [U-Net: Text Regions Detected/Segmented] → [Crop Regions] → [Tesseract: OCR Text Extraction] → [Transformer: Multilingual Text Translation]

Its NMT component is trained on 2.2 million translation pairs from Tatoeba and OPUS-100, and the complete pipeline is evaluated via text detection accuracy, text recognition quality, and BLEU scores (Sahay et al., 27 Oct 2025).

End-to-end IIMT began from more restrictive prototypes. “Towards End-to-End In-Image Neural Machine Translation” offered a preliminary investigation in which an image containing source-language text is transformed directly into an image containing translated text using only pixel-level supervision. The paper compared a U-net style convolutional baseline with a fuller modular-autoregressive-differentiable pipeline, reporting a development BLEU of 7.7 for the full model versus 0.5 for the convolutional baseline on synthetic German-English data (Mansimov et al., 2020).

Translatotron-V systematized the end-to-end approach by introducing four modules: an image encoder, a target text decoder, an image decoder, and an image tokenizer. The target text decoder alleviates the language alignment burden, and the image tokenizer converts long pixel sequences into shorter sequences of visual tokens. The model is trained with a two-stage framework, and the paper also proposes Structure-BLEU, a location-aware evaluation metric that matches OCR-extracted text regions by IoU before computing BLEU (Lan et al., 2024).

Subsequent architectures increasingly decompose the problem into text and background subproblems. DebackX separates the background and text-image from the source image, performs translation on the text-image directly, and fuses the translated text-image with the background to generate the target image. Its architecture includes a text-image background separation module, an image translation module built around vector quantization and a Title Image Translation auxiliary task, and a text-image background fusion module (Tian et al., 21 May 2025).

VisTrans, proposed for PRIM, also separates foreground text and background information. It uses Vision Transformers for visual feature extraction, a discrete codebook to encode textual regions, and a two-pass sequence modeling design: a 1-Pass Decoder generates target translation at subword level, a Subword-to-Char decoder refines to character level, and a 2-Pass Decoder generates the discrete code sequence reconstructing the output image. The model is trained in two stages and automatically adjusts font size to fit text and image dimensions (Tian et al., 5 Sep 2025).

UniTranslator pushes unification further by coupling translation understanding and visual text editing in a single end-to-end architecture. After multimodal encoding, it contains an Understanding Branch that autoregressively decodes target-language text, a Generation Branch that synthesizes the translated image using conditional generation, and a Mask Branch that predicts a text-region mask for pixel-level supervision. Its two key modules are the Understand-Generation Alignment Module (UGAM), which bridges the representation gap between understanding and generation, and the Spatial Mask Decoder (SMD), which improves spatial grounding, geometric alignment, and layout controllability during generation (Lyu et al., 23 Jun 2026).

4. Reasoning, alignment, and consistency as core design problems

A major recent development is the treatment of IIMMT and TIMT as reasoning-intensive multimodal tasks rather than simple OCR-plus-translation problems. MMTIT-Bench argues that existing thinking paradigms for TIMT are immature: some cascade parsing and translation sequentially, while others focus on language-only Chain-of-Thought and overlook visual cognition central to VLLMs. Its response is Cognition–Perception–Reasoning for Translation (CPR-Trans), a data paradigm that integrates scene cognition, text perception, and translation reasoning within a unified reasoning process (Li et al., 25 Mar 2026).

The CPR-Trans reasoning chain is defined as:

<think>
  <cognition>...</cognition>
  <perception>...</perception>
  <trans>...</trans>
</think>
<answer>...</answer>

Within this structure, scene cognition describes global visual context without relying on text recognition, text perception reasons about spatial layout, reading order, and structure using both image and OCR cues, and translation reasoning synthesizes the translation by justifying choices from the preceding steps. Experiments reported in the benchmark show that all three components contribute, and even inference-time guidance via multi-turn prompting boosts translation quality without further fine-tuning (Li et al., 25 Mar 2026).

Consistency is another organizing concept. HCIIT introduces two kinds of consistency: translation consistency and image generation consistency. In its first stage, a multimodal multilingual LLM translates per-region text using OCR-derived positions and Chain-of-Thought prompting to encourage scene-aware translation. In its second stage, a diffusion model backfills the translated text into the original image with style-consistent text-image generation, preserving background details. For this purpose, the paper curates over 400,000 style-consistent pseudo text-image pairs (Fu et al., 2024).

Alignment between understanding and rendering is also the central issue in UniTranslator. UGAM conditions the generation branch on an aligned condition embedding derived from the understanding branch and the shared multimodal representation, explicitly targeting semantic consistency between translated content prediction and text rendering. SMD injects translation semantics into visual features and predicts a target text-region mask with pixel-level supervision, addressing spatial position misalignment. The paper describes these modules as a response to two bottlenecks: semantic consistency between translation and visual rendering, and precise spatial placement with layout preservation (Lyu et al., 23 Jun 2026).

These design trends indicate that practical IIMMT is no longer modeled purely as string transduction plus inpainting. Instead, current systems treat scene cognition, translation, region localization, typography, and rendering coherence as coupled inference problems.

5. Evaluation protocols and reported empirical findings

Evaluation protocols have diversified because text-only scores are insufficient for image-output systems. MMTIT-Bench evaluates with VLLM-based human-aligned scores—Faithfulness, Fluency, Readability, and Terminology—alongside COMET, in both zero-shot and fine-tuned settings. PRIM evaluates translation quality with BLEU and COMET obtained after EasyOCR extraction from generated images, and image quality with Fréchet Inception Distance (FID). Translatotron-V adds Structure-BLEU to incorporate region location. IMTBench uses translation quality, background faithfulness, visual rendering quality, and a cross-modal alignment score, combined through the unified score

$S = \frac{1}{4}(S_{\text{text}} + S_{\text{bg}} + S_{\text{align}} + S_{\text{vis}})$

to normalize comparison across systems and scenarios (Li et al., 25 Mar 2026, Tian et al., 5 Sep 2025, Lan et al., 2024, Lyu et al., 11 Mar 2026).

On PRIM, the reported average test-set results are: Golden 66.7 BLEU, 85.2 COMET, 0.0 FID; EasyOCR-NLLB-Render 23.0 BLEU, 62.7 COMET, 100.2 FID; QwenVL-Render 17.1 BLEU, 58.4 COMET, 102.2 FID; AnyTrans 0.1 BLEU, 31.0 COMET, 204.1 FID; PARSeq-mTrans-Render 9.9 BLEU, 46.8 COMET, 103.8 FID; PEIT-Render 10.4 BLEU, 48.0 COMET, 101.4 FID; TranslatotronV 1.4 BLEU, 32.2 COMET, 69.1 FID; and VisTrans 11.3 BLEU, 47.0 COMET, 28.8 FID. The paper interprets these results as showing that VisTrans exceeds other end-to-end baselines and achieves the lowest FID, while strong cascades retain higher BLEU because they use explicit OCR and NMT (Tian et al., 5 Sep 2025).

Translatotron-V reports a different comparison on synthetic De↔En settings: OCR+MT+T2I obtains 15.4 BLEU, 14.9 Structure-BLEU, 0.778 SSIM, and 247M parameters; Translatotron-V obtains 15.4 BLEU, 15.3 Structure-BLEU, 0.783 SSIM, and 175M parameters. The paper emphasizes competitive performance with only 70.9% of the parameters of the strongest cascaded model, and a large gap over the earlier pixel-level end-to-end baseline (Lan et al., 2024).

UniTranslator reports state-of-the-art performance across multiple benchmarks. In the representative summary table, on Translatotron-V De→En test the best prior score is 15.39 BLEU and UniTranslator reports 25.03; on IIMT30k De→En test the best prior is 12.8 BLEU and 9.0 FID for DebackX, while UniTranslator reports 14.7 and 8.9; on PRIM average results the best prior is 11.3 and 28.8 for VisTrans, while UniTranslator reports 12.8 and 22.9 (Lyu et al., 23 Jun 2026).

HCIIT reports gains from image-aware reasoning on ambiguous translation. On the CoMMuTE test set, En-De improves from 16.04 to 17.31 BLEU and from 72.56 to 73.84 COMET; En-Fr improves from 19.06 to 20.18 BLEU and from 72.80 to 75.23 COMET; En-Zh improves from 18.88 to 19.65 BLEU and from 80.51 to 81.56 COMET. For image generation consistency on synthetic En-Zh, HCIIT reports 0.744 SSIM and 0.402 L1, compared with AliyunTrans at 0.605 and 0.436, YoudaoTrans at 0.516 and 0.561, and Anytrans at 0.381 and 0.561 (Fu et al., 2024).

Comparative evaluation across system families produces yet another picture. On English–German image-to-text translation, M2M100-1.2B reaches 27.7 BLEU, 53.3 chrF, and 62.3 TER; EuroLLM-9B reaches 31.2 BLEU, 56.4 chrF, and 58.4 TER; Gemini-2.5-lite reaches 32.1 BLEU, 57.3 chrF, and 58.0 TER; Gemini-2.5-pro reaches 35.6 BLEU, 59.9 chrF, and 52.5 TER; Translatotron-V, available only for some pairs, reports 17.9 BLEU. The study concludes that modular pipelines outperform the evaluated end-to-end approach, while MLLMs achieve the best overall performance (Puchol et al., 28 May 2026).

IMTBench highlights strong scenario dependence. In the Document scenario, Tencent reports $S_{\text{text}} = 63.1$ , $S_{\text{bg}} = 88.2$ , $S_{\text{align}} = 90.1$ , $S_{\text{vis}} = 75.5$ , and $S_{\text{avg}} = 80.2$ ; GPT-Image reports 61.0, 57.8, 17.4, 72.1, and 57.8; Qwen-Image reports 62.6, 48.8, 6.7, 75.6, and 52.7. The benchmark observes that cascaded pipelines remain best at structure and alignment in controlled layouts, proprietary UMMs offer strong translation quality and visual naturalness, and open-source UMMs lag markedly in alignment (Lyu et al., 11 Mar 2026).

6. Practical utility, unresolved limitations, and current research tensions

Practical utility is now described along three axes: evaluation, model design, and deployment. MMTIT-Bench provides a benchmark for realistic multilingual assessment and a blueprint for supervised data and prompts that foster multimodal reasoning; the paper argues that CPR-Trans-style supervision yields more robust handling of visual distractions, complex layouts, underrepresented languages, and more interpretable reasoning traces. PRIM and DebackX explicitly target translation apps, AR, video subtitles, and e-commerce; HCIIT situates the task in film poster translation and everyday scene image translation; IMTBench extends the scope to documents, web pages, scenes, and slides (Li et al., 25 Mar 2026, Tian et al., 5 Sep 2025, Fu et al., 2024, Lyu et al., 11 Mar 2026).

The literature also converges on several unresolved limitations. PRIM remains limited to single-line text, and the paper identifies multi-line and highly complex document layouts as still challenging. DebackX emphasizes the need to generalize from real-world background scenarios while still relying on synthetic pretraining for scale. MMTIT-Bench states that effective reasoning-oriented paradigms for TIMT are still nascent, and that training/annotation complexity, noise in uncontrolled reasoning, and extension to video and document-level translation remain open problems. IMTBench reports that Arabic, Russian, and Japanese remain difficult across systems, with gaps due largely to rendering and typography rather than input recognition (Tian et al., 5 Sep 2025, Tian et al., 21 May 2025, Li et al., 25 Mar 2026, Lyu et al., 11 Mar 2026).

A common misconception is that one paradigm already dominates all practical conditions. The benchmark evidence is more heterogeneous. On synthetic or text-output evaluations, MLLMs can outperform modular pipelines and clearly exceed the evaluated end-to-end image generators (Puchol et al., 28 May 2026). On structured visual rewriting benchmarks, cascaded commercial systems can retain the best alignment and background scores (Lyu et al., 11 Mar 2026). On some end-to-end image-output benchmarks, however, unified architectures such as UniTranslator report state-of-the-art results over prior open-source and specialist IIMMT systems (Lyu et al., 23 Jun 2026). This suggests that practical IIMMT is not yet a solved architectural problem; rather, performance depends on whether the primary requirement is text translation quality, layout-faithful rendering, cross-modal consistency, background preservation, multilingual generalization, or interpretability.

The present research trajectory therefore centers on broader multilingual coverage, stronger controllable editing, better structure-aware evaluation, and tighter coupling between scene understanding and text rendering. Within that trajectory, practical IIMMT has become a distinct multimodal systems problem: translating the text in an image is no longer sufficient unless the translated image remains faithful to the original scene, typography, and spatial organization.