VisTrans: In-Image Multilingual Translation

• VisTrans is a model for in-image multilingual translation that processes both visual text and background separately to maintain image integrity.
• It leverages the PRIM dataset, featuring real-world text images with diverse fonts, backgrounds, and five translation directions to enhance realism.
• The architecture uses a two-stage training strategy with a semi-autoregressive decoder, improving both translation quality and visual fidelity over cascade systems.

Searching arXiv for papers relevant to “VisTrans” to ground the article and resolve naming ambiguity. VisTrans is an end-to-end model introduced for Practical In-Image Multilingual Machine Translation (IIMMT), a setting in which an input image containing source-language text is translated into an output image containing target-language text while preserving the visual background. It was proposed together with the PRIM dataset, which emphasizes real-world captured one-line text images with complex background, various fonts, diverse text positions, and multilingual translation directions. In the recent literature, the near-homonymous term VisualTrans denotes a benchmark for Visual Transformation Reasoning rather than in-image translation; the two names are adjacent but refer to different problem formulations (Tian et al., 5 Sep 2025, Ji et al., 6 Aug 2025).

1. Terminology and task delimitation

VisTrans belongs to the literature on In-Image Machine Translation, where both input and output are images rather than plain text. Its immediate problem setting is Practical IIMMT: translating real images with embedded text while maintaining the integrity of the original image. This differs fundamentally from visual transformation reasoning benchmarks, which study how a scene changes from an initial image to a transformed image, often through paired-image inference or before/after reasoning.

A common source of confusion is the coexistence of VisTrans and VisualTrans in recent arXiv usage. VisualTrans is described as “the first comprehensive benchmark specifically designed for VTR in real-world human-object interaction scenarios,” with 472 high-quality question-answer pairs across 12 manipulation tasks and six subtasks grouped into spatial, procedural, and quantitative reasoning dimensions (Ji et al., 6 Aug 2025). Earlier transformation-driven visual reasoning work formalized the inference target as an atomic triplet $t=(o,a,v)$ or a sequence of such triplets $T=\{t_1,t_2,\dots,t_n\}$, and introduced TRANCE with Basic, Event, and View settings (Hong et al., 2020). By contrast, VisTrans addresses multilingual text translation inside images rather than transformation inference over scene states.

2. Problem setting and the PRIM dataset

The motivating claim behind Practical IIMMT is that prior end-to-end IIMT research mainly used synthetic data with simple background, single font, fixed text position, and bilingual translation, which does not fully reflect real-world usage. Practical IIMMT therefore emphasizes real-world captured images and multilingual translation. In this setting, source images contain one-line text, and the system must generate target images whose textual content is translated while the background remains visually coherent (Tian et al., 5 Sep 2025).

PRIM is the benchmark accompanying VisTrans. It contains real-world source images with one-line text, and each source image has a source image, source text, a manually constructed background image, target translated text, and target translated image. It supports five one-to-many translation directions: En $\rightarrow$ Ru, En $\rightarrow$ Fr, En $\rightarrow$ Ro, En $\rightarrow$ De, and En $\rightarrow$ Cs. Each direction contains 340 images, yielding $5 \times 340 = 1700$ source images, and each source image has two reference target images. The source images come from METIMT / E2E-TIT-related source images from video subtitles and MIT-10M / web-crawled textual images, especially e-commerce/ad boards. The construction procedure crops text regions from real-world source images to $32 \times 512$, manually inpaints the text region to create background-only images, translates the source English text into the five target languages using GPT-4 and Google Translate, and renders target text into the background to create the target image (Tian et al., 5 Sep 2025).

The dataset design is significant because it moves away from fully synthetic overlays toward real backgrounds, varied fonts and font sizes, diverse text positions, and manually annotated target images. The paper also reports that translation quality of PRIM references is evaluated using wmt22-cometkiwi-da and is about the level of human-annotated MTed data. A plausible implication is that PRIM is intended to function both as a translation benchmark and as a visual fidelity benchmark, since the target output is itself an image rather than a text string (Tian et al., 5 Sep 2025).

3. Architecture and decoding pipeline

VisTrans is defined by the explicit separation of visual text modeling from background modeling. Given a source image

$$I_{\text{src}} \in \mathbb{R}^{H \times W \times C},$$

the model uses two parallel pathways. The background pathway applies ViT, BackEncoder, BackDecoder1, and BackDecoder2, producing a background representation

$T=\{t_1,t_2,\dots,t_n\}$0

where $T=\{t_1,t_2,\dots,t_n\}$1 is patch size and $T=\{t_1,t_2,\dots,t_n\}$2 is hidden size. The translation / visual-text pathway applies another ViT and an MT Encoder, producing

$T=\{t_1,t_2,\dots,t_n\}$3

This architectural split is the central mechanism by which the model attempts to preserve background integrity while still supporting multilingual translation (Tian et al., 5 Sep 2025).

The textual decoding stack is staged. The 1-Pass Decoder is a multilingual text decoder that generates the target-language sentence autoregressively from shifted target text embeddings, a language tag prefix such as $T=\{t_1,t_2,\dots,t_n\}$4 or $T=\{t_1,t_2,\dots,t_n\}$5, and $T=\{t_1,t_2,\dots,t_n\}$6. It outputs the target subword sequence and hidden states

$T=\{t_1,t_2,\dots,t_n\}$7

where $T=\{t_1,t_2,\dots,t_n\}$8 is the subword length. VisTrans then introduces an S2C Decoder that converts subword-level hidden states into character-level hidden states

$T=\{t_1,t_2,\dots,t_n\}$9

where $\rightarrow$0 is the character length. The S2C Decoder is implemented as a Semi-Autoregressive Transformer, and the paper uses $\rightarrow$1 as the best trade-off. Its relaxed causal mask is

$\rightarrow$2

The motivation is that the image generation codebook operates at a much finer granularity than subwords, so a subword representation is too coarse to align well with visual codes (Tian et al., 5 Sep 2025).

The 2-Pass Decoder then autoregressively generates a sequence of discrete visual codes from the character-level representation. These codes are looked up in a codebook, producing $\rightarrow$3, which is combined with $\rightarrow$4 and passed to ImgDecoder and transposed convolutions to generate the final target image. The codebook quantization step is written as

$\rightarrow$5

where $\rightarrow$6 is the learnable codebook. The design claim is that visual text comes from the translated code sequence and background integrity comes from the background branch. The reported advantages over cascade render-based systems are no hard text removal, no manual rendering after translation, less background damage, and better handling of font size / layout consistency (Tian et al., 5 Sep 2025).

4. Training strategy and optimization objectives

VisTrans uses a two-stage training strategy with multi-task learning and auxiliary branches. The image reconstruction loss is

$\rightarrow$7

with

$\rightarrow$8

The VQ loss is

$\rightarrow$9

where $\rightarrow$0 is the quantized code vector, $\rightarrow$1 is the encoder output, and $\rightarrow$2 is stop-gradient. Cross-entropy loss is used for target text generation, code sequence generation, OCR auxiliary task, and translation auxiliary task, with label smoothing of 0.1 (Tian et al., 5 Sep 2025).

Stage 1 is described as vision training and contains two branches. Branch 1 is the visual text branch, which learns image-text representation and codebook quantization. Its loss is

$\rightarrow$3

Branch 2 is the background plus auxiliary translation branch, which learns background reconstruction, multilingual target text generation, and OCR. Its loss is

$\rightarrow$4

The total Stage 1 loss is

$\rightarrow$5

Stage 2 is translation training, in which the 1-Pass Decoder and MT Encoder are reused, the S2C Decoder produces character-level features, and the 2-Pass Decoder predicts code sequences. The Stage 2 loss is

$\rightarrow$6

This training decomposition suggests a deliberate separation between learning to reconstruct image appearance and learning to infer code sequences from multilingual text-side representations (Tian et al., 5 Sep 2025).

5. Evaluation protocol and empirical results

Evaluation is performed on PRIM. For translation quality, generated images are first recognized by EasyOCR and then compared to references using BLEU and COMET. For visual quality, the metric is FID between generated images and reference images. The baselines include pre-trained cascade systems such as EasyOCR-NLLB-Render, QwenVL-Render, and AnyTrans; cascade baselines such as PARSeq-mTransformer-Render and PEIT-Render; the end-to-end baseline TranslatotronV; and the proposed VisTrans. A “Golden” score is also reported by running OCR on the ground-truth images, giving an approximate upper bound because OCR noise affects evaluation (Tian et al., 5 Sep 2025).

The main numerical findings separate translation quality from image fidelity. EasyOCR-NLLB-Render gives the best BLEU/COMET among the baselines overall, due to strong NLLB translation. Among non-pretrained or structured methods, VisTrans and PEIT-Render are strongest. The reported average scores are 11.3 BLEU / 47.0 COMET for VisTrans, 10.4 BLEU / 48.0 COMET for PEIT-Render, and 1.4 BLEU / 32.2 COMET for TranslatotronV. In visual quality, VisTrans reaches 28.8 FID, compared with 69.1 for TranslatotronV, around 100+ for cascade baselines, and 0.0 for Golden. The paper interprets this as evidence that cascade systems can translate text well but degrade visual quality because text removal and rendering damage the image, whereas end-to-end methods preserve image structure better. This suggests that VisTrans improves on TranslatotronV specifically by separating background and text representations (Tian et al., 5 Sep 2025).

The S2C Decoder is a central ablation point. The reported average BLEU / speedup values are: None, 5.27 / $\rightarrow$7; CTC, 6.06 / $\rightarrow$8; AT, 11.87 / $\rightarrow$9; SAT ($\rightarrow$0), 11.32 / $\rightarrow$1; SAT ($\rightarrow$2), 8.97 / $\rightarrow$3; and SAT ($\rightarrow$4), 7.16 / $\rightarrow$5. The paper’s stated takeaway is that removing S2C hurts badly, AT gives the best BLEU but is slow, and SAT with $\rightarrow$6 is the best trade-off. Although PRIM is a one-line benchmark, a robustness study on IIMT30k reports that VisTrans outperforms DebackX on multi-line text: De-En valid/test 14.7 / 12.3 versus 10.8 / 8.6, and En-De valid/test 16.5 / 12.2 versus 9.5 / 6.9. The limitations noted by the paper are that training is computationally expensive, the model depends on large-scale training data, it uses a relatively basic codebook / decoder design, and it was mainly tested on one-line text images (Tian et al., 5 Sep 2025).

6. Relation to transformation reasoning and broader significance

Despite the similar name, VisTrans is not a benchmark for visual transformation reasoning. That neighboring literature studies paired-image reasoning problems of the form

$\rightarrow$7

where $\rightarrow$8 is an initial scene image, $\rightarrow$9 is a transformed scene image, $\rightarrow$0 is a task-specific reasoning question, and $\rightarrow$1 is the answer. VisualTrans evaluates three reasoning dimensions—spatial, procedural, and quantitative—through six subtasks, and zero-shot evaluations of 17 mainstream VLMs show that strong performance on static spatial tasks does not transfer to dynamic, multi-step reasoning scenarios. The best overall score reported there is 59.96% for o3, followed by 54.93% for Gemini 2.5 Pro, while the best open-source model, InternVL3-78B, reaches 35.01% (Ji et al., 6 Aug 2025).

The conceptual overlap between VisTrans and VisualTrans is therefore limited to the word “transformation.” VisualTrans concerns real-world human-object interaction and before/after reasoning; earlier TVR work likewise framed transformation inference as the recovery of $\rightarrow$2 or sequences of such triplets and showed that models perform well on Basic but are far from human-level intelligence on Event and View (Hong et al., 2020). VisTrans instead concerns practical image-conditioned multilingual generation in which visual text and background information are processed separately. A common misconception is that these names identify variants of a single framework; the literature instead indicates two distinct research programs, one centered on in-image multilingual translation and the other on transformation reasoning over scene change.

Within its own domain, VisTrans is significant because it shifts evaluation away from synthetic text overlays and toward real-world captured images, multilingual directions, and explicit measurement of both translation quality and visual quality. The broader implication is that end-to-end systems for image translation must be judged not only by OCR-recovered BLEU or COMET, but also by whether they maintain background integrity, handle font size and layout, and avoid the brittle OCR $\rightarrow$3 MT $\rightarrow$4 rendering error cascade that characterizes pipeline baselines (Tian et al., 5 Sep 2025).