PRIM Dataset: In-Image Multilingual Translation Benchmark

• The paper introduces PRIM as a benchmark for in-image multilingual translation using real-world one-line text images with complex backgrounds.
• PRIM features realistic conditions such as varied fonts, text positions, and multilingual translation directions compared to synthetic datasets.
• The benchmark employs BLEU, COMET, and FID metrics to assess challenges like OCR error propagation and accurate image rendering.

Searching arXiv for the relevant PRIM papers and related context. PRIM is a benchmark for Practical In-Image Multilingual Machine Translation (IIMMT), introduced in “PRIM: Towards Practical In-Image Multilingual Machine Translation” (Tian et al., 5 Sep 2025). It is designed for the task of translating an image containing text in English into a new image in another language, under conditions intended to reflect practical deployment rather than synthetic laboratory settings. The dataset consists of real-world captured one-line text images with complex background, various fonts, and diverse text positions, and supports multilingual translation directions rather than a single bilingual pair (Tian et al., 5 Sep 2025). In the associated paper, PRIM functions primarily as a benchmark test set for evaluating end-to-end IIMMT systems, especially with respect to both translation quality and visual quality.

1. Definition and task formulation

PRIM was introduced to support research on end-to-end in-image translation under realistic conditions (Tian et al., 5 Sep 2025). The task is not ordinary machine translation of extracted strings. Instead, the output is a translated image in which the original text has been replaced by translated text while preserving the plausibility of the background and the visual integrity of the image. The paper frames this as a shift from conventional in-image machine translation toward Practical In-Image Multilingual Machine Translation, emphasizing the simultaneous demands of multilingual translation, text rendering, and background preservation (Tian et al., 5 Sep 2025).

The dataset focuses on one-line text images, which the paper describes as “fundamental practical cases” and also as frequent in applications such as e-commerce (Tian et al., 5 Sep 2025). This focus narrows the visual scope while retaining substantial difficulty, because the images are drawn from real-world settings rather than synthesized with uniform backgrounds or fixed layouts. A plausible implication is that PRIM is intended to isolate a common industrial subproblem—single-line visual text translation—without reducing it to the overly simplified conditions used in earlier datasets.

2. Motivation and relation to earlier IIMT resources

The stated motivation for PRIM is that prior end-to-end IIMT research largely relied on synthetic data with simple background, single font, fixed text position, and often bilingual translation only, which “can not fully reflect real world” (Tian et al., 5 Sep 2025). The paper explicitly contrasts PRIM with several earlier resources: E2E-IIMT, SegPixel, TranslatotronV, UMTIT, and DebackX (Tian et al., 5 Sep 2025). According to the comparison summarized in the paper, earlier datasets were typically missing one or more of four properties: multilingual translation support, real-world backgrounds, font diversity, and text position diversity (Tian et al., 5 Sep 2025).

The paper identifies three practical difficulties that motivate the benchmark design. First, the OCR → MT → rendering cascade suffers from error propagation. Second, rendering translated text often damages the background. Third, real images exhibit variability in text style, layout, and background that synthetic datasets do not capture well (Tian et al., 5 Sep 2025). The authors therefore position PRIM as a benchmark that makes those failure modes measurable.

This design also distinguishes PRIM from work that uses the acronym “PriM” for unrelated purposes. In particular, “PriM: Principle-Inspired Material Discovery through Multi-Agent Collaboration” (Lai et al., 9 Apr 2025) is a framework/system paper for automated materials discovery and explicitly does not introduce or release a standalone public dataset or benchmark called ‘PriM Dataset’. That distinction matters because the names are similar but refer to different research objects.

3. Data sources and annotation pipeline

PRIM is built from real-world captured one-line text images drawn from two sources (Tian et al., 5 Sep 2025). The source images come from:

• Ma et al. (E2ETIT): images from video subtitles, with English text in different fonts, sizes, and positions.
• Li et al. (MIT-10M): text images crawled from websites, many from e-commerce advertising boards.

The authors crop text regions to a fixed size of $32 \times 512$ and state that they primarily select images with non-solid-color backgrounds, thereby increasing realism and difficulty (Tian et al., 5 Sep 2025). The benchmark thus combines standardized image dimensions with heterogeneous visual conditions.

The annotation process is explicitly described as a three-step pipeline (Tian et al., 5 Sep 2025):

1. Manually inpaint the source text region in the real image to obtain the background.
2. Translate the source English text into the target languages.
3. Render the target translation into the inpainted background to create the target image.

This pipeline is central to the dataset’s role as a benchmark. Because the background is manually recovered and the translated text is rendered into that background, PRIM supports evaluation not only of linguistic adequacy but also of the visual realism of generated images. The paper states that this manual target-image annotation enables metrics such as FID to be used by comparing system outputs with reference target images (Tian et al., 5 Sep 2025).

4. Language coverage, scale, and reference structure

PRIM supports five one-to-many translation directions from English (Tian et al., 5 Sep 2025):

Direction	Images	References per source image
En-Ru	340	2
En-Fr	340	2
En-Ro	340	2
En-De	340	2
En-Cs	340	2

According to the appendix summarized in the paper, the dataset contains 340 images per translation direction, for a total test-set size of $5 \times 340 = 1{,}700$ source images (Tian et al., 5 Sep 2025). Each source image is paired with 2 reference target images, produced from two translation sources: GPT-4 and Google Translate (Tian et al., 5 Sep 2025).

The dual-reference design directly affects evaluation. The paper states that BLEU is computed using 2 references, while COMET and FID are averaged across the two references (Tian et al., 5 Sep 2025). This structure is unusual relative to many image-text benchmarks and reflects the fact that the target is not merely a token sequence but a rendered image. A plausible implication is that the benchmark tolerates some variation in acceptable translations while still anchoring evaluation to concrete visual targets.

The authors also report a quality check using reference-free wmt22-cometkiwi-da, comparing PRIM translations to human-annotated MTed dev/test sets (Tian et al., 5 Sep 2025). The reported scores are:

Direction	PRIM-Google	PRIM-GPT4
En-De	0.8271	0.8246
En-Fr	0.8386	0.8359
En-Cs	0.8388	0.8397
En-Ru	0.8310	0.8308
En-Ro	0.8392	0.8459

The paper uses these values to argue that PRIM translation quality is comparable to human-annotated data (Tian et al., 5 Sep 2025). This concerns the textual translation layer of the references rather than the difficulty of the end-to-end image generation task.

5. Evaluation protocol and benchmark role

PRIM is used in the paper as the principal test bed for the proposed VisTrans model (Tian et al., 5 Sep 2025). The benchmark evaluates systems using three metrics:

• BLEU: computed on OCR-recognized output text versus reference text.
• COMET: semantic translation quality.
• FID: visual realism of generated images.

OCR is performed with EasyOCR (Tian et al., 5 Sep 2025). The paper notes that OCR introduces noise, and therefore also evaluates the golden reference outputs to approximate an upper bound (Tian et al., 5 Sep 2025). This is methodologically important: BLEU and COMET are not computed on latent ground-truth strings directly, but on strings recovered from generated images, so OCR quality becomes part of the benchmark pathway.

PRIM is not the paper’s training corpus. Because large-scale real-world parallel image pairs are difficult to collect, the authors construct a synthetic training set separately, using MTed texts, TRDG for source rendering, Arial via PIL for target rendering, backgrounds extracted from video frames, and the same fixed image size of $32 \times 512$ (Tian et al., 5 Sep 2025). The reported synthetic training/validation sizes are:

• En-Ru: 1,629,790 training / 3,404 validation
• En-Fr: 1,594,303 / 3,434
• En-Ro: 1,507,993 / 3,544
• En-De: 1,418,009 / 3,424
• En-Cs: 848,894 / 3,555

This separation between synthetic training data and real-world benchmark evaluation is a key property of PRIM’s experimental role. A common misunderstanding is to treat PRIM as the large-scale training corpus of the paper; the data indicate instead that PRIM is the realistic benchmark set, while training is conducted on a distinct synthetic corpus (Tian et al., 5 Sep 2025).

6. Difficulty profile, empirical findings, and significance

The experiments in the paper portray PRIM as a difficult benchmark. Average performance across the five directions is reported as follows (Tian et al., 5 Sep 2025):

System	BLEU	FID
Golden	66.7	0.0
EasyOCR-NLLB-Render	23.0	100.2
QwenVL-Render	17.1	102.2
AnyTrans	0.1	204.1
PARSeq-mTransformer-Render	9.9	103.8
PEIT-Render	10.4	101.4
TranslatotronV	1.4	69.1
VisTrans	11.3	28.8

The full table in the paper also reports COMET, with Golden = 85.2, EasyOCR-NLLB-Render = 62.7, QwenVL-Render = 58.4, AnyTrans = 31.0, PARSeq-mTransformer-Render = 46.8, PEIT-Render = 48.0, TranslatotronV = 32.2, and VisTrans = 47.0 (Tian et al., 5 Sep 2025). The comparison indicates that cascade systems can obtain better text translation quality than some end-to-end systems while producing weak image realism, whereas end-to-end models can improve visual fidelity but often struggle with translation quality. The paper states that VisTrans improves over TranslatotronV from 1.4 to 11.3 average BLEU and from 69.1 to 28.8 FID (Tian et al., 5 Sep 2025).

The paper interprets these results as evidence that PRIM exposes several practical failure modes: background damage, incomplete rendering of long translated text, and the general difficulty of multilingual rendering under realistic visual conditions (Tian et al., 5 Sep 2025). The case study further notes that even when a translation is correct, rendering may fail if the translated string is too long or the font size is inappropriate. This suggests that PRIM measures a compound competence: OCR-free language transfer, layout adaptation, and image synthesis.

In that sense, PRIM’s significance lies less in raw scale than in its benchmark design. It combines real-world visual complexity with multiple translation directions, manual background recovery, and paired visual references, thereby making it possible to evaluate in-image multilingual translation as an image-generation problem rather than as a pure text-generation task (Tian et al., 5 Sep 2025).