---
title: 'PRIM Dataset: In-Image Multilingual Translation Benchmark'
url: https://www.emergentmind.com/topics/prim-dataset
type: topic
---

# PRIM Dataset: In-Image Multilingual Translation Benchmark

Searching arXiv for the relevant PRIM papers and related context.
PRIM is a benchmark for **Practical In-Image Multilingual Machine Translation (IIMMT)**, introduced in “PRIM: Towards Practical In-Image Multilingual Machine Translation” [2509.05146]. It is designed for the task of translating an image containing text in English into a new image in another language, under conditions intended to reflect practical deployment rather than synthetic laboratory settings. The dataset consists of **real-world captured one-line text images** with **complex background**, **various fonts**, and **diverse text positions**, and supports **multilingual translation directions** rather than a single bilingual pair [2509.05146]. In the associated paper, PRIM functions primarily as a **benchmark test set** for evaluating end-to-end IIMMT systems, especially with respect to both **translation quality** and **visual quality**.

## 1. Definition and task formulation

PRIM was introduced to support research on **end-to-end in-image translation under realistic conditions** [2509.05146]. The task is not ordinary machine translation of extracted strings. Instead, the output is a **translated image** in which the original text has been replaced by translated text while preserving the plausibility of the background and the visual integrity of the image. The paper frames this as a shift from conventional in-image machine translation toward **Practical In-Image Multilingual Machine Translation**, emphasizing the simultaneous demands of multilingual translation, text rendering, and background preservation [2509.05146].

The dataset focuses on **one-line text images**, which the paper describes as “fundamental practical cases” and also as frequent in applications such as e-commerce [2509.05146]. This focus narrows the visual scope while retaining substantial difficulty, because the images are drawn from real-world settings rather than synthesized with uniform backgrounds or fixed layouts. A plausible implication is that PRIM is intended to isolate a common industrial subproblem—single-line visual text translation—without reducing it to the overly simplified conditions used in earlier datasets.

## 2. Motivation and relation to earlier IIMT resources

The stated motivation for PRIM is that prior end-to-end IIMT research largely relied on **synthetic data** with **simple background**, **single font**, **fixed text position**, and often **bilingual translation** only, which “can not fully reflect real world” [2509.05146]. The paper explicitly contrasts PRIM with several earlier resources: **E2E-IIMT**, **SegPixel**, **TranslatotronV**, **UMTIT**, and **DebackX** [2509.05146]. According to the comparison summarized in the paper, earlier datasets were typically missing one or more of four properties: **multilingual translation support**, **real-world backgrounds**, **font diversity**, and **text position diversity** [2509.05146].

The paper identifies three practical difficulties that motivate the benchmark design. First, the OCR → MT → rendering cascade suffers from **error propagation**. Second, rendering translated text often **damages the background**. Third, real images exhibit variability in **text style, layout, and background** that synthetic datasets do not capture well [2509.05146]. The authors therefore position PRIM as a benchmark that makes those failure modes measurable.

This design also distinguishes PRIM from work that uses the acronym “PriM” for unrelated purposes. In particular, “PriM: Principle-Inspired Material Discovery through Multi-Agent Collaboration” [2504.08810] is a **framework/system paper** for automated materials discovery and explicitly **does not introduce or release a standalone public dataset or benchmark called ‘PriM Dataset’**. That distinction matters because the names are similar but refer to different research objects.

## 3. Data sources and annotation pipeline

PRIM is built from **real-world captured one-line text images** drawn from two sources [2509.05146]. The source images come from:

- **Ma et al. (E2ETIT)**: images from **video subtitles**, with English text in different fonts, sizes, and positions.
- **Li et al. (MIT-10M)**: text images crawled from websites, many from **e-commerce advertising boards**.

The authors crop text regions to a fixed size of **\(32 \times 512\)** and state that they **primarily select images with non-solid-color backgrounds**, thereby increasing realism and difficulty [2509.05146]. The benchmark thus combines standardized image dimensions with heterogeneous visual conditions.

The annotation process is explicitly described as a three-step pipeline [2509.05146]:

1. **Manually inpaint the source text region** in the real image to obtain the background.
2. **Translate the source English text** into the target languages.
3. **Render the target translation** into the inpainted background to create the target image.

This pipeline is central to the dataset’s role as a benchmark. Because the background is manually recovered and the translated text is rendered into that background, PRIM supports evaluation not only of linguistic adequacy but also of the visual realism of generated images. The paper states that this manual target-image annotation enables metrics such as **FID** to be used by comparing system outputs with reference target images [2509.05146].

## 4. Language coverage, scale, and reference structure

PRIM supports **five one-to-many translation directions** from English [2509.05146]:

| Direction | Images | References per source image |
|---|---:|---:|
| En-Ru | 340 | 2 |
| En-Fr | 340 | 2 |
| En-Ro | 340 | 2 |
| En-De | 340 | 2 |
| En-Cs | 340 | 2 |

According to the appendix summarized in the paper, the dataset contains **340 images per translation direction**, for a total test-set size of **\(5 \times 340 = 1{,}700\)** source images [2509.05146]. Each source image is paired with **2 reference target images**, produced from two translation sources: **GPT-4** and **Google Translate** [2509.05146].

The dual-reference design directly affects evaluation. The paper states that **BLEU** is computed using **2 references**, while **COMET** and **FID** are **averaged across the two references** [2509.05146]. This structure is unusual relative to many image-text benchmarks and reflects the fact that the target is not merely a token sequence but a rendered image. A plausible implication is that the benchmark tolerates some variation in acceptable translations while still anchoring evaluation to concrete visual targets.

The authors also report a quality check using **reference-free wmt22-cometkiwi-da**, comparing PRIM translations to human-annotated **MTed** dev/test sets [2509.05146]. The reported scores are:

| Direction | PRIM-Google | PRIM-GPT4 |
|---|---:|---:|
| En-De | 0.8271 | 0.8246 |
| En-Fr | 0.8386 | 0.8359 |
| En-Cs | 0.8388 | 0.8397 |
| En-Ru | 0.8310 | 0.8308 |
| En-Ro | 0.8392 | 0.8459 |

The paper uses these values to argue that PRIM translation quality is **comparable to human-annotated data** [2509.05146]. This concerns the textual translation layer of the references rather than the difficulty of the end-to-end image generation task.

## 5. Evaluation protocol and benchmark role

PRIM is used in the paper as the principal **test bed** for the proposed **VisTrans** model [2509.05146]. The benchmark evaluates systems using three metrics:

- **BLEU**: computed on OCR-recognized output text versus reference text.
- **COMET**: semantic translation quality.
- **FID**: visual realism of generated images.

OCR is performed with **EasyOCR** [2509.05146]. The paper notes that OCR introduces noise, and therefore also evaluates the **golden reference outputs** to approximate an upper bound [2509.05146]. This is methodologically important: BLEU and COMET are not computed on latent ground-truth strings directly, but on strings recovered from generated images, so OCR quality becomes part of the benchmark pathway.

PRIM is not the paper’s training corpus. Because large-scale real-world parallel image pairs are difficult to collect, the authors construct a **synthetic training set** separately, using **MTed** texts, **TRDG** for source rendering, **Arial** via PIL for target rendering, backgrounds extracted from video frames, and the same fixed image size of **\(32 \times 512\)** [2509.05146]. The reported synthetic training/validation sizes are:

- **En-Ru:** 1,629,790 training / 3,404 validation  
- **En-Fr:** 1,594,303 / 3,434  
- **En-Ro:** 1,507,993 / 3,544  
- **En-De:** 1,418,009 / 3,424  
- **En-Cs:** 848,894 / 3,555  

This separation between **synthetic training data** and **real-world benchmark evaluation** is a key property of PRIM’s experimental role. A common misunderstanding is to treat PRIM as the large-scale training corpus of the paper; the data indicate instead that PRIM is the realistic benchmark set, while training is conducted on a distinct synthetic corpus [2509.05146].

## 6. Difficulty profile, empirical findings, and significance

The experiments in the paper portray PRIM as a difficult benchmark. Average performance across the five directions is reported as follows [2509.05146]:

| System | BLEU | FID |
|---|---:|---:|
| Golden | 66.7 | 0.0 |
| EasyOCR-NLLB-Render | 23.0 | 100.2 |
| QwenVL-Render | 17.1 | 102.2 |
| AnyTrans | 0.1 | 204.1 |
| PARSeq-mTransformer-Render | 9.9 | 103.8 |
| PEIT-Render | 10.4 | 101.4 |
| TranslatotronV | 1.4 | 69.1 |
| VisTrans | 11.3 | 28.8 |

The full table in the paper also reports **COMET**, with **Golden = 85.2**, **EasyOCR-NLLB-Render = 62.7**, **QwenVL-Render = 58.4**, **AnyTrans = 31.0**, **PARSeq-mTransformer-Render = 46.8**, **PEIT-Render = 48.0**, **TranslatotronV = 32.2**, and **VisTrans = 47.0** [2509.05146]. The comparison indicates that cascade systems can obtain better text translation quality than some end-to-end systems while producing weak image realism, whereas end-to-end models can improve visual fidelity but often struggle with translation quality. The paper states that **VisTrans** improves over **TranslatotronV** from **1.4** to **11.3** average BLEU and from **69.1** to **28.8** FID [2509.05146].

The paper interprets these results as evidence that PRIM exposes several practical failure modes: **background damage**, **incomplete rendering of long translated text**, and the general difficulty of **multilingual rendering** under realistic visual conditions [2509.05146]. The case study further notes that even when a translation is correct, rendering may fail if the translated string is too long or the font size is inappropriate. This suggests that PRIM measures a compound competence: OCR-free language transfer, layout adaptation, and image synthesis.

In that sense, PRIM’s significance lies less in raw scale than in its benchmark design. It combines **real-world visual complexity** with **multiple translation directions**, **manual background recovery**, and **paired visual references**, thereby making it possible to evaluate in-image multilingual translation as an image-generation problem rather than as a pure text-generation task [2509.05146].

Source: https://www.emergentmind.com/topics/prim-dataset