m-GenEval: Multilingual AI Evaluation

• m-GenEval is a multilingual extension of GenEval that rigorously evaluates text-to-image prompt alignment and visual consistency across languages.
• It employs human-verified prompt translations and metrics like Cross-Lingual Consistency (CLC) to measure semantic fidelity and robustness in generative outputs.
• Empirical findings reveal that models such as NeoBabel achieve high alignment scores, demonstrating the protocol’s role in advancing multilingual generative AI research.

m-GenEval is a multilingual, fine-grained evaluation protocol and benchmark designed to assess prompt-to-image alignment for text-to-image generative models across multiple languages. It extends the established GenEval evaluation suite—originally developed for English—into a multilingual context, providing rigorous methodologies and standardized protocols for quantifying both semantic and visual consistency in generative outputs. m-GenEval plays a critical role in the evaluation of contemporary multilingual generative frameworks, such as NeoBabel, by enabling direct assessment of crosslingual generalization, cultural fidelity, and robustness to code-mixed prompts (2507.06137).

1. Definition and Scope

m-GenEval is defined as a multilingual extension of the GenEval benchmark for evaluating generative models, particularly text-to-image systems, in settings where prompts appear in diverse languages. The protocol involves the translation of original GenEval prompts into multiple target languages—Chinese, Dutch, French, Hindi, and Persian—while maintaining semantic equivalence verified through human moderation and correction. This ensures that the same conceptual content is tested uniformly across languages, rendering m-GenEval suitable for evaluating both language-specific and crosslingual generative performance (2507.06137).

m-GenEval is used to systematically benchmark how well text-to-image models preserve prompt meaning and visual fidelity when the linguistic input varies, and it introduces metrics explicitly designed to measure alignment, consistency, and robustness in multilingual settings.

2. Methodology and Evaluation Protocol

The m-GenEval evaluation protocol involves several key stages:

• Prompt Construction and Translation: Each original (English) GenEval prompt is translated into the five additional supported languages, undergoing human verification and manual correction to ensure semantic and syntactic appropriateness for each linguistic context. This universal prompt set is central to the multilingual paradigm of the benchmark.
• Model Output Generation: A candidate text-to-image model generates images for each prompt in all supported languages. This results in a set of images per prompt per language.
• Assessment Procedures: Evaluation metrics are applied to quantify both prompt-image alignment (how well the image matches the intended meaning) and crosslingual consistency (how similar the outputs are across language versions of the same prompt).

Two specialized metrics are introduced in the m-GenEval framework to formalize these procedures:

• Cross-Lingual Consistency (CLC) measures the visual embedding similarity between reference images (generated from English prompts) and target images (generated from equivalent prompts in other languages). For a prompt $p$, with reference set $\mathcal{R}_p$ and target set $\mathcal{T}_p$, the CLC is computed as:

$$\mathrm{CLC}_p = \frac{1}{|\mathcal{R}_p| \cdot |\mathcal{T}_p|} \sum_{x_i \in \mathcal{R}_p} \sum_{x_j \in \mathcal{T}_p} \cos\big(f(x_i), f(x_j)\big)$$

where $f(\cdot)$ denotes the image embedding obtained from a robust vision encoder, such as EVA-CLIP or DINOv2.

• Code-Switching Similarity (CSS) assesses the fidelity of images generated from prompts that mix languages within the same input. For two variants—English-First (EF) and English-Second (ES)—the CSS for prompt $p$ in the EF case is:

$$\mathrm{CSS}_p^{\text{EF}} = \frac{1}{L-1} \sum_{l=1}^{L-1} \cos\left(f(x_{\text{ref}}), f(x_{\text{EF}}^{(l)})\right)$$

The overall CSS metric is taken as the average across all prompts and all supported languages.

These metrics rigorously quantify the requirements for true multilingual generation: not only that content is correctly generated from any single language, but also that the system's outputs are invariant under translation and robust to real-world linguistic code-mixing phenomena (2507.06137).

3. Empirical Findings and Benchmark Results

Experiments employing m-GenEval have demonstrated that targeted multilingual models, such as NeoBabel, achieve state-of-the-art performance in both semantic and visual alignment across the six evaluated languages. Notably, NeoBabel attains an m-GenEval score of 0.75, surpassing larger models such as BLIP3-o 8B, which scored 0.64 on the same benchmark. This was achieved without reliance on translation pipelines, highlighting the advantage of direct multilingual modeling in prompt-to-image architectures (2507.06137).

The usage of m-GenEval also reveals that:

• Model size and parameter count are not the sole determinants of multilingual alignment; training strategies and dataset curation play a crucial role.
• Models trained with a unified multilingual protocol can exhibit both high fidelity in English and superior generalization to non-English prompts, crucial for reducing digital inequities across linguistic groups.

4. Underlying Technologies and Implementation

The technical implementation of m-GenEval involves several components:

• Vision Encoders for Evaluation: Robust visual encoders (EVA-CLIP, DINOv2) are employed to produce image representations invariant to superficial variations, allowing for reliable computation of similarity metrics (CLC, CSS). These encoders are chosen for their demonstrated cross-domain and cross-language robustness.
• Standardized Prompt Sets: The multilingual prompts are meticulously constructed to avoid semantic drift and bias that can arise from automated translations. Human curation ensures prompts are both culturally and contextually appropriate.
• Automated and Reproducible Protocols: The provision of open-source code and datasets, as exemplified by NeoBabel’s toolkit, enables reproducible research and benchmarking using the m-GenEval protocol.

The use of fully aligned, high-quality text-image pairs in multiple languages further supports the reliability of evaluation and the training of subsequent generative or evaluative models under the m-GenEval standard.

5. Impact, Applications, and Significance

m-GenEval’s introduction marks a shift toward inclusive and rigorous evaluation for generative AI systems. Its principles and methodology have immediate applications in:

• Benchmarking and Leaderboards: Providing fair and transparent comparison of multilingual generative models, facilitating progress in the development of globally relevant AI systems.
• Model Development and Diagnosis: Enabling the diagnosis of systematic weaknesses—such as diminished image fidelity or semantic drift—in underrepresented languages or in code-switched inputs.
• Research into Robustness and Generalization: Serving as a testbed for probing how multilingual capability affects robustness, efficiency, and cultural adaptation in generative AI.

The introduction of crosslingual and code-switching metrics ensures that future models are evaluated in settings reflecting the complexities of real-world usage, including mixed-language environments and fine-grained visual requirements (2507.06137).

6. Future Directions and Open Challenges

While m-GenEval sets a new standard for multilingual evaluation, several avenues for further development remain:

• Scaling to More Languages: Current protocols support six languages; expanding coverage without sacrificing evaluative rigor will require scalable curation processes and new automatic verification tools.
• Cultural and Contextual Adaptation: Ensuring semantic equivalence and cultural appropriateness across an even broader range of linguistic and sociocultural contexts remains an open challenge.
• Integration with Other Modalities: Extending the m-GenEval paradigm to video, audio, and complex interleaved modalities will be necessary as generative models grow more capable.
• Evaluation Depth: While prompt-image alignment is central, future m-GenEval variants may include deeper subjective criteria such as cultural nuance or socio-ethical alignment, requiring careful metric definition and validation.

A plausible implication is that the structured evaluation standards introduced by m-GenEval will inform not only benchmarking but also training regimes, dataset construction, and the deployment standards for next-generation generative and evaluative AI systems.

---

Metric/Protocol	Purpose	Implementation Notes
m-GenEval Score	Aggregate prompt-to-image alignment	Evaluates semantic and visual accuracy across all languages
Cross-Lingual Consistency (CLC)	Measures cross-language output similarity	Averaged cosine similarity of image embeddings per prompt
Code-Switching Similarity (CSS)	Assesses robustness to mixed-language prompts	Cosine similarity between codemixed and monolingual outputs