MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation

Abstract: While multilingual LLMs perform well on high-level tasks like translation and question answering, their ability to handle grammatical gender and morphological agreement remains underexplored. In morphologically rich languages, gender influences verb conjugation, pronouns, and even first-person constructions with explicit and implicit mentions of gender. We introduce MORPHOGEN, a morphologically grounded large-scale benchmark dataset for evaluating gender-aware generation in three typologically diverse grammatically gendered languages: French, Arabic, and Hindi. The core task, GENFORM, requires models to rewrite a first-person sentence in the opposite gender while preserving its meaning and structure. We construct a high-quality synthetic dataset spanning these three languages and benchmark 15 popular multilingual LLMs (2B-70B) on their ability to perform this transformation. Our results reveal significant gaps and interesting insights into how current models handle morphological gender. MORPHOGEN provides a focused diagnostic lens for gender-aware language modeling and lays the groundwork for future research on inclusive and morphology-sensitive NLP.

• The paper presents the MORPHOGEN benchmark, a novel resource for evaluating gender-aware morphological generation in French, Arabic, and Hindi.
• It employs a controlled GENFORM task, using extensive morphological rules and thousands of sentence pairs to test first-person gender counterfactual rewriting.
• Experiments reveal scaling trends, persistent masculine bias, and variable performance across languages, highlighting significant challenges for LLMs.

MORPHOGEN: A Multilingual Benchmark for Gender-Aware Morphological Generation

Introduction and Motivation

The "MORPHOGEN" benchmark addresses the systematic evaluation of multilingual LLMs on gender-aware morphological generation in French, Arabic, and Hindi (2604.18914). Current multilingual evaluation suites overwhelmingly focus on high-level semantic tasks; yet, grammatical gender and morphological agreement pose significant challenges for LLMs, especially in morphologically complex languages where speaker gender subtly or overtly triggers changes across verbs, adjectives, and pronouns. MORPHOGEN provides the first large-scale, balanced, linguistically-controlled resource for probing these phenomena with a focus on first-person gender counterfactual rewriting—the GENFORM task. Given a sentence and speaker gender, LLMs must produce its opposite-gender counterpart while maintaining fluency and all agreement relations.

Benchmark Design

Linguistic Scope and Rules

The dataset targets three typologically diverse gendered languages—French, Arabic, and Hindi—intentionally selected for their contrasting strategies in morphological gender realization. Each language is represented via thousands of sentence pairs with parallel masculine/feminine forms and English translations. A canonical set of morphological rules governs the creation of counterfactuals, covering:

• Verb and tense-dependent inflections (e.g., French compound tenses, Arabic suffixation)
• Adjectival and occupational agreement
• Pronoun and possessive concord
• Clause-level and multi-entity interference scenarios

(Figure 1)

Figure 1: General morphological rules governing grammatical gender and agreement across French, Arabic, and Hindi.

Sentence templates and rule selection ensure coverage of both high-frequency and edge-case morphosyntactic patterns, with explicit focus on minimal-pair gender alternations. This construction is crucial for isolating fine-grained model deficiencies that might be obscured by surface-level semantic evaluation.

Dataset Statistics and Coverage

MORPHOGEN provides 9,999 French, 7,610 Hindi, and 2,719 Arabic first-person sentence pairs, with gendered term counts varying by language. Figure 2 visualizes the per-language distribution of sentence frequency by morphological rule, illustrating the resource's depth in rule diversity.

(Figure 3)

Figure 3: Distribution of sentence frequency across morphological rules in French, Arabic, and Hindi, evidencing linguistic depth.

Task Definition

The core GENFORM task requires the model to reformulate a first-person sentence in the opposite gender, editing only speaker-referring words. The synthetic, controlled nature of the corpus mitigates annotation noise and supports precise quantitative analysis of model behavior, especially under challenging multi-entity and agreement scenarios.

Evaluation Protocol

Three complementary automatic metrics are introduced:

• Sentence-Level Gender Accuracy (SGA): Measures the token-level accuracy of gender edits per sentence.
• Gender IoU (GIoU): Intersection-over-union metric that penalizes both spurious and missed transformations, capturing interference phenomena.
• Corpus-Level Gender Accuracy (CGA): Aggregated token-level correctness over the corpus.

Directional bias is diagnosed via disaggregated masculine-to-feminine and feminine-to-masculine statistics and their difference, as shown in $\triangle \text{SGA}$ analyses.

Experimental Study

Model Selection and Inference

Fifteen multilingual models are benchmarked, spanning LLAMA, Qwen, Gemma, and Phi series from 2B–70B parameters, including both open and closed-source checkpoints. All evaluations are performed under a zero-shot prompting regime with deterministic decoding and minimal intervention prompts.

Key Findings

Model Scale and Morphological Competence

Strong scaling trends are evident: smaller models (<4B) exhibit extremely poor performance on complex gendered morphology, with SGA and GIoU metrics often below 10% for Arabic and French. Larger models (27B–70B) achieve maximal CGA values near 90% in Hindi and >80% in French and Arabic only at the upper parameter spectrum.

Language-Dependent Morphological Challenge

Hindi exhibits the highest baseline performance across all scales, attributable to its more regular and semantically transparent gender system. In contrast, French and Arabic expose LLMs to the confounds of irregular agreement and pervasive gender marking, resulting in lower and more variable scores.

Figure 4: Comparison of $\triangle \text{SGA}$ (accuracy gap) for masculine→feminine vs. feminine→masculine conversions across French, Arabic, and Hindi.

Gender Bias and Directionality

A consistent masculine bias persists, particularly in French and Arabic, with $\triangle \text{SGA}$ values exceeding 10% in several LLMs, including large-scale models. Notably, LLAMA-3.3-70B and Qwen3-32B frequently default to masculine forms even when feminine is required, especially in plural and ambiguous contexts. Model scale reduces but does not eliminate this bias; see Figure 5.

Gender Interference in Multi-Entity Contexts

A persistent error mode is overgeneration: models alter gender agreement of non-speaker entities—an interference effect. Qualitative analysis with the LLAMA family demonstrates that while the 70B model stabilizes speaker-specific inflection, both the 3B and 8B variants frequently drift, leading to high GIoU penalties. These findings are visualized in concrete sentence-level analyses.

Figure 2: Example model predictions in French for multi-entity sentences illustrate speaker/non-speaker gender interference, especially in smaller models.

Figure 5: Arabic multi-entity scenario shows overgeneralization and misalignment of gender marking among LLM variants.

Figure 6: Hindi multi-entity modeling exhibits the least interference, especially for larger models, but lower-capacity LLMs still exhibit errors in pronoun/adjective agreement.

Implications and Future Work

MORPHOGEN exposes persistent limitations of multilingual LLMs in controlled morphological reasoning, even at high parameter counts. The presence of masculine bias in both poorly- and well-performing models highlights entrenched issues originating from training corpora and tokenization artifacts. In practical NLP applications, these deficits directly compromise the inclusivity and grammaticality of generated content for morphologically rich languages, hampering reliability in translation, summarization, and dialogue systems.

Extending the framework to additional languages, dialectal variants, and non-binary gender systems remains a critical next step. Furthermore, the diagnostic power of per-rule and per-context analysis—facilitated by the synthetic, linguistically motivated construction—suggests its use for guiding model debugging, finetuning, and error analysis pipelines in industrial contexts. Systematic inclusion of such tests in LLM evaluation protocols is essential as models are deployed in environments where morphosyntactic fidelity is critical.

Conclusion

MORPHOGEN establishes a new gold standard for probing gender-aware morphological competence in multilingual LLMs. Through a battery of nuanced tasks, it reveals sharp limitations and bias trends in the current paradigm, demonstrating that semantic proficiency does not guarantee morphosyntactic adequacy. Large-scale, structure-aware resources such as MORPHOGEN will be indispensable for both fair evaluation and the future development of linguistically robust, contextually sensitive LLMs.