MORPHOGEN: Multilingual Gender-Aware Morphology Benchmark
- MORPHOGEN is a multilingual benchmark dataset focused on gender-aware morphological generation in grammatically gendered languages.
- The benchmark uses the GENFORM task to transform first-person sentences by altering only gender-specific elements while preserving overall meaning and structure.
- It employs detailed annotation protocols and evaluation metrics across French, Arabic, and Hindi to diagnose models' control over morphological agreement.
Searching arXiv for the MORPHOGEN benchmark paper and closely related benchmark context. MORPHOGEN is a multilingual benchmark and dataset for evaluating gender-aware morphological generation in grammatically gendered languages. It is designed to test whether LLMs can rewrite a first-person sentence in the opposite gender while preserving meaning and structure, thereby isolating a specific competence that broad multilingual evaluations often leave underexamined: fine-grained control of grammatical gender and morphological agreement. The benchmark covers French, Arabic, and Hindi, and centers on the task GENFORM, with gold male/female counterfactuals and parallel English sentences (Agarwal et al., 20 Apr 2026).
1. Definition and diagnostic scope
MORPHOGEN is presented as a morphologically grounded large-scale benchmark dataset rather than a general-purpose generation corpus. Its purpose is not to evaluate translation, summarization, or open-ended fluency in the abstract, but to provide a focused test of whether a model can perform a constrained gender-conditioned rewrite in languages where speaker gender affects morphology in nontrivial ways (Agarwal et al., 20 Apr 2026).
This focus follows from a specific critique of existing multilingual evaluation practice. High-level benchmarks can make multilingual LLMs appear strong while leaving unresolved whether they actually control grammatical gender, morphological agreement, and speaker-conditioned inflection. MORPHOGEN therefore functions as a diagnostic instrument for a narrower failure mode: a model may preserve semantics while still missing the morphosyntactic transformations required by gendered first-person utterances. In that sense, the benchmark is deliberately closer to controlled morphology testing than to broad natural-language understanding.
A common misunderstanding is to treat MORPHOGEN as a translation benchmark. The benchmark is not organized around cross-lingual transfer as its primary object. Instead, it evaluates whether a model can identify which parts of a sentence are speaker-linked and gender-sensitive, modify those forms, and leave the rest unchanged. The paper frames this as a test of morphology-sensitive generation rather than generic multilingual competence (Agarwal et al., 20 Apr 2026).
2. GENFORM and the transformation target
The core task, GENFORM, requires a model to take a first-person sentence and rewrite it in the opposite speaker gender while preserving meaning and structure. The transformation is explicitly minimal: the model should adjust only the words that refer to the speaker and should not alter unrelated content (Agarwal et al., 20 Apr 2026).
The prompt specification makes this operational. Given a sentence in the target language and the gender of the speaker, the model must “adjust only the words that refer to the speaker to match the specified gender,” return only the modified sentence, and leave the sentence unchanged if no modification is required. This makes GENFORM distinct from paraphrasing and distinct from style transfer in the loose sense. The target is not a new sentence with equivalent meaning; it is a constrained counterfactual form.
The task is bidirectional: masculine-to-feminine and feminine-to-masculine. The paper states that results are reported separately through and , and that their difference is analyzed as a bias indicator (Agarwal et al., 20 Apr 2026). This framing is important because the benchmark is not only about aggregate correctness. It also probes asymmetry between the two transformation directions.
Another common misconception is that GENFORM is reducible to pronoun replacement. The benchmark is built precisely to show that this is false. In the covered languages, speaker gender can surface in verbs, adjectives, participles, role nouns, possessives, and constructions where first-person gender is morphologically expressed even when the speaker is not overtly named. The intended transformation is therefore structural and morphological, not merely lexical.
3. Linguistic coverage and typological design
MORPHOGEN covers French, Arabic, and Hindi, chosen as “three typologically diverse grammatically gendered languages.” The benchmark uses this typological spread to test different configurations of gender marking rather than treating grammatical gender as a single homogeneous phenomenon (Agarwal et al., 20 Apr 2026).
French is described as combining semantic cues, morphological cues, and phonological cues. The paper emphasizes that French is often irregular and context-dependent: present-tense verbs are generally gender-invariant, but past participles in compound tenses can agree in gender with the subject. This makes French particularly relevant for testing context-sensitive agreement.
Arabic is characterized as having a highly regular and pervasive agreement system, with gender morphology broadly expressed across verbs, adjectives, and pronouns. The appendix notes additional complexity in relative clauses, conditionals, multi-entity contexts, and plural forms. Arabic thus tests whether models can handle dense and systematic agreement rather than only isolated lexical alternations.
Hindi is presented as having a more natural-gender / semantically grounded system with partial morphological marking, often involving contrasts such as -ā and -ī. The paper treats Hindi as comparatively more regular and less context-dependent than French and Arabic, while still requiring substantial agreement control.
The linguistic phenomena covered by GENFORM include verb inflection, tense and aspect effects, adjectives, role nouns, pronouns, possessives, participles, and first-person constructions with explicit and implicit gender marking. The benchmark also includes multi-clause structures, passives, object-fronting, indirect speech, pseudo-cleft constructions, and multi-entity sentences designed to test gender interference (Agarwal et al., 20 Apr 2026).
The inclusion of multi-entity sentences is especially important. In such cases, only the speaker’s gender should control the rewrite. This means MORPHOGEN tests referential precision as well as morphology: the model must avoid propagating gender changes to other human referents in the sentence.
4. Dataset structure and quantitative profile
The dataset is organized around paired masculine and feminine counterfactuals, with the differing items defining the benchmark’s notion of gendered terms. A sentence pair therefore specifies exactly which forms should change under speaker-gender inversion (Agarwal et al., 20 Apr 2026).
The main paper reports the following dataset-level statistics.
| Language | Main table statistics | Appendix validation statistics |
|---|---|---|
| Arabic | 2,719 unique sentences; 14 rules; avg. gender terms 2.02; avg. word count 12.34 | 5,413 generated; 2,638 discarded; DVS 0.9733; Inter-Annotator Score 0.9526 |
| French | 9,999 unique sentences; 12 rules; avg. gender terms 1.78; avg. word count 26.76 | 16,415 generated; 6,416 discarded; DVS 0.9651; Inter-Annotator Score 0.9366 |
| Hindi | 7,610 unique sentences; 13 rules; avg. gender terms 1.43; avg. word count 15.46 | 10,248 generated; 2,694 discarded; DVS 0.9731; Inter-Annotator Score 0.9594 |
The main table also reports a maximum of 7 gendered elements for all three languages, and maximum word counts of 38 for Arabic, 67 for French, and 87 for Hindi (Agarwal et al., 20 Apr 2026). The benchmark therefore contains both short and structurally elaborate examples, and it is explicitly designed so that some items require multiple coordinated agreement changes within a single sentence.
The appendix statistics are also notable because they make the data construction process visible: generated sentences, discarded sentences, and validation scores are reported separately by language. The visible text explicitly notes an inconsistency between the main dataset table and the appendix table for Arabic and Hindi counts, which suggests that the paper distinguishes between generated material, filtered material, and final unique sentence counts in more than one way (Agarwal et al., 20 Apr 2026).
5. Construction pipeline and annotation protocol
MORPHOGEN is a synthetic but carefully controlled resource. The construction pipeline begins by identifying language-specific gender phenomena and then creating language-specific templates rather than forcing a single multilingual template inventory (Agarwal et al., 20 Apr 2026).
English source sentences were generated using GPT-4o-mini with structured prompts specifying the target rule, lexical arguments, discourse context, and required template structure. These English sentences were then translated into the benchmark languages using different systems: IndicTrans2 and GPT-4o-mini for Hindi, Grok-3 for Arabic, and NLLB-200 for French (Agarwal et al., 20 Apr 2026).
The translated material was not accepted as final output. It was refined by bilingual annotators proficient in English and the target language, who were instructed to discard bad examples, manually correct sentences, and produce both masculine and feminine forms. The guidelines required fidelity, fluency, template compliance, natural conversational phrasing, and guaranteed gender specificity in the target language.
The correction protocol also imposed an important invariance constraint: non-speaker gendered terms must remain the same across the male and female versions if they do not depend on the speaker. This rule is crucial for the logic of GENFORM, since the task is not supposed to rewrite the whole sentence into a globally re-gendered discourse. It is supposed to rewrite only those forms licensed by the speaker’s gender.
Validation was multi-stage. The paper states that every sentence pair was independently reviewed by two annotators, and the appendix reports Number of Annotators as 2 for Arabic and 3 for French and Hindi (Agarwal et al., 20 Apr 2026). It also reports a Data Validation Score and an Inter-Annotator Score for each language, with values above 0.93 in all cases shown. This suggests a high degree of annotation consistency, although the visible text does not present a full methodological discussion of disagreement resolution.
6. Benchmarking setting, interpretive value, and limits
The benchmark was used to evaluate 15 popular multilingual LLMs (2B-70B) on their ability to perform GENFORM. The paper states that the results reveal “significant gaps and interesting insights” in how current models handle morphological gender (Agarwal et al., 20 Apr 2026). In the visible text, however, the emphasis is more on benchmark design and error categories than on a fully enumerated leaderboard.
The interpretive value of MORPHOGEN lies in its granularity. Because the task is narrowly defined and gold counterfactuals are available, model failures can be read as failures of speaker-sensitive agreement control, context-sensitive morphology, or reference tracking rather than being obscured by broader semantic variation. This gives the benchmark what the paper calls a “focused diagnostic lens.”
The benchmark also bears on a broader issue in multilingual NLP: the relation between inclusive language technology and formal morphology. MORPHOGEN does not reduce gender-aware generation to social bias measurement, nor does it treat it as purely a fairness problem. Instead, it locates gender-aware generation at the intersection of morphological competence, agreement resolution, and speaker-conditioned rewriting. A plausible implication is that progress on inclusive or gender-sensitive NLP in grammatically gendered languages requires stronger control of morphology, not only better discourse-level calibration.
At the same time, MORPHOGEN is intentionally specialized. Its core unit is the first-person sentence; its target is opposite-gender rewriting; and its language set is limited to French, Arabic, and Hindi. This suggests breadth limitations, but those limits are part of the benchmark’s design philosophy rather than an accident. The paper positions MORPHOGEN not as a complete theory of multilingual gender generation, but as a controlled benchmark that isolates a morphologically precise subproblem and lays the groundwork for future research on inclusive and morphology-sensitive NLP (Agarwal et al., 20 Apr 2026).