GENFORM: Gender-Aware Morphological Generation

• GENFORM is a diagnostic task that tests models' ability to reverse speaker-dependent morphology in first-person sentences while preserving semantics and word order.
• It benchmarks multilingual performance in French, Arabic, and Hindi using detailed metrics like Sentence-Level Gender Accuracy, GIoU, and Corpus-Level Gender Accuracy.
• Empirical evaluations reveal that while increased model scale improves performance, challenges such as directional bias and collateral gender interference persist.

GENFORM is the core task introduced in MORPHOGEN for evaluating gender-aware morphological generation in multilingual LLMs. In GENFORM, a model receives a first-person sentence together with its original speaker gender and must produce a well-formed sentence in the same language that preserves meaning and word order while switching all speaker-dependent morphology to the opposite gender. The benchmark is defined for three grammatically gendered languages—French, Arabic, and Hindi—and is designed as a focused diagnostic setting for measuring whether multilingual LLMs can manipulate grammatical gender without distorting semantics or over-applying gender changes to unrelated tokens (Agarwal et al., 20 Apr 2026).

1. Formal task specification

Let $\mathcal{L}$ denote one of the target languages and let $G=\{M,F\}$ denote the binary speaker-gender feature set. A first-person source sentence is written as $s\in\Sigma^*$ with original speaker gender $g\in G$. GENFORM requires a transformation

$f:(s,g)\mapsto s'$

such that $s'$ is well-formed in $\mathcal{L}$, preserves the semantic content and word order of $s$, and reflects the opposite speaker gender $\bar g$, where $\bar M=F$ and $G=\{M,F\}$0 (Agarwal et al., 20 Apr 2026).

The task is defined tokenwise through the set $G=\{M,F\}$1, which contains the tokens whose form depends on speaker gender, including verbs, adjectives, pronouns, participles, and related morphology. If $G=\{M,F\}$2 maps each token to its gender feature or to $G=\{M,F\}$3 for gender-invariant tokens, then GENFORM enforces

$G=\{M,F\}$4

$G=\{M,F\}$5

This formulation makes GENFORM more restrictive than unconstrained rewriting. It is not merely a style-transfer task: the intended transformation is localized to speaker-dependent morphology, while non-speaker material is required to remain unchanged (Agarwal et al., 20 Apr 2026).

A central consequence of this definition is that correctness depends both on agreement realization and on boundary control. A model must identify exactly which tokens encode speaker gender and must not alter other entities or grammatical material. This is reflected later in the benchmark’s distinction between token-level agreement success and penalties for collateral errors.

2. Dataset construction and linguistic coverage

MORPHOGEN is built from synthetic English first-person templates spanning domains including academic, healthcare, politics, and everyday conversation. Example template patterns include sentences such as “I am a ⚤occupation⚤.” and “Yesterday I wrote a letter to my friend.” The construction pipeline begins with template-driven English generation using GPT-4o-mini, producing 16,415 French-style structures, 10,248 Hindi, and 5,413 Arabic English sentences (Agarwal et al., 20 Apr 2026).

These English templates are then translated automatically with language-specific systems: French via NLLB-200, Hindi via IndicTrans2 plus GPT-4o-mini, and Arabic via Grok-3. After translation, three native or near-native annotators per language refine the outputs for naturalness and produce both masculine and feminine renditions of each sentence while strictly following language-specific morphological rules. Each masculine–feminine pair is then independently rated valid or invalid by two annotators. Validation is summarized by

$G=\{M,F\}$6

and

$G=\{M,F\}$7

indicating high agreement and validity in the annotated pairs (Agarwal et al., 20 Apr 2026).

The released dataset statistics are language-specific. French contains 9,999 unique sentences, covers 12 rules, and has an average of 1.78 gendered terms per sentence, with $G=\{M,F\}$8 and $G=\{M,F\}$9. Arabic contains 2,719 unique sentences, covers 14 rules, and has an average of 2.02 gendered terms, with $s\in\Sigma^*$0 and $s\in\Sigma^*$1. Hindi contains 7,610 unique sentences, covers 13 rules, and has an average of 1.43 gendered terms, with $s\in\Sigma^*$2 and $s\in\Sigma^*$3 (Agarwal et al., 20 Apr 2026).

The appendix further breaks down per-rule frequencies and shows that coverage includes verb tenses, adjective agreement, pronoun and possessive rules, clause-level effects, and multi-entity contexts. This suggests that GENFORM is intended not only to test local agreement morphology but also to probe compositional phenomena in which gender marking interacts with syntax, reference, and discourse structure.

3. Evaluation metrics and what they measure

GENFORM is evaluated with three gender-focused metrics defined over the gold set of speaker-dependent tokens $s\in\Sigma^*$4 and the set of generated tokens that do not match gold, $s\in\Sigma^*$5 (Agarwal et al., 20 Apr 2026).

The first metric is Sentence-Level Gender Accuracy:

$s\in\Sigma^*$6

MORPHOGEN also reports direction-specific scores $s\in\Sigma^*$7 and $s\in\Sigma^*$8 for masculine-to-feminine and feminine-to-masculine transformations, together with the directional bias measure

$s\in\Sigma^*$9

A positive or negative $g\in G$0 quantifies asymmetry between the two conversion directions (Agarwal et al., 20 Apr 2026).

The second metric is Gender Intersection-over-Union:

$g\in G$1

GIoU is stricter than SGA because it penalizes not only missed gender changes but also spurious modifications. This is particularly important for GENFORM, where a model may correctly update some speaker-dependent tokens yet incorrectly propagate the speaker’s gender to other entities or nouns (Agarwal et al., 20 Apr 2026).

The third metric is Corpus-Level Gender Accuracy:

$g\in G$2

CGA aggregates correctness across the entire corpus rather than sentencewise. Taken together, SGA, GIoU, and CGA separate several phenomena that would be conflated by a generic exact-match or BLEU-style evaluation. A high SGA with a lower GIoU, for example, indicates that a model often identifies the relevant speaker-dependent positions but also introduces collateral gender interference.

4. Benchmark protocol and empirical performance

MORPHOGEN evaluates 15 popular multilingual LLMs in a zero-shot setting with low-temperature $g\in G$3 deterministic generation (Agarwal et al., 20 Apr 2026). The evaluated model families include LLAMA, Qwen, Gemma, Phi4-14B, and GPT-4o-mini. Cross-language reporting focuses on GIoU, $g\in G$4, and CGA.

The reported results show a clear scale effect, though not a complete resolution of the task. Small models in the 2B–4B range often score below 60% CGA in French and Arabic and under 50% in complex cases, while generally faring better in Hindi. Larger models at 14B and above improve steadily. GPT-4o-mini reaches above 90% CGA in French and Hindi and approximately 80% in Arabic. Directional bias remains pronounced in French and Arabic, where masculine defaults are common, while it is smaller in Hindi (Agarwal et al., 20 Apr 2026).

Concrete cross-language examples illustrate the spread. Qwen2.5-0.5B records French GIoU 5.5 and CGA 4.2, Arabic GIoU 4.1 and CGA 4.6, and Hindi GIoU 0.35 and CGA 0.21, with positive $g\in G$5 in all three languages. Gemma2-2B reaches French GIoU 39.7 and CGA 37.5, Arabic GIoU 14.7 and CGA 14.1, and Hindi GIoU 71.4 and CGA 65.4. LLAMA-3.3-70B reaches French GIoU 76.7 and CGA 76.1, Arabic GIoU 59.2 and CGA 64.4, and Hindi GIoU 93.3 and CGA 91.4. GPT-4o-mini reaches French GIoU 86.4 and CGA 90.3, Arabic GIoU 71.0 and CGA 80.3, and Hindi GIoU 88.8 and CGA 93.4 (Agarwal et al., 20 Apr 2026).

These results support two narrow conclusions present in the benchmark. First, model scale greatly assists with gender-aware morphological rewriting. Second, stronger performance on aggregate metrics does not eliminate directional bias or guarantee correct handling of multi-entity contexts. A plausible implication is that GENFORM stresses capabilities that are not captured by high-level multilingual benchmarks such as translation and question answering.

5. Error structure and recurrent failure modes

The benchmark identifies several recurring failure modes. All models struggle with complex syntax, including French compound tenses with past participle agreement, Arabic relative clauses, and Hindi indirect speech. In these categories, GIoU drops by 20–40 percentage points. This indicates that the difficult cases are not limited to local inflectional alternations; they involve dependencies that may span clauses or require structural conditioning (Agarwal et al., 20 Apr 2026).

A second failure mode is gender interference. In sentences with multiple human referents, medium-sized models in the 8B–12B range often over-apply the speaker’s gender to non-speaker entities. The benchmark notes that this behavior appears as a large gap between SGA and GIoU. This matters because a model can appear competent if judged only by whether it edits the obvious speaker-marked tokens, while still failing to preserve entity boundaries (Agarwal et al., 20 Apr 2026).

Pronoun and possessive errors are also recurrent. Smaller models default to masculine forms for possessives, exemplified in French by contrasts such as “son professeur” versus “sa professeure,” and they also mis-assign Hindi possessor gender. Directional bias is systematic in French and Arabic, which consistently drift toward masculine, whereas Hindi occasionally shows a mild feminine bias in female-to-male conversions for some models (Agarwal et al., 20 Apr 2026).

The LLaMA family case study makes the progression with scale explicit. At 3.2B, models default much of the morphology and obtain approximately 20–50% SGA. At 8B, they capture local agreement but “bleed” gender onto other nouns, which suppresses GIoU. At 70B, they show more robust application and clearer entity boundaries, with SGA and GIoU in the approximate range of 75–93% (Agarwal et al., 20 Apr 2026). This pattern cautions against equating larger parameter counts with full morphological control: the remaining errors are qualitatively different from simple omission and instead concern scope, reference, and syntactic conditioning.

6. Significance, scope, and projected extensions

GENFORM occupies a specific position within morphology-sensitive NLP. It is neither open-ended generation nor generic paraphrasing; rather, it is a constrained transformation task that isolates speaker-gender morphology in first-person utterances. That design gives MORPHOGEN a diagnostic role: it tests whether a model can preserve meaning and structure while applying precisely targeted grammatical changes (Agarwal et al., 20 Apr 2026).

The benchmark’s conclusions are correspondingly specific. Model scale helps in capturing long-distance morphological dependencies, but does not eliminate bias. Architectures require explicit control of entity boundaries to avoid over-generalization. Even state-of-the-art models underperform in languages with rich, context-dependent morphology, including French past-participle agreement and Arabic plurals (Agarwal et al., 20 Apr 2026). A common misconception would be that strong multilingual performance on broad tasks implies reliable handling of fine-grained gender morphology; GENFORM provides evidence against that assumption.

The future directions proposed in MORPHOGEN are concrete. They include extending GENFORM to second-person and third-person settings and to non-binary gender markings; incorporating explicit morphological supervision or gender tags during fine-tuning; developing loss functions or constraints, including on GIoU, that penalize collateral gender interference; enriching training data with more multi-entity and discourse-level gender contexts; and expanding MORPHOGEN to additional gendered languages and dialects (Agarwal et al., 20 Apr 2026).

In this sense, GENFORM serves as both a benchmark and a task formulation. As a benchmark, it provides a multilingual, morphologically grounded test bed. As a formulation, it makes explicit that gender-aware generation is not only about selecting the correct inflectional form, but also about preserving semantic invariance, respecting word order, and localizing agreement changes to the appropriate speaker-dependent tokens.