MuST-SHE: Gender Bias in Speech Translation
- MuST-SHE is a gender bias evaluation benchmark that measures how well systems handle grammatical gender by comparing a Correct Reference with a Wrong Reference.
- It isolates gender realization challenges by separating speaker-dependent cues (Category 1) from content-based signals (Category 2) using a controlled paired-reference design.
- Empirical results show that while audio cues partially aid speaker-self reference, a persistent masculine bias remains across systems, highlighting the need for explicit gender conditioning.
In the literature on speech translation, the benchmark most directly associated with the designation “SHE” is MuST-SHE, a gender-bias evaluation benchmark for speech translation and machine translation built from MuST-C to test whether systems correctly realize grammatical gender when translating English into gender-marked target languages, specifically Italian and French. Its distinguishing feature is that it isolates gender realization from general translation quality by pairing each segment with both a Correct Reference (C-Ref) and a Wrong Reference (W-Ref) that differs only in gender-marked forms, while also annotating the source of gender evidence, speaker gender, and all gender-marked target expressions (Bentivogli et al., 2020).
1. Definition and scope
MuST-SHE is a multilingual natural benchmark derived from MuST-C. It is a subset containing examples in which at least one English gender-neutral word must be translated into a masculine or feminine target form, and in which that formal distinction corresponds to a real sex/gender distinction for a human referent. Each benchmark instance is an (audio, transcript, translation) triplet, so the resource is applicable to both speech translation (ST) and machine translation (MT), although its initial use is centered on ST (Bentivogli et al., 2020).
The benchmark was motivated by a structural asymmetry in translation from English into languages with productive grammatical gender. English frequently underspecifies gender morphologically, whereas Italian and French often require explicit masculine or feminine realization. The paper identifies two reasons this remains difficult for machine systems: source ambiguity and training-data bias, including a tendency to prefer masculine forms. MuST-SHE was introduced to determine whether speech input can provide additional information—especially about the speaker—that helps reduce this bias.
The benchmark is explicitly described as satisfying the requirements of a GBET (Gender Bias Evaluation Testset). In practice, that means it aims at a balanced distribution of masculine and feminine references, balanced speakers, and evaluation focused on gender translation accuracy rather than only generic sentence-level adequacy. A central implication is that MuST-SHE is not a general-purpose ST benchmark; it is a targeted evaluation framework for gender-sensitive translation behavior.
2. Corpus construction and annotation design
MuST-SHE was extracted from MuST-C, selected because it is large, multilingual, based on TED talks, contains high-quality speech, covers a variety of speakers, and includes reasonable representation of women. Construction began with a manual analysis of 2,500 parallel sentences sampled from MuST-C in order to identify how gender phenomena arise in naturally occurring speech-translation data. Automatic extraction then used regular expressions encoding gender-agreement rules, language-pair-specific search patterns, and manually compiled lexical resources, including 50 human-referring adjectives in French, 50 human-referring adjectives in Italian, and 1,000+ English occupation nouns from U.S. Department of Labor statistics (Bentivogli et al., 2020).
The extracted candidates were then manually filtered. For each language direction, the procedure removed noisy examples, retained only sentence pairs containing at least one gender phenomenon, preserved corresponding En-It and En-Fr pairs to build a common multilingual subset, and selected examples to balance feminine versus masculine forms and female versus male speakers. After the textual subset was finalized, the corresponding audio segments were manually checked to correct possible misalignments.
Each segment is manually annotated with four elements: the category of gender phenomenon, the required masculine/feminine form, the speaker gender, and all gender-marked expressions in the reference translation. The most consequential annotation decision is the creation of two reference translations per segment. The Correct Reference is the ordinary translation with the intended gender marking; the Wrong Reference is an almost identical translation in which all gender-marked words are swapped to the opposite gender. This design allows gender behavior to be measured while holding the rest of the sentence effectively constant.
| Aspect | En-It | En-Fr |
|---|---|---|
| Triplets | 1,062 | 1,074 |
| Category 1 / Category 2 | 560 / 502 | 612 / 462 |
| Annotated gender-marked words | 1,940 | 2,010 |
The full benchmark contains 2,136 triplets spoken by 273 different speakers, plus a common subset of 696 instances for direct cross-language comparison. The paper reports total gender distribution as 516 feminine and 546 masculine for En-It, and 542 feminine and 532 masculine for En-Fr. It also reports speaker balance as 558 female / 513 male for En-It and 577 female / 498 male for En-Fr.
Annotation quality was checked by a second linguist, who independently re-annotated each segment’s category and produced an additional wrong reference. The reported outcome is no disagreement on Category 1, approximately 0.03% disagreement on Category 2—those cases were removed—and about 11% disagreement on wrong references, attributed to oversights and subsequently reconciled.
3. Linguistic coverage and category structure
MuST-SHE distinguishes two benchmark categories according to where gender-disambiguating evidence is available. Category 1 contains cases in which the correct target-side gender depends on the speaker’s own gender, typically because the speaker refers to themselves. In these examples, gender information is assumed recoverable from the audio signal. Category 2 contains cases in which the relevant gender is recoverable from the utterance content, such as pronouns, gender-exclusive lexical items, or proper names, rather than from the speaker’s voice (Bentivogli et al., 2020).
This distinction is methodologically central. If speech contributes useful information beyond text, it should matter primarily for Category 1. By contrast, Category 2 functions as a control condition: the relevant cue is already available in the transcript semantics, so audio should not intrinsically confer an advantage. The benchmark therefore does not merely test gender translation in the abstract; it decomposes the problem into speaker-conditioned and content-conditioned subproblems.
The benchmark also avoids a common simplification in gender-sensitive ST evaluation: equating speaker gender with referent gender. In Category 1, the two coincide by construction because the speaker is referring to themselves. In Category 2, however, they differ in about 50% of segments. This design choice is analytically important because it prevents systems from solving the task by using speaker gender as a universal proxy for target morphological gender.
Illustrative examples make the distinction concrete. “I was born and brought up in Mumbai” requires Italian Sono nata e cresciuta or Sono nato e cresciuto depending on speaker gender, and is therefore Category 1. “She'd get together with two of her dearest friends, these older women...” requires feminine realizations triggered by lexical content, not by the current speaker’s voice, and is therefore Category 2.
4. Evaluation protocol and benchmark metrics
MuST-SHE was designed in response to the inadequacy of standard corpus-level metrics such as BLEU and TER for diagnosing gender realization. The benchmark’s primary evaluation innovation is to score the same system outputs against both the Correct Reference and the Wrong Reference. Because the two differ only in gender-marked forms, the gap between those scores becomes informative about gender handling rather than overall sentence quality. The paper reports this as
A larger positive Diff indicates that the system is closer to the correctly gendered reference than to the gender-swapped alternative (Bentivogli et al., 2020).
This BLEU-with-swapped-references protocol is complemented by a more targeted measure, Accuracy, defined as the proportion of gender-marked words in the references that are correctly translated by the system. The benchmark applies an upper bound of one match for each gender-marked word, so overgeneration is not rewarded. Accuracy is computed globally, against correct and wrong references, and by category.
The paper emphasizes that these two metrics serve different analytic purposes. BLEU-based Diff reveals whether gender contributes meaningfully to total translation behavior, whereas Accuracy directly targets the lexical items where gender morphology is realized. This distinction matters because MuST-SHE contains only about 2,000 gender-marked words out of ~30,000 total words, with an average of 1.8 such words per sentence. As a result, corpus-level BLEU alone can understate gender-specific differences that become pronounced under the targeted accuracy metric.
The benchmark therefore operationalizes gender bias and gender competence through a paired-reference design rather than through a new bespoke scorer. Methodologically, this is one of MuST-SHE’s most consequential contributions: it transforms a generic MT/ST metric into a controlled probe of grammatical gender realization.
5. Benchmark systems and empirical findings
The initial benchmark study compares three systems: an End2End ST model, a standard Cascade ASR→MT system, and Cascade+Tag, which augments the cascade MT component with explicit speaker-gender tokens <toM> or <toF>. The End2End system is an S-Transformer using 40 MFCCs, a front-end stack of 2D CNNs with batch normalization and ReLU, followed by 2D self-attention and Transformer encoder layers with a distance penalty. The Cascade system uses a KALDI ASR component with a time-delay neural network and lattice-free maximum mutual information training, plus a Transformer MT model trained on OPUS and fine-tuned on MuST-C with BPE at 50K subword units (Bentivogli et al., 2020).
On general MuST-C test data, Cascade remains stronger in overall translation quality: 27.4 versus 21.5 BLEU for En-It, and 35.5 versus 31.0 for En-Fr. MuST-SHE, however, shows that global BLEU does not capture gender performance adequately. On gender-sensitive BLEU measured by Diff, End2End performs on par with Cascade in both directions, while Cascade+Tag shows the largest gains. For En-It, End2End and Cascade both obtain Diff 1.8, whereas Cascade+Tag reaches 2.9; for En-Fr, End2End and Cascade both obtain 2.1, whereas Cascade+Tag reaches 3.6.
The more targeted accuracy metric sharpens the picture. For En-It, End2End attains 43.3 accuracy against correct references versus Cascade’s 41.1, while for En-Fr Cascade leads with 49.6 versus End2End’s 46.0. In both language pairs, Cascade+Tag is clearly best, reaching 48.0 for En-It and 57.2 for En-Fr on correct-reference accuracy. The benchmark therefore separates three phenomena that ordinary evaluation conflates: overall translation strength, gender sensitivity, and the specific contribution of externally supplied speaker-gender information.
Across systems, the paper finds a persistent bias toward masculine forms. On both BLEU-based Diff and gender-word Accuracy, masculine subsets are handled better than feminine subsets. For example, En-It End2End has feminine accuracy Diff 10.2 versus masculine 41.7; En-Fr Cascade has feminine 13.4 versus masculine 43.5. Only En-Fr Cascade+Tag is close to balanced in BLEU terms, with feminine Diff 3.6 and masculine 3.5.
The most important category-specific result concerns whether speech helps. On Category 1, where gender information is available in the audio because the speaker refers to themselves, End2End substantially outperforms plain Cascade on feminine examples, though both remain poor relative to Cascade+Tag. For En-It feminine Category 1, End2End yields Diff -0.5, Cascade -18.6, and Cascade+Tag 32.6; for En-Fr the corresponding figures are -4.1, -17.1, and 46.7. The negative values indicate a strong tendency to output masculine forms even when feminine self-reference is required, but they also show that audio-conditioned End2End is materially less biased than a transcript-only cascade.
On Category 2, where the relevant gender is encoded in the utterance content rather than the speaker voice, End2End is consistently weakest on the feminine class. The paper attributes this not to a failure of audio conditioning per se, but to the system’s lower overall translation capacity given more limited training data. MuST-SHE therefore supports a precise conclusion: speech helps partially for speaker-dependent gender, but does not solve gender translation in general.
6. Significance, limitations, and interpretation
MuST-SHE occupies a distinctive position among gender-bias test sets because it is built on natural data rather than purely templated challenge sets, and because it is explicitly designed for both ST and MT. Its main significance lies in converting gender-sensitive translation from a diffuse fairness concern into a technically operational benchmark. By combining audio, transcript, multilingual references, category labels, and paired correct/wrong targets, it provides a framework in which errors can be localized to specific mechanisms: speaker-gender inference, referent tracking, lexical gender realization, or generic translation weakness (Bentivogli et al., 2020).
Several common misunderstandings are addressed by the benchmark’s design. One is the belief that access to speech should broadly solve gender bias. MuST-SHE shows that this is only partially true: audio helps mainly for Category 1, while Category 2 often depends on linguistic context rather than speaker identity. Another is the assumption that a system can succeed by relying on speaker gender alone. The benchmark explicitly prevents this shortcut by ensuring that in about 50% of Category 2 segments the speaker gender and referent gender differ. A third misconception is that standard BLEU is sufficient for fairness analysis; MuST-SHE demonstrates that BLEU must be paired with reference swapping or token-level gender accuracy to expose systematic masculine bias.
The benchmark also has clear limitations. It adopts a binary female/male framing, which the paper explicitly acknowledges as a simplification. It is relatively small, with roughly 1,000 examples per language, though densely annotated. Its practical speaker-gender oracle baseline, Cascade+Tag, is not realistic in settings where speaker metadata are unavailable. Finally, it shows that even when audio is available, current end-to-end systems do not recover or exploit gender information as effectively as systems given explicit gender tags.
Taken together, these properties make MuST-SHE less a general ST leaderboard than a specialized instrument for fine-grained evaluation of grammatical-gender realization under speech input. Its enduring importance lies in the fact that it separates generic translation quality from gender accuracy, and in showing that speech translation systems remain affected by gender bias even when the acoustic signal offers potentially useful cues.
