StereoBias-Stories: Evaluating Gender Stereotypes

• StereoBias-Stories is a benchmark measuring narrative-level gender bias by evaluating gender contribution metrics in open-ended story generation across various prompting regimes.
• It employs diverse prompting settings and a detailed inventory of gender-coded traits to assess how stereotype attributes shape the overall narrative composition.
• The analysis reveals that non-gender cues mitigate bias and larger models align more with human stereotype structures, offering nuanced, actionable insights.

StereoBias-Stories, often abbreviated SBS, is a benchmark for analyzing gender bias in open-ended story generation by LLMs. It was introduced to move beyond short-form bias probes such as sentence completion, coreference, and multiple-choice question answering, and instead to measure how stereotypes shape the overall composition of a generated narrative: who the story appears to be about, which gendered characters are introduced, and how psychologically grounded stereotype attributes shift the balance of male- and female-coded references. SBS contains about 148,082 generated stories from five LLMs, organized across No-Attribute, Single-attribute, Two-attribute, and Multi-attribute prompting settings, and uses a story-level gender-contribution metric rather than a token-level stereotype preference score (Masoudian et al., 5 Aug 2025).

1. Conceptual setting within stereotype and bias evaluation

StereoBias-Stories emerged in response to a limitation in earlier bias evaluation: many established benchmarks assess whether a model prefers a stereotypical completion over an anti-stereotypical or unrelated alternative, but do so in relatively short or fixed formats. StereoSet, for example, measures stereotypical bias through Context Association Tests (CATs) in intrasentence and intersentence form, with metrics such as Language Modeling Score, Stereotype Score, and ICAT (Nadeem et al., 2020). Cross-lingual extensions of StereoSet further showed that stereotypical bias varies across English, German, French, Spanish, and Turkish, and that English-only evaluation misses important differences (Öztürk et al., 2023). Contextual StereoSet then demonstrated that stereotype selection changes substantially when the same item is reframed by location, year, style, or observer framing, indicating that measured bias is highly context-conditional rather than a fixed scalar model property (Basu et al., 15 Jan 2026).

SBS adopts a different unit of analysis. Instead of asking whether a model picks a stereotypical continuation in a controlled contrast set, it asks how open-ended narrative generation changes when prompted with stereotype-associated attributes. This places it near other story-centered evaluations, but with a distinct emphasis on psychology-grounded gender stereotypes rather than multilingual stereotype discovery or sociocultural personalization. In parallel, StereoBias introduced a sentence-level dataset of 5,012 sentences annotated for both bias and stereotype labels to test whether stereotype detection can act as an auxiliary signal for bias detection (Tomar et al., 2 Jul 2025). SBS and StereoBias therefore address related phenomena at different levels: the former studies narrative generation, while the latter studies sentence-level classification.

The broader literature reinforces the importance of this shift. StereoTales studies multilingual open-ended stereotype discovery by aggregating attribute distributions over more than 650,000 generated stories across 10 languages, showing that harmful stereotypes emerge as statistically over-represented associations rather than as single offensive sentences (Jeune et al., 11 May 2026). Biased Tales similarly shows that children’s stories can be low-toxicity yet still embed strong cultural and gender stereotypes, including a 55.26% increase in appearance-related attributes when the protagonist is described as a girl rather than a boy (Rooein et al., 9 Sep 2025). Together, these results situate SBS within a movement from fixed contrast testing toward distributional and narrative bias analysis.

2. Benchmark design, prompting regimes, and attribute inventory

SBS contains short stories generated by GPT-4o-mini, GPT-4o, DeepSeek-R1-7B, DeepSeek-R1-32B, and DeepSeek-R1-70B under four prompting settings (Masoudian et al., 5 Aug 2025). The benchmark is based on 25 personality/behavior traits associated with gendered expectations, together with 3 story-ending conditions, yielding a 28-item pool from which conditioning attributes are sampled.

The four prompting settings are as follows:

Setting	Description	Size
No-Attribute	Simple story prompt with no stereotype conditioning	28,668
Single-attribute	One random attribute is added	16,661
Two-attribute	Two random attributes are sampled	19,539
Multi-attribute	Six random attributes are sampled	83,214

The paper notes that during each generation setting, 20% of the data is No-Attribute control, so unconditioned baselines are embedded throughout the benchmark. Prompts are deliberately simple and child-oriented, inspired by TinyStories-style generation: the model is asked to write a short story using very simple words, understandable by a 3-year-old child, usually in 1–2 paragraphs. The authors also state that they intentionally did not add a persona so that the model could distribute gender as freely as possible (Masoudian et al., 5 Aug 2025).

The stereotype inventory is organized into female-associated traits, male-associated traits, and a smaller neutral / less stereotypically gendered or used for diversification subset. Female-associated traits include Caring, Communication, Empathetic, Gossiping, Happy Ending, Indecisive, Intuitive, Over-emotional, Resilient, Self-Sacrificing, Sensitivity, and Supportive. Male-associated traits include Aggressive, Assertiveness, Bad Ending, Emotionally Suppressed, Leadership, Logic, Manipulative, Obligation, Overbearing, Reckless, Strategic Thinking, and Tyrannical. The neutral or control-oriented items are Neglectful, Guardian, Mentorship, and Neutral Ending (Masoudian et al., 5 Aug 2025).

Each attribute is additionally labeled by lexical sentiment as positive, negative, or neutral. The paper is explicit that these sentiment labels are an analytical lens and are “lexical and somewhat simplified.” Sampling is not purely random: attributes are sampled to keep gender stereotype and sentiment appearance balanced, and the benchmark avoids contradictory combinations such as over-emotional vs. emotionally suppressed and bad ending vs. neutral ending (Masoudian et al., 5 Aug 2025).

Generation used temperature = 0.7, beam size = 2, and max new tokens = 3000 for DeepSeek-R1 models. R1-7B was run locally on a single Nvidia 3090 RTX in float16, while OpenAI models and the other DeepSeek-R1 models were accessed through APIs (Masoudian et al., 5 Aug 2025).

3. Story-level measurement and dataset validation

The central methodological contribution of SBS is its gender contribution metric. Rather than scoring whether a model chooses a stereotype over an anti-stereotype in a fixed item, the benchmark counts gendered mentions in the generated story text itself. The authors construct a lexicon of 14,255 gendered terms in total, later described in the appendix as 7,307 male words and 6,948 female words (Masoudian et al., 5 Aug 2025).

For a generated story, the contribution of each gender is defined as

$C_G = \frac{N_G}{\sum_G N_G}$

where $N_G$ is the number of gendered mentions for gender $G \in \{\text{Male, Female}\}$. The main bias metric is then

$Gap = C_{Male} - C_{Female}$

with $Gap = 0$ indicating balanced gender contribution, $Gap > 0$ indicating male over-representation, and $Gap < 0$ indicating female over-representation. To make conditioning effects comparable across models, SBS further defines

$\Delta_{Gap_m} = Gap - \mu_{Gap^{No-Attribute}_m}$

where $\mu_{Gap^{No-Attribute}_m}$ is the average unconditioned gap for model $m$. Positive $N_G$0 means that conditioning shifts the story further toward male representation, while negative $N_G$1 means that conditioning shifts it toward female representation (Masoudian et al., 5 Aug 2025).

Stories with no gender identifiers are excluded from the gap analysis. The paper reports that this was rare: only 127 stories in R1-7B lacked any gendered terms (Masoudian et al., 5 Aug 2025).

The benchmark also includes a dataset-quality evaluation. Lexical measures include Perplexity as a fluency proxy, N-gram diversity, and a Redundancy ratio defined as

$N_G$2

where $N_G$3 is the number of unique sentences and $N_G$4 is the total number of sentences. Human evaluation recruited 58 Prolific participants and sampled 280 stories from the Single-attribute setting, with 2 samples per attribute per model and 5 stories rated per participant. Annotators rated overall story quality and attribute expression on 1–5 Likert scales. A second quality check asked models to rate 700 samples, using the same five models as evaluators except that R1-32B was unavailable and replaced by Qwen-32B. The paper reports that higher-quality models generally receive better ratings and that GPT-4o aligns best with human judgments, supporting the validity of the dataset (Masoudian et al., 5 Aug 2025).

4. Main empirical findings on gender bias in stories

The benchmark’s first major finding is that unconditioned prompts are strongly male-biased. In the No-Attribute condition, most models produce stories with more male identifiers than female identifiers. From the single-attribute plot, the paper reports an average male contribution of about 0.61 and an average female contribution of about 0.39 across models. The table of gender gaps gives positive unconditioned $N_G$5 values for most models: R1-32B: 0.426, R1-70B: 0.386, GPT4o-mini: 0.468, and GPT4o: 0.432. R1-7B is an exception with a slightly negative baseline gap of -0.057, but the paper treats this cautiously because that model also performs worst on story quality and attribute adherence (Masoudian et al., 5 Aug 2025).

The second major finding is that conditioning on non-gender stereotypes mitigates male bias. Single-attribute conditioning lowers the gap for most models, two-attribute conditioning lowers it further, and multi-attribute conditioning still remains less biased than the No-Attribute baseline, although the paper describes the multi-attribute trend as less clean because of sparsity and complexity (Masoudian et al., 5 Aug 2025). This is a notable result because the conditioning attributes are not explicit gender instructions; they are personality, behavior, or structural cues.

The third finding is that stereotype direction matters. Using the psychology-grounded labels, the paper shows that male-coded attributes tend to increase male representation, while female-coded attributes tend to decrease male representation and thus increase female representation. Examples include male-associated prompts such as aggressive, leadership, and reckless, which generally push stories toward male identifiers, and female-associated prompts such as empathetic, caring, and supportive, which generally push stories toward female identifiers. The paper states that this effect is statistically significant for most stereotypes, with one-sample t-tests usually giving $N_G$6 against a zero baseline (Masoudian et al., 5 Aug 2025).

A fourth result is the compositionality of stereotype pressure. When models receive two female stereotypes together, the mitigating effect becomes stronger; when they receive two male stereotypes together, the male-amplifying effect becomes stronger; and when they receive mixed-gender combinations, the effects can cancel each other out. The authors extend this observation to the six-attribute regime as well, while noting that the multi-attribute setting is sparser and harder to interpret (Masoudian et al., 5 Aug 2025).

Sentiment also interacts with gender bias. Under negative sentiment, female stereotypes mitigate bias while male stereotypes amplify it. Under neutral sentiment, female stereotypes still mitigate strongly, whereas male stereotypes have a weaker amplifying effect. Under positive sentiment, most models except R1-7B show mitigation even when male stereotypes are present. The paper relates this to the women-are-wonderful effect, suggesting that positive evaluation may generally favor female-coded portrayals (Masoudian et al., 5 Aug 2025).

5. Alignment with psychological ground-truth and model scaling

A distinctive claim of SBS is that model behavior increasingly matches psychological stereotype categories as model size increases. The paper defines alignment as whether the sign of $N_G$7 matches the literature’s gender association for the attribute (Masoudian et al., 5 Aug 2025). This makes the benchmark not only a measurement tool for bias magnitude, but also a tool for testing whether model bias is structured in ways that resemble established psychological stereotype inventories.

Across the 24 gendered attributes, mean agreement is reported as 60.1% overall, which the paper states is significantly above random chance at 50%. By model, the average alignment is GPT-4o: 64.7%, R1-70B: 64.5%, GPT4o-mini: 60.5%, R1-32B: 58.3%, and R1-7B: 51.3% and not significant (Masoudian et al., 5 Aug 2025). The model-family comparison is therefore not simply that larger models are “less biased.” Rather, the reported pattern is that larger models are more aligned with psychologically established stereotype structure.

The paper adds a majority-vote sanity check: 19 out of 24 attributes, or 79.1%, show majority agreement between models and the literature. Within the DeepSeek-R1 family, the reported correlation between model size and alignment is Spearman $N_G$8, with a similarly strong correlation for stereotype sensitivity in $N_G$9. The OpenAI family also shows an increase from GPT4o-mini to GPT4o (Masoudian et al., 5 Aug 2025).

This finding is methodologically important because it changes how scaling is interpreted. A plausible implication is that larger models can become more consistent with human social priors, rather than simply neutralizing them. The data support the narrower claim that scaling increases agreement with the benchmark’s psychology-grounded categorization.

6. Relation to adjacent benchmarks, misconceptions, and limitations

StereoBias-Stories sits within a broader ecosystem of stereotype-sensitive evaluation, but its design differs materially from adjacent benchmarks. StereoBias, the sentence-level dataset introduced in “Stereotype Detection as a Catalyst for Enhanced Bias Detection,” contains 5,012 sentences annotated for both Bias vs No Bias and Stereotype vs No Stereotype, with Fleiss’ $G \in \{\text{Male, Female}\}$0 for bias and Fleiss’ $G \in \{\text{Male, Female}\}$1 for stereotype. Its principal result is that joint training on bias and stereotype detection significantly improves bias detection relative to separate training, including encoder-only and QLoRA-tuned decoder-only models (Tomar et al., 2 Jul 2025). SBS does not perform joint classification; instead, it measures how stereotype attributes reshape narrative gender composition. The two benchmarks are therefore complementary rather than interchangeable.

Compared with StereoSet, SBS replaces triplet preference scoring with story-level generation and lexicon-based contribution measurement. Compared with Contextual StereoSet, it does not vary location, year, style, or observer framing, but the contextual sensitivity results of that benchmark indicate that story-like framing can materially change stereotype expression, with 1990 vs 2030 raising stereotype selection in all tested models on that contrast and out-group observer framing shifting it by up to 13 percentage points (Basu et al., 15 Jan 2026). Compared with StereoTales, SBS is narrower in social scope and language coverage: StereoTales studies 10 languages, 79 socio-demographic attributes, and identifies more than 1,500 over-represented associations through contingency-table analysis, Fisher tests, effect-size filtering, and harmfulness judgments (Jeune et al., 11 May 2026).

Several related works also clarify common misconceptions. Biased Tales shows that stories can be age-appropriate and low-toxicity while still encoding strong cultural and gender stereotypes; it reports AoA = 5.86, FKRE = 75.5, and average toxicity = 0.06, alongside substantial stereotype patterns in personalized children’s stories (Rooein et al., 9 Sep 2025). This directly undermines the notion that low toxicity implies low bias. Another misconception is that narrative harms are reducible to isolated offensive sentences. StereoTales instead shows that harmful stereotypes often emerge only at the distributional level after aggregating many outputs (Jeune et al., 11 May 2026).

SBS itself is explicit about its limitations. The analysis is restricted to binary gender only; non-binary and gender-diverse identities are not addressed. Gender contribution is estimated by counting gendered identifiers, which may oversimplify portrayal, agency, or role structure. The benchmark is English-only. The mapping from psychological constructs to prompt attributes is necessarily simplified. The multi-attribute regime is expressive but sparse and harder to interpret. Sentiment labels are lexical rather than fully contextual. Finally, the authors intentionally do not filter offensive or stereotypical content because the dataset is intended for research use rather than deployment (Masoudian et al., 5 Aug 2025).

A broader implication, suggested collectively by SBS and neighboring work, is that stereotype evaluation of LLM-generated stories requires at least three complementary lenses: story-level composition, context sensitivity, and distributional analysis across many generations. SBS contributes the first of these by showing that psychologically grounded prompt attributes can systematically shift the gender balance of open-ended narratives in ways that short-form benchmarks may miss.