---
title: 'StereoBias-Stories: Evaluating Gender Stereotypes'
url: https://www.emergentmind.com/topics/stereobias-stories
type: topic
---

# StereoBias-Stories: Evaluating Gender Stereotypes

StereoBias-Stories, often abbreviated **SBS**, is a benchmark for analyzing **gender bias in open-ended story generation** by large language models. It was introduced to move beyond short-form bias probes such as sentence completion, coreference, and multiple-choice question answering, and instead to measure how stereotypes shape the **overall composition of a generated narrative**: who the story appears to be about, which gendered characters are introduced, and how psychologically grounded stereotype attributes shift the balance of male- and female-coded references. SBS contains about **148,082 generated stories** from **five LLMs**, organized across **No-Attribute**, **Single-attribute**, **Two-attribute**, and **Multi-attribute** prompting settings, and uses a story-level gender-contribution metric rather than a token-level stereotype preference score [2508.03292].

## 1. Conceptual setting within stereotype and bias evaluation

StereoBias-Stories emerged in response to a limitation in earlier bias evaluation: many established benchmarks assess whether a model prefers a stereotypical completion over an anti-stereotypical or unrelated alternative, but do so in relatively short or fixed formats. **StereoSet**, for example, measures stereotypical bias through **Context Association Tests (CATs)** in **intrasentence** and **intersentence** form, with metrics such as **Language Modeling Score**, **Stereotype Score**, and **ICAT** [2004.09456]. Cross-lingual extensions of StereoSet further showed that stereotypical bias varies across English, German, French, Spanish, and Turkish, and that English-only evaluation misses important differences [2307.07331]. **Contextual StereoSet** then demonstrated that stereotype selection changes substantially when the same item is reframed by **location**, **year**, **style**, or **observer framing**, indicating that measured bias is highly context-conditional rather than a fixed scalar model property [2601.10460].

SBS adopts a different unit of analysis. Instead of asking whether a model picks a stereotypical continuation in a controlled contrast set, it asks how **open-ended narrative generation** changes when prompted with stereotype-associated attributes. This places it near other story-centered evaluations, but with a distinct emphasis on **psychology-grounded gender stereotypes** rather than multilingual stereotype discovery or sociocultural personalization. In parallel, **StereoBias** introduced a sentence-level dataset of **5,012 sentences** annotated for both **bias** and **stereotype** labels to test whether stereotype detection can act as an auxiliary signal for bias detection [2507.01715]. SBS and StereoBias therefore address related phenomena at different levels: the former studies **narrative generation**, while the latter studies **sentence-level classification**.

The broader literature reinforces the importance of this shift. **StereoTales** studies multilingual open-ended stereotype discovery by aggregating attribute distributions over more than **650,000 generated stories** across **10 languages**, showing that harmful stereotypes emerge as statistically over-represented associations rather than as single offensive sentences [2605.10442]. **Biased Tales** similarly shows that children’s stories can be low-toxicity yet still embed strong cultural and gender stereotypes, including a **55.26%** increase in appearance-related attributes when the protagonist is described as a girl rather than a boy [2509.07908]. Together, these results situate SBS within a movement from **fixed contrast testing** toward **distributional and narrative bias analysis**.

## 2. Benchmark design, prompting regimes, and attribute inventory

SBS contains short stories generated by **GPT-4o-mini**, **GPT-4o**, **DeepSeek-R1-7B**, **DeepSeek-R1-32B**, and **DeepSeek-R1-70B** under four prompting settings [2508.03292]. The benchmark is based on **25 personality/behavior traits associated with gendered expectations**, together with **3 story-ending conditions**, yielding a **28-item pool** from which conditioning attributes are sampled.

The four prompting settings are as follows:

| Setting | Description | Size |
|---|---|---:|
| No-Attribute | Simple story prompt with no stereotype conditioning | 28,668 |
| Single-attribute | One random attribute is added | 16,661 |
| Two-attribute | Two random attributes are sampled | 19,539 |
| Multi-attribute | Six random attributes are sampled | 83,214 |

The paper notes that during each generation setting, **20% of the data is No-Attribute control**, so unconditioned baselines are embedded throughout the benchmark. Prompts are deliberately simple and child-oriented, inspired by TinyStories-style generation: the model is asked to write a short story using very simple words, understandable by a **3-year-old child**, usually in **1–2 paragraphs**. The authors also state that they intentionally **did not add a persona** so that the model could distribute gender as freely as possible [2508.03292].

The stereotype inventory is organized into **female-associated traits**, **male-associated traits**, and a smaller **neutral / less stereotypically gendered or used for diversification** subset. Female-associated traits include **Caring**, **Communication**, **Empathetic**, **Gossiping**, **Happy Ending**, **Indecisive**, **Intuitive**, **Over-emotional**, **Resilient**, **Self-Sacrificing**, **Sensitivity**, and **Supportive**. Male-associated traits include **Aggressive**, **Assertiveness**, **Bad Ending**, **Emotionally Suppressed**, **Leadership**, **Logic**, **Manipulative**, **Obligation**, **Overbearing**, **Reckless**, **Strategic Thinking**, and **Tyrannical**. The neutral or control-oriented items are **Neglectful**, **Guardian**, **Mentorship**, and **Neutral Ending** [2508.03292].

Each attribute is additionally labeled by **lexical sentiment** as **positive**, **negative**, or **neutral**. The paper is explicit that these sentiment labels are an analytical lens and are “lexical and somewhat simplified.” Sampling is not purely random: attributes are sampled to keep **gender stereotype and sentiment appearance balanced**, and the benchmark avoids contradictory combinations such as **over-emotional vs. emotionally suppressed** and **bad ending vs. neutral ending** [2508.03292].

Generation used **temperature = 0.7**, **beam size = 2**, and **max new tokens = 3000** for DeepSeek-R1 models. **R1-7B** was run locally on a **single Nvidia 3090 RTX** in float16, while OpenAI models and the other DeepSeek-R1 models were accessed through APIs [2508.03292].

## 3. Story-level measurement and dataset validation

The central methodological contribution of SBS is its **gender contribution** metric. Rather than scoring whether a model chooses a stereotype over an anti-stereotype in a fixed item, the benchmark counts **gendered mentions in the generated story text itself**. The authors construct a lexicon of **14,255 gendered terms** in total, later described in the appendix as **7,307 male words** and **6,948 female words** [2508.03292].

For a generated story, the contribution of each gender is defined as

\[
C_G = \frac{N_G}{\sum_G N_G}
\]

where \(N_G\) is the number of gendered mentions for gender \(G \in \{\text{Male, Female}\}\). The main bias metric is then

\[
Gap = C_{Male} - C_{Female}
\]

with \(Gap = 0\) indicating balanced gender contribution, \(Gap > 0\) indicating male over-representation, and \(Gap < 0\) indicating female over-representation. To make conditioning effects comparable across models, SBS further defines

\[
\Delta_{Gap_m} = Gap - \mu_{Gap^{No-Attribute}_m}
\]

where \(\mu_{Gap^{No-Attribute}_m}\) is the average unconditioned gap for model \(m\). Positive \(\Delta_{Gap}\) means that conditioning shifts the story further toward male representation, while negative \(\Delta_{Gap}\) means that conditioning shifts it toward female representation [2508.03292].

Stories with no gender identifiers are excluded from the gap analysis. The paper reports that this was rare: only **127 stories in R1-7B** lacked any gendered terms [2508.03292].

The benchmark also includes a dataset-quality evaluation. Lexical measures include **Perplexity** as a fluency proxy, **N-gram diversity**, and a **Redundancy ratio** defined as

\[
RR = 1 - \frac{|N_u|}{|N_t|}
\]

where \(N_u\) is the number of unique sentences and \(N_t\) is the total number of sentences. Human evaluation recruited **58 Prolific participants** and sampled **280 stories** from the **Single-attribute** setting, with **2 samples per attribute per model** and **5 stories** rated per participant. Annotators rated **overall story quality** and **attribute expression** on **1–5 Likert scales**. A second quality check asked models to rate **700 samples**, using the same five models as evaluators except that **R1-32B** was unavailable and replaced by **Qwen-32B**. The paper reports that higher-quality models generally receive better ratings and that **GPT-4o aligns best with human judgments**, supporting the validity of the dataset [2508.03292].

## 4. Main empirical findings on gender bias in stories

The benchmark’s first major finding is that **unconditioned prompts are strongly male-biased**. In the **No-Attribute** condition, most models produce stories with more male identifiers than female identifiers. From the single-attribute plot, the paper reports an average male contribution of about **0.61** and an average female contribution of about **0.39** across models. The table of gender gaps gives positive unconditioned \(Gap\) values for most models: **R1-32B: 0.426**, **R1-70B: 0.386**, **GPT4o-mini: 0.468**, and **GPT4o: 0.432**. **R1-7B** is an exception with a slightly negative baseline gap of **-0.057**, but the paper treats this cautiously because that model also performs worst on story quality and attribute adherence [2508.03292].

The second major finding is that **conditioning on non-gender stereotypes mitigates male bias**. Single-attribute conditioning lowers the gap for most models, two-attribute conditioning lowers it further, and multi-attribute conditioning still remains less biased than the No-Attribute baseline, although the paper describes the multi-attribute trend as less clean because of sparsity and complexity [2508.03292]. This is a notable result because the conditioning attributes are not explicit gender instructions; they are personality, behavior, or structural cues.

The third finding is that **stereotype direction matters**. Using the psychology-grounded labels, the paper shows that **male-coded attributes** tend to increase male representation, while **female-coded attributes** tend to decrease male representation and thus increase female representation. Examples include male-associated prompts such as **aggressive**, **leadership**, and **reckless**, which generally push stories toward male identifiers, and female-associated prompts such as **empathetic**, **caring**, and **supportive**, which generally push stories toward female identifiers. The paper states that this effect is statistically significant for most stereotypes, with one-sample t-tests usually giving **\(p < 0.01\)** against a zero baseline [2508.03292].

A fourth result is the **compositionality of stereotype pressure**. When models receive **two female stereotypes together**, the mitigating effect becomes stronger; when they receive **two male stereotypes together**, the male-amplifying effect becomes stronger; and when they receive mixed-gender combinations, the effects can cancel each other out. The authors extend this observation to the six-attribute regime as well, while noting that the multi-attribute setting is sparser and harder to interpret [2508.03292].

Sentiment also interacts with gender bias. Under **negative sentiment**, female stereotypes mitigate bias while male stereotypes amplify it. Under **neutral sentiment**, female stereotypes still mitigate strongly, whereas male stereotypes have a weaker amplifying effect. Under **positive sentiment**, most models except **R1-7B** show mitigation even when male stereotypes are present. The paper relates this to the **women-are-wonderful effect**, suggesting that positive evaluation may generally favor female-coded portrayals [2508.03292].

## 5. Alignment with psychological ground-truth and model scaling

A distinctive claim of SBS is that model behavior increasingly matches **psychological stereotype categories** as model size increases. The paper defines **alignment** as whether the sign of \(\Delta_{Gap}\) matches the literature’s gender association for the attribute [2508.03292]. This makes the benchmark not only a measurement tool for bias magnitude, but also a tool for testing whether model bias is structured in ways that resemble established psychological stereotype inventories.

Across the **24 gendered attributes**, mean agreement is reported as **60.1% overall**, which the paper states is significantly above random chance at **50%**. By model, the average alignment is **GPT-4o: 64.7%**, **R1-70B: 64.5%**, **GPT4o-mini: 60.5%**, **R1-32B: 58.3%**, and **R1-7B: 51.3% and not significant** [2508.03292]. The model-family comparison is therefore not simply that larger models are “less biased.” Rather, the reported pattern is that larger models are **more aligned with psychologically established stereotype structure**.

The paper adds a majority-vote sanity check: **19 out of 24 attributes**, or **79.1%**, show majority agreement between models and the literature. Within the **DeepSeek-R1** family, the reported correlation between model size and alignment is **Spearman \(\rho = 0.98\)**, with a similarly strong correlation for stereotype sensitivity in \(\Delta_{Gap}\). The OpenAI family also shows an increase from **GPT4o-mini** to **GPT4o** [2508.03292].

This finding is methodologically important because it changes how scaling is interpreted. A plausible implication is that larger models can become **more consistent with human social priors**, rather than simply neutralizing them. The data support the narrower claim that scaling increases agreement with the benchmark’s psychology-grounded categorization.

## 6. Relation to adjacent benchmarks, misconceptions, and limitations

StereoBias-Stories sits within a broader ecosystem of stereotype-sensitive evaluation, but its design differs materially from adjacent benchmarks. **StereoBias**, the sentence-level dataset introduced in “Stereotype Detection as a Catalyst for Enhanced Bias Detection,” contains **5,012 sentences** annotated for both **Bias vs No Bias** and **Stereotype vs No Stereotype**, with **Fleiss’ \(\kappa = 0.6239\)** for bias and **Fleiss’ \(\kappa = 0.7714\)** for stereotype. Its principal result is that **joint training on bias and stereotype detection significantly improves bias detection** relative to separate training, including encoder-only and QLoRA-tuned decoder-only models [2507.01715]. SBS does not perform joint classification; instead, it measures how stereotype attributes reshape narrative gender composition. The two benchmarks are therefore complementary rather than interchangeable.

Compared with **StereoSet**, SBS replaces triplet preference scoring with story-level generation and lexicon-based contribution measurement. Compared with **Contextual StereoSet**, it does not vary **location**, **year**, **style**, or **observer framing**, but the contextual sensitivity results of that benchmark indicate that story-like framing can materially change stereotype expression, with **1990 vs 2030** raising stereotype selection in all tested models on that contrast and **out-group observer framing** shifting it by up to **13 percentage points** [2601.10460]. Compared with **StereoTales**, SBS is narrower in social scope and language coverage: StereoTales studies **10 languages**, **79 socio-demographic attributes**, and identifies more than **1,500 over-represented associations** through contingency-table analysis, Fisher tests, effect-size filtering, and harmfulness judgments [2605.10442].

Several related works also clarify common misconceptions. **Biased Tales** shows that stories can be **age-appropriate and low-toxicity** while still encoding strong cultural and gender stereotypes; it reports **AoA = 5.86**, **FKRE = 75.5**, and **average toxicity = 0.06**, alongside substantial stereotype patterns in personalized children’s stories [2509.07908]. This directly undermines the notion that low toxicity implies low bias. Another misconception is that narrative harms are reducible to isolated offensive sentences. StereoTales instead shows that harmful stereotypes often emerge only at the **distributional level** after aggregating many outputs [2605.10442].

SBS itself is explicit about its limitations. The analysis is restricted to **binary gender only**; non-binary and gender-diverse identities are not addressed. Gender contribution is estimated by **counting gendered identifiers**, which may oversimplify portrayal, agency, or role structure. The benchmark is **English-only**. The mapping from psychological constructs to prompt attributes is necessarily simplified. The **multi-attribute** regime is expressive but sparse and harder to interpret. Sentiment labels are **lexical** rather than fully contextual. Finally, the authors intentionally do not filter offensive or stereotypical content because the dataset is intended for research use rather than deployment [2508.03292].

A broader implication, suggested collectively by SBS and neighboring work, is that stereotype evaluation of LLM-generated stories requires at least three complementary lenses: **story-level composition**, **context sensitivity**, and **distributional analysis across many generations**. SBS contributes the first of these by showing that psychologically grounded prompt attributes can systematically shift the gender balance of open-ended narratives in ways that short-form benchmarks may miss.

Source: https://www.emergentmind.com/topics/stereobias-stories