BIASEDTALES-ML: A Multilingual Dataset for Analyzing Narrative Attribute Distributions in LLM-Generated Stories

Abstract: LLMs are increasingly used to generate narrative content, including children's stories, which play an important role in social and cultural learning. Despite growing interest in AI safety and alignment, most existing evaluations focus primarily on English, leaving the cross-lingual generalization of aligned behavior underexplored. In this work, we introduce BiasedTales-ML, a large-scale parallel corpus of approximately 350,000 children's stories generated across eight typologically and culturally diverse languages using a full-permutation prompting design. We propose a structured generator-extractor pipeline and a multi-dimensional distributional analysis framework to examine how narrative attributes vary across languages, models, and social conditions. Our analysis reveals substantial cross-lingual variability in narrative generation patterns, indicating that distributions observed in English do not always exhibit similar characteristics in other languages, particularly in lower-resource settings. At the narrative level, we identify recurring structural patterns involving character roles, settings, and thematic emphasis, which manifest differently across linguistic contexts. These findings highlight the limitations of English-centric evaluation for characterizing socially grounded narrative generation in multilingual settings. We release the dataset, code, and an interactive visualization tool to support future research on multilingual narrative analysis and evaluation.

• The paper introduces a multilingual dataset and extraction pipeline to analyze narrative biases in LLM-generated children's stories, revealing systematic gender and class stereotypes.
• It employs a controlled, full-permutation prompt design across eight languages using three LLMs and quantifies bias with metrics such as log-probability ratios and Jensen–Shannon Divergence.
• Findings highlight model- and language-dependent differences, underscoring the need for alignment protocols that address linguistic diversity and structural biases in narrative generation.

Multilingual Analysis of Narrative Attribute Distributions in LLM-Generated Stories with BiasedTales-ML

Introduction

The increased integration of LLMs into creative and educational content generation, especially for children, necessitates rigorous analyses of their representational and social biases. While prior work on bias in LLM-generated narratives has predominantly focused on English and short-form text, this paper introduces "BIASEDTALES-ML: A Multilingual Dataset for Analyzing Narrative Attribute Distributions in LLM-Generated Stories" (2604.17008), addressing a significant gap in evaluating how social attributes and biases propagate in long-form narratives across diverse linguistic and cultural contexts. The authors develop a parallel, large-scale corpus paired with a systematic extraction and analysis pipeline, enabling controlled comparison and fine-grained inspection of narrative bias beyond Anglocentric settings.

Dataset Construction and Linguistic Scope

The BiasedTales-ML corpus comprises approximately 350,000 machine-generated children's stories spanning eight typologically diverse languages: English, Chinese, Japanese, Korean, Spanish, Russian, Arabic, and Swahili. Three LLMs—Qwen-3-8B, Llama-3.1-8B, and Llama-3.2-1B—were used for generation, with a full-permutation prompt design instantiating all combinations of nationality, religion, social class, parent role, and child gender.

Key properties of the dataset include:

• Typological and Resource Diversity: The selection of languages targets both presence/absence of grammatical gender and varying resource levels, enhancing typological and sociocultural coverage.
• Controlled Prompt Localization: Prompts were localized to ensure semantic equivalence and naturalness, removing confounds from backtranslation artifacts and preserving language-specific grammatical and cultural conventions.
• Parallel Generation: The structure supports pairwise and cross-lingual comparison under parallel conditioning, facilitating isolation of linguistic versus cultural effects.

  Figure 1: Geographic and typological reach of the BiasedTales-ML dataset, highlighting inclusion of high-resource, gendered, and culturally distinct language regions.

Evaluation Framework and Extraction Pipeline

To dissect narrative attribute distributions, the study deploys a generator–extractor pipeline. Narrative features—protagonist adjectives, setting/environment keywords, and explicit cultural references—are extracted using a high-capacity instruction-following LLM. Extraction precision is validated (Cohen’s $\kappa$ = 0.618, 85.6% precision), confirming robust applicability across the covered languages.

Complementary metrics include:

• Directional Bias (log-probability ratios for feature prevalence under each conditioning variable—e.g., gender, social class).
• Distributional Divergence (Jensen–Shannon Divergence between group-wise attribute distributions).
• Cross-Lingual Consistency (cosine similarity of bias vectors across languages).
• Quality Controls (Valid Story Rate based on language identification and refusal detection).

Cross-Lingual and Model-Level Findings

Gendered and Social Attribute Biases

Analysis of log-probability ratios and lexical fingerprints identifies consistent patterns:

• Male-Conditioned Narratives: Emphasize intellect, agency, and activity—especially in languages with grammatical gender (Arabic, Russian, Spanish).
• Female-Conditioned Narratives: Pronounced communality and relational attributes, with systematic association of domestic and affective environments.

  Figure 2: Radar plots illustrating bias fingerprints for narrative dimensions, with consistent patterns emerging across languages and models.

Grammatical Gender as an Amplifier

Languages with grammatical gender show higher distributional divergence in gendered adjective use, particularly with Llama-3.1-8B, indicating that surface grammatical features amplify representational bias. Qwen-3-8B demonstrates less sensitivity to grammatical gender, signifying model-dependent mediation of such structural effects.

Figure 3: Grammatical gender group languages exhibit higher divergence in gendered narrative descriptors.

Lexical Bias Markers

Log-odds analysis identifies robust, distinctive lexical markers for both gender and social class:

• Male/Working-Class Narratives: Action-oriented, labor-related, and outdoor terms dominate.
• Female/Wealthy Narratives: Relational, domestic, and leisure-oriented language occurs more frequently.

  Figure 4: Distinctive keywords with elevated association by gender and class, partitioned by narrative context.

Model Capacity and Resource Limitations

Comparisons across model sizes reveal specific deficits:

• Llama-3.2-1B: Markedly reduced lexical diversity and flattened attribute distributions, leading to apparent—but misleading—mitigation of distributional bias.
• Low-Resource Languages: Both larger and smaller models exhibit declines in story quality (as measured by Valid Story Rate) and increases in distributional divergence, most pronounced in Swahili.

  Figure 5: Tradeoff between story validity rates and bias strength, with larger models maintaining quality but not fully reducing divergences.

Cross-Lingual Alignment Gaps

Pairwise similarity analyses reveal substantial variation in distributional patterns across languages:

• Qwen-3-8B: Exhibits higher cross-lingual alignment, indicating more stable bias transfer across typologies.
• Llama-3.1-8B: Increased divergence, particularly with lower-resource and typologically distinct languages.

  Figure 6: Cosine similarity matrices for bias vectors; lighter cells denote stronger cross-lingual consistency.

Theoretical Implications and Alignment Considerations

These findings stress that English-centric evaluation of narrative bias substantially underestimates the heterogeneity of LLM behavior. Grammatical structure, training data breadth, and model architecture interact with narrative attribute expression in complex ways, affecting both magnitude and lexical realization of biases. Heritage in the alignment and safety pipeline does not guarantee cross-lingual generalization of distributional properties; results demonstrate "mismatched generalization" and partial decoupling of quality and bias for high-capacity models.

The approach also underscores the inadequacy of static benchmarks in capturing emergent, narrative-level biases, advocating for distributional and structural analysis as complements to surface-level toxicity or stereotype detection.

Practical Relevance and Future Directions

• Dataset and Visualization Tools: Public release of BiasedTales-ML and the Biased Tales Explorer supports programmable and interactive inspection, making intersectional and qualitative analysis tractable for the broader community.
• Alignment Protocols: Results motivate language- and capacity-aware alignment objectives, especially for systems deployed in multilingual educational and media applications.
• Extension to Higher-Order Intersections: The full-permutation design enables, but the current work does not exhaust, systematic investigation of interactions among multiple social attributes.

Limitations

• Language Family Exclusion: Indo-Aryan, African, and Indigenous languages remain unexamined; results may not generalize to all typological spaces.
• Genre Specificity: Focus on children's stories may limit transferability of findings to other narrative genres or discourse settings.
• Extraction Bias: Reliance on LLM-based extraction introduces model-based biases in feature detection.
• Static Attribute Focus: The framework does not account for character interactions or narrative causality.

Conclusion

BiasedTales-ML provides a robust empirical foundation for cross-lingual, distributional analysis of LLM-generated narratives. The framework reveals that social attribute biases are neither monolithic nor English-bound: structural patterns in character roles, settings, and thematics persist, but their instantiation is highly sensitive to linguistic, cultural, and model-specific factors. Evaluations and alignment strategies in generative NLP must evolve to encompass fine-grained, multilingual, and structurally aware approaches to adequately characterize and mitigate narrative bias (2604.17008).