BiasTrace: Linking Reasoning Behaviours to Biased Outputs in LLMs

Abstract: LLMs exhibit social biases that can produce inaccurate and discriminatory inferences, posing risks in high-stakes applications. While prior work has made progress in measuring and mitigating bias, it largely focuses on final outputs of models, with limited understanding of the mechanisms that produce biased outcomes. Recent advances in LLM reasoning offers a new lens for investigating bias, yet the link between reasoning and bias remains poorly understood. Existing approaches focus primarily on final answer correctness or explicitly biased language, overlooking different behaviours in reasoning that can drive biased outcomes. We introduce BiasTrace, an annotation scheme for labelling reasoning behaviours in model-generated traces and linking them to biased outcomes. BiasTrace captures bias-specific behaviours (e.g., unsupported demographic assumptions) as well as general reasoning patterns that may implicitly contribute to bias (e.g. overthinking). We apply BiasTrace to reasoning traces in bias-sensitive contexts, scaled using validated LLM-as-a-judge methods, producing a large annotated dataset. Our analysis shows that biased outputs often stem from subtle reasoning behaviours rather than explicitly biased language, and that reasoning-level annotations improve bias detection. We further show that BiasTrace behaviours can be exploited for inference-time mitigation. These findings underscore the importance of examining a broader range of reasoning patterns to better understand bias in LLMs.

• The paper introduces a validated framework that labels six observable reasoning behaviours across 250,976 traces and links them to stereotype-aligned errors in LLM outputs.
• The paper finds that overthinking is the strongest independent predictor of biased outputs (OR = 2.34–2.47), while group assumptions alone are not significant predictors and outside demographic knowledge increases bias odds.
• The paper shows that reasoning-level bias detection improves evaluation and enables training-free mitigation, with filtered majority voting raising Llama-3-8B-Instruct accuracy from 72.64% to 77.27% while reducing bias from 12.36% to 7.36%.

BiasTrace is an annotation scheme and analysis framework that links observable behaviours in LLM chain-of-thought (CoT) reasoning traces to biased final outputs (2608.14161). The work departs from prior bias evaluation, which concentrates on final answers or explicitly biased language, by decomposing reasoning into fine-grained behaviours — some bias-specific (e.g., unsupported demographic assumptions), others seemingly neutral (e.g., overthinking) — and quantifying which of these predict stereotype-aligned errors. The central empirical claim is that biased outputs frequently arise from subtle reasoning patterns rather than overtly stereotypical language, with overthinking emerging as the strongest predictor.

Motivation and research questions

The authors position the work against a gap in the bias literature: mitigation methods such as RLHF largely train models to avoid explicitly biased language, while biases persist implicitly through other pathways. Existing reasoning-bias studies either evaluate only final-answer correctness or rely on coarse labels for explicitly biased text, and one cited study reports that biases expressed in CoT are not strongly predictive of biased final outputs. BiasTrace addresses two questions: (RQ1) whether reasoning behaviours beyond explicit bias contribute to biased outputs and which are most predictive; and (RQ2) whether these behaviours can improve detection of traces likely to yield biased outcomes. The authors acknowledge that CoT is not guaranteed to be a faithful representation of internal computation, but argue it remains a valuable lens for hypothesis generation and monitoring.

The annotation scheme

The scheme comprises six binary labels across four categories: bias-specific behaviours (Group Assumption — attributing traits to a demographic group without contextual justification; Bias Acknowledgment — recognising potential bias in one's own reasoning); meta-cognitive behaviour (Meta Reflection on task framing or intent); out-of-context knowledge (Outside Demographic Knowledge and Outside Topical Knowledge); and reasoning under uncertainty (Overthinking, operationalised as excessive doubt or revisiting answer options more than three times). The categories are grounded in cognitive science constructs such as motivated reasoning and implicit stereotypes.

The scheme was developed through manual inspection of roughly 40–50 Qwen3-8B/14B traces on BBQ, augmented with Claude Opus 4 analysis of 40 weighted-sampled traces. Two annotators independently labelled 100 traces after guideline refinement, achieving perfect agreement ($\kappa = 1.00$) on most labels after refinement. A validated LLM-as-a-judge (DeepSeek-V3.2) was then used for scalable annotation: strong human agreement was obtained for Outside Demographic Knowledge, Group Assumption, Overthinking, and Meta Reflection ($\kappa = 0.64$–$0.84$), moderate agreement for Bias Acknowledgment and Outside Topical Knowledge ($\kappa = 0.30$–$0.46$), and two labels (Missing Logic, Unresolved) were excluded due to $\kappa < 0.3$. This validation discipline is a stated improvement over prior work using unvalidated annotations at scale.

Dataset and outcome definition

The authors annotate 250,976 reasoning traces: all 31,372 BBQ questions across nine demographic categories, generated by Qwen3-8B, Qwen3-14B, and GPT-OSS-120B under simple and guided prompts (and low/medium reasoning effort for GPT-OSS-120B). A biased outcome is defined strictly as a final answer that is both incorrect and aligned with the question's targeted stereotype, excluding neutral "unknown" errors. Notably, error rates are very low overall (0.16%–2.28%) in ambiguous contexts, yet among those errors, 48.4%–82.7% align with stereotypes — peaking at 100% in some configurations. In disambiguated contexts, error rates are higher (6.98%–29.15%) but mostly neutral. Guided prompts reduce ambiguous-context errors substantially (e.g., Qwen3-14B: 1.30% → 0.16%) while inflating disambiguated errors, indicating that prompt-induced conservatism trades one failure mode for another without eliminating stereotype-aligned ones.

Predictive analysis (RQ1)

On $N = 125{,}487$ Qwen3 traces, logistic regression (pseudo-$R^2 = 0.51$) yields PR-AUC of 0.44 — 60 times above random given the extreme class imbalance (biased outcomes at 0.72%) — and ROC-AUC of 0.98. Key findings:

• Overthinking is the strongest predictor ($OR = 2.34$, $p < .01$ in ambiguous contexts). The authors show this effect survives controls for reasoning length: adding log length to the model leaves overthinking significant ($\kappa = 0.64$0), and a length-residualised overthinking signal remains predictive ($\kappa = 0.64$1, $\kappa = 0.64$2).
• Group Assumption alone is not a significant predictor — a notable result, since this label most closely corresponds to what baseline evaluations measure. Its effect appears only through interaction with ambiguous context ($\kappa = 0.64$3).
• Outside Demographic Knowledge significantly increases bias odds across all prompts and contexts.
• Guided prompting has non-uniform effects: it reduces overall bias odds but amplifies the overthinking effect ($\kappa = 0.64$4 interaction) while suppressing group assumption ($\kappa = 0.64$5). This directly contradicts the assumption that bias-aware prompting uniformly suppresses bias pathways.
• Combinatorial analysis shows Overthinking appears in every high-lift subset; combinations such as overthinking + group assumption reach bias rates of 0.30 with lift exceeding 40 relative to baseline.

Improved bias evaluation (RQ2)

The authors distil the predictive findings into a dedicated evaluation prompt that defines key behaviours, flags high-risk combinations, and requests a binary bias-likelihood score from DeepSeek-V3.2. Although derived from Qwen3 traces, the evaluation signal generalises to GPT-OSS-120B: correlations with biased outcomes range from $\kappa = 0.64$6 to $\kappa = 0.64$7, versus $\kappa = 0.64$8–$\kappa = 0.64$9 for baselines (a 0–5 ordinal scale, binary labelling, and a trained fairness reward model). On COMPAS recidivism classification with GPT-OSS-120B (EO gap 0.28, EOpp gap 0.14), the BiasTrace prompt achieves the highest correlation with per-sample fairness contributions ($0.84$0) and captures 52–62% of top-10–20% fairness-critical samples, though Baseline 0–5 slightly outperforms on the top-5% slice. An ablation isolating prompt structure shows structure alone already outperforms baselines ($0.84$1 vs. $0.84$2–$0.84$3), with the behavioural rubric adding further gain ($0.84$4) — so part of the improvement stems from better task formulation rather than rubric content exclusively.

Inference-time mitigation

The evaluation prompt is repurposed as a filter: for each BBQ question, eight reasoning chains are sampled, and majority voting is restricted to chains flagged as unbiased (Maj-BT), falling back to full majority voting if none qualify. Across four models spanning two families not used in earlier analyses (Qwen3-1.7B/4B, Llama-3.2-3B-Instruct, Llama-3-8B-Instruct), Maj-BT improves both accuracy and bias rate over single sampling and unrestricted majority voting — e.g., Llama-3-8B-Instruct improves from 72.64% accuracy / 12.36% bias (Maj-All) to 77.27% / 7.36%. This demonstrates that reasoning-level signals can be operationalised as a training-free intervention that reduces biased outputs without sacrificing accuracy.

Limitations and open questions

The paper concedes several constraints. It does not claim CoT faithfulness, and results are framed as associations between observable behaviours and outcomes rather than causal mechanisms; the scheme may not transfer to models that reason differently or produce no visible traces. Human validation covered only 100 samples due to resource limits. Evaluation is restricted to open-weight English-LLMs, and the authors note that GPT-OSS-120B does not exhibit overthinking even at medium effort, which explains its lower evaluation correlations — raising the open question of which reasoning–bias relationships generalise across models versus being model-specific. The overthinking threshold (revisiting options more than three times) was chosen manually, and alternative continuous measures remain unexplored. Finally, the behavioural categories were developed on structured multiple-choice QA (BBQ), and the COMPAS fairness linkage is presented only as initial evidence; extending the framework to free-form settings and diverse tasks is left unresolved.

Conclusion

BiasTrace provides a validated, scalable methodology for connecting specific reasoning behaviours to stereotype-aligned outputs, and its strongest empirical contributions are threefold: overthinking predicts biased outcomes independently of verbosity; explicitly stereotypical language (group assumption) is not by itself a significant predictor; and bias-aware prompting suppresses some bias pathways while amplifying others. The downstream demonstrations — improved bias evaluation correlation, identification of fairness-critical COMPAS samples, and training-free inference-time mitigation across unseen model families — indicate practical utility of reasoning-level annotations. The work's scope remains bounded by structured QA benchmarks, English-only evaluation, and the interpretability caveat inherent to any CoT-based analysis.