BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence

Abstract: Bias audits of LLMs now operate within governance frameworks such as the EU AI Act, making benchmark reliability a security concern in its own right. Many current benchmarks, however, collapse bias into a single scalar from one prompt format and one surface label. This design misses two failure modes that can be exploited without changing model weights. Across prompts, meaning-preserving format changes shift bias endorsement by more than $0.7$ on a fixed statement pool. Within a response, the discrete Selection and free-text Elaboration can take opposing stances, so an apparently clean aggregate may hide substantial internal inconsistency (a ``cancellation trap''). Selection-only and elaboration-only rankings are therefore nearly uncorrelated across eight LLMs (Spearman $ρ= 0.238$, $p = 0.570$): LLaMA3-70B ranks in the middle under selection-only scoring but highest under elaboration-only scoring on the same responses. We introduce \textsc{BiAxisAudit}, a protocol that reports each bias score together with a reliability estimate on two orthogonal axes. The across-prompt axis evaluates each statement under a factorial grid of task format, perspective, role, and sentiment, treating bias as a distribution rather than a point estimate. The within-response axis uses Split Coding to recover Selection and Elaboration as separate signals, measured by the Inconsistency Rate and Divergence Net Imbalance. Across eight LLMs with $80{,}200$ coded responses each, task format alone explains as much variance as model choice; $63.6\%$ of pooled bias signals (up to $85.2\%$ per model) appear in only one coding layer, and prompt-dimension interactions exceed main effects. The instrument also separates real bias reductions from apparent reductions caused by cross-layer redistribution: some prompt configurations reduce both BER and IR, whereas others suppress only selection-layer bias.

• The paper introduces a dual-axis framework that evaluates LLM bias by quantifying both prompt sensitivity and intra-response divergence.
• The methodology decomposes responses into Selection and Elaboration layers, utilizing metrics like BER, IR, and DNI to capture bias variability.
• Empirical results reveal significant prompt and layer effects, emphasizing the need for multidimensional bias evaluation in LLM auditing.

BiAxisAudit: A Dual-Axis Framework for Robust LLM Bias Evaluation

Introduction and Motivation

Conventional auditing protocols for LLM bias operationalize bias as a unidimensional property, typically reporting a single scalar—commonly the rate at which a model selects a stereotyped option for a fixed prompt template. This paradigm, as instantiated in CrowS-Pairs, StereoSet, BBQ, and their descendants, overlooks two structurally orthogonal modes of unreliability: (1) prompt sensitivity, where the measured bias varies drastically with prompt format even for semantically identical statements, and (2) response-layer divergence, wherein the explicit answer (Selection) and free-text explanation (Elaboration) within the same response can encode contradictory stances. The authors of "BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence" (2605.09041) rigorously dissect these limitations and propose a measurement-theoretically principled, adversarially robust audit instrument that transforms LLM bias assessment into a two-axis protocol.

Methodology

Two-Axis Audit Structure

BiAxisAudit implements bias evaluation along two axes:

• Across-Prompt Axis: Each stereotype statement is embedded into a factorial grid of prompt conditions, systematically varying task format, perspective, role, and sentiment. The audit therefore generates distributions over (rather than points estimates of) bias scores.
• Within-Response Axis: Each response is analytically decomposed into Selection (structured discrete choice, e.g., forced choice, rating) and Elaboration (free-text reasoning). Both channels are labeled separately; divergence between them is quantified.

Core Metrics

The framework introduces the following metrics:

• BER (Bias Endorsement Rate): Calculated independently on the Selection layer (BER_sel), Elaboration layer (BER_elab), both jointly (BER_cor), and their union (BER_union). BER_union serves as the deployment metric, giving the proportion of responses where at least one layer exhibits stereotyped endorsement.
• Inconsistency Rate (IR): The mean Hamming distance between binary bias indicators from the Selection and Elaboration layers. Quantifies the magnitude of intra-response disagreement.
• Divergence Net Imbalance (DNI): Measures the signed direction of disagreement (BER_sel - BER_elab), indicating whether audits are more likely to over- or under-estimate bias depending on which layer is used.

These metrics are computed on a divergence-eligible base, ensuring auditing evaluates only those responses where both layers emit non-abstain stances.

Experimental Design

Evaluations span eight LLMs (covering closed- and open-source), each scored on 200 stereotype statements (10 social bias axes) × 401 prompt templates (21 OAT conditions × 20 paraphrasings), totaling 80,200 coded responses per model. Coding is conducted by a three-model LLM judge ensemble (majority-vote), rigorously validated for inter-annotator agreement.

Key protocol features:

• Prompt Variation: The factorial prompt grid orthogonalizes the impact of task format, perspective, role, and sentiment.
• Split Coding: Selection and Elaboration are explicitly coded and compared on every divergence-eligible instance.

Empirical Findings

Impact of Prompt Sensitivity

Task format alone explains as much variance in BER as the model choice (e.g., $n^2=0.395$). For a fixed stereotype statement set and a fixed model, BER_union swings from 0.06 to 0.78 depending solely on task format. The duality is stark: e.g., LLaMA3-70B ranks mid-pack under selection-only BER but is highest under elaboration-only.

Implication: Audit results are non-stable properties of models; they are prompt-model interaction artifacts. Compliance scores (e.g., for the EU AI Act or NIST AI RMF) are sensitive to prompt selection, making the choice of prompt template a security-relevant attack vector (prompt-shopping).

Cross-Layer Disagreement

Split coding exposes a pervasive selection-elaboration dissociation:

• On average, 63.6% of bias signals appear in only one layer across models.
• IR (mean 0.176; up to 0.318 depending on the model) indicates that a single-layer audit typically misrepresents one in five responses.
• Models exhibit both systematic over-estimation (Selection endorses, Elaboration denies) and under-estimation (Selection denies, Elaboration endorses), with cancellation traps (DNI ≈ 0, but IR > 0) making aggregate scores coincidentally appear calibrated while masking structural unreliability.

Contradictory Claim: Rankings by BER_sel and BER_elab are nearly uncorrelated (Spearman $\rho$ = 0.238, $p = 0.570$); there is no reliable scalar summary for bias rankings.

Prompt-Dimension Interactions

Cross-factor interactions—particularly task × sentiment—explain more variance in BER than main effects; e.g., skeptical framing decreases BER_sel by 26pp in one task, yet increases it by 6pp in another. Aggregating scores across prompt configurations does not yield a calibrated measure but pools structurally antagonistic errors.

Bias Mitigation

Factorial analysis of prompt-level interventions (task reformulation, expert-role assignment, sentiment manipulation) under the two-axis metrics distinguishes genuine bias reduction (manifests as concurrent BER and IR decrease) from superficial layer rerouting (apparent BER reduction on one layer, bias persists or shifts to the other). The Pareto-optimal configuration (binary judgment + ai_ethicist role + neutral sentiment) achieves a 96% BER_sel reduction without increasing DNI.

Theoretical and Practical Implications

Audit Security Model: BiAxisAudit formalizes the measurement surface of bias audits under adversarial settings. A vendor need not alter model weights to pass audits—selecting audit protocols/benchmarks is sufficient. Traditional benchmarks are not instrumentally robust and should not be relied on for regulatory evidence or model-card reporting absent dual-axis auditing.

Standardization and Compliance: Regulatory and standards bodies (NIST, EU, ISO/IEC 42001:2023) should not accept single-scalar, single-prompt bias scores as evidence. Robust audit protocols must report distributions across prompt conditions and expose (IR, DNI).

Methodology for Future Interventions: The axes that present attack surfaces (prompt-shopping, coding-layer selection) are also levers for deployment-time bias mitigation. The framework enables fine-grained diagnosis and reliable mitigation selection.

Limitations and Future Directions

• Scope: Evaluation is restricted to English-language text LLMs, 10 bias dimensions, and 401 prompt templates. Extending to multimodal, multilingual, or longitudinal conditions is left open.
• Judging Artifacts: LLM-based judging is not guaranteed to mimic human annotation, though substantial judge agreement is empirically demonstrated.
• Harm Measurement: The protocol quantifies bias signal divergence and audit reliability, not downstream social harm.

Research Trajectories:

• Full factorial prompt cross-analysis, including higher-order interactions, would refine prompt-effect modeling and debiasing strategies.
• Extending the methodology to RLHF-aligned and instruction-tuned models under real-world deployment interfaces.
• Adoption in regulatory compliance pipelines as an auditable, reproducible protocol.

Conclusion

BiAxisAudit fundamentally recasts LLM bias auditing as a bi-dimensional measurement problem—one axis quantifying prompt-induced surface variance, and the other response-layer coherence. It operationalizes benchmark reliability as a first-class security and compliance property, directly exposing and quantifying the extent to which certificate-scale audits can be gamed by prompt-shopping or single-layer coding. The contributions are not new bias rankings, but a robust, reproducible, and actionable diagnostic framework that should serve as the minimal operational standard for trustworthy LLM bias auditing, regulatory reporting, and benchmark artifact release.