VoiceBBQ: Spoken Bias Benchmark

• VoiceBBQ is a benchmark that extends BBQ by introducing controlled acoustic variations to diagnose both content and acoustic biases in spoken language models.
• It employs a 2×2 factorial design over speaker gender and accent, synthesizing 935,872 audio files from 58,492 examples under 16 distinct voice conditions.
• VoiceBBQ offers a framework for isolating bias sources, enabling direct comparisons with text benchmarks and guiding fairer deployments of spoken language models.

Searching arXiv for the cited paper and the original BBQ benchmark to ground the article with primary sources. VoiceBBQ is a spoken extension of BBQ, the Bias Benchmark for Question Answering, designed to measure social bias in spoken LLMs (SLMs) by preserving BBQ’s ambiguous and disambiguated item structure while adding controlled acoustic variation to the context. Its central premise is that bias in SLMs can arise from two distinct sources: the content aspect, corresponding to stereotypes embedded in linguistic content and inherited from the backbone LLM, and the acoustic aspect, corresponding to sensitivity to speaker-specific attributes such as gender presentation and accent. By converting each BBQ context into multiple spoken renditions while keeping the question and answer choices in text, VoiceBBQ preserves item semantics and original BBQ scoring, while enabling joint diagnosis of content-axis and acoustic-axis bias (Choi et al., 25 Sep 2025).

1. Motivation and conceptual scope

VoiceBBQ is motivated by the need for social-bias evaluation in end-to-end SLMs. As speech becomes a primary interface for AI systems, bias can manifest immediately in interactive settings and cause harm. Prior work had focused either on text-only LLMs or on task-specific speech systems such as ASR and TTS, leaving end-to-end SLMs comparatively underdiagnosed. The benchmark therefore targets a gap that is specific to speech-based inference: identical semantic content can be delivered through different voices, and those acoustic differences may alter model behavior even when the underlying linguistic signal is unchanged (Choi et al., 25 Sep 2025).

The benchmark inherits BBQ’s two evaluation regimes. In ambiguous contexts, there is insufficient information and an unbiased system should select “UNKNOWN.” In disambiguated contexts, there is sufficient information for a unique correct answer $y_i^*$. VoiceBBQ preserves this structure verbatim from BBQ and adds controlled acoustic conditions to the spoken context only. This makes it possible to ask two analytically separate questions: whether a model reproduces social stereotypes from content, and whether its predictions vary when only speaker acoustics change.

A central contribution is thus not merely a speech version of an existing benchmark, but a framework for separating content bias from acoustic bias without altering the canonical BBQ item semantics. The paper characterizes VoiceBBQ as a compact, drop-in testbed for jointly diagnosing these two sources of bias across spoken LLMs.

2. Dataset design, controlled acoustic factors, and validation

VoiceBBQ converts all BBQ items into speech at the level of the context alone; the question and the three answer choices remain textual in order to preserve comparability and avoid added acoustic confounds in short spans. The source set comprises 58,492 BBQ examples rendered into speech. Each item is synthesized under 16 acoustic conditions, yielding 935,872 audio files in total. Files are stored as 24 kHz WAV and organized by BBQ category, with filenames including item IDs and speaker IDs. Average context duration is 13.2 s, with a range of 2.9–40.0 s, corresponding to approximately 12.35 million seconds of audio, or about 3,432 hours (Choi et al., 25 Sep 2025).

The text-to-speech pipeline uses Kokoro-TTS, described as StyleTTS2-based and multispeaker, with no fine-tuning. Synthesis is sentence-level via nltk.sent_tokenize, and waveform concatenation is performed via numpy to form the full spoken context. Synthesis and concatenation were run on an NVIDIA A6000 ADA GPU under PyTorch 2.2.1+cu118. Prosody, speaking rate, pitch, and timbre are not explicitly parameterized beyond the voice identity provided by Kokoro’s speakers. No background noise, microphone variation, or SNR manipulation is introduced; the benchmark is intentionally studio-clean to isolate gender and accent as the primary acoustic axes.

The controlled voice conditions form a $2 \times 2$ factorial over speaker gender presentation and accent, with four distinct voices per cell to average over timbre and idiosyncrasies:

• American Male: am_puck, am_eric, am_liam, am_adam
• American Female: af_heart, af_sarah, af_nova, af_alloy
• British Male: bm_george, bm_fable, bm_lewis, bm_daniel
• British Female: bf_emma, bf_isabella, bf_alice, bf_lily

The content dimensions are inherited from BBQ’s 11 sensitive domains: age, disability status, gender identity, nationality, physical appearance, race/ethnicity, race×SES, race×gender, religion, SES, and sexual orientation. Labels for stereotypical versus anti-stereotypical answer choices are carried directly from BBQ, with the third choice fixed as “UNKNOWN.”

Validation is automated rather than human-annotated. Using pretrained classifiers, 96.5% of synthesized contexts matched the intended gender presentation, with Female at 100.00% and Male at 93.16%. Accent validation reached 94.8% overall, with GB at 99.49% and US at 90.15%. No human annotator checks or inter-rater agreement are reported; validation is limited to acoustic identity verification to ensure condition integrity. A plausible implication is that the benchmark’s control over accent is stronger for GB than for US, and that residual ambiguity in US-labeled synthesis should be considered when interpreting accent-conditioned effects.

3. Evaluation formalism and fairness diagnostics

VoiceBBQ evaluates free-form model outputs produced from an audio context together with a textual question and textual answer choices. Responses are normalized and mapped to $A/B/C$ via exact match or sequence similarity; if necessary, a dedicated “classification assistant” prompt collapses a generative output to one of $A/B/C$ (Choi et al., 25 Sep 2025).

Let items be indexed by $i \in D$, with ambiguous subset $D_{\mathrm{amb}}$ and disambiguated subset $D_{\mathrm{dis}}$. The “UNKNOWN” option index is $u_i$, fixed as the third choice. For disambiguated items, the correct label is $y_i^*$, and the model prediction is $\hat{y}_i \in \{A,B,C\}$. Each item also has a bias sign variable $2 \times 2$0, where $2 \times 2$1 if the chosen substantive option is stereotypical and $2 \times 2$2 if anti-stereotypical. For “UNKNOWN,” bias is undefined and excluded from bias computation.

Per-axis accuracy on disambiguated items is defined by category $2 \times 2$3 and acoustic condition $2 \times 2$4. If $2 \times 2$5, then

$2 \times 2$6

Bias on ambiguous items measures stereotype endorsement when the model fails to choose “UNKNOWN”:

$2 \times 2$7

Values lie in $2 \times 2$8. Near 0 indicates no preference; negative values indicate stereotypical preference.

Bias on disambiguated items is computed only over mistakes:

$2 \times 2$9

Again, $A/B/C$0 means all wrong answers are stereotypical, $A/B/C$1 means all wrong answers are anti-stereotypical, and 0 indicates no directional preference.

To isolate acoustic effects, the same bias metrics are restricted to a condition $A/B/C$2, yielding $A/B/C$3 and $A/B/C$4. Contrasts across two acoustic settings $A/B/C$5 and $A/B/C$6 are

$A/B/C$7

$A/B/C$8

and category-specific accuracy contrasts are

$A/B/C$9

Voice invariance for fixed content is captured through consistency. For item $A/B/C$0 with acoustic renders $A/B/C$1, one definition is

$A/B/C$2

An alternative item-level invariance fraction is

$A/B/C$3

with $A/B/C$4.

Statistical sensitivity to acoustics is tested using McNemar’s test. For paired outcomes across two acoustic conditions, with $A/B/C$5 the number of items correct in $A/B/C$6 but incorrect in $A/B/C$7, and $A/B/C$8 the reverse, the continuity-corrected statistic is

$A/B/C$9

VoiceBBQ reports significance using the thresholds $i \in D$0, $i \in D$1, and $i \in D$2. In addition, Pearson correlation

$i \in D$3

is used to quantify content-bias inheritance from the backbone LLM, reported separately for ambiguous and disambiguated settings across categories.

The paper defines the expected behavior of an unbiased model as follows: in ambiguous items, high “UNKNOWN” selection, $i \in D$4, and small acoustic contrasts; in disambiguated items, high accuracy, $i \in D$5 among errors, small $i \in D$6 and $i \in D$7, and high consistency across voices.

4. Experimental setup and evaluated spoken LLMs

Two spoken LLMs are evaluated. LLaMA-Omni is described as a modular SLM consisting of a frozen Whisper-large-v3 encoder, a lightweight speech adapter, and a LLaMA 3.1 LLM, specifically the meta-llama/Llama-3.1-8B-Instruct backbone family. It operates on audio plus text, with decoding configured as do_sample=False, num_beams=1, top_p=None, max_new_tokens=1024, batch size 1, and single-turn inference. Qwen2-Audio-7B-Instruct is described as an end-to-end model integrating a Whisper-initialized audio encoder trained jointly with the QwenLM backbone. It also uses audio plus text, via AutoProcessor and Qwen2AudioForConditionalGeneration, with model.generate(max_length=1024), batch size 1, and single-turn inference (Choi et al., 25 Sep 2025).

Both systems are evaluated on single NVIDIA A6000-class GPUs in Python 3.9–3.10 using Hugging Face Transformers. The evaluation uses direct audio understanding; no external ASR is inserted, and each model processes audio features through its integrated encoder. Answer extraction follows a two-stage normalization procedure: stripping non-alphanumerics and performing exact or approximate string matching to the answer choices, optionally followed by a prompt-based classifier that maps the response to $i \in D$8.

The benchmark’s ablations are voice-based rather than architecture-internal. Acoustic-condition ablations are implemented via the 16 TTS voices, allowing comparisons across gender and accent and their four-speaker averages per cell. No explicit removal of acoustic cues is performed beyond these controlled conditions; there is no noise insertion and no encoder replacement. This setup makes the reported differences attributable to controlled changes in speaker presentation rather than to perturbation-induced degradation.

5. Empirical results and architectural contrasts

VoiceBBQ reports that Qwen2-Audio exhibits stronger content-bias alignment with its backbone than LLaMA-Omni does with its own. The Pearson correlation with the backbone for the Qwen family is $i \in D$9 on ambiguous items and $D_{\mathrm{amb}}$0 on disambiguated items, both with $D_{\mathrm{amb}}$1. For the LLaMA family, the corresponding values are $D_{\mathrm{amb}}$2 on ambiguous items with $D_{\mathrm{amb}}$3, and $D_{\mathrm{amb}}$4 on disambiguated items, reported as non-significant (Choi et al., 25 Sep 2025).

Selected category-level bias values illustrate the difference in content-axis behavior. For LLaMA-Omni, ambiguous/disambiguated bias examples include Age $D_{\mathrm{amb}}$5, Disability $D_{\mathrm{amb}}$6, Gender identity $D_{\mathrm{amb}}$7, Nationality $D_{\mathrm{amb}}$8, Race/Ethnicity $D_{\mathrm{amb}}$9, and SES $D_{\mathrm{dis}}$0. For Qwen2-Audio, reported values include Age $D_{\mathrm{dis}}$1, Disability $D_{\mathrm{dis}}$2, Gender identity $D_{\mathrm{dis}}$3, Nationality $D_{\mathrm{dis}}$4, Race/Ethnicity $D_{\mathrm{dis}}$5, Race×SES $D_{\mathrm{dis}}$6, and SES $D_{\mathrm{dis}}$7. Negative values indicate stereotypical preference. Disambiguated accuracy is also markedly different: LLaMA-Omni typically falls in the range 0.53–0.65 across categories, while Qwen2-Audio typically falls in 0.75–0.93, indicating stronger content understanding.

The paper interprets these results as showing that Qwen2-Audio largely inherits backbone biases while being more accurate, whereas LLaMA-Omni shows lower bias magnitudes than Qwen2-Audio in many categories, likely reflecting its instruction-style speech training data, InstructS2S-200K, which reshapes biases relative to the backbone. This suggests that post-backbone speech training can alter content-bias profiles even when it does not maximize semantic accuracy.

On the acoustic axis, LLaMA-Omni is substantially more sensitive than Qwen2-Audio. For disambiguated items, McNemar statistics for LLaMA-Omni include SES with $D_{\mathrm{dis}}$8 for Gender and $D_{\mathrm{dis}}$9 for Accent, Gender identity with $u_i$0 for Gender and $u_i$1 for Accent, and Race/Ethnicity with $u_i$2 for Gender. Other categories also vary, including Age with $u_i$3. For Qwen2-Audio, the reported McNemar values are mostly small and non-significant; examples include Gender identity with $u_i$4 for Gender and $u_i$5 for Accent, and SES with $u_i$6 for Gender and $u_i$7 for Accent.

Bias-score contrasts reinforce this asymmetry. For LLaMA-Omni, disambiguated bias shifts reported in percentage points include Race×SES with $u_i$8 for Gender and $u_i$9 for Ambiguous, SES with $y_i^*$0 for Gender and $y_i^*$1 for Ambiguous, Religion with $y_i^*$2 for Gender, and Nationality with $y_i^*$3 for Gender and $y_i^*$4 for Ambiguous. Several additional categories show 1–3% shifts, and accent effects are also present, though somewhat distinct from gender effects. By contrast, Qwen2-Audio’s $y_i^*$5 values are near zero across categories and regimes, consistent with dampened acoustic sensitivity.

The paper attributes these contrasts to architecture. For LLaMA-Omni, the frozen Whisper encoder and lightweight adapter are hypothesized to preserve residual acoustic information that can leak into decisions, producing stronger gender and accent sensitivity in both bias and choice variability. For Qwen2-Audio, joint training of the audio encoder with the LLM is hypothesized to encourage representations that are less tied to speaker attributes, thereby reducing acoustic bias while maintaining content-driven tendencies. The proposed mechanism is explicitly presented as a hypothesis rather than as a demonstrated causal account.

Common failure modes also differ. On ambiguous items, a recurrent error is failure to choose “UNKNOWN,” often with a tendency toward stereotypical substantive choices, reflected in negative $y_i^*$6. On disambiguated items, LLaMA-Omni’s wrong answers are more likely to be stereotypical in certain categories and more likely to vary across voices, while Qwen2-Audio’s wrong-answer bias is more stable. Gender identity, SES, and Race/Ethnicity show particularly pronounced voice-conditioned variability in LLaMA-Omni. Noise robustness is not evaluated, because all audio is clean TTS.

6. Comparability, limitations, and practical use

VoiceBBQ is explicitly designed to remain compatible with BBQ. Items, categories, ambiguity regimes, and scoring are preserved, and only the context is spoken while questions and answers remain textual. Bias definitions and free-form output mapping to $y_i^*$7 follow the official BBQ protocol. The benchmark is therefore intended to support direct comparison of accuracy and bias scores between the spoken benchmark and the original text benchmark (Choi et al., 25 Sep 2025).

Its novelty relative to prior speech-bias evaluation lies in controlled separation of acoustic effects from content effects. The benchmark jointly diagnoses content and acoustic bias through factorial voice conditions, enabling acoustic-isolation tests such as $y_i^*$8 and McNemar analysis while maintaining BBQ comparability. A plausible implication is that VoiceBBQ can function as an audit layer for SLM deployment pipelines that would otherwise rely only on text-based bias benchmarks.

The paper also states several limitations and ethical cautions. TTS identity and timbre may leak unmodeled cues, such as perceived age or sociolect, alongside gender and accent. Accent labeling validity depends on classifier performance, and the 90.15% US accent validation result indicates some residual ambiguity. The benchmark assumes binary gender presentation in TTS labels, while real-world gender identity and presentation are more nuanced. Demographic inference from acoustics is characterized as ethically sensitive, and VoiceBBQ is framed as a diagnostic tool rather than a demographic classifier. More broadly, clean TTS without noise, microphone variation, or SNR shifts may not reflect human-recorded, noisy, or code-switched speech, and only two SLMs are evaluated, so architectural conclusions are indicative rather than definitive. No mitigation method is proposed.

The benchmark is also presented as practically deployable. The recommended workflow is to download the dataset from Hugging Face, select one or more of the 16 voice conditions, present the 24 kHz WAV context to an SLM together with the textual question and textual answer choices, collect the model’s generative answer, normalize and map it to $y_i^*$9, and then compute disambiguated accuracy, $\hat{y}_i \in \{A,B,C\}$0, $\hat{y}_i \in \{A,B,C\}$1, acoustic contrasts, and optionally consistency across the 16 voices. Category-level aggregation is then compared with the reported baselines for LLaMA-Omni and Qwen2-Audio. The dataset includes item metadata, category labels, and pointers to audio files per voice condition, while example scripts are reported in a PyTorch-plus-Transformers environment. The paper does not specify the dataset license in the main text and directs readers to the Hugging Face page.

The practical conclusions reported by the paper are correspondingly specific. Multi-voice renders of identical content should be used to isolate acoustic bias. Reporting $\hat{y}_i \in \{A,B,C\}$2 and McNemar results across gender and accent is recommended when auditing SLMs. End-to-end audio encoders trained to abstract away speaker identity are suggested when the goal is to reduce acoustic bias, whereas freezing external encoders may preserve speaker-dependent features. Instruction-tuning and specialized speech datasets may reshape content bias, so post-training audit remains necessary. For ambiguous items, explicit training to select “UNKNOWN” is recommended, together with monitoring of $\hat{y}_i \in \{A,B,C\}$3 among non-“UNKNOWN” responses. The paper further suggests adversarial or contrastive objectives to suppress speaker-dependent information while preserving semantics, though it does not experimentally evaluate such mitigations.