RePOPE-Spk: Speech Query Hallucination Benchmark

• RePOPE-Spk is a multimodal benchmark that extends RePOPE by converting text questions into spoken queries to study object hallucination.
• It demonstrates that switching from text to speech increases hallucination errors by 3% under clean audio and up to 20% under noisy conditions.
• Controlled experiments on noise levels, query length, and modality order show that speech input acts as a reliability stressor for multimodal models.

RePOPE-Spk is a speech-augmented benchmark for measuring object hallucination in multimodal LLMs when the query is spoken rather than typed. It is introduced as an extension of RePOPE, itself a corrected version of the earlier POPE benchmark for visual object hallucination, and is designed to isolate the effect of changing only the query modality while keeping the underlying image-question benchmark fixed. The benchmark targets multimodal systems that can jointly accept image, audio, and text inputs, and its central empirical claim is that spoken queries make multimodal hallucination worse: under clean speech the reported error increase is about 3%, and under environmental noise it rises to as much as 20% (Park et al., 19 Sep 2025).

1. Benchmark lineage and task definition

RePOPE-Spk inherits the structure of RePOPE exactly. RePOPE asks yes/no-style questions about whether objects are present in an image, with carefully selected absent objects to probe false positive tendencies. RePOPE-Spk preserves the same images, questions, and evaluation task, but converts the written questions into spoken audio so that text-query and spoken-query conditions can be compared under a controlled modality substitution (Park et al., 19 Sep 2025).

The inherited benchmark contains three subsets. The Adversarial subset has 2,684 QA pairs, the Popular subset has 2,727 QA pairs, and the Random subset has 2,774 QA pairs. The subsets differ in how the absent object is selected. In Random, objects are sampled randomly from the vocabulary. In Popular, they are frequent objects in the dataset that are not present in the current image. In Adversarial, absent objects are selected by co-occurrence statistics with the image’s true objects, making them likely distractors that a model may wrongly infer (Park et al., 19 Sep 2025).

A recurrent misconception is to treat RePOPE-Spk as a new image benchmark from scratch. The benchmark instead changes only the query modality. This matters methodologically because the construction aims to attribute performance differences specifically to speech input rather than to changes in image selection or QA design. The reported degradation is therefore interpreted as a property of multimodal spoken-query processing rather than of a different visual benchmark.

2. Construction of the spoken benchmark

All text questions from RePOPE are converted into speech using a high-quality TTS model. The paper specifies Parler-TTS, a sampling rate of 16 kHz, and a consistent speaker profile. The speech generation was controlled to ensure natural prosody, clear articulation, minimal confounds from speaker variability, and avoidance of unnatural artifacts or mispronunciations. This design choice is central to the benchmark’s internal validity: the intention is for hallucination changes to reflect the spoken modality itself rather than poor synthesis quality or uncontrolled speaker variation (Park et al., 19 Sep 2025).

RePOPE-Spk includes both clean and noisy spoken conditions. Clean speech is a direct TTS rendering of the original text question with no added noise. Noisy speech is generated by injecting environmental noise from the ESC-50 dataset. The paper names five categories used in the main experiments: Animals, Natural, Human, Interior, and Exterior. Speech and noise are mixed at two explicit signal-to-noise ratios, +5 dB and +0 dB, with lower SNR producing stronger corruption (Park et al., 19 Sep 2025).

Two further controlled variations are built into the benchmark. The first is modality order, written as $I \rightarrow S$ for image followed by spoken query and $S \rightarrow I$ for spoken query followed by image. The second is query length, with conditions using the original duration or versions padded to 5 s and 10 s. These perturbations let the benchmark probe not only acoustic corruption but also multimodal sequencing and temporal robustness.

3. Experimental protocol and reported metrics

The main evaluation covers one proprietary and one open-source model: Gemini-1.5-Flash and Gemma-3n. For chain-of-thought experiments, the paper additionally uses Gemini-2.5-Flash. The benchmark task remains RePOPE’s object hallucination evaluation: the model receives an image and a question about object presence, and performance is scored as a binary classification problem (Park et al., 19 Sep 2025).

The reported metrics are Accuracy (A), Recall (R), Precision (P), and F1. The paper does not provide explicit formulas for these metrics and does not define a separate hallucination-rate equation. Instead, increased hallucination is inferred operationally from degraded classification performance, especially from precision loss, because false positive object assertions reduce precision. This is an important interpretive detail: the headline claims about hallucination growth are high-level summaries, whereas the tables themselves report only $A$, $R$, $P$, and $F1$.

The experimental scope is also structured. Tables 1 and 2 evaluate across all three RePOPE subsets. All other experiments, unless otherwise noted, use the Adversarial subset with Natural-category noise conditions. The paper evaluates two mitigation families in addition to the base benchmark. On Gemini-1.5-Flash, it studies 1-shot, 5-shot, and 10-shot many-shot prompting under clean audio, +5 dB noise, and +0 dB noise. On Gemini-2.5-Flash, it compares Zero-shot (text) and Zero-shot (+0 dB) with four chain-of-thought-style prompts: Step-by-Step, Visual-Description, Speech-Description, and Cross-Modal (Park et al., 19 Sep 2025).

4. Quantitative behavior under spoken input and noise

The central empirical finding is that switching from text to speech worsens performance even before noise is added, and that environmental noise amplifies the degradation sharply. The dominant failure signature under noise is asymmetric: recall remains very high or even rises, while precision drops substantially. In a yes/no absent-object benchmark, this pattern indicates that the model is saying “yes” too often and is therefore hallucinating object presence more frequently (Park et al., 19 Sep 2025).

Condition	Gemini-1.5-Flash on Adversarial (A / P / F1)	Gemma-3n on Adversarial (A / P / F1)
Text	92.6 / 95.9 / 92.0	89.1 / 82.8 / 88.6
Clean audio	90.9 / 92.7 / 90.2	83.7 / 75.8 / 83.5
+5 dB average	82.3 / 73.4 / 82.5	68.7 / 59.1 / 72.9
+0 dB average	74.9 / 64.6 / 76.7	62.1 / 54.0 / 69.2

Across all three subsets, the clean-audio condition already reduces performance relative to text. For Gemini-1.5-Flash, the Adversarial subset moves from $A=92.6, P=95.9, F1=92.0$ under text to $A=90.9, P=92.7, F1=90.2$ under clean audio. On the Random subset, the drop is larger: $A=96.8, R=96.2, P=96.2, F1=96.2$ under text becomes $A=92.5, R=89.1, P=93.5, F1=91.2$ under clean audio. For Gemma-3n, the degradation is stronger. On Adversarial, text $S \rightarrow I$0 falls to clean-audio $S \rightarrow I$1, and on Random it falls from $S \rightarrow I$2 to $S \rightarrow I$3 (Park et al., 19 Sep 2025).

Noise then produces much larger precision collapse. On the Adversarial subset, Gemini-1.5-Flash moves from clean-audio $S \rightarrow I$4 to +5 dB average $S \rightarrow I$5 and +0 dB average $S \rightarrow I$6. Gemma-3n moves from clean-audio $S \rightarrow I$7 to +5 dB average $S \rightarrow I$8 and +0 dB average $S \rightarrow I$9. Among the noise types, Natural and Exterior are repeatedly among the harshest conditions for both model families.

The paper also studies modality order. Gemini is nearly order-invariant: $A$0 gives $A$1, whereas $A$2 gives $A$3. Gemma, by contrast, is highly order-sensitive: $A$4 gives $A$5, while $A$6 falls to $A$7. The paper interprets this as evidence that Gemma’s training may be less effective for reversed modality sequences (Park et al., 19 Sep 2025).

5. Robustness ablations and attempted mitigations

A notable result is that longer spoken queries partially restore robustness under severe noise. On the Adversarial subset with Natural noise at +0 dB, Gemini-1.5-Flash improves from original noisy $A$8 to 5 s $A$9 and 10 s $R$0. Gemma-3n improves from original noisy $R$1 to 5 s $R$2 and 10 s $R$3. However, neither model returns to clean-audio performance, so longer input acts as a partial mitigation rather than a remedy (Park et al., 19 Sep 2025).

Many-shot prompting also helps, but only up to a point. On Gemini-1.5-Flash, moving from 1 shot to 5 shots produces large gains under clean audio and under both noisy conditions, whereas moving from 5 shots to 10 shots slightly hurts. Under clean audio, the reported values are $R$4 for 1 shot, $R$5 for 5 shots, and $R$6 for 10 shots. Under +0 dB, they are $R$7, $R$8, and $R$9, respectively. The paper’s interpretation is that speech-based prompting is less scalable than text-based prompting (Park et al., 19 Sep 2025).

Chain-of-thought reasoning on Gemini-2.5-Flash raises performance under noisy speech but leaves a large residual gap relative to text. The model’s Zero-shot (text) condition gives $P$0, while Zero-shot (+0 dB) falls to $P$1. The four CoT-style prompts improve this to $P$2 for Step-by-Step, $P$3 for Visual-Description, $P$4 for Speech-Description, and $P$5 for Cross-Modal. The strongest noisy condition therefore remains $P$6 F1 points below text zero-shot. The authors conclude that much of the available gain comes from explicitly handling the speech input rather than from generic reasoning scaffolds (Park et al., 19 Sep 2025).

These ablations support a consistent interpretation. Speech itself acts as a reliability stressor, environmental noise weakens robustness further, cross-modal integration can be brittle, and stronger reasoning models do not automatically solve speech-grounded hallucination. The vulnerabilities appear in both proprietary and open-source systems, which the paper presents as evidence that the issue is systemic rather than isolated.

6. Relation to the broader RePOPE ecosystem and terminological scope

RePOPE-Spk is best understood as a spoken-query extension of RePOPE rather than as a replacement for the original benchmark. Subsequent work on standard RePOPE, without speech, has used it for repair-oriented evidence recovery in faithful visual attribution. In that setting, RePOPE is evaluated through image-region masking and answer-token scoring on models such as Qwen2.5-VL-3B-Instruct and LLaVA-v1.5-7B, and compact top-$P$7 masks have been shown to support strong single-point repair rates. The reported TRACE Direct repair rates are 94.44% on Qwen2.5-VL-3B-Instruct and 96.00% on LLaVA-v1.5-7B, but that line of work is strictly visual-language and does not discuss speech or RePOPE-Spk (Gu et al., 7 Jul 2026).

The name also requires terminological care across domains. In one distinct usage in the provided literature, “RePOPE-Spk” denotes the behavior of the peak flux densities of the 44.06943 GHz and 95.16946 GHz class I methanol maser transitions, especially the ratio $P$8 and its variation with maser strength (Yang et al., 2020). The label should also be distinguished from SP$P$9, which denotes Strictly $F1$0-Piecewise languages in formal-language studies of long-distance dependencies (Mahalunkar et al., 2019), and from SpK, an atomic and microphysics code extended to generate global equation-of-state data for HEDP and ICF workflows (Fraser et al., 2024).

Within multimodal hallucination research, however, RePOPE-Spk has a specific and narrower meaning. It names a benchmark that preserves RePOPE’s visual-question structure and replaces written queries with spoken ones under controlled acoustic conditions. Its main contribution is not a new scoring formalism, but an experimental demonstration that image-grounded hallucination behavior changes materially when the query arrives through speech, especially under environmental noise. A plausible implication is that robust voice-based multimodal systems will require methods targeted at speech uncertainty handling, cross-modal alignment, and spoken-query anti-hallucination, because straightforward reuse of text-centric prompting does not restore text-level reliability (Park et al., 19 Sep 2025).