---
title: 'RePOPE-Spk: Speech Query Hallucination Benchmark'
url: https://www.emergentmind.com/topics/repope-spk
type: topic
---

# RePOPE-Spk: Speech Query Hallucination Benchmark

RePOPE-Spk is a speech-augmented benchmark for measuring object hallucination in multimodal large language models when the query is spoken rather than typed. It is introduced as an extension of RePOPE, itself a corrected version of the earlier POPE benchmark for visual object hallucination, and is designed to isolate the effect of changing only the query modality while keeping the underlying image-question benchmark fixed. The benchmark targets multimodal systems that can jointly accept image, audio, and text inputs, and its central empirical claim is that spoken queries make multimodal hallucination worse: under clean speech the reported error increase is about 3%, and under environmental noise it rises to as much as 20% [2510.08581].

## 1. Benchmark lineage and task definition

RePOPE-Spk inherits the structure of RePOPE exactly. RePOPE asks yes/no-style questions about whether objects are present in an image, with carefully selected absent objects to probe false positive tendencies. RePOPE-Spk preserves the same images, questions, and evaluation task, but converts the written questions into spoken audio so that text-query and spoken-query conditions can be compared under a controlled modality substitution [2510.08581].

The inherited benchmark contains three subsets. The **Adversarial** subset has 2,684 QA pairs, the **Popular** subset has 2,727 QA pairs, and the **Random** subset has 2,774 QA pairs. The subsets differ in how the absent object is selected. In **Random**, objects are sampled randomly from the vocabulary. In **Popular**, they are frequent objects in the dataset that are not present in the current image. In **Adversarial**, absent objects are selected by co-occurrence statistics with the image’s true objects, making them likely distractors that a model may wrongly infer [2510.08581].

A recurrent misconception is to treat RePOPE-Spk as a new image benchmark from scratch. The benchmark instead changes only the query modality. This matters methodologically because the construction aims to attribute performance differences specifically to speech input rather than to changes in image selection or QA design. The reported degradation is therefore interpreted as a property of multimodal spoken-query processing rather than of a different visual benchmark.

## 2. Construction of the spoken benchmark

All text questions from RePOPE are converted into speech using a high-quality TTS model. The paper specifies **Parler-TTS**, a sampling rate of **16 kHz**, and a **consistent speaker profile**. The speech generation was controlled to ensure natural prosody, clear articulation, minimal confounds from speaker variability, and avoidance of unnatural artifacts or mispronunciations. This design choice is central to the benchmark’s internal validity: the intention is for hallucination changes to reflect the spoken modality itself rather than poor synthesis quality or uncontrolled speaker variation [2510.08581].

RePOPE-Spk includes both clean and noisy spoken conditions. Clean speech is a direct TTS rendering of the original text question with no added noise. Noisy speech is generated by injecting environmental noise from the **ESC-50** dataset. The paper names five categories used in the main experiments: **Animals**, **Natural**, **Human**, **Interior**, and **Exterior**. Speech and noise are mixed at two explicit signal-to-noise ratios, **+5 dB** and **+0 dB**, with lower SNR producing stronger corruption [2510.08581].

Two further controlled variations are built into the benchmark. The first is modality order, written as \(I \rightarrow S\) for image followed by spoken query and \(S \rightarrow I\) for spoken query followed by image. The second is query length, with conditions using the original duration or versions padded to **5 s** and **10 s**. These perturbations let the benchmark probe not only acoustic corruption but also multimodal sequencing and temporal robustness.

## 3. Experimental protocol and reported metrics

The main evaluation covers one proprietary and one open-source model: **Gemini-1.5-Flash** and **Gemma-3n**. For chain-of-thought experiments, the paper additionally uses **Gemini-2.5-Flash**. The benchmark task remains RePOPE’s object hallucination evaluation: the model receives an image and a question about object presence, and performance is scored as a binary classification problem [2510.08581].

The reported metrics are **Accuracy (A)**, **Recall (R)**, **Precision (P)**, and **F1**. The paper does not provide explicit formulas for these metrics and does not define a separate hallucination-rate equation. Instead, increased hallucination is inferred operationally from degraded classification performance, especially from precision loss, because false positive object assertions reduce precision. This is an important interpretive detail: the headline claims about hallucination growth are high-level summaries, whereas the tables themselves report only \(A\), \(R\), \(P\), and \(F1\).

The experimental scope is also structured. **Tables 1 and 2** evaluate across all three RePOPE subsets. All other experiments, unless otherwise noted, use the **Adversarial subset with Natural-category noise conditions**. The paper evaluates two mitigation families in addition to the base benchmark. On **Gemini-1.5-Flash**, it studies **1-shot**, **5-shot**, and **10-shot** many-shot prompting under clean audio, +5 dB noise, and +0 dB noise. On **Gemini-2.5-Flash**, it compares **Zero-shot (text)** and **Zero-shot (+0 dB)** with four chain-of-thought-style prompts: **Step-by-Step**, **Visual-Description**, **Speech-Description**, and **Cross-Modal** [2510.08581].

## 4. Quantitative behavior under spoken input and noise

The central empirical finding is that switching from text to speech worsens performance even before noise is added, and that environmental noise amplifies the degradation sharply. The dominant failure signature under noise is asymmetric: recall remains very high or even rises, while precision drops substantially. In a yes/no absent-object benchmark, this pattern indicates that the model is saying “yes” too often and is therefore hallucinating object presence more frequently [2510.08581].

| Condition | Gemini-1.5-Flash on Adversarial (A / P / F1) | Gemma-3n on Adversarial (A / P / F1) |
|---|---|---|
| Text | 92.6 / 95.9 / 92.0 | 89.1 / 82.8 / 88.6 |
| Clean audio | 90.9 / 92.7 / 90.2 | 83.7 / 75.8 / 83.5 |
| +5 dB average | 82.3 / 73.4 / 82.5 | 68.7 / 59.1 / 72.9 |
| +0 dB average | 74.9 / 64.6 / 76.7 | 62.1 / 54.0 / 69.2 |

Across all three subsets, the clean-audio condition already reduces performance relative to text. For **Gemini-1.5-Flash**, the **Adversarial** subset moves from \(A=92.6, P=95.9, F1=92.0\) under text to \(A=90.9, P=92.7, F1=90.2\) under clean audio. On the **Random** subset, the drop is larger: \(A=96.8, R=96.2, P=96.2, F1=96.2\) under text becomes \(A=92.5, R=89.1, P=93.5, F1=91.2\) under clean audio. For **Gemma-3n**, the degradation is stronger. On **Adversarial**, text \(A=89.1, R=95.2, P=82.8, F1=88.6\) falls to clean-audio \(A=83.7, R=92.9, P=75.8, F1=83.5\), and on **Random** it falls from \(A=91.6, R=95.6, P=85.8, F1=90.4\) to \(A=82.0, R=92.9, P=72.1, F1=81.2\) [2510.08581].

Noise then produces much larger precision collapse. On the **Adversarial** subset, **Gemini-1.5-Flash** moves from clean-audio \(P=92.7, F1=90.2\) to +5 dB average \(P=73.4, F1=82.5\) and +0 dB average \(P=64.6, F1=76.7\). **Gemma-3n** moves from clean-audio \(P=75.8, F1=83.5\) to +5 dB average \(P=59.1, F1=72.9\) and +0 dB average \(P=54.0, F1=69.2\). Among the noise types, **Natural** and **Exterior** are repeatedly among the harshest conditions for both model families.

The paper also studies modality order. **Gemini** is nearly order-invariant: \(I \rightarrow S\) gives \(A=90.9, R=87.4, P=92.7, F1=90.2\), whereas \(S \rightarrow I\) gives \(A=90.1, R=88.5, P=89.2, F1=88.8\). **Gemma**, by contrast, is highly order-sensitive: \(I \rightarrow S\) gives \(A=83.7, R=92.9, P=75.8, F1=83.5\), while \(S \rightarrow I\) falls to \(A=69.0, R=82.0, P=61.2, F1=70.1\). The paper interprets this as evidence that Gemma’s training may be less effective for reversed modality sequences [2510.08581].

## 5. Robustness ablations and attempted mitigations

A notable result is that longer spoken queries partially restore robustness under severe noise. On the Adversarial subset with **Natural noise at +0 dB**, **Gemini-1.5-Flash** improves from original noisy \(A=71.2, R=94.9, P=61.0, F1=74.3\) to **5 s** \(A=79.7, R=92.4, P=70.6, F1=80.1\) and **10 s** \(A=82.9, R=94.0, P=74.2, F1=82.9\). **Gemma-3n** improves from original noisy \(A=60.4, R=97.1, P=52.8, F1=68.4\) to **5 s** \(A=71.1, R=95.5, P=61.1, F1=74.5\) and **10 s** \(A=74.7, R=95.1, P=64.5, F1=76.9\). However, neither model returns to clean-audio performance, so longer input acts as a partial mitigation rather than a remedy [2510.08581].

Many-shot prompting also helps, but only up to a point. On **Gemini-1.5-Flash**, moving from **1 shot** to **5 shots** produces large gains under clean audio and under both noisy conditions, whereas moving from **5 shots** to **10 shots** slightly hurts. Under clean audio, the reported values are \(A=71.8, R=92.7, P=62.2, F1=74.4\) for 1 shot, \(A=91.2, R=88.3, P=91.4, F1=89.8\) for 5 shots, and \(A=90.2, R=89.7, P=88.3, F1=89.0\) for 10 shots. Under +0 dB, they are \(A=55.3, R=95.2, P=49.7, F1=65.3\), \(A=78.8, R=91.0, P=70.0, F1=79.1\), and \(A=78.1, R=92.3, P=68.5, F1=78.7\), respectively. The paper’s interpretation is that speech-based prompting is less scalable than text-based prompting [2510.08581].

Chain-of-thought reasoning on **Gemini-2.5-Flash** raises performance under noisy speech but leaves a large residual gap relative to text. The model’s **Zero-shot (text)** condition gives \(A=94.6, R=94.7, P=93.3, F1=94.0\), while **Zero-shot (+0 dB)** falls to \(A=73.4, R=49.6, P=83.4, F1=62.2\). The four CoT-style prompts improve this to \(F1=73.5\) for **Step-by-Step**, \(76.6\) for **Visual-Description**, \(77.8\) for **Speech-Description**, and \(77.9\) for **Cross-Modal**. The strongest noisy condition therefore remains \(16.1\) F1 points below text zero-shot. The authors conclude that much of the available gain comes from explicitly handling the speech input rather than from generic reasoning scaffolds [2510.08581].

These ablations support a consistent interpretation. Speech itself acts as a reliability stressor, environmental noise weakens robustness further, cross-modal integration can be brittle, and stronger reasoning models do not automatically solve speech-grounded hallucination. The vulnerabilities appear in both proprietary and open-source systems, which the paper presents as evidence that the issue is systemic rather than isolated.

## 6. Relation to the broader RePOPE ecosystem and terminological scope

RePOPE-Spk is best understood as a spoken-query extension of RePOPE rather than as a replacement for the original benchmark. Subsequent work on standard RePOPE, without speech, has used it for repair-oriented evidence recovery in faithful visual attribution. In that setting, RePOPE is evaluated through image-region masking and answer-token scoring on models such as **Qwen2.5-VL-3B-Instruct** and **LLaVA-v1.5-7B**, and compact top-\(k\) masks have been shown to support strong single-point repair rates. The reported **TRACE Direct** repair rates are **94.44%** on Qwen2.5-VL-3B-Instruct and **96.00%** on LLaVA-v1.5-7B, but that line of work is strictly visual-language and does not discuss speech or RePOPE-Spk [2607.06726].

The name also requires terminological care across domains. In one distinct usage in the provided literature, “RePOPE-Spk” denotes the behavior of the peak flux densities of the 44.06943 GHz and 95.16946 GHz class I methanol maser transitions, especially the ratio \(S_{\rm pk,95}/S_{\rm pk,44}\) and its variation with maser strength [2005.00392]. The label should also be distinguished from **SP\(k\)**, which denotes **Strictly \(k\)-Piecewise languages** in formal-language studies of long-distance dependencies [1907.06048], and from **SpK**, an atomic and microphysics code extended to generate global equation-of-state data for HEDP and ICF workflows [2405.13814].

Within multimodal hallucination research, however, RePOPE-Spk has a specific and narrower meaning. It names a benchmark that preserves RePOPE’s visual-question structure and replaces written queries with spoken ones under controlled acoustic conditions. Its main contribution is not a new scoring formalism, but an experimental demonstration that image-grounded hallucination behavior changes materially when the query arrives through speech, especially under environmental noise. A plausible implication is that robust voice-based multimodal systems will require methods targeted at speech uncertainty handling, cross-modal alignment, and spoken-query anti-hallucination, because straightforward reuse of text-centric prompting does not restore text-level reliability [2510.08581].

Source: https://www.emergentmind.com/topics/repope-spk