World-of-Whale Benchmark (WoW-Bench)

• World-of-Whale Benchmark (WoW-Bench) is a comprehensive benchmark designed to evaluate low-level acoustic perception and cognition in audio-language models using marine mammal vocalizations.
• It measures both perception, via species and vocalization classification in a zero-shot setting, and cognition aligned with Bloom’s taxonomy through tasks like Remember, Understand, Apply, and Analyze.
• The benchmark employs distractor questions to probe auditory grounding, revealing significant model weaknesses compared to human performance in fine-grained acoustic tasks.

to=arxiv_search ＿天天json code state? Searching arXiv for the benchmark paper and closely related evaluation context. The World-of-Whale Benchmark (WoW-Bench) is a benchmark for evaluating fine-grained acoustic perception and cognition in large audio-LLMs (LALMs) using marine mammal vocalizations (Kim et al., 28 Aug 2025). It was introduced to address a specific gap: although LALMs perform strongly on high-level audio reasoning such as captioning and question answering, their capacity for low-level listening—detecting pitch, duration, and related acoustic cues—had not been systematically evaluated. WoW-Bench therefore measures both perceptual generalization to unfamiliar sounds and cognitive processing built on fine-grained listening, with experiments showing performance far below human levels (Kim et al., 28 Aug 2025).

1. Motivation and conceptual scope

WoW-Bench is motivated by the claim that low-level auditory perception is foundational for robust audio understanding and reasoning, and that perceptual failures often cascade into reasoning errors (Kim et al., 28 Aug 2025). The benchmark is explicitly framed around out-of-distribution conditions in which a model must reason about unfamiliar sounds on the basis of acoustic structure rather than semantic familiarity alone.

Its high-level structure has two parts. The Perception benchmark tests whether a model can categorize unfamiliar marine mammal calls in a zero-shot setting. The Cognition benchmark is inspired by Bloom’s four lower levels—Remember, Understand, Apply, and Analyze—and is intended to assess cognitive operations grounded in listening rather than in language priors alone (Kim et al., 28 Aug 2025).

A central design choice is the use of marine mammal vocalizations. The benchmark includes clicks, whistles, calls, pulses, hums, and related categories drawn from the Watkins Marine Mammal Sound Database, with species including humpback whale, killer whale, melon-headed whale, beluga, dolphin, and seal (Kim et al., 28 Aug 2025). This choice supports evaluation on acoustically diverse material spanning a wide frequency and duration range.

2. Dataset composition and construction

WoW-Bench contains 1,777 multiple-choice questions in total (Kim et al., 28 Aug 2025). Its composition is summarized below.

Component	Subdivision	Questions
Perception	99 Species, 97 Vocalization, 100 Both	296
Cognition non-distractor	200 Remember, 190 Understand, 200 Apply-Frequency, 200 Apply-Duration, 200 Analyze-Acoustics, 191 Analyze-Vocalization	1,181
Cognition distractor	50 per task	300
Total	—	1,777

The acoustic range is broad: sampling rates span from 8 kHz up to $>48$ kHz, and frequencies span 20 Hz–100 kHz (Kim et al., 28 Aug 2025). Durations vary from short clicks of $<0.1$ s to long calls of $>2$ s. The reported analysis found no systematic performance drop at any particular duration or sampling-rate threshold (Kim et al., 28 Aug 2025).

Data collection and preprocessing combine automated generation with human review. The source material comes from the Watkins Marine Mammal Sound Database, used with explicit permission. For question generation, Perception items were created by prompting GPT-4o with clip metadata; Cognition items other than Remember were created with GPT-4o plus spectrograms through a vision-language pipeline; and Remember items were built through manual selection of triplets with silence gaps (Kim et al., 28 Aug 2025). Each question-answer pair was then reviewed by three annotators, who could accept, edit, or discard it. Annotators also ensured that distractor choices were acoustically plausible, unambiguous, and balanced in lexical difficulty (Kim et al., 28 Aug 2025).

3. Perception benchmark

The Perception benchmark evaluates categorization of unfamiliar sounds in a zero-shot regime. It contains three task types (Kim et al., 28 Aug 2025).

Species classification requires assigning a clip to a species label.

Vocalization classification requires selecting a natural-language description of the call type, such as “high-pitched whistle.”

Both requires joint identification of species and vocalization, such as “Leopard Seal – Long Call.”

The primary evaluation metric is accuracy:

$$\mathrm{Accuracy}=\frac{\text{Number of correct answers}}{\text{Total questions}}.$$

For completeness, the benchmark description also gives classwise definitions of precision, recall, and $F_1$:

$$\mathrm{Precision}_i=\frac{TP_i}{TP_i+FP_i}, \qquad \mathrm{Recall}_i=\frac{TP_i}{TP_i+FN_i},$$

$$F_{1,i}=2\cdot\frac{\mathrm{Precision}_i\cdot\mathrm{Recall}_i}{\mathrm{Precision}_i+\mathrm{Recall}_i}.$$

However, because all tasks are four-option MCQs with one correct answer, the paper states that micro-averaged accuracy suffices (Kim et al., 28 Aug 2025).

The reported results show an asymmetry between semantic targets. Models do slightly better on Vocalization than on Species, with performance on Vocalization reaching up to $\sim64\%$ for Qwen2.5, whereas Species performance reaches only $\sim28\%$ (Kim et al., 28 Aug 2025). The authors interpret this as suggesting some sensitivity to intuitive acoustic labels, even when species-level categorization remains weak.

4. Cognition benchmark and distractor methodology

The Cognition benchmark operationalizes four lower levels of Bloom’s taxonomy through acoustically grounded tasks (Kim et al., 28 Aug 2025).

Remember presents a reference sound and three test sounds and asks for the identical one, without semantic content.

Understand asks for the description—pitch label plus frequency range—that best fits a single clip.

Apply is divided into Apply-Frequency, in which the model selects the highest- or lowest-pitch clip among three, and Apply-Duration, in which it selects the longest or shortest duration.

Analyze is divided into Analyze-Acoustics, which requires identifying an acoustic transition such as low tone $\rightarrow$ broadband pulse in a dual-segment clip, and Analyze-Vocalization, which requires detecting a change in species or vocalization in a seamless concatenation.

The benchmark’s most distinctive methodological feature is its use of distractor questions to measure whether models are truly solving problems through listening rather than through heuristics (Kim et al., 28 Aug 2025). Distractor construction deliberately inverts common patterns so that naive shortcut strategies fail. The examples given in the benchmark description are specific: when all three sounds are identical, the correct answer is “All indistinguishable”; when a reference is replaced by noise, the correct answer is “None match” (Kim et al., 28 Aug 2025). Understand and Analyze-Acoustics use noise-based distractors, and Analyze-Vocalization uses same-species concatenations.

Scoring remains simple: distractor and non-distractor items are both evaluated by MCQ accuracy, with no partial credit. The paper reports the two splits separately in order to measure auditory grounding (Kim et al., 28 Aug 2025).

5. Evaluation protocol, models, and empirical results

The evaluated systems include SALMONN (13B), LTU (7B), LTU-AS (7B), GAMA (7B), GAMA-IT (7B), Qwen-Audio-Chat (8.4B), Qwen2-Audio-Instruct (8.4B), Qwen2.5-Omni (10.7B), AudioFlamingo2 (3.3B), Gemini 1.5-Pro, Gemini 2.0-Flash, and Gemini 2.5-Flash (Kim et al., 28 Aug 2025). These are described as encoder-LLM hybrids trained on large audio-text corpora.

Overall model performance remains substantially below human performance. The best model, Gemini 2.5, achieves 45.5\% overall, whereas human results range from approximately 70–97\% depending on task (Kim et al., 28 Aug 2025). On Cognition-Remember, the gap is especially large: humans 97.1\% versus Gemini 2.5 57.1\% (Kim et al., 28 Aug 2025).

The reported statistical protocol includes bootstrap resampling with 1,000 replications to compute 95\% confidence intervals on model accuracies (Kim et al., 28 Aug 2025). For Cognition-Remember, the difference between model and human scores is reported as significant at $<0.1$0 under a two-tailed McNemar’s test (Kim et al., 28 Aug 2025).

Task-specific results identify several persistent bottlenecks. Pitch discrimination in Apply-Frequency is particularly difficult, with Gemini 2.5 reaching only 21.4\%–33.6\%, whereas Apply-Duration can exceed 60\% (Kim et al., 28 Aug 2025). Analyze performance is also low, often below 35\%, indicating weaknesses in temporal-spectral segmentation (Kim et al., 28 Aug 2025).

6. Interpretation, limitations, and nomenclature

The paper’s analysis emphasizes that distractors sharply degrade model scores, with a reported negative correlation of $<0.1$1 between standard and distractor performance (Kim et al., 28 Aug 2025). This is taken as evidence that many models rely on linguistic heuristics rather than direct acoustic reasoning. The authors further describe a recurring classify-first strategy, in which a model first assigns a semantic label and then infers acoustic properties from that label; this strategy produces errors on both non-distractor and distractor items (Kim et al., 28 Aug 2025).

The benchmark therefore functions not only as a leaderboard instrument but also as a diagnostic probe of auditory grounding. The reported recommendations are correspondingly architectural and training-oriented: incorporate explicit low-level feature extraction modules such as pitch trackers and envelope detectors; train on synthetic tasks requiring direct spectral comparisons through contrastive listening objectives; and augment pretraining with out-of-distribution bioacoustic data to strengthen perceptual generalization (Kim et al., 28 Aug 2025). A plausible implication is that progress on high-level audio reasoning alone may not transfer to settings that depend on fine-grained sensory discrimination.

The name WoW-Bench can also be a source of ambiguity. In the arXiv literature, “WoW-bench” also names a benchmark in “World of Workflows: a Benchmark for Bringing World Models to Enterprise Systems” (Gupta et al., 29 Jan 2026), and “WoWBench” appears again in “WoW: Towards a World omniscient World model Through Embodied Interaction” (Chi et al., 26 Sep 2025). These are unrelated to the World-of-Whale Benchmark, whose domain is low-level auditory perception and cognition over marine mammal vocalizations (Kim et al., 28 Aug 2025).