BAGEL: Benchmarking Animal Knowledge Expertise in Language Models

Abstract: LLMs have shown strong performance on broad-domain knowledge and reasoning benchmarks, but it remains unclear how well LLMs handle specialized animal-related knowledge under a unified closed-book evaluation protocol. We introduce BAGEL, a benchmark for evaluating animal knowledge expertise in LLMs. BAGEL is constructed from diverse scientific and reference sources, including bioRxiv, Global Biotic Interactions, Xeno-canto, and Wikipedia, using a combination of curated examples and automatically generated closed-book question-answer pairs. The benchmark covers multiple aspects of animal knowledge, including taxonomy, morphology, habitat, behavior, vocalization, geographic distribution, and species interactions. By focusing on closed-book evaluation, BAGEL measures animal-related knowledge of models without external retrieval at inference time. BAGEL further supports fine-grained analysis across source domains, taxonomic groups, and knowledge categories, enabling a more precise characterization of model strengths and systematic failure modes. Our benchmark provides a new testbed for studying domain-specific knowledge generalization in LLMs and for improving their reliability in biodiversity-related applications.

• The paper introduces BAGEL, a closed-book, multicategory MCQ benchmark to assess animal knowledge in LLMs across diverse domains.
• BAGEL leverages four heterogeneous sources—Wikipedia, GloBI, bioRxiv, and Xeno-canto—with balanced answer shuffling to rigorously evaluate domain-specific reasoning.
• Results show significant domain gaps, non-monotonic scaling, and persistent challenges in transferring animal expertise across different LLM architectures.

BAGEL: Benchmarking Animal Knowledge Expertise in LLMs

Motivation and Context

The proliferation of LLMs has led to impressive results on broad-domain benchmarks and general scientific evaluations, but their competence in specialized domains—particularly the expansive and complex domain of animal knowledge—remains poorly understood. "BAGEL: Benchmarking Animal Knowledge Expertise in LLMs" (2604.16241) addresses this evaluative gap by introducing a closed-book, multicategory, multiple-choice benchmark specifically targeting animal-related and natural history knowledge. BAGEL leverages four heterogeneous sources—Wikipedia, Global Biotic Interactions (GloBI), bioRxiv, and Xeno-canto—to probe factual, ecological, scientific, and acoustics-related reasoning without granting models access to source materials at inference, thereby saturating the evaluation with parametric knowledge and eliminating retrieval shortcuts.

Figure 1: Overview of the BAGEL benchmark curation pipeline: four source-specific preparation tracks feed domain prompts into a shared generator, followed by quality checks, four-option formatting, and option-order shuffling.

Benchmark Design and Curation Pipeline

BAGEL assembles 11,852 four-option multiple-choice questions, distributed across its four domains to probe distinct dimensions of animal expertise:

• Wikipedia: Questions target species-level encyclopedic facts, taxonomy, behavior, communication, morphology, habitat, cognition, distribution, and diet. Generation is constrained to strictly use information explicit in article text, with GPT-4o-mini orchestrating the controlled synthesis to produce up to eight balanced items per taxon per theme.
• GloBI: Focuses on ecological interaction reasoning via questions that require identifying masked interaction participants or inferring interaction types based on natural-language summaries derived from structured GloBI records. Emphasis is placed on ensuring that items demand ecological inference, avoiding lexical cues or trivial label retrieval.
• bioRxiv: Contains items centered on interpreting results from recent animal-related scientific preprints. Each question requires integrating contextual information from the article to select the correct interpretation of findings, thus testing literature-based probabilistic reasoning.
• Xeno-canto: Uniquely targets text-only bioacoustic knowledge (acoustic properties, vocalization structure), deriving items from community-contributed audio metadata and visual spectrograms. Models do not receive any non-text input; instead, generation involves GPT-4o-mini creating MCQs about dominant frequency, call duration, modulation, and harmonic structure.

The benchmark enforces answer-option shuffling with balanced distribution, mitigating known MC answer-position biases observed in earlier versions and ensuring that models cannot exploit superficial ordering heuristics.

Xeno-canto Generation Process

The Xeno-canto domain is notable for requiring models to reason about bioacoustic properties using strictly text-based cues, based on spectrogram-derived features.

Figure 2: Xeno-canto question generation pipeline. The process involves feeding a log-frequency spectrogram of a bioacoustic recording into GPT-4o-mini to generate structured, multiple-choice questions based on visual acoustic features.

Evaluation Protocol and Models

All models—spanning two closed-source references (GPT-5.4, Claude Opus 4.6) and a range of open-weight LLMs from SmolLM2-360M through Qwen3-32B and Gemma 3 27B IT—are evaluated in a standardized, closed-book protocol. The prompt presents only the question and the four answer choices, with no supporting passage or metadata, simulating parametric recall and source-agnostic reasoning.

Performance is scored both at the aggregate and per-domain level, enabling granular analysis of domain strengths and deficits.

Results: Domain Heterogeneity and Scaling

Aggregate results reveal substantial domain-wise variation even for leading closed-source LLMs. Specifically, while proprietary frontiers achieve high accuracy on Wikipedia and bioRxiv domains, their performance lags considerably on Xeno-canto, with similar cross-domain fragility observed across open-weight models.

Mid- to large-sized open models (Gemma 3 27B IT, Llama 3.1 Instruct-8B, Phi-4, Qwen3-14B/32B) exhibit notable scaling with respect to Wikipedia, bioRxiv, and GloBI, but frequently display non-monotonic or plateauing performance on Xeno-canto. In the Qwen3 family, strikingly, Xeno-canto performance peaks at 14B and declines at 32B, demonstrating non-monotonic scaling on this acoustics-oriented track.

Figure 3: Qwen3 instruct family on BAGEL (seed 0, greedy): accuracy versus model size for each source domain and Overall. GloBI, Wikipedia, and bioRxiv generally improve up to 32B, while Xeno-canto peaks at 14B and drops at 32B under our protocol—the same non-monotonicity discussed in the text.

This result challenges the assumption that model size or general language modeling competence alone confers robustness across biodiversity knowledge—highlighting persistent gaps in domain transfer and the limits of scaling for specialized subfields.

Fine-Grained Performance Dissection

• Xeno-canto Gap: Across all models, Xeno-canto remains the hardest domain. Even top-performing LLMs only attain 0.59–0.71 accuracy at best on the easiest Xeno-canto subtask (“dominant frequency range”), dropping below 0.30 for more specialized or underrepresented topics (e.g., “modulation pattern”). Lexical analysis establishes that Xeno-canto items have higher density of rare tokens and bioacoustic-specific terminology, but item statistics alone cannot explain the accuracy gap.
• Distractor Quality and Option Ambiguity: Manual audit of GloBI items uncovers recurring issues with answer discriminability, taxonomic ambiguity, and multiple plausible correct options, highlighting the challenge of achieving unambiguous evaluation signals in complex, structured knowledge domains.
• Scaling Effects: Smaller models cluster near random chance, with large open models closing much (but not all) of the gap to closed-source frontiers on the easiest domains. Domain-divergent scaling further emphasizes that parameter growth does not equivalently lift all axes of animal expertise.

Implications and Future Directions

BAGEL's design and results hold significant implications for future LLM evaluation and development:

• Domain-Targeted Evaluation: Testing specialized knowledge domains like biodiversity reveals deficits and heterogeneity often invisible to broad academic benchmarks. BAGEL enables tracking of scientific and natural history knowledge acquisition in LLMs, with immediate relevance to ecological automation, conservation biology, and organismal informatics.
• Limits of Scaling and Generalization: The non-monotonic and inconsistent scaling observed, especially in the Xeno-canto domain, demonstrates the need for specialized, domain-adapted pretraining or augmentation workflows. Standard scaling schedules do not guarantee expertise transfer to rare or technical domains.
• Benchmark Limitations: BAGEL is strictly closed-book and English-only. Its results reflect a combination of parametric recall, potential pretraining overlap, and MCQ test-taking behavior. Disentangling genuine domain understanding from “testmanship” and training set leakage remains a challenge for the field.
• Guidance for AI for the Biosciences: The empirical gaps highlighted by BAGEL suggest that foundation models for nature, environmental science, and animal behavior need to move beyond simple aggregation of existing LLM architectures and embrace domain-tailored data curation, model conditioning, and error analysis frameworks.

Conclusion

BAGEL sets a new standard for evaluating LLMs on structured, multi-domain animal knowledge. By introducing questions sampled and synthesized from encyclopedic, ecological, scientific, and bioacoustic domains and curating a challenging, answer-shuffling-robust MCQ testbed, the benchmark surfaces robustly the persistent limitations of current LLMs in domain-specific generalization. Substantial headroom remains for both general-purpose and specialized models, with the accuracy gap to proprietary frontiers and the Xeno-canto performance drop both underscoring the need for further architecture, data, and evaluation research in parametric animal expertise. Future developments will require advances in prompt fidelity, distractor generation, and multimodal transfer mechanisms to close these gaps and deliver more reliable automated biological expertise.