BioBench: Ecology-Driven Vision Benchmark

• BioBench is an ecology-focused vision benchmark that unifies nine ecological tasks and diverse imaging modalities to assess the transferability of frozen visual backbones.
• The benchmark reveals that high ImageNet accuracy does not reliably predict performance on ecological imagery, with rank correlations as low as 0.25 leading to frequent mis-rankings.
• It provides a standardized Python API and uncertainty-aware evaluation protocol, setting a blueprint for robust, application-driven AI-for-science benchmarks.

BioBench is an ecology-focused vision benchmark introduced to replace the increasingly unreliable practice of using ImageNet-1K transfer performance as a proxy for “general visual representation quality” in scientific applications. It consolidates nine publicly released ecological vision tasks across four taxonomic kingdoms and six acquisition modalities, totaling 3.1 million images and about 337 GB of data, and evaluates frozen visual backbones through lightweight task-level probes over embeddings $f:\text{image}\to\mathbb{R}^d$. Its central claim is that ImageNet accuracy no longer predicts which modern vision models transfer best to realistic ecological imagery, particularly once models exceed the now-common high-accuracy regime on ImageNet (Stevens, 20 Nov 2025).

1. Conception and rationale

BioBench was motivated by an empirical failure of ImageNet as a scientific-model-selection proxy. Across 46 modern vision checkpoints spanning 11 model families and several training paradigms—supervised models, self-supervised models such as DINOv2, and image-text models such as CLIP, SigLIP, and AIMv2—ImageNet-1K top-1 accuracy explains only 34% of the variance in BioBench performance, with $R^2 = 0.34$. Rank agreement is also weak: the reported Spearman correlation between ImageNet ranking and BioBench ranking is $\rho = 0.55$, with 95% confidence interval $[0.45, 0.64]$ and $p < 0.0005$ under 5000-permutation randomization. The paper gives the ranking-error interpretation

$$\text{mis-ranking probability} = \frac{1}{2}(1-\rho),$$

so $\rho = 0.55$ implies that the ImageNet-preferred model is worse on BioBench about 22% of the time (Stevens, 20 Nov 2025).

The mismatch intensifies at the frontier. Restricting attention to models above 75% ImageNet top-1 accuracy, rank concordance drops to $\rho = 0.42$, with 95% confidence interval $[0.15, 0.65]$ and $p < 0.01$, implying mis-ranking about 30% of the time. The introduction further reports that, across three representative ecology tasks, Spearman correlation falls below 0.25 once models exceed the 75% threshold. The paper describes this threshold behavior as a “ranking cliff”: among stronger models, better ImageNet performance ceases to predict better performance on real ecological tasks (Stevens, 20 Nov 2025).

Two causes are emphasized. First, there is distribution mismatch: ImageNet consists of web photos, whereas ecological imagery includes camera-trap infrared, drone imagery, microscope images, specimen photos, and other regimes with different spectra, noise, context, and viewpoint. Second, there is task mismatch: ecological applications often involve extremely fine-grained and long-tailed distinctions, including rare species and subtle traits, whereas ImageNet’s 1000 object categories do not reflect this granularity or imbalance. BioBench is therefore framed as a benchmark for application-driven transfer “where it matters,” rather than a proxy benchmark whose predictive validity has broken down (Stevens, 20 Nov 2025).

2. Corpus and ecological coverage

BioBench unifies nine publicly released tasks spanning animals, plants, fungi, and protists/protozoa. The six acquisition regimes are described as drone RGB footage or frames, curated web video, microscope micrographs, in-situ photos, specimen photos, and camera-trap frames; another summary phrases these as camera-trap RGB/IR, drone video frames, museum specimens, in-situ macro, web video, and micrographs. The benchmark is deliberately heterogeneous along both biological and imaging dimensions (Stevens, 20 Nov 2025).

Task	Benchmark role
Beluga	Individual re-identification of beluga whales
FishNet	Fish functional trait prediction from specimen images
FungiCLEF	Fungi species identification
Herbarium19	Plant species classification from herbarium specimen images
iWildCam	Mammal species classification from camera-trap images
KABR	Behavior recognition for Kenyan ungulates from drone videos
MammalNet	Mammal behavior recognition from web video
Plankton	Protist/plankton species classification from micrographs
Pl@ntNet	Plant species identification from in-situ photos

This task mixture spans species identification, individual re-identification, behavior classification, and trait prediction. The associated applications include biodiversity monitoring, long-tail species recognition, camera-trap-based wildlife survey, plant identification, fungal identification, individual animal re-identification, animal behavior analysis from drones and video, and functional trait inference from natural history specimens. The authors emphasize that the suite includes both “proxy-driven” tasks and “mission-driven” tasks that correspond directly to conservation and ecological workflows (Stevens, 20 Nov 2025).

The benchmark’s breadth is methodological as well as biological. It was designed in response to fragmentation across prior ecology datasets, each with its own repository, file layout, split definitions, and metric scripts. BioBench instead provides a unified interface over a shared task collection, enabling cross-backbone comparison under a standardized evaluation regime (Stevens, 20 Nov 2025).

3. Evaluation protocol and metrics

A core design goal is to isolate backbone representation quality from task-specific engineering. Every evaluated model need only expose a frozen embedding function

$R^2 = 0.34$0

BioBench then fits “linear or logistic probes per task,” described throughout as lightweight classifiers or linear probes, on top of fixed embeddings rather than fine-tuning the backbone end to end. This is intended to produce a cleaner and more reproducible measure of transferability across pretraining families (Stevens, 20 Nov 2025).

The default metric is class-balanced macro-F1, chosen because the benchmark tasks are long-tailed and ecology applications often care about rare classes rather than only aggregate accuracy. The standard definition used is

$R^2 = 0.34$1

and

$R^2 = 0.34$2

Two tasks are exceptions: FishNet and FungiCLEF use their domain-standard metrics rather than the benchmark default. The paper also notes that some prior state-of-the-art comparison numbers use non-macro-F1 metrics, specifically macro-accuracy for MammalNet and micro-accuracy for Herbarium19, whereas BioBench reports results under its own uniform evaluation convention (Stevens, 20 Nov 2025).

Uncertainty reporting is built into the protocol. BioBench reports bootstrap confidence intervals, and its key benchmark-validity analyses use permutation or randomization tests. The correlation-versus-threshold figures include 95% bootstrapped confidence intervals, and significance for the rank-validity analyses is assessed with 5000-permutation randomization tests. This emphasis on confidence estimates and ranking robustness is part of the benchmark’s stated design philosophy: benchmark scores should not be treated as exact absent uncertainty quantification (Stevens, 20 Nov 2025).

Computationally, the benchmark is intended to be practical. A single Python API handles downloading data, fitting lightweight classifiers, and reporting metrics. The abstract states that ViT-L models evaluate in about 6 hours on an NVIDIA A6000 GPU; the main body states that evaluation can run on a single A6000 or parallelize across SLURM clusters, and that ViT-B/16 and ViT-L/14 checkpoints finish in about one hour on an A6000, with larger models scaling predictably. The paper also states, “We use a single Nvidia A6000 GPU to evaluate all models.” Code and predictions are released at https://github.com/samuelstevens/biobench, with benchmark results at https://samuelstevens.me/biobench (Stevens, 20 Nov 2025).

4. Empirical findings and leaderboard behavior

BioBench’s empirical results reinforce its critique of ImageNet proxying. Image-text models are especially strong on the benchmark. Among the listed models, SigLIP 2 ViT-1B/16 achieves the best reported BioBench mean at 43.5. DINOv2 ViT-g/14 and BioCLIP 2 ViT-L/14 both reach 41.7 mean. By contrast, AIMv2, despite 86.7 ImageNet top-1, attains only 36.9 mean on BioBench. SAM 2 and V-JEPA perform poorly on ImageNet but better than random on some tasks, though they remain weak overall on BioBench (Stevens, 20 Nov 2025).

The benchmark therefore changes model ranking in ways ImageNet does not anticipate. This is not only a matter of absolute transfer drop; it is a reordering of which model families appear preferable for scientific deployment. The paper’s time-series discussion summarizes this strongly: most newly released “generalist” models do not improve the benchmark, and only CLIP, SigLIP, and SigLIP 2 are reported as setting new state-of-the-art BioBench scores over time (Stevens, 20 Nov 2025).

A plausible implication is that BioBench is not merely harder than ImageNet-style transfer evaluation; it is sensitive to representational properties that ImageNet largely fails to distinguish in modern high-performing models. The paper treats this as evidence that progress claimed on generic web-photo leaderboards should not be assumed to transfer to ecology, even when the candidate models are all frontier checkpoints (Stevens, 20 Nov 2025).

5. BioBench as a benchmark-construction blueprint

BioBench is presented not only as an ecology benchmark but as a template for building reliable AI-for-science benchmarks more broadly. Its recommended design principles are explicit: benchmarks should be application-driven rather than proxy-driven; they should span meaningful axes of variation in the scientific domain, such as acquisition modality, task type, and subdomain diversity; they should use class-balanced or otherwise domain-appropriate metrics rather than defaulting to top-1 accuracy; they should separate representation quality from problem-specific tuning through a consistent evaluation protocol; and they should report uncertainty, significance tests, and ranking stability in order to avoid what the paper, citing prior work, frames as “benchmark lottery” effects (Stevens, 20 Nov 2025).

This blueprint places BioBench in a broader scientific-benchmarking discourse. Omnibenchmark, for example, is an infrastructure-oriented system for continuous and open benchmarking in bioinformatics rather than a single benchmark suite (Mallona et al., 2024). LAB-Bench and LABBench2 target practical biology research capabilities for LLMs and agents, including literature work, figures, protocols, database access, and molecular biology assistance, rather than visual backbone transfer (Laurent et al., 2024, Laurent et al., 4 Feb 2026). BixBench focuses on open-ended computational biology workflows in notebook-like analytical environments (Mitchener et al., 28 Feb 2025), BioProBench concentrates on biological protocol understanding and reasoning (Liu et al., 11 May 2025), and BioXArena evaluates LLM agents on end-to-end biomedical machine-learning pipeline construction across heterogeneous modalities (Li et al., 15 May 2026). This suggests that BioBench occupies a distinct niche within the broader benchmark landscape: ecology vision transfer benchmarking for frozen visual representations, rather than agentic, text-centric, or workflow-execution evaluation.

The term has also appeared in other, unrelated contexts. In knowledge-graph systems research, BioBench-Allie denotes a fixed-size biological RDF benchmark with 94,420,988 triples, used as one component of a broader comparative study of data-management systems and query types (Salehpour et al., 2020). That usage is historically separate from the ecology benchmark introduced in 2025.

6. Limitations and interpretive boundaries

The paper is explicit that BioBench has important limits. First, it focuses on ecology; other scientific domains such as medicine or manufacturing may require different modalities, task structures, and evaluation priorities. Second, frozen-feature probing is a deliberate simplification: it isolates representation quality, but it may underestimate the gains available from task-specific fine-tuning. Third, macro-F1 emphasizes tail-class performance, which is often appropriate in ecology, but some operational settings may instead prefer metrics tied to specific decision thresholds, such as precision at fixed recall. Fourth, the benchmark concentrates on classification-like tasks rather than detection, segmentation, or calibration-heavy scientific tasks (Stevens, 20 Nov 2025).

These limitations clarify what BioBench measures and what it does not. It is a benchmark for comparative evaluation of visual backbones under standardized ecological transfer, not a universal ecology benchmark, not a full-spectrum scientific-vision benchmark, and not a substitute for downstream task-specific development. Its main contribution is to show that ImageNet-driven model selection is no longer reliable for scientific vision in ecology, and to replace that proxy with a heterogeneous, application-grounded, uncertainty-aware benchmark suite (Stevens, 20 Nov 2025).

The benchmark’s larger argument is methodological. If model evaluation in scientific machine learning is to be trustworthy, then benchmarking must be performed on scientific data, scientific task structure, and scientific metrics themselves. BioBench is therefore both a benchmark and a benchmarking position: scientific ML should be evaluated on science, rather than on generic leaderboards whose ranking validity has eroded for real scientific use cases (Stevens, 20 Nov 2025).