BioBench: Ecology Vision Benchmark

• BioBench is an ecology-focused vision benchmark unifying nine ecological tasks and 3.1 million images to assess scientific visual representations.
• It replaces unreliable ImageNet-1K transfer performance by employing lightweight probing with class-balanced macro-F1 metrics and domain-specific measures.
• The benchmark offers a reproducible, single-file API framework that standardizes evaluation across diverse modalities and taxonomic groups.

BioBench is an ecology-focused vision benchmark introduced to replace the increasingly unreliable practice of using ImageNet-1K transfer performance as a proxy for “general visual representation quality” in scientific applications. It is presented both as a practical benchmark for ecological computer vision and as a blueprint for building more reliable benchmarks for scientific machine learning more broadly. BioBench unifies nine publicly released, application-driven tasks spanning four taxonomic kingdoms and six acquisition modalities, totaling 3.1 million images, and evaluates frozen visual backbones through lightweight probing with class-balanced macro-F1 as the default metric, plus domain-standard metrics for selected tasks (Stevens, 20 Nov 2025).

1. Motivation and the breakdown of ImageNet as a proxy

BioBench is motivated by the claim that ImageNet-1K top-1 accuracy no longer predicts which visual models will work best on realistic scientific imagery. Across 46 modern vision checkpoints spanning 11 model families and multiple training paradigms, ImageNet top-1 explains only 34% of the variance in BioBench performance, with $R^2 = 0.34$. Rank agreement is also limited: the reported Spearman correlation between ImageNet ranking and BioBench ranking is $\rho = 0.55$, with a 95% confidence interval of $[0.45, 0.64]$ and significance at $p < 0.0005$ using 5000-permutation randomization. The paper gives the ranking-error interpretation

$$\text{mis-ranking probability} = \frac{1}{2}(1-\rho),$$

so $\rho = 0.55$ implies that the ImageNet-preferred model is worse on BioBench about 22% of the time (Stevens, 20 Nov 2025).

The mismatch becomes more severe among stronger models. Restricting attention to checkpoints above 75% ImageNet top-1, rank concordance drops to $\rho = 0.42$, with 95% confidence interval $[0.15, 0.65]$ and $p < 0.01$ by 5000-permutation randomization, implying mis-ranking about 30% of the time. The introduction also reports that across three representative ecology tasks, Spearman correlation falls below 0.25 once models exceed the now-common 75% ImageNet threshold. This is summarized as a “ranking cliff”: among stronger models, better ImageNet performance ceases to predict better performance on real ecological tasks (Stevens, 20 Nov 2025).

The paper attributes this failure to two linked mismatches. First is distribution mismatch: ImageNet is built from web photos, whereas ecological imagery includes camera-trap infrared, drone imagery, microscope images, specimen photos, and related regimes with very different spectra, noise, context, and viewpoint. Second is task mismatch: scientific applications often require extremely fine-grained and long-tailed discrimination, such as rare species or subtle traits, which differs from ImageNet’s 1000 object categories. BioBench is therefore designed to measure representation quality directly on application-driven ecological tasks rather than through a proxy benchmark whose predictive validity has degraded (Stevens, 20 Nov 2025).

2. Benchmark composition and ecological coverage

BioBench consolidates nine publicly released ecological vision tasks spanning four taxonomic kingdoms—animals, plants, fungi, and protists/protozoa—and six acquisition modalities, for a total of 3.1 million images and about 337 GB of data. The modalities are described across the paper as drone RGB footage or frames, curated web video, microscope micrographs, in-situ photos, specimen photos, and camera-trap frames; another summary phrases them as camera-trap RGB/IR, drone video frames, museum specimens, in-situ macro, web video, and micrographs. The benchmark is deliberately heterogeneous along both biological and imaging dimensions (Stevens, 20 Nov 2025).

The task set spans species identification, individual re-identification, behavior classification, and trait prediction. The paper emphasizes that this mix includes both “proxy-driven” tasks and “mission-driven” tasks corresponding directly to conservation and ecological workflows such as biodiversity monitoring, camera-trap-based wildlife survey, plant and fungal identification, animal behavior analysis, and functional trait inference (Stevens, 20 Nov 2025).

Task	Description	Task type
Beluga	individual re-identification of beluga whales	individual re-identification
FishNet	fish functional trait prediction from specimen images	trait prediction
FungiCLEF	fungi species identification	species identification
Herbarium19	plant species classification from herbarium specimen images	species classification
iWildCam	mammal species classification from camera-trap images	species classification
KABR	behavior recognition for Kenyan ungulates from drone videos	behavior recognition
MammalNet	mammal behavior recognition from web video	behavior recognition
Plankton	protist/plankton species classification from micrographs	species classification
Pl@ntNet	plant species identification from in-situ photos	species identification

This breadth is central to the benchmark’s design. BioBench is not a single-dataset replacement for ImageNet transfer; it is a standardized suite intended to expose failures of proxy evaluation across diverse ecological settings, label granularities, and acquisition regimes (Stevens, 20 Nov 2025).

3. Evaluation protocol, metrics, and uncertainty reporting

A central design goal is to isolate backbone representation quality from task-specific engineering. Every evaluated model only needs to expose a frozen embedding function

$f:\text{image}\to\mathbb{R}^d.$

BioBench then fits lightweight classifiers on top of those fixed representations. The paper describes this as fitting “linear or logistic probes per task,” and in practice evaluation proceeds by extracting features from all images with the frozen backbone and training a simple task-level classifier rather than fine-tuning the entire model. This yields a more reproducible comparison of transferability across pretraining families (Stevens, 20 Nov 2025).

The default metric is class-balanced macro-F1, chosen because the tasks are long-tailed and ecology often cares about rare classes rather than aggregate accuracy alone. The paper’s intended definition is the standard one:

$\rho = 0.55$0

and

$\rho = 0.55$1

Two tasks are exceptions: FishNet and FungiCLEF use their domain-standard metrics rather than default macro-F1. The paper also notes that some prior state-of-the-art comparisons in the literature use non-macro-F1 metrics, specifically macro-accuracy for MammalNet and micro-accuracy for Herbarium19, whereas BioBench reports results under its own unified evaluation convention (Stevens, 20 Nov 2025).

Uncertainty estimation is part of the benchmark rather than an afterthought. BioBench reports bootstrap confidence intervals and uses permutation or randomization tests for the benchmark-validity analyses. The correlation-vs-threshold analysis includes 95% bootstrapped confidence intervals around the Spearman curves, and the main text reports 95% confidence intervals for both $\rho = 0.55$2 and $\rho = 0.55$3, with significance assessed by 5000-permutation randomization tests. This emphasis on confidence estimates and ranking robustness is part of the benchmark’s stated design philosophy, intended to reduce what the paper, citing prior work, frames as “benchmark lottery” effects (Stevens, 20 Nov 2025).

Computationally, the benchmark is designed to be routine rather than prohibitive. A single Python API handles downloading data, fitting lightweight classifiers, and reporting metrics. The abstract states that ViT-L models evaluate in about 6 hours on an NVIDIA A6000 GPU, while the main text also states that ViT-B/16 and ViT-L/14 checkpoints finish in about one hour on an A6000, with larger models scaling predictably; the paper further states, “We use a single Nvidia A6000 GPU to evaluate all models.” Evaluation can run on local hardware or parallelize across SLURM clusters (Stevens, 20 Nov 2025).

4. Empirical findings and benchmark rankings

The benchmark results reinforce the paper’s critique of ImageNet-based model selection. Among the listed checkpoints, image-text models are especially strong on BioBench. SigLIP 2 ViT-1B/16 achieves the best reported BioBench mean at 43.5. DINOv2 ViT-g/14 and BioCLIP 2 ViT-L/14 each score 41.7 mean. By contrast, AIMv2 reaches only 36.9 mean despite 86.7 ImageNet top-1. The paper also notes that SAM 2 and V-JEPA perform poorly on ImageNet but better than random on some BioBench tasks, although they remain weak overall on the benchmark (Stevens, 20 Nov 2025).

The time-series discussion sharpens the same point. The paper states that most newly released “generalist” models do not improve BioBench over time; only CLIP, SigLIP, and SigLIP 2 are said to set new state-of-the-art BioBench scores. This is used to argue that progress on general-domain computer-vision leaderboards should not be assumed to transfer into ecological or science-facing deployment (Stevens, 20 Nov 2025).

A plausible implication is that BioBench is not primarily a leaderboard for ecology alone, but a validity test for claims about “general” visual representation quality. Its headline results show that once models reach the modern high-accuracy ImageNet regime, frontier ordering on ImageNet and frontier ordering on ecological vision diverge materially, and sometimes systematically (Stevens, 20 Nov 2025).

5. Software, reproducibility, and the benchmark as a construction recipe

The software and reproducibility story is a major part of BioBench’s contribution. The benchmark is framed as a response to fragmentation in ecological ML, where prior datasets often came with incompatible repositories, file layouts, split definitions, and metric scripts. BioBench provides a single-file Python downloader and a unified API around the minimal model contract $\rho = 0.55$4. It standardizes evaluation through frozen embeddings, task-specific probing, macro-F1 reporting, and bootstrap uncertainty, while reducing the integration overhead for new backbones (Stevens, 20 Nov 2025).

The paper states that code and predictions are available at https://github.com/samuelstevens/biobench, and benchmark results are available at https://samuelstevens.me/biobench. It also states that individual model predictions and aggregate statistics will be released, explicitly supporting transparent post hoc analysis and independent reproduction of rankings (Stevens, 20 Nov 2025).

Conceptually, BioBench is presented not only as an ecology benchmark but as a recipe for scientific benchmark construction. The benchmark-design recommendations that emerge from the paper are explicit. Scientific ML benchmarks should be application-driven rather than proxy-driven; they should span meaningful axes of variation in the target domain, such as acquisition modality, task type, and subdomain diversity; they should use class-balanced or otherwise domain-appropriate metrics rather than defaulting to top-1 accuracy; they should separate backbone representation quality from problem-specific tuning when backbone comparison is the goal; and they should report uncertainty, confidence intervals, significance tests, and ranking stability (Stevens, 20 Nov 2025).

6. Scope, limitations, and terminological distinctions

The paper is explicit about BioBench’s scope and limits. First, the benchmark is ecology-focused. Other sciences, including medicine or manufacturing, may require very different modalities, task structures, and evaluation priorities. Second, frozen-feature probing is a deliberate simplification: it isolates representation quality, but it may underestimate the gains achievable through task-specific fine-tuning. Third, macro-F1 emphasizes tail-class performance, which is often appropriate in ecology, but some operational settings may instead prefer threshold-dependent metrics such as precision at fixed recall. The benchmark also concentrates on classification-like tasks rather than detection, segmentation, or calibration-heavy scientific tasks (Stevens, 20 Nov 2025).

A recurring misconception is to treat “BioBench” as a generic label for any biology-facing benchmark. In the literature, similarly named resources refer to substantially different objects. “BioBench-Allie,” for example, is a fixed-size biological RDF benchmark used in knowledge-graph query-performance studies rather than an ecology vision suite (Salehpour et al., 2020). Other biology benchmarks with superficially similar names address different modalities and objectives, including LABBench2 for AI systems performing biology research tasks (Laurent et al., 4 Feb 2026), BioProBench for biological protocol understanding and reasoning (Liu et al., 11 May 2025), BixBench for open-ended computational biology analysis by LLM agents (Mitchener et al., 28 Feb 2025), and BioXArena for end-to-end biomedical ML coding by LLM agents (Li et al., 15 May 2026). BioBench, in its strict sense, refers to the ecology-vision benchmark introduced to move beyond ImageNet as a model-selection proxy for scientific imagery (Stevens, 20 Nov 2025).

The benchmark’s main takeaway is straightforward. ImageNet-driven model selection is no longer reliable for scientific vision, at least in ecology. BioBench responds by measuring transfer directly on diverse, application-grounded ecological tasks with heterogeneous modalities, biologically realistic long tails, and domain-relevant metrics. In that sense it functions simultaneously as a benchmark, a critique of proxy evaluation, and a template for domain-faithful benchmark design in scientific machine learning (Stevens, 20 Nov 2025).