OceanBenchmark: Marine Science Evaluation
- OceanBenchmark is a comprehensive evaluation benchmark that measures marine science reasoning and visual perception across text and multimodal inputs.
- It employs expert-curated questions and a strict consensus-based validation process to assess performance on ocean science, sonar, and image tasks.
- Empirical results show significant domain adaptation gains, especially in sonar interpretation and marine organism recognition, highlighting its research impact.
OceanBenchmark is the evaluation arm of the OceanPile suite, introduced as a manually curated benchmark for rigorous assessment of marine science capabilities in LLMs and multimodal foundation models (Xue et al., 25 Apr 2026). Within OceanPile, it is the evaluation-only counterpart to OceanCorpus, which supplies large-scale multimodal pre-training data, and OceanInstruction, which provides domain-specific instruction-tuning data. OceanBenchmark is designed to assess text-only ocean science knowledge and reasoning, multimodal interpretation of marine scientific figures and underwater imagery, sonar image understanding, and fine-grained marine organism recognition under a single benchmark structure (Xue et al., 25 Apr 2026).
1. Definition and position within OceanPile
OceanBenchmark is defined as “a comprehensive benchmark for evaluating the marine science capabilities of LLMs and MLLMs,” consisting of carefully curated questions spanning both textual and multimodal tasks (Xue et al., 25 Apr 2026). Its role is evaluative rather than generative: no training splits are described, and the benchmark is used to measure how well models perform on realistic marine science tasks after generic pre-training or domain adaptation.
Its function is best understood relative to the other two OceanPile components. OceanCorpus is the raw multimodal reservoir integrating sonar data, underwater imagery, marine science visuals, and scientific text. OceanInstruction is the instruction-tuning dataset synthesized via a pipeline guided by the hierarchical Ocean Concept Knowledge Graph. OceanBenchmark is the test suite that measures whether those upstream resources translate into usable marine capabilities in downstream models (Xue et al., 25 Apr 2026).
The benchmark targets four capability areas explicitly emphasized in the OceanPile description: ocean science knowledge and reasoning, visual understanding of marine imagery, sonar image understanding, and fine-grained marine organism recognition. This gives it a narrower but more domain-specific remit than generic QA or VQA benchmarks. A plausible implication is that OceanBenchmark was designed to expose failure modes that broad-domain MLLMs do not reveal, especially in underwater sensing and biodiversity-related recognition.
2. Construction and curation methodology
OceanBenchmark is organized as two specialized sub-benchmarks: a Textual Benchmark for text-only comprehension and a Multimodal Benchmark for multimodal reasoning (Xue et al., 25 Apr 2026). Its construction begins from “authoritative marine science documents and aligned multimodal samples,” after which marine science professionals design multiple-choice questions based on curated content. The retained benchmark items are then filtered by expert validation.
The formal retention criterion is a strict majority rule across annotators. Let there be annotators for a question–answer pair, each providing a binary judgment . A pair is retained only if
This criterion operationalizes expert consensus and is the benchmark’s main explicit quality-control condition (Xue et al., 25 Apr 2026). In the supplied description, this is linked to scientific correctness and agreement across multiple domain experts.
The underlying source material overlaps with OceanCorpus. The benchmark draws from oceanographic textbooks and papers, marine-related web pages, sonar detection datasets, underwater image datasets, and field-collected underwater sonar and optical data captured by AUVs in the Zhoushan region (Xue et al., 25 Apr 2026). OceanBenchmark is therefore not an independent corpus but a filtered, expert-annotated evaluation slice of the broader OceanPile data reservoir.
An important clarification concerns answer format. During construction, questions are described as multiple-choice questions designed by marine science professionals. During evaluation, however, models generate free-form textual answers, and correctness is judged against the ground truth by an LLM-as-a-Judge rather than by exact option matching (Xue et al., 25 Apr 2026). This suggests that the multiple-choice structure is part of authoring and quality control, whereas the final benchmark is operationalized as open-ended QA/VQA.
3. Benchmark structure, modalities, and coverage
OceanBenchmark contains 1,469 samples in total and divides them across one textual task and three multimodal tasks (Xue et al., 25 Apr 2026). Each instance includes a question, the correct answer, an image for multimodal tasks, and detailed metadata.
| Task | Inputs | Samples |
|---|---|---|
| Ocean Science QA | Text-only question | 102 |
| Ocean Science VQA | Marine-themed image or scientific diagram + question | 99 |
| Sonar VQA | Sonar or acoustic image + question | 796 |
| Marine Organisms VQA | Underwater RGB image + question | 472 |
The modality design is notably heterogeneous. The textual component targets ocean science concepts and reasoning. The multimodal component spans scientific figures and marine-themed images, sonar imagery including side-scan, multibeam echosounders, and forward-looking sonar, and underwater RGB photographs of marine organisms (Xue et al., 25 Apr 2026). The paper frames all of these as question answering rather than detection, segmentation, retrieval, or regression.
The task distribution is highly asymmetric. Sonar VQA and Marine Organisms VQA dominate the sample count, which the supplied description interprets as a reflection of emphasis on underwater perception and biodiversity tasks (Xue et al., 25 Apr 2026). Ocean Science VQA, by contrast, is much smaller, suggesting that the benchmark’s multimodal core is less about generic scientific chart reading than about marine-specific visual understanding.
Conceptually, the benchmark spans marine biology, physical and geological oceanography, underwater sensing and robotics, and general ocean science knowledge. The paper does not enumerate a formal concept list for OceanBenchmark itself, but the benchmark is described as aligned in spirit with the Ocean Concept Knowledge Graph used elsewhere in OceanPile (Xue et al., 25 Apr 2026). This suggests that its coverage mirrors the larger taxonomy of marine biology, physical oceanography, marine chemistry, and related subdomains without being explicitly generated from the graph in the same way as OceanInstruction.
4. Evaluation protocol and reported empirical behavior
OceanBenchmark is evaluated with an LLM-as-a-Judge protocol. Model outputs are compared against the corresponding ground-truth answers by another LLM, and the reported scores are percentages of correctly judged answers (Xue et al., 25 Apr 2026). Although the paper does not state a formal accuracy equation, the reported quantity is effectively
Scores are reported separately for Ocean Science QA, Ocean Science VQA, Sonar VQA, and Marine Organisms VQA, together with an Overall score for the multimodal benchmark (Xue et al., 25 Apr 2026). Human experts are central to benchmark construction, but model scoring in the reported experiments depends on LLM-as-a-Judge rather than human raters.
The evaluated models include open-source baselines, the same open-source models fine-tuned “with OceanPile,” and closed-source MLLMs. In the text-only setting, Qwen3-30B improves from 25.49 to 26.47 on Ocean Science QA after OceanPile fine-tuning, exceeding GPT-5 at 16.67, GPT-4o at 6.86, and slightly surpassing Gemini-3-Flash at 24.51 (Xue et al., 25 Apr 2026). In the multimodal setting, Qwen3-VL-8B rises from 21.21 to 29.29 on Ocean Science VQA, from 8.04 to 19.97 on Sonar VQA, from 9.96 to 48.52 on Marine Organisms VQA, and from 13.07 to 32.59 Overall after fine-tuning with OceanPile (Xue et al., 25 Apr 2026).
These results expose a pronounced task asymmetry. Sonar VQA is especially difficult: GPT-5 scores 0.71, GPT-4o 5.71, baseline Qwen3-VL-8B 8.04, and the OceanPile-adapted version 19.97 (Xue et al., 25 Apr 2026). Marine Organisms VQA shows the largest domain-adaptation gain, with Qwen3-VL-8B moving from 9.96 to 48.52. Ocean Science VQA and Ocean Science QA improve more moderately. The supplied interpretation is that generic models may already encode some ocean science concepts, but sonar understanding and fine-grained marine biodiversity remain poorly represented in broad-domain pre-training (Xue et al., 25 Apr 2026).
Gemini-3-Flash remains competitive, obtaining 32.32 on Ocean Science VQA, 11.11 on Sonar VQA, 50.21 on Marine Organisms VQA, and 31.21 Overall, but Qwen3-VL-8B with OceanPile slightly exceeds it overall at 32.59 (Xue et al., 25 Apr 2026). This comparison is central to the benchmark’s empirical role: it functions not only as a test set, but as evidence that marine-specific pre-training and instruction-tuning can alter the ranking of models relative to generic-domain baselines.
5. Scientific significance, scope, and limitations
OceanBenchmark is significant because it concentrates on ocean-centric QA/VQA tasks that generic benchmarks underspecify. The benchmark directly targets marine biodiversity understanding, underwater sensing via sonar interpretation, and conceptual ocean science grounded in scientific literature and figures (Xue et al., 25 Apr 2026). The practical applications identified in the supplied description include marine AI research, autonomous underwater vehicles and robotics, biodiversity monitoring and habitat mapping, and support for scientific interpretation of marine figures and literature.
The benchmark’s design also exposes a recurring misconception in ocean AI evaluation: that general-purpose multimodal performance transfers smoothly to marine settings. The reported results argue against that assumption, especially in sonar and marine organism recognition (Xue et al., 25 Apr 2026). OceanBenchmark therefore serves as a stress test for domain mismatch between mainstream foundation-model training data and oceanic sensing modalities.
Its limitations are largely implicit rather than formalized in a separate section. The benchmark size, 1,469 samples, is modest relative to large general-purpose evaluation suites. All tasks are QA/VQA; there are no dedicated tasks for segmentation, detection, retrieval, trajectory prediction, time-series forecasting, or map-based reasoning (Xue et al., 25 Apr 2026). The modality coverage focuses on text and images, including sonar and RGB imagery, but does not include explicit time-series or gridded climate-state tasks.
Additional limitations arise from data provenance and evaluation strategy. The field-collected imagery includes data from the Zhoushan region, which may bias organism and sonar distributions. Model scoring depends on LLM-as-a-Judge, so the evaluation inherits the judge model’s biases and competence boundaries (Xue et al., 25 Apr 2026). The paper does not report a human-in-the-loop audit of model outputs during final scoring. A plausible implication is that future revisions could broaden geographic and taxonomic coverage, add new modalities, and incorporate stronger human validation of generated answers.
6. Relation to the broader ocean benchmarking landscape and resource access
OceanBenchmark belongs to a broader ecosystem of ocean-specific benchmarks, but its task formulation is distinct. OceanBench focuses on sea surface height interpolation and data-fusion pipelines for SSH reconstruction from satellite altimetry (Johnson et al., 2023). IceBench addresses sea ice type classification from multi-channel satellite and reanalysis data (Taleghan et al., 22 Mar 2025). ZooplanktonBench targets geo-aware zooplankton detection, classification, and tracking in cluttered in situ imagery (Liu et al., 24 May 2025). OceanForecastBench standardizes data-driven global ocean forecasting with observation-based evaluation (Jia et al., 24 Nov 2025). IOMB is a benchmarking package for evaluating ocean and biogeochemical fields in CMIP Earth system models (Fu et al., 2022). OceanBenchmark differs from all of these by centering open-ended QA and VQA over text, scientific figures, sonar imagery, and underwater organism imagery (Xue et al., 25 Apr 2026).
This naming proximity can be misleading. OceanBenchmark is not the same resource as OceanBench: the latter is an SSH-oriented framework and benchmark suite, whereas OceanBenchmark is the evaluation-only component of OceanPile (Johnson et al., 2023, Xue et al., 25 Apr 2026). The distinction is substantive as well as terminological. OceanBench evaluates field reconstruction quality with RMSE and spectral diagnostics; OceanBenchmark evaluates marine-language and multimodal reasoning with judged answer accuracy.
The benchmark is publicly released through the OceanPile collection on Hugging Face, with related code linked through the OceanGPT repository and an associated project homepage (Xue et al., 25 Apr 2026). OceanBenchmark is distributed in CSV format, with rows containing question, answer, image for multimodal tasks, and additional metadata fields. Its intended usage is straightforward: load the CSV, prompt a model with the question and image when applicable, and compute per-task and overall judged accuracy with an LLM-as-a-Judge protocol matching the paper’s evaluation setup (Xue et al., 25 Apr 2026).
In that wider landscape, OceanBenchmark occupies the niche of marine scientific reasoning and perception assessment for foundation models. It does not replace ocean forecasting, sea-ice classification, SSH reconstruction, zooplankton detection, or Earth-system model benchmarking. Instead, it complements them by providing a standardized testbed for whether foundation models can answer scientifically grounded marine questions across text, figures, sonar, and underwater imagery (Xue et al., 25 Apr 2026).