SailorFog-QA: Deep-Sea Species Recognition
- SailorFog-QA is a deep-sea species identification benchmark built from authentic JAMSTEC imagery.
- The dataset consists of 100 high-resolution images with multiple-choice questions and expert annotations.
- Reported accuracies of 34% to 50% highlight current multimodal LLM limitations in specialized visual taxonomy.
Searching arXiv for the specified paper and the requested dataset name to ground the article in current research. J-EDI QA is a benchmark for evaluating multimodal LLMs on deep-sea organism recognition in Japanese, built from the JAMSTEC Earth Deep-sea Image archive and introduced in the paper "J-EDI QA: Benchmark for deep-sea organism-specific multimodal LLM" (Yoshida et al., 2024). It targets a narrow but technically demanding task: species-oriented visual question answering on authentic deep-sea imagery. The benchmark consists of 100 images paired with multiple-choice questions and expert-authored answers, with an emphasis on marine organisms rather than generic visual reasoning. Its stated purpose is to measure how well multimodal models can identify deep-sea taxa and operate with Japanese marine-biology terminology, while also exposing the limitations of state-of-the-art systems in a specialized scientific domain.
1. Archival basis and domain scope
J-EDI QA is derived from J-EDI, the JAMSTEC Earth Deep-sea Image archive maintained by the Japan Agency for Marine-Earth Science and Technology. J-EDI is a publicly accessible archive of deep-sea videos and still images collected from 1982 to 2019, comprising over one million still frames and videos, mostly from deep waters around Japan (Yoshida et al., 2024). The archive includes marine organisms such as fish, crustaceans, echinoderms, cnidarians, and molluscs, including many rare or hydrothermal-vent species, as well as benthic substrates and physical phenomena such as undersea volcanic activity and hydrothermal vents. Each item is accompanied by capture location and general taxonomic tags.
This archival basis is central to the benchmark’s character. Rather than drawing on synthetic, textbook, or web-curated imagery, the benchmark uses actual deep-sea photographs from a research archive. A plausible implication is that the task inherits the observational constraints of underwater imaging, including difficult orientations, background clutter, and domain-specific morphology, which are less prominent in broad-coverage multimodal benchmarks.
2. Motivation and relationship to existing QA benchmarks
The benchmark is motivated by a gap in multimodal evaluation. Existing multimodal QA benchmarks such as MMMU, TextVQA, GQA, and Perception Test evaluate broad image or video reasoning, while scientific-domain QA datasets such as ScienceQA and college-level image tests such as MMMU exist but do not probe expert-level, species-specific image understanding in deep-sea contexts (Yoshida et al., 2024).
J-EDI QA therefore narrows the problem definition to the question of species identification in real deep-sea photographs. Its stated aims are threefold: to measure how well multimodal LLMs recognize and name deep-sea taxa, to provide a testbed in Japanese for marine-biology terminology and visual identification skills, and to seed future domain-adapted LLMs for underwater survey automation, education, and outreach.
A common misconception in multimodal evaluation is that high performance on general-purpose visual reasoning implies competence in specialized scientific recognition tasks. The reported results in J-EDI QA directly counter that assumption: the benchmark is explicitly designed around expert-level, species-specific image understanding rather than generic object recognition or broad commonsense reasoning.
3. Dataset construction and annotation protocol
The benchmark contains images selected by JAMSTEC researchers to represent a variety of fishes, crustaceans, and “other invertebrates” (Yoshida et al., 2024). Individual image files are approximately 0.5–3 MB, described as sufficiently high in resolution to reveal diagnostic features. All images are drawn from the public J-EDI archive under non-commercial terms of use.
The question format is uniform. Each item is a four-option multiple-choice question in Japanese asking, “画像の生物は何でしょうか?” (“What organism is depicted?”), followed by four Japanese species names labeled A–D. For evaluation, a model must both select one of A/B/C/D and provide a short explanation, or 解説, justifying the choice. Expert labels and brief diagnostic commentaries were authored by JAMSTEC deep-sea biologists.
The benchmark evaluates species-level visual identification. The paper gives as an example the distinction between similar sharks such as Somniosus pacificus and Hexanchus griseus. The required discrimination depends on recognition of anatomical traits including number of gill slits, body shape, coloration patterns, and appendage lengths, together with knowledge of deep-sea habitus and, occasionally, geological context such as the presence of hydrothermal vents. This suggests that the task is not limited to nominal classification; it also probes whether a model can ground its answer in visually and biologically diagnostic evidence.
4. Evaluation design and task definition
The evaluation protocol is deliberately simple. Accuracy is the sole metric, defined as
The benchmark is conducted in Japanese, so it simultaneously tests visual recognition and Japanese taxonomic vocabulary (Yoshida et al., 2024). During model runs, no auxiliary metadata are provided: only the image and the question with answer options are given, and species names from the original J-EDI page are withheld. The models reported in the paper are OpenAI o1, identified in the summary as “GPT-4o vision,” and GPT-4o. The prompt template is likewise standardized: “画像を見て、以下の問題に答えてください。… 画像の生物は何でしょうか? A… B… C… D…”
The benchmark’s design choices emphasize controlled measurement rather than elaborate scoring. The single-metric setup reduces interpretive ambiguity, while the absence of archive metadata prevents trivial lookup behavior. At the same time, because models must also provide a short explanation, the task retains a qualitative dimension even though the reported score is based on answer correctness alone.
5. Quantitative results and category-wise performance
The paper reports results for 100 question-answer items. OpenAI o1 achieved 50% correct, corresponding to 50/100, and GPT-4o achieved 39% correct, corresponding to 39/100 (Yoshida et al., 2024). Humans with some domain knowledge scored approximately 40%, which is roughly on par with GPT-4o but below o1.
For o1, category-wise accuracy was reported as 47.2% on fishes across 36 items, 70.0% on crustaceans across 20 items, and 43.2% on other invertebrates across 44 items. GPT-4o showed lower performance overall, particularly on other invertebrates, where the reported accuracy was 34.1%.
| Evaluation slice | Count | Accuracy |
|---|---|---|
| o1 overall | 100 | 50% |
| GPT-4o overall | 100 | 39% |
| Humans with some domain knowledge | — | ~40% |
| o1 fishes | 36 | 47.2% |
| o1 crustaceans | 20 | 70.0% |
| o1 other invertebrates | 44 | 43.2% |
| GPT-4o other invertebrates | 44 | 34.1% |
The category distribution is informative. The paper observes that crustacean morphology tends to be more distinctive, and associates this with higher model accuracy. A plausible implication is that benchmark difficulty is shaped not only by taxonomic rarity but by the density and reliability of visual diagnostic cues available in a single underwater image.
6. Error patterns, interpretive significance, and limitations
Several failure modes are identified in the reported evaluation (Yoshida et al., 2024). One is visual ambiguity, including cases in which the animal orientation is poor. Another is overreliance on non-diagnostic cues, such as the presence of a “JAMSTEC” logo or background geology. A third is confusion among morphologically similar taxa, including different deep-sea holothurians.
These error modes clarify what the benchmark measures. The task is not merely a test of coarse object detection; it stresses discriminative morphology under domain-specific imaging conditions. The overreliance on incidental cues further indicates that current multimodal systems may produce superficially plausible identifications without anchoring their judgments in the most taxonomically relevant anatomical features. This suggests a mismatch between broad visual-language competence and expert biological recognition.
The reported ceiling of around 50% for the strongest evaluated model is presented as evidence that, as of December 2024, even state-of-the-art models lack expert-level deep-sea species comprehension. The paper frames this as a limitation of current systems rather than a deficiency of the benchmark. In that sense, J-EDI QA functions as a stress test for domain specialization.
7. Scientific role and proposed future extensions
The benchmark is presented as significant for several reasons. First, it offers domain depth: unlike general multimodal benchmarks, it focuses on expert-level species identification in a scientifically important environment (Yoshida et al., 2024). Second, it has a Japanese-language focus, addressing a gap in benchmarks for non-English scientific vocabularies. Third, it serves as a test of real-world utility, since performance around 50% indicates that leading multimodal LLMs of late 2024 do not yet exhibit robust visual taxonomy skills in specialized marine contexts.
The paper also positions J-EDI QA relative to broader datasets. MMMU, GQA, and TextVQA are described as assessing general reasoning, object attributes, and spatial relations, while ScienceQA is described as probing scientific diagrams, often at textbook complexity. J-EDI QA is narrower in scope, concentrating on species identification, but richer in domain nuance through its emphasis on rare oceanic taxa and subtle morphological cues.
Several future extensions are explicitly proposed: expanding the benchmark to more images and video clips, increasing difficulty by adding more answer options in the manner of MMMU-Pro, translating the QA into English for cross-lingual analysis, and incorporating Retrieval-Augmented Generation over specialized catalogs or illustrated field guides. These proposals indicate that the benchmark is intended not only as an evaluative artifact but also as an initial foundation for domain-adapted multimodal models in marine science.
