OceanInstruction for Marine AI

• OceanInstruction is an instruction-tuning layer that bridges raw ocean data and model behavior using curated question–answer pairs tailored to marine science.
• It leverages a hierarchical Ocean Concept Knowledge Graph to systematically organize and align diverse marine content from textbooks, sonar data, and images.
• Empirical validation shows significant performance gains in multimodal tasks like sonar analysis and marine organism reasoning after fine-tuning with OceanInstruction.

Searching arXiv for the cited OceanPile paper and closely related ocean-AI context papers. arXiv search: OceanPile OceanInstruction ocean multimodal corpus foundation models OceanInstruction is the instruction-tuning component of OceanPile, a large-scale multimodal ocean corpus for foundation models, and functions as the layer that teaches ocean-oriented LLMs and MLLMs how to use the corpus in instruction-following settings (Xue et al., 25 Apr 2026). It is a high-quality set of question–answer instruction pairs designed to train models to follow marine-science prompts, interpret multimodal ocean content, and produce scientifically grounded responses. Within OceanPile’s three-part structure—OceanCorpus for pretraining, OceanInstruction for supervised fine-tuning and instruction tuning, and OceanBenchmark for evaluation—OceanInstruction is the bridge between raw ocean content and usable model behavior.

1. Position within OceanPile

OceanInstruction is explicitly described as the instruction-tuning heart of OceanPile. OceanPile itself is introduced as a response to a severe data bottleneck in ocean AI: ocean data are fragmented, noisy, weakly labeled, and distributed across heterogeneous sources such as textbooks, papers, sonar datasets, underwater images, and field-collected data. In that framework, OceanInstruction provides domain-specific instruction-following data for supervised fine-tuning of ocean foundation models, instruction tuning for both text-only and multimodal models, helping models learn ocean-science reasoning, and bridging the gap between raw ocean content and usable model behavior (Xue et al., 25 Apr 2026).

The distinction between OceanInstruction and OceanCorpus is functional. OceanCorpus is the pretraining substrate, whereas OceanInstruction is the “teach the model how to use it” layer. This division situates OceanInstruction within the now-familiar foundation-model pipeline of corpus construction, instruction tuning, and benchmarked evaluation, but the paper emphasizes that generic instruction corpora do not capture the conceptual structure, terminology, and cross-modal reasoning patterns of ocean science. The dataset is therefore positioned not as a generic synthetic instruction resource, but as a knowledge-augmented synthetic dataset that is more scientifically structured than mainstream instruction corpora.

2. Motivation and scientific rationale

The motivation for OceanInstruction follows directly from the paper’s diagnosis of ocean AI. Ocean data are said to be highly fragmented across disparate sources and to exhibit multi-modal, high-noise, and weakly labeled characteristics, lacking unified schemas and semantic alignment. Although Multimodal LLMs have achieved strong results in general domains, their application to ocean science is described as severely constrained by the absence of large-scale, well-aligned multimodal datasets tailored to marine environments (Xue et al., 25 Apr 2026).

OceanInstruction was created to address this gap at the instruction-following stage. Its intended uses are explicitly enumerated: supervised fine-tuning of ocean foundation models, instruction tuning for both text-only and multimodal models, helping models learn ocean-science reasoning rather than simple recognition, and connecting raw ocean content to practically useful model behavior. The paper further argues that existing synthetic instruction generation methods tend to miss the depth and structure needed for specialized scientific domains.

A central implication of this framing is that ocean instruction tuning is treated as a semantic alignment problem, not merely a data-scaling problem. The dataset is designed to reflect the conceptual organization of marine science and to preserve links between source content, disciplinary context, and authoritative external knowledge. This suggests that OceanInstruction is meant to encode not only task format, but also scientific ontology.

3. Synthesis pipeline and the Ocean Concept Knowledge Graph

The defining methodological feature of OceanInstruction is its synthesis pipeline, which is grounded in a hierarchical Ocean Concept Knowledge Graph rather than generated by unconstrained prompting. The paper presents three broad stages.

First, marine knowledge is organized into a hierarchical concept inventory. Starting from a text corpus of marine textbooks and expert-curated materials, the pipeline defines a set of primary disciplines,

$\mathcal{P} = \{P_k\}_{k=1}^{K},$

where each $P_k$ is a core discipline such as marine biology, physical oceanography, or marine chemistry. For each primary discipline, GPT-4o is used to extract candidate subcategories from the marine text corpus conditioned on that discipline,

$$\mathcal{S}_k = \mathcal{M}(\mathcal{D}_{\text{text} \mid P_k}).$$

These subcategories are then refined by merging similar items and filtering out infrequent ones using a threshold $\tau_f$,

$$\hat{\mathcal{S}_k} = \mathcal{M}(\mathcal{S}_k; \tau_f).$$

The result is a cleaner hierarchical concept graph spanning primary disciplines and refined subtopics (Xue et al., 25 Apr 2026).

Second, each multimodal input is mapped into this concept structure. For each input $\mathcal{X}_i$, which may be a text document $D_j$, a visual element $V_j$, or a visual item with associated description or label $T_j$, the pipeline assigns the most relevant primary discipline $P_k$ and refined subcategory $P_k$0. It also retrieves supporting authoritative external knowledge,

$P_k$1

This stage anchors each sample in both a semantic hierarchy and curated contextual knowledge.

Third, GPT-4o generates the instruction–answer pairs:

$P_k$2

with output

$P_k$3

The generator therefore uses the source content, the discipline label, the refined subtopic, and authoritative supporting knowledge to produce a scientifically grounded question and answer.

This pipeline is presented as novel because it is not simply “ask a model to invent questions.” Its design principle is systematic coverage of the conceptual landscape of ocean science. A plausible implication is that the Ocean Concept Knowledge Graph serves simultaneously as a coverage mechanism, a semantic alignment mechanism, and a guardrail against shallow synthetic diversity.

4. Instruction modalities, task families, and data schema

The paper identifies three main categories of instructions in OceanInstruction. Textual instructions are generated from textbooks and papers and probe key marine science concepts and foundational knowledge. Visual instructions are generated from diagrams and images and focus on interpreting visual scientific content. Task-specific multimodal instructions are generated from labeled underwater detection data and target applications such as species identification, object analysis, sonar interpretation, and marine scene understanding (Xue et al., 25 Apr 2026).

This task design makes OceanInstruction broader than a simple VQA dataset. It spans knowledge questions, image reasoning, and task-oriented multimodal supervision. The paper also specifies the task families implied by the dataset even though it does not print individual examples: knowledge questions from textbooks and papers, visual interpretation of scientific diagrams and images, sonar analysis, species recognition, and object detection interpretation.

OceanInstruction is released in two aligned variants.

Variant	Size	Additional field
Text-only OceanInstruction	69,192 instruction–answer pairs	none
Multimodal OceanInstruction	71,932 instruction–answer pairs	image

The multimodal version includes an additional image field pointing to marine-themed visual content such as underwater photographs, scientific diagrams, or sonar images. Both variants are stored in structured CSV format. Each instance includes a question field and an answer field; the multimodal version also includes an image field containing the filename or path to associated visual content.

The schema is deliberately simple. In practical terms, this simplicity directly supports fine-tuning of both LLMs and MLLMs, while the upstream synthesis pipeline carries the burden of semantic structuring and scientific grounding.

5. Quality control, filtering, and reliability

OceanInstruction incorporates a multi-stage quality control process intended to ensure scientific validity and alignment across modalities. The first stage is automated verification. For a generated pair $P_k$4, multiple MLLMs act as verification agents. Each verifier $P_k$5 assigns a score

$P_k$6

based on factual correctness, relevance, and clarity. The final score is the average across the $P_k$7 verifiers,

$P_k$8

and pairs below a threshold $P_k$9 are removed (Xue et al., 25 Apr 2026).

The second stage is human verification. After automated filtering, domain experts manually review sampled instances through a multi-round process: independent evaluators score quality, instances with divergent scores are flagged, flagged items are discussed in a consensus meeting, and unresolved cases are removed. The final reported inter-annotator agreement is 0.86.

The paper presents this IAA value as evidence of strong reliability. More importantly, the combination of multi-MLLM scoring and consensus-based expert review indicates that OceanInstruction is not treated as a purely synthetic resource whose quality is assumed from the generator alone. The quality-control design reflects the scientific status of the target domain: correctness, relevance, and clarity are all explicitly operationalized as filtering criteria.

6. Role in model training and empirical validation

Within OceanPile, OceanInstruction is the instruction-tuning layer used after pretraining on OceanCorpus and before evaluation on OceanBenchmark. In the paper’s framing, it teaches the model to follow marine-science instructions, connect concepts to observations, reason over sonar and imagery, and answer domain-specific questions accurately (Xue et al., 25 Apr 2026).

The paper validates OceanInstruction by fine-tuning baseline models and measuring improvement on OceanBenchmark. For the text-only setting, Qwen3-30B-A3B-Instruct on Ocean Science QA improves from 25.49 before OceanPile to 26.47 after fine-tuning with OceanPile, a gain of +0.98. For the multimodal setting, Qwen3-VL-8B-Instruct improves on Ocean Science VQA from 21.21 to 29.29 (+8.08), on Sonar VQA from 8.04 to 19.97 (+11.93), on Marine Organisms VQA from 9.96 to 48.52 (+38.56), and overall from 13.07 to 32.59 (+19.52).

The paper further states that the fine-tuned multimodal model performs better overall than GPT-5 and GPT-4o on the benchmark and is slightly above Gemini-3-Flash on the overall multimodal score. These gains are presented as the principal empirical evidence for the utility of OceanInstruction.

The performance pattern is notable. Improvements are especially large on sonar and marine organism reasoning, which are precisely the areas where generic instruction corpora would be expected to be weakest. This suggests that OceanInstruction’s contribution lies not only in adding more supervision, but in aligning supervision with marine-science semantics, modality structure, and task-specific reasoning demands.