OceanPile: Multimodal Ocean Foundation Models

• OceanPile is a comprehensive multimodal data infrastructure unifying marine texts, images, sonar, and field data for foundation model development.
• It is organized into three components—OceanCorpus, OceanInstruction, and OceanBenchmark—that support pretraining, instruction tuning, and evaluation.
• Empirical results show significant performance gains on marine tasks, particularly in sonar analysis and marine species recognition.

Searching arXiv for OceanPile and closely related marine multimodal foundation-model papers. OceanPile is a publicly released, large-scale multimodal corpus for ocean foundation models, introduced to address what its authors identify as the central bottleneck in marine artificial intelligence: the absence of large, well-aligned, scientifically valid datasets suitable for training and evaluating ocean-domain LLMs and multimodal LLMs. It is organized as a three-part data stack—OceanCorpus for pretraining, OceanInstruction for instruction tuning, and OceanBenchmark for evaluation—and is explicitly designed to integrate scientific text, scientific visuals, sonar imagery, underwater optical imagery, and field-collected multimodal data into a common framework for marine reasoning (Xue et al., 25 Apr 2026).

1. Problem setting and conceptual motivation

OceanPile is motivated by a mismatch between the needs of ocean foundation models and the structure of existing marine data. The paper argues that ocean science possesses abundant raw material—textbooks, papers, web resources, sonar measurements, underwater imagery, scientific figures, and field observations—but that these materials are fragmented across isolated repositories, expressed in incompatible formats, weakly labeled, noisy, and often semantically disconnected across modalities (Xue et al., 25 Apr 2026).

The resource is framed against four interlocking difficulties. First, ocean data are distributed across scientific literature, engineering reports, institutional repositories, observational systems, and niche datasets. Second, marine knowledge is intrinsically multimodal: sonar acoustic imagery, underwater optical images, scientific diagrams, tables, mathematical notation, and technical prose encode complementary but heterogeneous information. Third, these modalities occupy different semantic spaces; sonar signatures, optical features, and domain-specific scientific language do not align naturally without explicit curation. Fourth, many marine datasets are weakly labeled or inconsistently annotated, while document-derived text often carries layout artifacts, duplicated sections, and boilerplate (Xue et al., 25 Apr 2026).

Within that framing, OceanPile is presented as the first comprehensive attempt to unify such sources into a single corpus that supports the full lifecycle of ocean-domain model development. The intended use is not restricted to a single task such as classification or detection. Instead, the corpus targets pretraining, supervised fine-tuning or instruction tuning, and rigorous evaluation for models that must interpret marine scientific text, diagrams, underwater imagery, and sonar data in a scientifically grounded way (Xue et al., 25 Apr 2026).

A common misconception is to treat OceanPile as merely a benchmark or merely a text archive. In the paper’s formulation, it is neither. It is a coordinated multimodal data infrastructure, spanning corpus construction, instruction synthesis, and evaluation. A plausible implication is that the authors view marine foundation-model development as a data-engineering problem as much as a modeling problem.

2. Corpus architecture and scope

OceanPile consists of three coordinated components: OceanCorpus, OceanInstruction, and OceanBenchmark (Xue et al., 25 Apr 2026).

Component	Function	Reported scale
OceanCorpus	Foundational multimodal pretraining corpus	over 300,000 raw PDF documents; over 5 billion tokens of preprocessed multimodal documents
OceanInstruction	Instruction-tuning dataset	69,192 text-only instruction-answer pairs; 71,932 multimodal instruction-answer pairs
OceanBenchmark	Manually curated evaluation suite	1,469 total samples

OceanCorpus serves as the foundational data collection for broad domain adaptation. It integrates five major categories of data: oceanographic textbooks and papers, marine-related web pages, sonar detection datasets, underwater image datasets, and field-collected underwater data from autonomous underwater vehicle deployments. The text side includes textbooks, papers, and web pages; the visual side includes scientific figures, diagrams, tables extracted from documents, underwater photographs, and sonar imagery; and the field collection provides synchronized sonar and optical imagery captured in natural environments (Xue et al., 25 Apr 2026).

OceanInstruction is the supervised fine-tuning component. It appears in two versions: a text-only instruction dataset and a multimodal instruction dataset. Each example contains at least a question and answer field, while multimodal examples also contain an image field storing the path or filename of the visual input. The multimodal portion includes task-specific material for sonar analysis and marine species recognition, as well as marine science visuals such as underwater photographs and scientific diagrams (Xue et al., 25 Apr 2026).

OceanBenchmark is the evaluation component. It is described as a manually curated benchmark for rigorous assessment of LLMs and MLLMs on marine science tasks. It contains a Textual Benchmark and a Multimodal Benchmark, with each instance including a question, the correct answer, an image for multimodal tasks, and metadata (Xue et al., 25 Apr 2026).

The breadth of modality is central to the project’s identity. OceanPile includes scientific text from textbooks, papers, and web pages; scientific figures and visuals extracted from documents; underwater optical imagery of marine organisms and scenes; sonar and acoustic imagery from side-scan and multibeam systems; and synchronized sonar-optical field data collected by AUVs. This suggests that the corpus is designed not merely to expand sample count, but to create cross-modal semantic alignment in a domain where such alignment is ordinarily absent.

3. OceanCorpus: sources, representation, and preprocessing

OceanCorpus is assembled from authoritative and domain-specific sources rather than broad general web crawl material. The textbooks are drawn from academic publishers and institutional repositories and cover chemical, biological, geological, and physical oceanography. Papers come from open-access platforms such as arXiv and Nature portfolio journals, selected using marine-specific keywords, subject categories, and LLM-assisted abstract analysis. Web pages come from marine news sites, educational portals, specialized forums, and expert-recommended links. Sonar data come from public side-scan sonar and multibeam echosounder datasets. Underwater images come from public biodiversity datasets such as WildFish, WildFish++, SCoralDet, and CoralVQA. The field-collected component comes from AUV deployments in the Zhoushan marine region of China, using sonar imaging and high-resolution optical cameras (Xue et al., 25 Apr 2026).

The document-processing pipeline emphasizes source fidelity. When structured sources such as LaTeX and Markdown are available, they are converted directly to clean text while preserving logical hierarchy. PDFs without structured source are converted with a specialized PDF-to-Markdown tool that retains text, images, tables, headings, captions, scientific symbols, formulas, and domain notations. Cleaning then removes headers, footers, page numbers, publication metadata, boilerplate, and non-essential reference sections. LLMs are subsequently used for intelligent filtering and semantic deduplication (Xue et al., 25 Apr 2026).

Web pages are handled through enhanced HTML extraction, with navigation menus, advertisements, and scripts stripped away, followed by textual-quality filtering and deduplication by textual similarity. Images associated with web pages are evaluated by MLLMs for relevance and quality. For target-detection datasets, heterogeneous annotation formats are normalized into a consistent bounding box convention, explicitly given as $[x1, y1, x2, y2]$, and synonymous labels such as “cube” and “square box” are merged. When sonar annotations are sparse, vision-LLMs generate localized descriptions if bounding boxes are present and scene-level descriptions if only image-level labels exist (Xue et al., 25 Apr 2026).

The processed data are stored in CSV format together with associated image files. The paper states that OceanCorpus preserves over 300,000 raw PDF documents and produces over 5 billion tokens of preprocessed multimodal documents, but it does not provide a finer-grained breakdown by source type and does not specify train, validation, or test splits for OceanCorpus (Xue et al., 25 Apr 2026).

This absence of per-source counts is consequential. It means the corpus scale is known at a high level, but modality balance remains unclear. The paper itself notes this indirectly by reporting a very large token count while not reporting counts of sonar images, underwater photographs, synchronized field-collected pairs, or per-source distributions. A plausible implication is that text may still dominate the raw volume of material even though the resource is explicitly multimodal.

4. OceanInstruction and the Ocean Concept Knowledge Graph

OceanInstruction is distinguished by its synthesis pipeline. Rather than generating instructions ad hoc from prompts alone, the paper states that the dataset is produced through a knowledge-augmented process guided by a hierarchical Ocean Concept Knowledge Graph, or OCG (Xue et al., 25 Apr 2026).

The OCG is organized around primary disciplines and subcategories. The set of primary disciplines is written as

$\mathcal{P} = \{P_k\}_{k=1}^{K},$

with examples including marine biology, physical oceanography, and marine chemistry. These top-level disciplines are identified by domain experts and textbook taxonomies. Letting $\mathcal{D}_{\text{text}}$ denote the corpus of marine textbooks and expert-curated materials, candidate subcategories are extracted with GPT-4o:

$$\mathcal{S}_k = \mathcal{M}(\mathcal{D}_{\text{text} \mid P_k}).$$

These candidates are then refined by merging similar items and filtering out those whose occurrence count falls below threshold $\tau_f$:

$$\hat{\mathcal{S}_k} = \mathcal{M}(\mathcal{S}_k; \tau_f).$$

The paper notes typesetting corruption in these formulas, but the intended operational meaning is clear (Xue et al., 25 Apr 2026).

Instruction generation maps each source item $\mathcal{X}_i$—which may be a text document, a visual element, or a visual element with description or label—to the most relevant primary discipline $P_k$ and refined subcategory $\hat{\mathcal{S}_k}$, then retrieves supporting authoritative knowledge $\mathcal{K}_i$. GPT-4o generates an instruction-answer pair through

$\mathcal{P} = \{P_k\}_{k=1}^{K},$0

The paper describes this synthesis as supporting three categories: textual instruction generation from textbooks and papers, visual instruction generation from diagrams and images, and task-specific instruction generation from detection-labeled underwater data for applications such as species recognition and object analysis (Xue et al., 25 Apr 2026).

Quality control is multi-stage. Multiple MLLMs serve as verification agents, each scoring a generated pair $\mathcal{P} = \{P_k\}_{k=1}^{K},$1 on factual correctness, relevance, and clarity. The final score is the average

$\mathcal{P} = \{P_k\}_{k=1}^{K},$2

and pairs with $\mathcal{P} = \{P_k\}_{k=1}^{K},$3 are filtered out. After automated filtering, marine domain experts manually review random samples via a dedicated platform and correct remaining ambiguities or errors. The reported final inter-annotator agreement is 0.86. The technical validation section adds another human-verification layer: multiple evaluators independently score each instance, large score variance triggers discussion in consensus meetings, and items are removed if agreement cannot be reached (Xue et al., 25 Apr 2026).

The role of the OCG is methodological rather than merely descriptive. It functions as a semantic scaffold for coverage and grounding. The paper does not report the final number of disciplines, nodes, or edges in the graph, so the graph’s structural scale is not specified. Nevertheless, the pipeline indicates that instruction diversity is intended to be systematically constrained by domain knowledge rather than left to unconstrained synthetic prompting.

5. OceanBenchmark and reported model performance

OceanBenchmark is a manually curated evaluation suite with 1,469 samples. It is divided into a Textual Benchmark and a Multimodal Benchmark. The textual component, Ocean Science QA, contains 102 samples and targets factual knowledge and reasoning in marine science through text-only questions. The multimodal component contains three sub-benchmarks: Ocean Science VQA with 99 samples, Sonar VQA with 796 samples, and Marine Organisms VQA with 472 samples (Xue et al., 25 Apr 2026).

Benchmark construction is expert-driven. Marine professionals design multiple-choice questions from authoritative marine science documents and aligned multimodal samples. Validation uses majority voting among $\mathcal{P} = \{P_k\}_{k=1}^{K},$4 annotators, with retention only if

$\mathcal{P} = \{P_k\}_{k=1}^{K},$5

The paper does not provide the exact number of annotators per item, nor does it enumerate the full metadata schema beyond stating that detailed metadata are included (Xue et al., 25 Apr 2026).

Empirical validation focuses on fine-tuning and benchmark evaluation rather than pretraining. Qwen3-30B-A3B-Instruct is fine-tuned on OceanInstruction for text tasks, and Qwen3-VL-8B-Instruct is fine-tuned on OceanInstruction for multimodal tasks. Closed-source models—Gemini-3-Flash, GPT-4o, and GPT-5—are also evaluated on OceanBenchmark for comparison. Correctness is determined through an LLM-as-a-Judge procedure that compares model outputs to ground-truth answers (Xue et al., 25 Apr 2026).

On the text benchmark, Qwen3-30B improves from 25.49% to 26.47% after fine-tuning on OceanPile, a gain of 0.98 points. The paper further states that this exceeds GPT-5 at 16.67% and GPT-4o at 6.86%, and is close to Gemini-3-Flash at 24.51%. On the multimodal benchmark, Qwen3-VL-8B improves from 21.21% to 29.29% on Ocean Science VQA, from 8.04% to 19.97% on Sonar VQA, and from 9.96% to 48.52% on Marine Organisms VQA. Its overall multimodal score rises from 13.07% to 32.59%, a gain of 19.52 points. The largest single increase is on Marine Organisms VQA, where the improvement is 38.56 points; Sonar VQA increases by 11.93 points (Xue et al., 25 Apr 2026).

These reported results support the paper’s central claim that domain-specific multimodal curation materially improves marine-task performance, especially for modalities and categories underrepresented in general corpora. At the same time, the paper does not report ablation studies, does not isolate the separate effects of OceanCorpus and OceanInstruction, and does not present experiments on pretraining directly from OceanCorpus. Consequently, OceanPile’s evaluation evidence is strongest for instruction tuning and benchmark performance, not for corpus-only pretraining effects.

6. Position within marine AI, limitations, and interpretive cautions

OceanPile is positioned against several classes of prior work. The paper argues that general multimodal datasets are insufficient because they underrepresent specialized marine concepts, acoustic sensing data, scientific visuals, and interdisciplinary ocean knowledge. Traditional sonar and underwater image datasets are described as task-specific and not designed for MLLM training. Simulated marine datasets are said not to capture the complexity of real-world conditions. Prior ocean large models such as OceanGPT are characterized as single-modal, while multimodal marine systems such as MarineGPT and NAUTILUS are described as focusing mainly on underwater scene understanding rather than the broader physical, chemical, biological, and engineering scope of marine science (Xue et al., 25 Apr 2026).

Its stated novelty therefore lies in breadth and integration: authoritative oceanographic sources, rare marine modalities such as sonar, graph-guided instruction synthesis, and a manually curated benchmark are brought together into a unified resource. This suggests that OceanPile should be understood less as an isolated dataset release than as a domain-specific training and evaluation substrate for marine foundation models.

The paper also leaves several limitations explicit or implicit. Many component-level statistics are absent: the numbers of sonar images, underwater photographs, synchronized field-collected pairs, concept-graph nodes, and per-source distributions are not reported. Modality balance is not quantified. Some instruction generation and validation steps depend on proprietary MLLMs such as GPT-4o, which may introduce model-dependent biases. Benchmark size remains modest by general foundation-model standards, especially for the textual and general marine VQA subsets. Evaluation itself uses an LLM-as-a-Judge framework, which may also introduce bias. The field-collected component is geographically centered on the Zhoushan marine region of China, so environmental and ecological coverage is not comprehensive (Xue et al., 25 Apr 2026).

Several misconceptions can therefore be corrected directly. OceanPile is not a purely text dataset; it is explicitly multimodal. It is not a benchmark-only release; benchmarking is only one of its three layers. It is not a paper reporting direct pretraining gains from OceanCorpus; the experiments center on instruction tuning with OceanInstruction and evaluation on OceanBenchmark. It is also not presented as an exhaustive census of marine data sources; rather, it is a curated, public, domain-specific corpus intended to provide the data infrastructure that marine MLLMs previously lacked (Xue et al., 25 Apr 2026).

In that sense, OceanPile occupies an infrastructural position within marine AI. Its central contribution is alignment: textbooks, papers, web knowledge, sonar imagery, underwater photographs, scientific figures, and expert-curated concepts are organized into a common training and evaluation framework. The paper’s reported results indicate that this alignment is especially valuable for difficult marine modalities such as sonar and for fine-grained marine-organism understanding. A plausible implication is that future marine foundation-model progress will depend as much on such domain-specific corpus engineering as on further scaling of general-purpose architectures.