---
title: 'NautData: Underwater Multimodal Dataset'
url: https://www.emergentmind.com/topics/nautdata
type: topic
---

# NautData: Underwater Multimodal Dataset

NautData denotes, in the strict sense, the underwater instruction-following dataset introduced for the NAUTILUS large multimodal model, comprising **1.45 million image-text pairs**, **158K images**, and support for **eight underwater scene understanding tasks** built from **10 public underwater datasets** [2510.27481]. The surrounding literature also suggests a broader editorial meaning: a family of marine, maritime, and aquatic data resources that transform heterogeneous observations into georeferenced, analysis-ready, uncertainty-aware products for oceanography, vessel analytics, perception, navigation, bathymetry, and infrastructure mapping [2201.08289][2105.00063][2507.13880][2502.13863][2302.14201]. In that broader sense, NautData is less a single homogeneous repository than a data paradigm centered on structured annotations, multimodal fusion, explicit uncertainty, and operational reuse.

## 1. Named dataset, task structure, and annotation logic

The dataset explicitly named NautData was introduced to support **coarse-grained classification, fine-grained classification, counting, visual question answering, detection, grounding, region caption, and image caption** in underwater scenes [2510.27481]. Its held-out benchmark contains **3,920 images** and **7,916 QA examples**, and its supervision spans **scene-level**, **region-level**, and **object-level** understanding. This multigranularity is central to the dataset’s design: image caption and VQA operate at scene scale; region caption and counting target localized or grouped content; and detection, grounding, and classification address object-centric perception.

Construction combines structured annotation conversion with large-model synthesis. Detection boxes, class labels, fish taxonomic classes, and fish counts are reformatted into instruction-response dialogs, while grounding descriptions, region captions, image captions, and VQA pairs are generated by multimodal models. The pipeline first uses **Gemini 2.0 Flash** to produce initial outputs, then uses **Qwen2.5-VL-72B** to evaluate them; low-quality responses are replaced, and for the NautData test set **GPT-4o** is used for further assessment, with flagged items undergoing **manual verification by the research team** [2510.27481]. This makes NautData a hybrid corpus: partly derived from source labels, partly synthesized, and partially human-checked.

The source distribution is uneven but explicit. NautData draws from **USIS10k, UIIS, RUOD, Deepfish, Brackish, IOCfish5k, UVOT-400, Aquarium, Underwater Trash, and FishNet**; **RUOD** contributes **326,068** QA pairs, while **Aquarium** contributes **15,076** [2510.27481]. The counting task is sourced only from **IOCfish5k**, and **FishNet** provides the **8 taxonomic classes** used in fine-grained fish classification. A plausible implication is that NautData is strongest where underwater ecological and detection datasets already exist, and thinner where source corpora are sparse.

## 2. Oceanographic and environmental data layers

A NautData-like marine data stack is exemplified by the **hourly Lagrangian sea-surface-temperature dataset** derived from the **NOAA Global Drifter Program**, which converts irregular drifter temperature telemetry into hourly SST estimates aligned with hourly drifter positions and velocities [2201.08289]. The workflow is explicitly layered: **Level-0** is the original unevenly timed drifter data; **Level-1** is quality-controlled SST at original times; **Level-2** is model-fitted SST at those same times; and **Level-3** is the final **top-of-the-hour** SST dataset. In the release described, Level-0 spans **20-Dec-1978 02:00:00 to 06-Jul-2020 22:59:31** with **285,886,818** SST tuples; Level-1 contains **197,916,695** SST-time tuples from **24,597 drifter trajectories**; and Level-3 contains **165,754,333** hourly target records from **17,324 trajectories** [2201.08289].

Its estimation model is a robust locally weighted regression adapted from **LOWESS**, with a **local linear trend** and **three harmonics of the daily cycle**, corresponding to **\(P=1\)** and **\(N=3\)**. The product separates **non-diurnal SST**, **diurnal SST anomaly**, and **total SST**, and propagates standard uncertainties for each component. The nominal half-bandwidth is **1 day**, extendable in **1-hour steps** to **2 days**, and fewer than **0.4%** of data points required a half-bandwidth longer than 1 day [2201.08289]. The practical significance is that the dataset is not simple interpolation: it is a trajectory-following, uncertainty-bearing SST analysis product coupled directly to drifter kinematics.

The same section of literature shows how regional environmental baselines can be integrated with in situ products. The **Northwestern Atlantic Ocean Reanalysis (NAOR)** provides a **30-year** ROMS-based regional reanalysis for **1993–2022** at **4 km** horizontal resolution and **50 vertical sigma levels**, assimilating **sea level anomaly**, **sea surface temperature**, and **subsurface temperature and salinity** through **Ensemble Optimal Interpolation** [2503.06907]. It is forced by **ERA5**, nested in **GLORYS**, includes **10 major tidal constituents** from **TPXO**, and explicitly incorporates freshwater from **120 major rivers**. The stated purpose is to provide a more accurate physical and dynamical baseline for the **North American east coast shelf seas, Gulf of Mexico, and Caribbean Sea** than coarser global products.

A related development in inland waters is the agentic monitoring system **NAIAD**, which orchestrates **Sentinel-2 retrieval**, **NDCI**, **NDWI**, chlorophyll-a estimation, weather queries, **CyFi**, and retrieval-augmented reasoning through a runtime **DAG** [2601.05256]. This suggests that NautData, in a broader sense, may encompass not only static datasets but also orchestration layers that turn heterogeneous Earth-observation and environmental services into queryable analytical workflows.

A recurring misconception is that such products are equivalent to raw observation archives. The drifter SST product is not raw telemetry and its non-diurnal estimate is **not exactly a “foundation temperature”**; its uncertainty also does **not** include individual-sensor bias or drift [2201.08289]. Likewise, NAOR is a regional reanalysis nested in GLORYS and is not presented as sufficient, without local validation, for very shallow/intertidal studies or extreme-event attribution [2503.06907].

## 3. AIS-derived vessel activity, journeys, and port efficiency

A second major NautData axis is AIS-based operational reconstruction. One strand focuses on validating unreliable AIS navigational status before deriving port metrics. In **“Vessel and Port Efficiency Metrics through Validated AIS data”**, the central target field is **navigational status**, especially **0** (underway using engine), **1** (at anchor), and **5** (moored), which the authors identify as frequently misreported [2105.00063]. Three validation strategies are compared: a rule-based method using location, speed, and manually defined anchorage/terminal polygons; a kinematic method using speed and heading/rotation behavior; and machine learning with **HDBSCAN**, **CatBoost**, and **KNN**. The reported “cleanest result” is produced by **KNN with a large number of neighbours**, specifically **at least 300 neighbours** [2105.00063].

That paper then defines a **voyage** as all movements a vessel makes inside a port area during a single arrival, including entry, optional anchorage waiting, terminal stay, and departure. Messages are grouped into a voyage if there is **less than 24 hours** between two messages from the same vessel in the port area; if the gap is **more than 5 hours** and the vessel moved **more than 100 meters**, the chain is split into **two voyages** [2105.00063]. This segmentation supports metrics such as anchorage waiting time, moored duration, movement duration inside port, and average movement speed. In validation against Port Community System data for Piraeus, the tool’s daily-arrival estimates achieved **MAE = 4.46 daily arrivals** against a ground truth average of **54 arrivals per day** [2105.00063].

A complementary open-access AIS reconstruction is provided by the Baltic Sea study based on **91,111,731 messages** and **14,620 unique MMSI numbers** collected over **91 days** from **2024-07-29** to **2024-10-27** [2511.23016]. After cleansing and journey reconstruction, the final analysis covers **8,378 vessels**. The system classifies vessels as moving or stationary, links movement legs through idle intervals, and aggregates vessel occupancy in **32,760** intervals of **4 minutes** each. The reported mean simultaneous activity in the Baltic ROI is **4061** active vessels, split into **774 moving** and **3287 stationary**, with more than **300** vessels entering or leaving the area each day; specifically, the default daily transit estimate is **313.1** [2511.23016].

Its spatial products are based on a grid of **\(15'' \times 30''\)**, corresponding to approximately **464 m** cell height and **378–559 m** cell width depending on latitude. Density is defined as time-normalized occupancy rather than simple crossing counts, allowing port detection via Gaussian smoothing, thresholding at **0.5 vessels/km²**, and watershed segmentation [2511.23016]. The method identifies **145 overdensities**, with **44%** overlap with World Port Index ports overall and **85%** overlap for large ports. The same study is careful about limitations: it is coastal, Class A–only in its main outputs, and does **not** solve spoofing or intentional AIS manipulation [2511.23016].

Taken together, these studies show a common NautData pattern: raw AIS streams become analytically useful only after state correction, trajectory reconstruction, gap interpretation, and explicit uncertainty modeling.

## 4. Visual, chart-linked, and synthetic maritime scene data

A further NautData dimension is maritime perception data that preserve explicit links between image space and geospatial structure. **“Real-Time Fusion of Visual and Chart Data for Enhanced Maritime Vision”** introduces a dataset of real-world maritime scenes in which visible navigational aids are not only bounded in the image but also linked to specific chart objects from the NOAA Marine Cadastre **Aids to Navigation** database [2507.13880]. The reported split is **6,257 training samples** and **1,052 validation samples**, with **10,977** distinct navigational-aid instances from **47 video sequences**. The proposed transformer replaces generic DETR object queries with chart-conditioned queries parameterized by **distance** and **bearing**, so that each decoder query corresponds to a specific buoy candidate.

This produces a markedly different data model: a frame is paired with vessel pose, camera geometry, chart candidates, and explicit chart-to-image correspondences. On the test sequence, the best reported model, an **RT-DETR-based fusion transformer**, reaches **Precision 0.905**, **Recall 0.881**, **F1 0.893**, **Mean-IoU 0.744**, and **22.8 FPS**, while the DETR-based variant is faster at **31.3 FPS** [2507.13880]. A plausible implication is that chart-linked annotations are not merely auxiliary metadata; they redefine the perception problem as chart-conditioned grounding.

For semantic segmentation, the **OASIs** benchmark provides maritime imagery collected **from 2017 to 2023** using the **SxSM200N** sensor module, with collection locations including **Ulsan and Busan in South Korea** and platform contexts **in ports and aboard ships** [2407.09005]. It uses a coarse **4-class semantic segmentation taxonomy**: **Others = 0**, **Sea = 50**, **Land = 100**, and **Sea Objects = 150**, and it groups scenes into **Type-1** (Normal Condition, Backlit, Cloudy), **Type-2** (Rain, Haze), and **Type-3** (Night, Night light, Early night and Dawn). OASIs is therefore best understood as a navigation-scene parsing benchmark rather than a fine-grained vessel ontology.

Synthetic data generation extends this vision stack. **Neptune-X** introduces the **Maritime Generation Dataset (MGD)** with **11,900 samples**, combining images, captions, water surface masks, and object boxes across five categories—**ship, buoy, person, floating object, fixed object**—and multiple viewpoint, location, and imaging-environment attributes [2509.20745]. Its **X-to-Maritime** generator uses text, object layouts, and water masks, while **Bidirectional Object-Water Attention** explicitly models boundary interactions between objects and water. To improve downstream detection, generated samples are filtered and then ranked by **Attribute-correlated Active Sampling**, which uses detector-informed difficulty across **object category**, **viewpoint**, **location**, and **imaging environment**. The full method reaches **FID = 18.05**, **CAS = 79.34**, and a **YOLO Score = 17.08 / 39.14 / 13.52** on generation evaluation, and improves detectors such as **YOLOv10** from **39.99** to **43.62 mAP** [2509.20745].

The named underwater NautData dataset sits within this same visual-data trajectory but emphasizes instruction following and multimodal semantics rather than only detection or generation [2510.27481]. Across these resources, a common design principle emerges: maritime vision data become substantially more reusable when they preserve localized structure—boxes, masks, chart entities, or referring expressions—rather than only global image labels.

## 5. Navigation, multisensor localization, and bathymetric platforms

NautData-like systems also include vehicle-centered navigation datasets and survey platforms. The **NavINST** dataset is designed for high-precision positioning, mapping, and multisensory fusion in urban environments, but its structure is instructive for maritime and aquatic platform design because it combines heterogeneous navigation sensors, dense maps, and precisely post-processed reference trajectories [2502.13863]. It spans **76.99 km** and **304.74 min** over **15 trajectories**, including **10 outdoor recordings** and **5 indoor recordings**. The suite includes **1 tactical-grade IMU**, **4 commercial-grade IMUs**, **2 GNSS receivers**, **1 mechanical 3D LiDAR**, **1 solid-state 3D LiDAR**, **4 electronically scanning 4D radars**, **1 bumper-mounted 1D Doppler radar**, **1 monocular camera**, **2 stereo cameras**, and vehicle odometry. Data are distributed as **ROS bag files**, with dense indoor garage maps in **PCD** format and calibration assets provided.

For surface hydrography, **“Design and Implementation of a Dual Uncrewed Surface Vessel Platform for Bathymetry Research under High-flow Conditions”** proposes a two-vessel architecture explicitly separating navigation-and-control experimentation from expensive sonar deployment [2502.12539]. The **NAC-USV** is a low-cost platform for autonomy, fail-safe behavior, velocity and heading control, waypoint following, and obstacle sensing; the **BEP-USV** mirrors its control and computing architecture but carries a **Norbit iWBMSe MBES** with **400 kHz** center frequency, **200–700 kHz** frequency range, and ping rate up to **60 Hz**. The BEP hull is the **Seafloor Systems Echoboat-160**, with **1.7 m** length, **0.8 m** width, **0.24 m** height, **50 kg** empty weight, and **27 kg** maximum payload capacity. The control stack combines **Cube Orange**, **Here 4 Multiband RTK GPS**, **ArduPilot**, **ROS 2 Humble**, **MAVLink**, and Jetson-class onboard computing [2502.12539].

The same paper makes explicit a systems insight that recurs across NautData-like infrastructures: bathymetric data quality is inseparable from platform motion quality. The design target for the BEP-USV was speed approaching **7 knots** with at least **1 hour** endurance, and the authors model required thrust from drag terms such as
$$
R_T = R_V + R_W + R_A,
$$
with air drag neglected in the final sizing [2502.12539]. This coupling of platform dynamics, sensing, and post-processing is characteristic of data systems built for measurement rather than only perception.

## 6. Cross-layer maritime infrastructure and recurring limits

A final NautData-relevant layer is infrastructure cartography. **Nautilus** performs cross-layer mapping between traceroute-observed IP links and submarine cables using public traceroutes, TeleGeography cable maps, **11 geolocation sources**, and **4 IP-to-ASN sources** [2302.14201]. It generates mappings for **3.05 million IPv4 links** and **1.43 million IPv6 links**, covering **91% of active cables** and **90% of submarine cable landing points**. Its final prediction score is
$$
S = f * [0.5 * (C_1 + C_2) + 0.4 * (2 - d_{1} - d_{2}) + 0.1 * (O_1 + O_2)],
$$
where cluster evidence, landing-point distance, and owner evidence are combined with a categorical factor \(f\) that discounts merely potential submarine links [2302.14201]. This broadens the notion of NautData beyond water-column and vessel observations to include maritime communications infrastructure.

Across the surveyed resources, several interpretive limits recur. First, analysis-ready outputs are not equivalent to ground truth. Drifter SST fields include local random-noise and quantization effects but not platform bias or sensor drift [2201.08289]. Baltic open-AIS reconstructions are accurate enough to agree within **20%** with studies using proprietary data, but remain constrained by receiver coverage, AIS-dark behavior, AIS-B exclusion in the main analysis, and unresolved spoofing [2511.23016]. Nautilus submarine-cable mappings are heuristic rather than ground-truth assignments and are limited by geolocation error, incomplete ownership data, and MPLS blindness [2302.14201]. Synthetic or model-generated maritime annotations improve coverage and controllability, but they do not eliminate long-tail sparsity or generator bias [2509.20745][2510.27481].

This suggests that NautData is best understood not as a claim of complete maritime observability, but as a design philosophy for constructing reusable aquatic data products: retain source provenance, preserve spatial and temporal structure, expose uncertainty, and convert raw measurements into forms that can support both scientific interpretation and operational systems.

Source: https://www.emergentmind.com/topics/nautdata