NautData: Underwater Multimodal Dataset
- NautData is a comprehensive underwater dataset featuring 1.45 million image-text pairs and support for eight detailed scene understanding tasks.
- It integrates heterogeneous marine data from public sources using advanced synthesis and validation methods like Gemini 2.0 Flash and GPT-4o.
- The paradigm emphasizes structured annotations, explicit uncertainty, and operational reuse, transforming raw observations into actionable marine insights.
NautData denotes, in the strict sense, the underwater instruction-following dataset introduced for the NAUTILUS large multimodal model, comprising 1.45 million image-text pairs, 158K images, and support for eight underwater scene understanding tasks built from 10 public underwater datasets (Xu et al., 31 Oct 2025). The surrounding literature also suggests a broader editorial meaning: a family of marine, maritime, and aquatic data resources that transform heterogeneous observations into georeferenced, analysis-ready, uncertainty-aware products for oceanography, vessel analytics, perception, navigation, bathymetry, and infrastructure mapping (Elipot et al., 2022, Martincic et al., 2021, Kreis et al., 18 Jul 2025, Araujo et al., 19 Feb 2025, Ramanathan et al., 2023). In that broader sense, NautData is less a single homogeneous repository than a data paradigm centered on structured annotations, multimodal fusion, explicit uncertainty, and operational reuse.
1. Named dataset, task structure, and annotation logic
The dataset explicitly named NautData was introduced to support coarse-grained classification, fine-grained classification, counting, visual question answering, detection, grounding, region caption, and image caption in underwater scenes (Xu et al., 31 Oct 2025). Its held-out benchmark contains 3,920 images and 7,916 QA examples, and its supervision spans scene-level, region-level, and object-level understanding. This multigranularity is central to the dataset’s design: image caption and VQA operate at scene scale; region caption and counting target localized or grouped content; and detection, grounding, and classification address object-centric perception.
Construction combines structured annotation conversion with large-model synthesis. Detection boxes, class labels, fish taxonomic classes, and fish counts are reformatted into instruction-response dialogs, while grounding descriptions, region captions, image captions, and VQA pairs are generated by multimodal models. The pipeline first uses Gemini 2.0 Flash to produce initial outputs, then uses Qwen2.5-VL-72B to evaluate them; low-quality responses are replaced, and for the NautData test set GPT-4o is used for further assessment, with flagged items undergoing manual verification by the research team (Xu et al., 31 Oct 2025). This makes NautData a hybrid corpus: partly derived from source labels, partly synthesized, and partially human-checked.
The source distribution is uneven but explicit. NautData draws from USIS10k, UIIS, RUOD, Deepfish, Brackish, IOCfish5k, UVOT-400, Aquarium, Underwater Trash, and FishNet; RUOD contributes 326,068 QA pairs, while Aquarium contributes 15,076 (Xu et al., 31 Oct 2025). The counting task is sourced only from IOCfish5k, and FishNet provides the 8 taxonomic classes used in fine-grained fish classification. A plausible implication is that NautData is strongest where underwater ecological and detection datasets already exist, and thinner where source corpora are sparse.
2. Oceanographic and environmental data layers
A NautData-like marine data stack is exemplified by the hourly Lagrangian sea-surface-temperature dataset derived from the NOAA Global Drifter Program, which converts irregular drifter temperature telemetry into hourly SST estimates aligned with hourly drifter positions and velocities (Elipot et al., 2022). The workflow is explicitly layered: Level-0 is the original unevenly timed drifter data; Level-1 is quality-controlled SST at original times; Level-2 is model-fitted SST at those same times; and Level-3 is the final top-of-the-hour SST dataset. In the release described, Level-0 spans 20-Dec-1978 02:00:00 to 06-Jul-2020 22:59:31 with 285,886,818 SST tuples; Level-1 contains 197,916,695 SST-time tuples from 24,597 drifter trajectories; and Level-3 contains 165,754,333 hourly target records from 17,324 trajectories (Elipot et al., 2022).
Its estimation model is a robust locally weighted regression adapted from LOWESS, with a local linear trend and three harmonics of the daily cycle, corresponding to and . The product separates non-diurnal SST, diurnal SST anomaly, and total SST, and propagates standard uncertainties for each component. The nominal half-bandwidth is 1 day, extendable in 1-hour steps to 2 days, and fewer than 0.4% of data points required a half-bandwidth longer than 1 day (Elipot et al., 2022). The practical significance is that the dataset is not simple interpolation: it is a trajectory-following, uncertainty-bearing SST analysis product coupled directly to drifter kinematics.
The same section of literature shows how regional environmental baselines can be integrated with in situ products. The Northwestern Atlantic Ocean Reanalysis (NAOR) provides a 30-year ROMS-based regional reanalysis for 1993–2022 at 4 km horizontal resolution and 50 vertical sigma levels, assimilating sea level anomaly, sea surface temperature, and subsurface temperature and salinity through Ensemble Optimal Interpolation (He et al., 10 Mar 2025). It is forced by ERA5, nested in GLORYS, includes 10 major tidal constituents from TPXO, and explicitly incorporates freshwater from 120 major rivers. The stated purpose is to provide a more accurate physical and dynamical baseline for the North American east coast shelf seas, Gulf of Mexico, and Caribbean Sea than coarser global products.
A related development in inland waters is the agentic monitoring system NAIAD, which orchestrates Sentinel-2 retrieval, NDCI, NDWI, chlorophyll-a estimation, weather queries, CyFi, and retrieval-augmented reasoning through a runtime DAG (Baltzi et al., 20 Oct 2025). This suggests that NautData, in a broader sense, may encompass not only static datasets but also orchestration layers that turn heterogeneous Earth-observation and environmental services into queryable analytical workflows.
A recurring misconception is that such products are equivalent to raw observation archives. The drifter SST product is not raw telemetry and its non-diurnal estimate is not exactly a “foundation temperature”; its uncertainty also does not include individual-sensor bias or drift (Elipot et al., 2022). Likewise, NAOR is a regional reanalysis nested in GLORYS and is not presented as sufficient, without local validation, for very shallow/intertidal studies or extreme-event attribution (He et al., 10 Mar 2025).
3. AIS-derived vessel activity, journeys, and port efficiency
A second major NautData axis is AIS-based operational reconstruction. One strand focuses on validating unreliable AIS navigational status before deriving port metrics. In “Vessel and Port Efficiency Metrics through Validated AIS data”, the central target field is navigational status, especially 0 (underway using engine), 1 (at anchor), and 5 (moored), which the authors identify as frequently misreported (Martincic et al., 2021). Three validation strategies are compared: a rule-based method using location, speed, and manually defined anchorage/terminal polygons; a kinematic method using speed and heading/rotation behavior; and machine learning with HDBSCAN, CatBoost, and KNN. The reported “cleanest result” is produced by KNN with a large number of neighbours, specifically at least 300 neighbours (Martincic et al., 2021).
That paper then defines a voyage as all movements a vessel makes inside a port area during a single arrival, including entry, optional anchorage waiting, terminal stay, and departure. Messages are grouped into a voyage if there is less than 24 hours between two messages from the same vessel in the port area; if the gap is more than 5 hours and the vessel moved more than 100 meters, the chain is split into two voyages (Martincic et al., 2021). This segmentation supports metrics such as anchorage waiting time, moored duration, movement duration inside port, and average movement speed. In validation against Port Community System data for Piraeus, the tool’s daily-arrival estimates achieved MAE = 4.46 daily arrivals against a ground truth average of 54 arrivals per day (Martincic et al., 2021).
A complementary open-access AIS reconstruction is provided by the Baltic Sea study based on 91,111,731 messages and 14,620 unique MMSI numbers collected over 91 days from 2024-07-29 to 2024-10-27 (Hütten, 28 Nov 2025). After cleansing and journey reconstruction, the final analysis covers 8,378 vessels. The system classifies vessels as moving or stationary, links movement legs through idle intervals, and aggregates vessel occupancy in 32,760 intervals of 4 minutes each. The reported mean simultaneous activity in the Baltic ROI is 4061 active vessels, split into 774 moving and 3287 stationary, with more than 300 vessels entering or leaving the area each day; specifically, the default daily transit estimate is 313.1 (Hütten, 28 Nov 2025).
Its spatial products are based on a grid of , corresponding to approximately 464 m cell height and 378–559 m cell width depending on latitude. Density is defined as time-normalized occupancy rather than simple crossing counts, allowing port detection via Gaussian smoothing, thresholding at 0.5 vessels/km², and watershed segmentation (Hütten, 28 Nov 2025). The method identifies 145 overdensities, with 44% overlap with World Port Index ports overall and 85% overlap for large ports. The same study is careful about limitations: it is coastal, Class A–only in its main outputs, and does not solve spoofing or intentional AIS manipulation (Hütten, 28 Nov 2025).
Taken together, these studies show a common NautData pattern: raw AIS streams become analytically useful only after state correction, trajectory reconstruction, gap interpretation, and explicit uncertainty modeling.
4. Visual, chart-linked, and synthetic maritime scene data
A further NautData dimension is maritime perception data that preserve explicit links between image space and geospatial structure. “Real-Time Fusion of Visual and Chart Data for Enhanced Maritime Vision” introduces a dataset of real-world maritime scenes in which visible navigational aids are not only bounded in the image but also linked to specific chart objects from the NOAA Marine Cadastre Aids to Navigation database (Kreis et al., 18 Jul 2025). The reported split is 6,257 training samples and 1,052 validation samples, with 10,977 distinct navigational-aid instances from 47 video sequences. The proposed transformer replaces generic DETR object queries with chart-conditioned queries parameterized by distance and bearing, so that each decoder query corresponds to a specific buoy candidate.
This produces a markedly different data model: a frame is paired with vessel pose, camera geometry, chart candidates, and explicit chart-to-image correspondences. On the test sequence, the best reported model, an RT-DETR-based fusion transformer, reaches Precision 0.905, Recall 0.881, F1 0.893, Mean-IoU 0.744, and 22.8 FPS, while the DETR-based variant is faster at 31.3 FPS (Kreis et al., 18 Jul 2025). A plausible implication is that chart-linked annotations are not merely auxiliary metadata; they redefine the perception problem as chart-conditioned grounding.
For semantic segmentation, the OASIs benchmark provides maritime imagery collected from 2017 to 2023 using the SxSM200N sensor module, with collection locations including Ulsan and Busan in South Korea and platform contexts in ports and aboard ships (Kim et al., 2024). It uses a coarse 4-class semantic segmentation taxonomy: Others = 0, Sea = 50, Land = 100, and Sea Objects = 150, and it groups scenes into Type-1 (Normal Condition, Backlit, Cloudy), Type-2 (Rain, Haze), and Type-3 (Night, Night light, Early night and Dawn). OASIs is therefore best understood as a navigation-scene parsing benchmark rather than a fine-grained vessel ontology.
Synthetic data generation extends this vision stack. Neptune-X introduces the Maritime Generation Dataset (MGD) with 11,900 samples, combining images, captions, water surface masks, and object boxes across five categories—ship, buoy, person, floating object, fixed object—and multiple viewpoint, location, and imaging-environment attributes (Guo et al., 25 Sep 2025). Its X-to-Maritime generator uses text, object layouts, and water masks, while Bidirectional Object-Water Attention explicitly models boundary interactions between objects and water. To improve downstream detection, generated samples are filtered and then ranked by Attribute-correlated Active Sampling, which uses detector-informed difficulty across object category, viewpoint, location, and imaging environment. The full method reaches FID = 18.05, CAS = 79.34, and a YOLO Score = 17.08 / 39.14 / 13.52 on generation evaluation, and improves detectors such as YOLOv10 from 39.99 to 43.62 mAP (Guo et al., 25 Sep 2025).
The named underwater NautData dataset sits within this same visual-data trajectory but emphasizes instruction following and multimodal semantics rather than only detection or generation (Xu et al., 31 Oct 2025). Across these resources, a common design principle emerges: maritime vision data become substantially more reusable when they preserve localized structure—boxes, masks, chart entities, or referring expressions—rather than only global image labels.
5. Navigation, multisensor localization, and bathymetric platforms
NautData-like systems also include vehicle-centered navigation datasets and survey platforms. The NavINST dataset is designed for high-precision positioning, mapping, and multisensory fusion in urban environments, but its structure is instructive for maritime and aquatic platform design because it combines heterogeneous navigation sensors, dense maps, and precisely post-processed reference trajectories (Araujo et al., 19 Feb 2025). It spans 76.99 km and 304.74 min over 15 trajectories, including 10 outdoor recordings and 5 indoor recordings. The suite includes 1 tactical-grade IMU, 4 commercial-grade IMUs, 2 GNSS receivers, 1 mechanical 3D LiDAR, 1 solid-state 3D LiDAR, 4 electronically scanning 4D radars, 1 bumper-mounted 1D Doppler radar, 1 monocular camera, 2 stereo cameras, and vehicle odometry. Data are distributed as ROS bag files, with dense indoor garage maps in PCD format and calibration assets provided.
For surface hydrography, “Design and Implementation of a Dual Uncrewed Surface Vessel Platform for Bathymetry Research under High-flow Conditions” proposes a two-vessel architecture explicitly separating navigation-and-control experimentation from expensive sonar deployment (Kumar et al., 18 Feb 2025). The NAC-USV is a low-cost platform for autonomy, fail-safe behavior, velocity and heading control, waypoint following, and obstacle sensing; the BEP-USV mirrors its control and computing architecture but carries a Norbit iWBMSe MBES with 400 kHz center frequency, 200–700 kHz frequency range, and ping rate up to 60 Hz. The BEP hull is the Seafloor Systems Echoboat-160, with 1.7 m length, 0.8 m width, 0.24 m height, 50 kg empty weight, and 27 kg maximum payload capacity. The control stack combines Cube Orange, Here 4 Multiband RTK GPS, ArduPilot, ROS 2 Humble, MAVLink, and Jetson-class onboard computing (Kumar et al., 18 Feb 2025).
The same paper makes explicit a systems insight that recurs across NautData-like infrastructures: bathymetric data quality is inseparable from platform motion quality. The design target for the BEP-USV was speed approaching 7 knots with at least 1 hour endurance, and the authors model required thrust from drag terms such as
with air drag neglected in the final sizing (Kumar et al., 18 Feb 2025). This coupling of platform dynamics, sensing, and post-processing is characteristic of data systems built for measurement rather than only perception.
6. Cross-layer maritime infrastructure and recurring limits
A final NautData-relevant layer is infrastructure cartography. Nautilus performs cross-layer mapping between traceroute-observed IP links and submarine cables using public traceroutes, TeleGeography cable maps, 11 geolocation sources, and 4 IP-to-ASN sources (Ramanathan et al., 2023). It generates mappings for 3.05 million IPv4 links and 1.43 million IPv6 links, covering 91% of active cables and 90% of submarine cable landing points. Its final prediction score is
where cluster evidence, landing-point distance, and owner evidence are combined with a categorical factor that discounts merely potential submarine links (Ramanathan et al., 2023). This broadens the notion of NautData beyond water-column and vessel observations to include maritime communications infrastructure.
Across the surveyed resources, several interpretive limits recur. First, analysis-ready outputs are not equivalent to ground truth. Drifter SST fields include local random-noise and quantization effects but not platform bias or sensor drift (Elipot et al., 2022). Baltic open-AIS reconstructions are accurate enough to agree within 20% with studies using proprietary data, but remain constrained by receiver coverage, AIS-dark behavior, AIS-B exclusion in the main analysis, and unresolved spoofing (Hütten, 28 Nov 2025). Nautilus submarine-cable mappings are heuristic rather than ground-truth assignments and are limited by geolocation error, incomplete ownership data, and MPLS blindness (Ramanathan et al., 2023). Synthetic or model-generated maritime annotations improve coverage and controllability, but they do not eliminate long-tail sparsity or generator bias (Guo et al., 25 Sep 2025, Xu et al., 31 Oct 2025).
This suggests that NautData is best understood not as a claim of complete maritime observability, but as a design philosophy for constructing reusable aquatic data products: retain source provenance, preserve spatial and temporal structure, expose uncertainty, and convert raw measurements into forms that can support both scientific interpretation and operational systems.