Neptune-X: Maritime Detection Framework

• Neptune-X is a data-centric maritime detection framework that uses controllable synthetic generation and active sampling to overcome annotated image scarcity and dataset imbalance.
• The framework employs key modules such as X-to-Maritime, Bidirectional Object-Water Attention, and Attribute-correlated Active Sampling to generate realistic images and improve detection under varied conditions.
• Empirical results demonstrate significant improvements in detection metrics on YOLO models and reduced training difficulty variance across maritime attributes.

Neptune-X is a data-centric generation-and-selection framework for maritime object detection designed to address two stated bottlenecks in the field: the scarcity of annotated maritime imagery and poor generalization across maritime attributes such as object category, viewpoint, location, and imaging environment. Its architecture couples controllable synthetic data generation with task-aware sample selection, centering on the X-to-Maritime generative model, the Bidirectional Object-Water Attention module, the Attribute-correlated Active Sampling strategy, and the Maritime Generation Dataset (Guo et al., 25 Sep 2025).

1. Definition and problem setting

Neptune-X is situated within maritime object detection, where imagery may originate from shipboard cameras, shore or coastal surveillance systems, and aerial platforms such as UAVs. The downstream tasks emphasized for this setting are navigation safety, surveillance, and autonomous operations. The framework is motivated by two concrete observations.

First, annotated maritime data is difficult and expensive to collect. Real deployments across maritime environments must be performed before labor-intensive box annotation can begin, and standard augmentations such as random flipping, Mixup, CutMix, and Mosaic do not create genuinely new semantic situations. Second, existing maritime datasets are imbalanced across several semantic dimensions. The paper repeatedly highlights variation in object category, viewpoint, location or environment type, and imaging environment, and it identifies underrepresented scenarios such as open-sea environments as particularly problematic.

Within this context, the phrase “universal maritime object detection” denotes robust generalization across the full diversity of maritime attributes rather than a new formally defined benchmark task. The intended target is a detector that performs reliably across category, viewpoint, location, weather, illumination, and imaging environment, rather than one tuned to a narrow subset of common cases.

The framework’s central claim is therefore not merely that more synthetic data is useful, but that synthetic data must be both controllable and selectively incorporated. Neptune-X is explicitly organized as a generate–filter–actively select–train pipeline.

2. System architecture

At the highest level, Neptune-X has three operational stages: synthetic maritime image generation, task-aware filtering and selection of the generated pool, and detector training on real data augmented with selected synthetic samples. The two principal technical components are X-to-Maritime and Attribute-correlated Active Sampling. A third indispensable element is the Maritime Generation Dataset, which supports both generative training and downstream benchmarking.

Component	Function	Core mechanism
X-to-Maritime	Synthesizes controllable maritime scenes	Multi-modality-conditioned latent diffusion
BiOW-Attn	Improves object–water realism	Bidirectional Object-Water Attention
AAS	Selects useful synthetic samples	Attribute-correlated difficulty scoring
MGD	Supports training and evaluation	Maritime Generation Dataset

The end-to-end workflow begins with a maritime training set and labeled conditions. Random transformations are applied to labels and conditions, including text descriptions, bounding-box resizing and flipping, and water-surface descriptions. X-to-Maritime then generates a large synthetic pool. A quality filter is applied using semantic consistency via CLIP and layout accuracy via a pre-trained ResNet classifier. A detector is pre-trained on real data, after which synthetic images are scored using Attribute-correlated Training Difficulty Factors and selected by AAS. The detector is then fine-tuned on real data plus the selected synthetic subset. In the supplement, the generated pool is stated to contain 100,000 synthetic images.

This decomposition is important because Neptune-X is not solely a generator. It is a data-centric framework in which generation, filtering, and active selection are co-designed around downstream detector improvement.

3. X-to-Maritime and multi-condition generation

X-to-Maritime is Neptune-X’s generative subsystem. The designation “X” refers to multiple conditioning modalities rather than a single source. In practice, the model supports a global text or caption condition, object conditions, and a water-surface condition. For each object, the conditioned inputs include class label $L_o^i$, spatial coordinates $P_o^i$, and binary mask $\mathcal{M}_o^i$. For the water region, the inputs are water description $L_w$, enclosing rectangle $P_w$, and binary mask $\mathcal{M}_w$. At image level, there is also a caption embedding $\mathcal{C}$.

The base architecture is Stable Diffusion as a latent diffusion model. An image $I \in \mathbb{R}^{H \times W \times 3}$ is encoded by a VAE into a latent code

$z \in \mathbb{R}^{h \times w \times c},$

where $h = H/m$ and $P_o^i$0, with compression factor $P_o^i$1. With only caption conditioning, the denoiser $P_o^i$2 is trained using the standard noise-prediction objective

$P_o^i$3

Neptune-X extends this objective so that denoising depends jointly on global caption, object conditions, and water-surface conditions: $P_o^i$4

The condition embedding scheme is inspired by GLIGEN. For object $P_o^i$5, the position is Fourier-encoded,

$P_o^i$6

the class text is encoded by the CLIP text encoder,

$P_o^i$7

and the final object condition token is produced by

$P_o^i$8

The same construction is used for the water condition by replacing object label and coordinates with $P_o^i$9 and $\mathcal{M}_o^i$0, yielding $\mathcal{M}_o^i$1.

A key implementation detail is that the Stable Diffusion weights are frozen. Only the layout condition embedders and the BiOW-Attn modules are trained. This preserves the base generative prior while specializing the model to maritime scene control.

4. Bidirectional Object-Water Attention and active sampling

The paper’s central domain insight is that maritime scenes differ from generic layout-to-image tasks because objects and water are tightly coupled. A ship must visually interact with the water surface, and object placement that ignores water geometry can produce implausible images, such as ships floating above water or object–water boundaries that are semantically inconsistent. Neptune-X addresses this with the Bidirectional Object-Water Attention module.

BiOW-Attn contains two stages. The first stage performs object-guided and water-guided cross-attention: $\mathcal{M}_o^i$2 For the object branch, the enhanced features are aggregated and spatially masked: $\mathcal{M}_o^i$3 The water branch is constructed analogously using $\mathcal{M}_o^i$4 and $\mathcal{M}_o^i$5.

The second stage performs bidirectional interaction: object features attend to water features and water features attend to object features. In the supplement, the module output is fused with gated residuals: $\mathcal{M}_o^i$6 The learnable scalars $\mathcal{M}_o^i$7 and $\mathcal{M}_o^i$8 are initialized at zero for stable fine-tuning.

The selection half of Neptune-X is Attribute-correlated Active Sampling. It begins by pretraining a detector on the real training set and then estimating attribute-wise training difficulty through Attribute-correlated Training Difficulty Factors. Box-level accuracy is measured as

$\mathcal{M}_o^i$9

where $L_w$0 is the ground-truth box, $L_w$1 is the predicted box, $L_w$2 is prediction confidence, and $L_w$3 is intersection-over-union.

For the $L_w$4-th attribute at iteration $L_w$5, the initial difficulty is

$L_w$6

This quantity is then updated by exponential moving average: $L_w$7 with attribute-wise momentum

$L_w$8

The method normalizes ATDF values by softmax within each attribute dimension so that larger probability corresponds to greater detector difficulty.

Image-level difficulty is then scored by combining context difficulty and object-category difficulty: $L_w$9 Images are ranked by $P_w$0, and the top-$P_w$1 samples are selected. This makes AAS neither pure uncertainty sampling nor pure diversity sampling; it is an attribute-aware selection rule driven by detector performance and semantic imbalance.

5. Maritime Generation Dataset and evaluation protocol

The Maritime Generation Dataset is presented as the first dataset tailored for generative maritime learning. It contains 11,900 samples aggregated from MaSTr1325, USVInland, MIT Sea Grant / Robowhaler, SMD, SeaShips, Seagull, Fvessel, LaRS, and additional “Others.” The dataset spans shore, ship, and aerial viewpoints, and each image includes an image-level description or caption, a water-surface mask, and object bounding boxes (Guo et al., 25 Sep 2025).

Dimension	Values	Scope
Object categories	ship, buoy, person, floating object, fixed object	5 classes
Viewpoints	shore-based, shipboard, aerial	3 views
Locations	sea, river, harbor, lake	4 settings
Imaging environments	sunny, cloudy, foggy, rainy, dawn/dusk, night	6 conditions

The dataset is split 3:1:1 into 7,140 training samples, 2,380 validation samples, and 2,380 test samples. The supplement states that the validation and test sets are combined for image-generation evaluation. It also adds that the dataset includes multi-level descriptions—image description, water-surface description, and object description—generated with assistance from T-Rex2 for boxes, SAM2 for water masks, and LLaVA-NeXT for text descriptions, with human verification.

MGD is intentionally imbalanced, and those imbalances are central to Neptune-X’s motivation. Category statistics are reported as ship 72.44%, buoy 13.16%, person 11.97%, floating object 1.53%, and fixed object 0.90%. Viewpoint statistics are shore 50.77%, ship 20.66%, and aerial 28.56%. Location statistics are sea 48.98%, river 46.48%, harbor 2.37%, and lake 2.17%. Imaging-environment statistics are sunny 54.55%, cloudy 23.48%, foggy 10.29%, rainy 4.33%, dawn/dusk 4.90%, and night 2.45%. The dataset thus encodes precisely the skew that Neptune-X is intended to counteract.

The implementation uses PyTorch 1.13 and Python 3.8 on 2 Intel Xeon Silver 4410Y CPUs and 4 NVIDIA 5880 Ada GPUs. The generator is trained with AdamW at learning rate $P_w$2 for 100,000 iterations, with patch size $P_w$3, batch size 8, and total training time about 100 hours. Generation quality is evaluated with FID, CAS, and YOLO Score, where CAS uses a ResNet-101 trained on MGD and YOLO Score uses a YOLOv10 trained on MGD. Downstream detection is evaluated on YOLOv10, YOLOv11, YOLOv12, and Grounding DINO using mAP and mAP$P_w$4.

6. Empirical performance and stated limitations

The empirical results emphasize two points: Neptune-X improves controllable maritime image generation, and those gains translate into improved downstream detection (Guo et al., 25 Sep 2025).

Metric	Neptune-X	Best prior baseline in table
FID $P_w$5	18.05	18.17
CAS $P_w$6	79.34	77.06
YOLO Score mAP $P_w$7	17.08	12.74

Against SD1.5, LayoutDiff, GLIGEN, InstDiff, and RC-L2I, Neptune-X records the best FID, the best CAS, and the best YOLO Score. The strongest separation appears in detection-relevant generation quality. Relative to GLIGEN, the paper reports YOLO Score improvements of $P_w$8 mAP and $P_w$9 mAP$\mathcal{M}_w$0, while relative to InstDiff it reports $\mathcal{M}_w$1 mAP$\mathcal{M}_w$2.

The downstream detector gains are also consistent across detector families.

Detector	Baseline	With Neptune-X
YOLOv10	39.99 / 61.13	43.62 / 65.50
YOLOv11	41.29 / 62.51	44.43 / 66.15
YOLOv12	39.06 / 60.53	42.91 / 63.85
Grounding DINO	65.03 / 86.12	68.04 / 89.86

The paper states that gains are especially strong in attribute categories that originally had lower detection accuracy. Figure 1 is summarized as showing improvements across all four semantic dimensions—category, viewpoint, location, and imaging environment—and the reported aggregate effect is a 13.77% increase in mean attribute accuracy and a 6.39% decrease in variance. This is the empirical basis for the claim that Neptune-X reduces cross-attribute training difficulty disparities.

The ablations identify both the maritime-specific generator design and the active-sampling policy as necessary. For generation, the full BiOW-Attn design outperforms object cross-attention alone, object-plus-water cross-attention, and one-way variants such as Obj2WatCA or Wat2ObjCA. For active sampling, AAS outperforms Entropy, Variance, Margin, Greedy K-Center, and K-Means Corset on YOLOv10. The sample-efficiency result is particularly notable: 5k AAS-selected samples produce 43.11 / 64.70, nearly matching 10k randomly selected samples at 43.31 / 64.95. The saturation analysis further indicates that gains plateau from 10k to 20k samples, suggesting that AAS front-loads the most useful synthetic data.

The paper’s stated limitation is that ATDF currently relies on a fixed discrete set of predefined attributes such as viewpoint, lighting, and object type. This may limit granularity, and future work is suggested on continuous or hierarchical attribute spaces. A plausible implication is that Neptune-X’s effectiveness is tied to the adequacy of the chosen semantic axes; if those axes omit important forms of maritime variation, the active-sampling prior may be incomplete. The code is reported as available at https://github.com/gy65896/Neptune-X.

7. Nomenclature and distinction from Neptune planetary studies

The name “Neptune-X” can invite confusion because a large surrounding literature uses “Neptune” for planetary science, mission architecture, small-body dynamics, or exoplanet classification. In the cited machine-learning work, however, Neptune-X refers specifically to “Active X-to-Maritime Generation for Universal Maritime Object Detection,” not to a Neptune-centered planetary mission, an ice-giant observatory, or a Neptune-like exoplanet.

This distinction is important because Neptune mission studies in the literature use different designations. For example, the Neptune-centered L-class multirole observatory and science platform discussed as a flagship-scale concept is named Arcanum rather than Neptune-X (McKevitt et al., 2021). Neptune-X is therefore best understood as a maritime computer-vision framework whose “X-to-Maritime” terminology denotes multi-modality-conditioned generation, with “X” standing for the supported conditioning signals rather than any astronomical association.