ShipSem-VL: Semantic-Aware Ship Detection

• ShipSem-VL is a remote-sensing vision-language dataset designed for semantic-aware ship detection with fine-grained attributes such as hull color, size class, and orientation.
• It leverages a multi-scale adaptive sliding window strategy coupled with a CLIP-style model and cross-modal attention to improve detection precision in diverse maritime scenes.
• Empirical evaluations demonstrate significant gains in recall and precision for complex semantic queries, highlighting the importance of fine-grained attribute supervision over traditional methods.

Searching arXiv for the primary paper and related context. ShipSem-VL is a remote-sensing vision-language dataset introduced for Semantic-Aware Ship Detection (SASD), together with an associated detection framework that combines a CLIP-style Vision-LLM, a multi-scale adaptive sliding window strategy, and cross-modal attention to capture fine-grained ship semantics in high-resolution satellite imagery (Li et al., 21 Aug 2025). It was proposed to address a specific limitation of existing ship-detection practice in remote sensing: coarse category labels and bounding-box localization do not adequately encode attributes such as hull color, size class, orientation, and structural details, even though such attributes are necessary for distinguishing visually similar vessels, tracking specific ships, or querying for distinctive markings.

1. Conceptual scope and problem formulation

ShipSem-VL is defined around the task of Semantic-Aware Ship Detection, in which the objective is not only to detect ships in remote-sensing imagery but also to ground fine-grained semantic descriptions in image regions. The motivating claim is that existing remote-sensing ship datasets emphasize coarse categories such as cargo versus fishing and bounding-box localization, while lacking fine-grained semantic labels including hull color, size class, orientation, and structural details such as deck markings and superstructure counts (Li et al., 21 Aug 2025).

The dataset is described as the first remote-sensing Vision-Language dataset tailored for SASD. Its central design choice is the use of multi-label, multi-granularity textual descriptions for each ship instance, with the explicit aim of avoiding manual box annotation, reducing labeling cost, and enabling fine-grained semantic grounding. This design places ShipSem-VL at the intersection of remote-sensing object detection, attribute recognition, and image-text alignment. A plausible implication is that the dataset is intended less as a conventional detection benchmark in the strict IoU-centric sense than as a benchmark for semantically conditioned retrieval and localization.

The associated SASD framework operationalizes this formulation through three evaluation types: general ship detection in cluttered backgrounds, detection of specific ship types via complex semantic queries, and disambiguation of visually similar ships sharing many attributes. This tripartite evaluation setup indicates that ShipSem-VL is not limited to category-level recognition; it is also constructed to test semantic specificity and attribute-based discrimination (Li et al., 21 Aug 2025).

2. Dataset composition and semantic annotation model

ShipSem-VL contains 2,611 images and approximately 4,800 ship instances, corresponding to an average of about 1.8 ships per image. The imagery has spatial resolutions from 0.30 m to 0.60 m ground sample distance, characterized as typical Google Earth imagery. Scene coverage spans 23 port types, including container terminals, oil terminals, fishing docks, and naval bases, together with adjacent open-sea regions. Environmental conditions include clear sky, overcast, sun-glint on water, calm sea, rough sea, and partial shadow (Li et al., 21 Aug 2025).

The annotation model is multi-label and multi-granularity. Each ship instance is represented by a short text composed from controlled attribute groups, with an optional free-text field for unique identifiers. The schema is summarized below.

Attribute group	Label type	Values
ShipType	Coarse, single-label	Cargo, Tanker, Fishing, Cruise, Military
SizeClass	Coarse, single-label	Small $(< 50\,\mathrm{m})$, Medium $(50$–$150\,\mathrm{m})$, Large $(> 150\,\mathrm{m})$
Orientation	Coarse, single-label	8 compass directions $$(\mathrm{N}, \mathrm{NE}, \mathrm{E}, \mathrm{SE}, \dots)$$
HullColor	Fine, multi-label	White, Red, Blue, Gray, Black
StructuralDetails	Fine, multi-label	SuperstructureCount $\in \{1,2,3+\}$, DeckMarkings $\in \{\mathrm{Yes}, \mathrm{No}\}$, MastType $\in \{\mathrm{Single}, \mathrm{Double}\}$
UniqueMarkers	Optional, free text	Examples include “hull number 4729” and “yellow stripe at bow”

An example annotation is given as: “Medium Cargo ship, oriented NE, hull color white and red stripe, 2 superstructures, yellow funnel marking.” This example illustrates the intended coupling of coarse vessel class with finer appearance and structure cues. It also shows that the textual descriptions are not merely class captions; they are compositional attribute statements suitable for text-conditioned matching.

A common misconception would be to treat ShipSem-VL as a conventional box-labeled ship-detection corpus. The paper instead emphasizes textual description per ship instance and explicitly states that the design avoids manual box annotation. This suggests that the dataset’s primary novelty lies in semantic expressivity rather than in dense geometric supervision (Li et al., 21 Aug 2025).

3. Data collection, pre-processing, and quality control

The source imagery is Google Earth high-resolution satellite data, sampled systematically across 23 port types and adjacent sea areas worldwide. Raw imagery is tiled into overlapping $1024 \times 1024$ pixel crops. The pre-processing pipeline removes crops with less than 75% nautical water or with excessive cloud cover, and normalizes contrast through histogram equalization (Li et al., 21 Aug 2025).

Annotation is performed through a three-stage expert workflow. Two trained remote-sensing experts independently inspect each crop using a web-based tool described as a custom CVAT extension supporting free-form text entry for each detected ship. Annotators assign attributes from the controlled vocabulary, after which a third expert reconciles disagreements. This procedure directly reflects the dataset’s emphasis on semantic precision rather than minimal-cost crowd annotation.

Inter-annotator agreement is measured using Cohen’s $\kappa$ on a 10% hold-out set of images. The reported values are $(50$0, $(50$1, $(50$2, and $(50$3. Disagreements are resolved by majority vote or group discussion, and final consistency is reported to exceed $(50$4 across all attribute groups (Li et al., 21 Aug 2025).

These agreement values are significant because they quantify the practical annotatability of fine-grained maritime semantics from overhead imagery. Structural details and color are intrinsically more difficult than coarse type and size, yet the reported consistency remains above the threshold the paper treats as acceptable. This supports the feasibility of attribute-rich annotation even under the view constraints of remote sensing.

4. SASD architecture and learning objective

The SASD framework combines three components: a Multi-Scale Adaptive Sliding Window for ROI proposal, a CLIP-style Vision-LLM with a ViT-B-32 backbone for image-text similarity, and cross-modal attention for fine-grained alignment (Li et al., 21 Aug 2025).

For an input image $(50$5 of size $(50$6, the framework defines window scales $(50$7 such as $(50$8, $(50$9, and $150\,\mathrm{m})$0, with strides $150\,\mathrm{m})$1. The sampling function generates a pool of cropped ROIs:

$150\,\mathrm{m})$2

Each window $150\,\mathrm{m})$3 is mapped by a visual encoder $150\,\mathrm{m})$4 to an embedding $150\,\mathrm{m})$5, while each attribute description $150\,\mathrm{m})$6 is mapped by a Transformer-based text encoder $150\,\mathrm{m})$7 to an embedding $150\,\mathrm{m})$8. Fine-grained alignment is then refined through a pairwise similarity matrix between $150\,\mathrm{m})$9 visual tokens and $(> 150\,\mathrm{m})$0 text tokens:

$(> 150\,\mathrm{m})$1

where $(> 150\,\mathrm{m})$2 is a learnable temperature. The attended joint feature is defined as

$(> 150\,\mathrm{m})$3

with $(> 150\,\mathrm{m})$4 denoting concatenation.

Training optimizes two objectives jointly. The semantic alignment loss is contrastive:

$(> 150\,\mathrm{m})$5

The detection confidence loss treats an ROI whose similarity exceeds threshold $(> 150\,\mathrm{m})$6 as a positive detection and applies binary cross-entropy:

$(> 150\,\mathrm{m})$7

where $(> 150\,\mathrm{m})$8 is the ground-truth indicator for $(> 150\,\mathrm{m})$9, $$(\mathrm{N}, \mathrm{NE}, \mathrm{E}, \mathrm{SE}, \dots)$$0 is the cosine similarity, and $$(\mathrm{N}, \mathrm{NE}, \mathrm{E}, \mathrm{SE}, \dots)$$1 is the sigmoid. The total loss is

$$(\mathrm{N}, \mathrm{NE}, \mathrm{E}, \mathrm{SE}, \dots)$$2

Architecturally, the framework is notable for coupling proposal generation and semantic grounding without introducing a conventional bounding-box regression stage. This explains both its strengths in query-conditioned semantic matching and its later-reported limitations in localization granularity.

5. Evaluation design and empirical performance

The evaluation protocol defines three task families, comprising 45 total subtasks: general ship detection in cluttered backgrounds, detection of specific ship types via complex semantic queries, and disambiguation of visually similar ships sharing many attributes (Li et al., 21 Aug 2025).

Performance is measured using precision and recall at similarity thresholds $$(\mathrm{N}, \mathrm{NE}, \mathrm{E}, \mathrm{SE}, \dots)$$3. The protocol does not use bounding-box IoU. Instead, each high-similarity ROI is treated as a hit if it overlaps the true ship region by more than 50% area, with this overlap assessed manually. This choice is important because it aligns the metric with the framework’s window-based detection mechanism and with the dataset’s non-box-centric annotation philosophy.

At $$(\mathrm{N}, \mathrm{NE}, \mathrm{E}, \mathrm{SE}, \dots)$$4, the reproduced results compare vanilla ViT-B-32 CLIP with ViT-ShipSemVL.

Task	ViT-B-32 R/P	ViT-ShipSemVL R/P
Task A	76.6 / 80.0	84.6 / 88.0
Task B	80.0 / 63.2	73.3 / 84.6
Task C	87.1 / 77.1	90.3 / 84.9

The paper explicitly interprets these numbers as follows. On Task A, ShipSem-VL fine-tuning yields +8 pp recall and +8 pp precision at $$(\mathrm{N}, \mathrm{NE}, \mathrm{E}, \mathrm{SE}, \dots)$$5. On Task B, precision improves by +21 pp $$(\mathrm{N}, \mathrm{NE}, \mathrm{E}, \mathrm{SE}, \dots)$$6, which is identified as critical for complex semantics. On Task C, the best balanced precision/recall is approximately $$(\mathrm{N}, \mathrm{NE}, \mathrm{E}, \mathrm{SE}, \dots)$$7 at $$(\mathrm{N}, \mathrm{NE}, \mathrm{E}, \mathrm{SE}, \dots)$$8, compared with a baseline of $$(\mathrm{N}, \mathrm{NE}, \mathrm{E}, \mathrm{SE}, \dots)$$9 (Li et al., 21 Aug 2025).

The ablation summary isolates the contributions of the main components. Removing the multi-scale window and using single-scale only drops Task B precision by 12 percentage points. Disabling cross-modal attention and relying only on global cosine similarity lowers Task C recall by 6 percentage points. These results support the interpretation that proposal scale diversity is especially important for complex semantic queries, while token-level cross-modal alignment is especially important for differentiating ships that share many coarse attributes.

6. Interpretation, limitations, and projected extensions

The paper identifies three principal empirical insights. First, fine-grained semantic supervision substantially boosts precision in attribute-specific detection tasks, confirming the value of multi-label annotations. Second, the multi-scale sliding windows adapt to varying ship sizes and imaging conditions, improving recall in cluttered scenes. Third, cross-modal attention helps the model focus on visually localized details such as hull numbering or funnel color, going beyond global image-text matching (Li et al., 21 Aug 2025).

Several limitations are also stated explicitly. In visually dense scenes corresponding to Task C, lowering the threshold $\in \{1,2,3+\}$0 can cause neighboring ships to merge into one ROI, reducing recall. Because the framework does not include explicit bounding-box regression, localization granularity is constrained by fixed window sizes. Manual evaluation remains necessary for final accuracy assessment. These are not incidental shortcomings: they follow directly from the design decision to prioritize semantic-aware matching over conventional dense localization.

The future directions proposed in the paper are correspondingly targeted. They include introducing a weakly supervised bounding-box regressor to refine ROI coordinates, expanding ShipSem-VL via semi-automatic annotation using self-training and LLM assistance to thousands of samples, and generalizing SASD to other maritime objects such as oil rigs and buoys while integrating temporal sequences for vessel tracking with vision-language cues (Li et al., 21 Aug 2025). This suggests a longer-term trajectory in which the current framework functions as a semantic foundation that may later be coupled with stronger spatial localization and temporal reasoning.

Taken together, ShipSem-VL occupies a specific niche within remote-sensing maritime analysis: it formalizes ship detection as a vision-language grounding problem with explicit fine-grained attributes. Its main contribution is therefore not simply higher ship-detection accuracy, but a reformulation of maritime remote sensing around attribute-rich semantic supervision and text-conditioned retrieval, with measurable gains on cluttered detection, complex query resolution, and disambiguation of similar vessels.