ShipsEar Benchmark: Underwater Acoustic Recognition

• The benchmark defines robust protocols and baselines for UATR using over 9 hours of ship noise and background data from the QiandaoEar22 dataset.
• It extracts 8 key acoustic features across 1D and 2D domains and evaluates multiple deep learning models, achieving near-perfect accuracy in binary and multi-target tasks.
• Results emphasize that careful feature extraction, particularly using spectrum and MFCC, is more crucial than deep network architectures for robust ship noise identification.

The ShipsEar Benchmark is a standardized framework for evaluating underwater acoustic target recognition (UATR) algorithms under realistic multi-target conditions, utilizing the QiandaoEar22 dataset. It establishes reproducible protocols, task definitions, and baselines for detecting and classifying ship-radiated noises, emphasizing performance under complex backgrounds and vessel diversity. Its creation addresses the lack of high-fidelity, multi-target, real-world underwater acoustic datasets and benchmarking tools for robust UATR research (Du et al., 2024).

1. Dataset Foundation and Acquisition Protocols

ShipsEar Benchmark is grounded in experimentally acquired data from QiandaoEar22, comprising 9 hours 28 minutes of ship-radiated noise and 21 hours 58 minutes of background noise, with signals recorded at two positions in Qiandao Lake. The signals were captured using DigitalHyd SR-1 self-capacitance hydrophones with a 52,734 Hz sampling rate, 16-bit resolution, covering a 1–25.8 kHz frequency band, and were deployed at 10–15 m in water depths of 30–50 m.

The benchmark encompasses 10,611 ship segments and 25,900 background noise segments, sampled in fixed 3-second windows aligned to ship-passage events. Each data segment is annotated with timestamped metadata including ship name/ID, target type (S/M), distance category, and audibility. Classification categories cover 20 vessel classes plus a “background” class. The dataset does not involve signal augmentation or denoising, preserving raw environmental features.

2. Feature Extraction and Representation

The ShipsEar Benchmark defines a comprehensive set of eight acoustic features spanning both 1D signal and 2D time-frequency domains:

• 1D Features:

1. Spectrum ($S[k] = |X[k]|$), using the discrete Fourier transform.
2. Power Spectral Density ($P_{xx}[k] = \frac{1}{N}|X[k]|^2$).
3. DEMON spectrum ($$D[k] = \bigl|\sum_{n=0}^{N-1}|x_f[n]| e^{-j2\pi kn/N}\bigr|$$), extracting the envelope of filtered signals.

• 2D Features:

1. LOFAR spectrogram ($L(n,f) = 20\log_{10}|\mathrm{STFT}\{x\}(n,f)|$).
2. Log-Mel spectrogram ($$M(n,m) = \log\Bigl(\sum_{k=0}^{K-1} H_m(k)|X(n,k)|^2\Bigr)$$).
3. MFCC (Mel-Frequency Cepstral Coefficients), as in: $$c_n = \sum_{k=1}^K \log S(k) \cos\left[\frac{n \pi}{K}(k-0.5)\right]$$.
4. PNCC (Power-Normalized Cepstral Coefficients): $$p_n = \sum_{k=1}^K W_n(k) \log\left(\frac{|X(k)|^2}{N(k)}\right)$$.
5. GFCC (Gammatone-Frequency Cepstral Coefficients): $g_n = \sum_{k=1}^K G_n(k) \log |X(k)|$.

Feature extraction is performed without additional signal conditioning beyond segmentation and down-sampling as required by feature computation.

3. Benchmark Tasks and Experimental Design

The ShipsEar Benchmark comprises two principal tasks:

• Task 1: Ship Presence Detection
  • Binary classification: “ship noise” versus “background.”
  • Dataset: 5,000 ship segments and 5,000 background segments.
  • Feature: 2D Log-Mel spectrogram.
  • Models: CNN2D, CRNN2D, BiLSTM, ResNet18, DenseNet121, ECAPA-TDNN.
  • Standard 70/15/15 split for training, validation, and testing.
• Task 2: Specific-Ship Identification in Multi-Target Contexts
  • Three one-vs-rest subsets: SpeedBoat, KaiYuan, UUV as “target class” versus all other ships.
  • Employed all eight feature types; input dimensionality constrained to ~16,400–16,900.
  • Classification networks included both 1D and 2D variants, as in Task 1.
  • Standardized training/validation/testing splits per target ship.

Evaluation Metrics: - Accuracy: $\mathrm{Acc} = \frac{TP + TN}{TP + TN + FP + FN}$ - Precision: $P = \frac{TP}{TP + FP}$ - Recall: $P_{xx}[k] = \frac{1}{N}|X[k]|^2$0 - $P_{xx}[k] = \frac{1}{N}|X[k]|^2$1 score: $P_{xx}[k] = \frac{1}{N}|X[k]|^2$2 - False Negative Rate (FNR), False Positive Rate (FPR)

4. Deep Learning Architectures and Training Regimens

The benchmark employs six canonical network architectures tailored for acoustic scene analysis and sequence modeling:

• CNN (1D/2D): Four convolutional blocks (kernel 5→3, stride 2, padding 1, batch norm, ReLU, dropout 0.1), followed by max pooling and fully connected output.
• CRNN (CNN + BiLSTM layers): CNN backbone identical to CNN2D, followed by a two-layer 512-unit bidirectional LSTM, fully connected output.
• BiLSTM: Two-layer bidirectional LSTM stacked on feature vector sequences, with fully connected output.
• ResNet18 and DenseNet121: Standard structures with final fully connected or softmax adjusted for binary tasks.
• ECAPA-TDNN: Five-channel architecture, 80→512 hidden units, designed for discriminative acoustic sequence modeling.

All models are trained with Adam optimizer, an initial learning rate of 0.001 (decayed on plateau), batch sizes of 32 (2D) or 64 (1D), and up to 100 epochs with early stopping governed by validation loss.

5. Performance Outcomes and Error Characteristics

Task 1 results demonstrate near-perfect separation of ship-radiated from background noise. CRNN2D achieves the highest accuracy (99.79 ± 0.05%), with FNR 0.00% and FPR 0.43%. All networks except BiLSTM surpass 99% accuracy.

In Task 2, the spectrum feature with BiLSTM attains the highest accuracy (99.56%) for UUV identification. For SpeedBoat and KaiYuan, optimal accuracies are 95.40% (spectrum + CNN1D) and 97.15% (spectrum + CRNN1D), respectively. Averaged over 10 runs, the spectrum feature outperforms all others (mean Acc 96.73%), with MFCC as the best 2D feature (95.47%). DEMON and LOFAR features underperform (<90% accuracy) in mixed scenarios.

Error analysis reveals the highest FPR in SpeedBoat identification (≈15%), attributed to the large “other” class with 13 vessel types; FNR is lowest (0–2%) in the UUV scenario. The data suggest that confusion increases when the “other” class is heterogeneous and large.

6. Implications for Feature and Network Selection

Empirical findings underscore that feature type has greater impact on UATR performance than network architecture depth or style. Spectrum and MFCC features consistently produce the most discriminative representations for ship-radiated noise, while DEMON and LOFAR are less effective under multi-source conditions.

Network selection guidelines emerging from this work are:

• For 1D spectrum features: CNN1D is preferable for small datasets (≤1,000 clips), CRNN1D for larger datasets.
• For 2D features (MFCC, Log-Mel): CRNN2D is effective for small-to-medium datasets, ECAPA-TDNN offers high accuracy for large datasets, while DenseNet121 achieves the highest accuracy if computational constraints are negligible.

This suggests that optimal performance is obtained less through deeper or more complex models than through adequate feature extraction aligned with task structure.

7. Future Directions and Research Opportunities

ShipsEar reveals several open avenues for advancing UATR:

• Multimodal fusion: Integrating 1D and 2D features via unified attention/transformer encoders.
• Data augmentation: Simulating multipath propagation and Doppler distortions to improve robustness.
• Transfer learning and domain adaptation: Extending the benchmark to new hydrophone types and aquatic environments (e.g., open ocean).
• Real-time deployment: Applying network pruning and quantization for embedded edge deployment in autonomous monitoring systems.

A plausible implication is that data-centric paradigms—including realistic data augmentation and semi-supervised learning—will become central to future growth in UATR accuracy and generalizability (Du et al., 2024).