Robust Cross-Modal Foundation Model Perception for Underwater Robots under Degraded Visual Conditions

Abstract: Reliable underwater robotic perception remains difficult because optical imagery degrades under turbidity, wavelength-dependent attenuation, low illumination, scattering, and blur. Although sonar provides complementary information that is less affected by optical visibility, prior visual-sonar research has largely focused on feature alignment and nominal detection performance. We investigate cross-modal robustness as visual reliability deteriorates and assess whether pretrained visual foundation-model representations can be complemented by sonar under severe degradation. We use frozen DINOv2 as the visual encoder and construct a controlled five-level benchmark ranging from clean to extreme visual conditions. We compare conventional visual detection, frozen foundation-model representations, sonar context, fixed multimodal fusion, clean-trained adaptive gating, and degradation-aware gated fusion. Our method trains the fusion mechanism across the full range of degradation while keeping the visual and sonar encoders frozen, allowing modality contributions to adapt without fine-tuning the pretrained backbone. Under extreme combined degradation, the DINOv2 baseline achieves 0.4610 balanced accuracy, while degradation-aware visual-sonar fusion reaches 0.6152, a 33.5% relative improvement. The learned sonar contribution increases from 14.2% under clean conditions to 41.3% under extreme degradation, demonstrating adaptive redistribution of cross-modal reliance. Fusion provides the largest gains under severe turbidity and blur, whereas color attenuation alone yields little additional benefit. These results show that foundation-model representations remain valuable but insufficient under severe information loss, and that explicitly adapting fusion to modality reliability can improve robust underwater multimodal perception.

• The paper introduces a degradation-aware gated fusion model that trains on multiple levels of synthetic optical corruption while keeping frozen DINOv2 visual and ResNet-18 sonar encoders.
• The model maintains 0.6152 balanced accuracy at the most severe degradation level, a 33.5% relative improvement over visual-only perception, with mean sonar weighting rising from 14.2% to 41.3%.
• Results show sonar assistance is most valuable when turbidity or blur destroys visual structure, while fixed fusion and clean-trained gating fail to adapt reliably without exposure to changing modality quality.

Motivation and problem statement

Optical perception underwater is limited by wavelength-dependent absorption, scattering-induced veiling, illumination loss, and blur, producing a distribution shift between training imagery and degraded deployment conditions. Sonar is largely insensitive to optical visibility but offers lower spatial resolution and weaker appearance information, so the two modalities are complementary rather than interchangeable. The paper addresses a question that prior visual-sonar work—focused on feature alignment and nominal detection accuracy—has not systematically examined: how should a multimodal model redistribute its reliance between modalities as visual reliability progressively collapses?

The study formulates this as a controlled reliability shift. Both modalities remain present at all times; only the information content of the optical modality is systematically corrupted across five ordered severity levels $D_0$–$D_4$, while the paired sonar observation is held fixed. This separates modality availability from modality reliability and enables paired comparisons of performance retention.

Benchmark and evaluation protocol

Experiments use a synchronized subset of the UMOD dataset (4,000 paired optical–sonar observations from a BlueView M900 forward-looking sonar and ZED2 stereo camera in a pool setting), restricted to five target categories for recognition tasks. A grouped block split prevents sequence-level leakage: contiguous 40-sample blocks are assigned to train/val/test partitions, and all degradation variants of an observation remain in the same partition as their clean counterpart.

The combined degradation operator applies severity-scaled instances of four perturbations—intensity scaling for illumination loss, per-channel coefficients with $\beta_{R,s} < \beta_{G,s} < \beta_{B,s}$ for wavelength attenuation, an attenuation–backscatter model $I_{\text{turb}} = J\,t_s + A_s(1-t_s)$ for turbidity, and Gaussian smoothing for blur. The authors are explicit that these are controlled robustness probes rather than physical simulations of underwater radiative transfer, and that severity levels do not map to specific water types or visibility distances.

Robustness is summarized by retained performance $R_m(s) = M_m(s)/M_m(0)$ relative to each model's own clean baseline, aggregated as mean relative robustness (MRR). Balanced accuracy serves as the primary recognition metric given class imbalance; detection uses mAP@0.5 and [email protected]:0.95. Absolute scores across the detection and recognition tasks are explicitly not treated as comparable.

Method

The architecture deliberately decouples representation from fusion. The visual branch uses a frozen DINOv2-small encoder on ground-truth object crops (with a 10% bounding-box expansion), yielding a 384-dimensional embedding. The acoustic branch encodes the full synchronized sonar frame with a frozen ImageNet-pretrained ResNet-18, yielding a 512-dimensional context vector—notably not an object-aligned sonar crop, since optical bounding boxes cannot be validly transferred to range-azimuth sonar geometry without calibration or learned correspondence.

Three fusion regimes are compared:

• Fixed concatenation: the two embeddings are concatenated and classified; no mechanism exists to adjust modality contributions.
• Clean-trained gated fusion: modality-specific projections into a shared space feed a gating network whose softmax weights combine the projected embeddings; trained only on $D_0$ data.
• Degradation-aware gated fusion: identical architecture, but trained across all five degradation levels, with every degraded variant paired to the same sonar representation and label.

Only projection layers, gate, and classifier head are trainable; both encoders stay frozen. Crucially, the gate never receives the severity index $s$ as input—it must infer reliability from the representations themselves. This design isolates whether exposure to reliability variation, rather than architectural adaptivity alone, drives robustness.

Results

Visual-only degradation behavior

YOLO11n detection collapses under severe corruption: [email protected] drops from 0.7020 ($D_0$) to 0.3359 ($D_3$) and 0.0281 ($D_4$)—approximately 4% of clean performance—with recall falling from 0.5127 to 0.0215, indicating loss of detectable evidence rather than localization precision. The frozen DINOv2 representation is markedly more stable: balanced accuracy moves from 0.6210 ($D_4$0) through a slight increase at $D_4$1 (0.6495) down to 0.4610 ($D_4$2), retaining roughly 74.2% of clean performance (MRR 0.8919 versus YOLO's 0.5531). Because the DINOv2 experiment uses ground-truth crops while YOLO localizes, this comparison supports only the narrower claim that pretrained representations are relatively invariant once localization is supplied.

Model	$D_4$3 BAcc	$D_4$4 retained	MRR
Visual FM (DINOv2)	0.4610	0.7423	0.8919
Sonar context	0.3581	1.0000	1.0000
Fixed fusion	0.4343	0.6726	0.8827
Clean-trained gate	0.3752	0.6043	0.9011
Degradation-aware gate	0.6152	0.9700	1.0278

Complementarity does not imply robust fusion

Sonar alone yields a constant balanced accuracy of 0.3581 across all severities—inferior to clean vision but immune to optical corruption. Fixed concatenation exceeds vision-only modestly under mild conditions (0.7143 vs. 0.6495 at $D_4$5) yet falls below the visual baseline at $D_4$6 (0.4343 vs. 0.4610). More strikingly, the clean-trained adaptive gate—the same architecture that succeeds when trained degradation-aware—collapses to 0.3752 at $D_4$7, worse than fixed fusion. This rules out insufficient architectural flexibility as the explanation: a learnable gating mechanism fails without training evidence of reliability shifts, consistent with findings from ModDrop and SMIL that explicit modality-corruption exposure during training is necessary for adaptation.

Degradation-aware fusion

Degradation-aware gating maintains 0.6438 at $D_4$8 and 0.6152 at $D_4$9—a 33.5% relative improvement over the DINOv2 baseline at extreme severity—while remaining competitive under clean conditions (0.6343). Its MRR of 1.0278 reflects approximate preservation of clean performance; the authors caution this should not be read as degradation improving perception, given small non-monotonic variations on a limited test set.

The learned gate exhibits the hypothesized redistribution without any monotonicity constraint: mean sonar weight rises from 14.2% ($\beta_{R,s} < \beta_{G,s} < \beta_{B,s}$0) through 17.1% ($\beta_{R,s} < \beta_{G,s} < \beta_{B,s}$1) to 29.1% ($\beta_{R,s} < \beta_{G,s} < \beta_{B,s}$2) and 41.3% ($\beta_{R,s} < \beta_{G,s} < \beta_{B,s}$3). The slight dip between $\beta_{R,s} < \beta_{G,s} < \beta_{B,s}$4 and $\beta_{R,s} < \beta_{G,s} < \beta_{B,s}$5 confirms the weights are driven by representations rather than severity index. The authors emphasize these coefficients are task-dependent fusion contributions, not calibrated sensor-reliability probabilities—a caveat that limits interpretation of the weight trajectory as a reliability estimate.

Degradation-type ablation

Isolating individual perturbations shows acoustic assistance is strongly condition-dependent. Severe turbidity produces the largest gain (20.8% relative improvement, 0.4667 → 0.5638), followed by severe blur (15.4%, 0.6190 → 0.7143). Brightness reduction yields smaller gains (5.9% at $\beta_{R,s} < \beta_{G,s} < \beta_{B,s}$6). Isolated color attenuation at $\beta_{R,s} < \beta_{G,s} < \beta_{B,s}$7 produces zero measurable benefit—both models reach 0.7295—because channel-statistics shifts leave spatial structure intact enough for DINOv2 to remain discriminative. This refines the central conclusion: sonar adds value specifically when structural visual evidence is destroyed, not uniformly across all optical shifts. Eight qualitative recovery cases at $\beta_{R,s} < \beta_{G,s} < \beta_{B,s}$8 illustrate the aggregate improvement at sample level, though the authors correctly note they carry little statistical weight.

Limitations

The study concedes several constraints that bound its claims. The test set is small (five of nine UMOD classes), increasing sensitivity to individual samples. Recognition experiments use ground-truth crops rather than end-to-end detection, so the dramatic YOLO11n localization failure under poor visibility is not addressed by the fusion results. The sonar branch provides scene-level context rather than object-aligned features, meaning some gains may stem from scene-level correlations rather than target-specific acoustic information. Only the visual modality is corrupted; real deployments can degrade both sensors jointly, and the synthetic degradations do not reproduce full underwater radiative transfer. Finally, the learned weights lack calibration against true sensor reliability—an open question the authors propose addressing via uncertainty-aware fusion—and transfer beyond the pool-collected UMOD setting to field conditions with naturally varying visibility remains undemonstrated.

Conclusion

The paper establishes that underwater multimodal robustness is fundamentally a reliability-adaptation problem rather than a representational one. Frozen foundation-model representations substantially delay—but do not prevent—performance collapse under severe optical degradation, and the mere presence of complementary sonar neither guarantees nor automatically confers robustness. The decisive factor is exposing the fusion mechanism to the full range of reliability conditions during training: doing so yields a 33.5% relative improvement over the visual baseline at extreme degradation, accompanied by an emergent redistribution of learned modality contributions from 14.2% to 41.3% sonar reliance, concentrated precisely where visual structure is destroyed (turbidity and blur). The principal open questions left by this preliminary study are whether degradation-aware weighting transfers to end-to-end detection with joint cross-modal correspondence learning, whether the emergent acoustic reliance persists under naturally varying visibility in field deployment, and how the approach extends to bidirectional corruption and missing-modality conditions.