SAMFusion: Sensor-Adaptive 3D Detection

• SAMFusion is a sensor-adaptive multimodal fusion framework that combines RGB, NIR, LiDAR, and radar for robust 3D detection in challenging conditions.
• It employs distance-aware blending and transformer-based refinement to dynamically weight sensor reliability during the detection process.
• Experimental evaluations highlight significant gains in long-range pedestrian detection under adverse weather, improving AP by up to 17.2 points.

SAMFusion is a sensor-adaptive multimodal fusion framework for 3D object detection in adverse weather and low-light conditions. It was introduced as a four-sensor detection stack that combines RGB camera, NIR gated camera, LiDAR, and radar through attentive, depth-based blending schemes, learned refinement on the Bird’s Eye View (BEV) plane, and a transformer decoder that weighs modalities based on distance and visibility (Palladin et al., 22 Aug 2025). In the cited formulation, the method is aimed at maintaining reliable multimodal perception when one or more sensing channels degrade under heavy fog, snow, night-time conditions, or obstructions due to soiling, and it is reported to improve average precision by 17.2 AP compared to the next best method for vulnerable pedestrians in long distances and challenging foggy scenes (Palladin et al., 22 Aug 2025).

1. Adverse-weather 3D detection problem and design motivation

SAMFusion is positioned against a failure mode of contemporary multimodal 3D detection: strong fusion systems often perform well in clear weather but become brittle when sensor quality degrades unevenly across modalities. The motivating observation is asymmetric degradation. LiDAR can lose points or become asymmetric in fog and snow; RGB cameras suffer from darkness, glare, backscatter, and occlusion; radar remains robust at long range and in bad weather but is sparse and noisy; and NIR gated cameras can maintain usable imagery in darkness, fog, snow, and other low-visibility settings through active illumination (Palladin et al., 22 Aug 2025).

The framework targets two specific weaknesses. The first is poor cross-modal projection quality: methods that rely heavily on projecting one modality into another, especially into BEV, become unreliable when depth estimates, point clouds, or image visibility are degraded. The second is over-reliance on a single modality for proposal generation, especially LiDAR. If proposals are largely LiDAR-derived, then fog- or snow-induced LiDAR degradation propagates directly into missed detections, with long-range pedestrians being especially vulnerable because they are small, partially occluded, and difficult to detect in poor visibility (Palladin et al., 22 Aug 2025).

SAMFusion responds by avoiding a single dominant sensing channel. Its core design principle is distance-aware and weather-aware sensor complementarity. This suggests that the framework should be understood less as generic feature concatenation and more as a reliability-weighted detection system in which different modalities dominate under different sensing regimes. The reported emphasis on pedestrians at 50–80 m reflects this rationale, since such cases concentrate the combined effects of sparsity, occlusion, and adverse-weather attenuation (Palladin et al., 22 Aug 2025).

2. Sensor stack and overall fusion architecture

The method uses four sensing modalities: RGB camera, gated NIR camera, LiDAR, and radar. These are encoded into modality-specific feature maps

$\varphi_C,\ \varphi_G,\ \varphi_L,\ \varphi_R,$

corresponding respectively to RGB camera, gated camera, LiDAR, and radar features (Palladin et al., 22 Aug 2025).

The fusion stack proceeds in three stages. First, it performs early cross-modal adaptive blending. This includes camera–LiDAR interactions, radar–camera interactions, and the use of the gated camera to strengthen visibility in low light and adverse weather. Second, it forms BEV proposals by fusing LiDAR and radar with distance-dependent weighting, followed by later injection of gated-camera information to add distance cues and improve localization. Third, it applies transformer-decoder refinement, using multimodal BEV proposals and modality-aware decoding (Palladin et al., 22 Aug 2025).

This organization distinguishes SAMFusion from fusion pipelines that merge modalities only once or only at a single representational level. The encoder-stage blending integrates image and range cues before proposal formation; the BEV-stage fusion establishes distance-adaptive proposal features; and the decoder stage refines detections from multimodal rather than LiDAR-only queries. A plausible implication is that robustness arises not from any one sensor addition alone, but from the repeated re-evaluation of modality utility across stages.

3. Cross-modal adaptive blending and BEV proposal formation

At encoder stage, SAMFusion performs camera-adaptive blending by treating camera features as queries and LiDAR features as keys and values after projection into corresponding camera points. Pixel-wise depth is used to lift image coordinates into 3D:

$$\begin{cases} z = d(u,v), \ x = (u - C_x)\times z/f_x, \ y = (v - C_y)\times z/f_y, \end{cases}$$

where $d(u,v)$ is pixel-wise depth, $(u,v)$ is image location, $(f_x,f_y)$ are focal lengths, and $(C_x,C_y)$ are principal point coordinates (Palladin et al., 22 Aug 2025). After projection, RGB- and gated-camera contexts derived from LiDAR are combined by

$$\varphi_{L,CG} = \varphi_{L,C} \oplus \varphi_{L,G},$$

with $\oplus$ denoting element-wise addition (Palladin et al., 22 Aug 2025).

Cross-modal attention then enriches the queried camera stream:

$$\varphi_{C;G}^* = \sum_{\varphi_{L,CG}\in J_s} \mathrm{softmax}\!\left( \frac{\varphi_{C;G}\,\varphi_{L,CG}^{T}}{\sqrt{d}} \right)\varphi_{L,CG},$$

where $J_s$ is a local attention window and $$\begin{cases} z = d(u,v), \ x = (u - C_x)\times z/f_x, \ y = (v - C_y)\times z/f_y, \end{cases}$$0 is the feature dimensionality for normalization (Palladin et al., 22 Aug 2025). In parallel, intra-modal attention is applied within the queried modality to preserve modality-specific structure. The same general strategy is reversed for LiDAR-adaptive blending, where LiDAR features query image-derived context, and is adapted for radar, where only RGB camera context is used and intra-modal attention is omitted because radar point clouds are sparse (Palladin et al., 22 Aug 2025).

The BEV proposal stage is a central component of the architecture. Instead of relying solely on LiDAR for proposals, SAMFusion creates multimodal BEV object proposals $$\begin{cases} z = d(u,v), \ x = (u - C_x)\times z/f_x, \ y = (v - C_y)\times z/f_y, \end{cases}$$1 using LiDAR, radar, and gated camera features (Palladin et al., 22 Aug 2025). LiDAR and radar are fused through a learned Gaussian distance gate:

$$\begin{cases} z = d(u,v), \ x = (u - C_x)\times z/f_x, \ y = (v - C_y)\times z/f_y, \end{cases}$$2

with

$$\begin{cases} z = d(u,v), \ x = (u - C_x)\times z/f_x, \ y = (v - C_y)\times z/f_y, \end{cases}$$3

where $$\begin{cases} z = d(u,v), \ x = (u - C_x)\times z/f_x, \ y = (v - C_y)\times z/f_y, \end{cases}$$4 is distance from the ego vehicle, $$\begin{cases} z = d(u,v), \ x = (u - C_x)\times z/f_x, \ y = (v - C_y)\times z/f_y, \end{cases}$$5 is a learned variance parameter, and $$\begin{cases} z = d(u,v), \ x = (u - C_x)\times z/f_x, \ y = (v - C_y)\times z/f_y, \end{cases}$$6 is learned MLP refinement (Palladin et al., 22 Aug 2025). In the paper’s interpretation, LiDAR is emphasized at close range, while radar is emphasized at long range.

The gated camera is then transformed into BEV using pillar-based conditioning, producing $$\begin{cases} z = d(u,v), \ x = (u - C_x)\times z/f_x, \ y = (v - C_y)\times z/f_y, \end{cases}$$7, and late fusion forms

$$\begin{cases} z = d(u,v), \ x = (u - C_x)\times z/f_x, \ y = (v - C_y)\times z/f_y, \end{cases}$$8

Class-dependent convolution layers are applied to $$\begin{cases} z = d(u,v), \ x = (u - C_x)\times z/f_x, \ y = (v - C_y)\times z/f_y, \end{cases}$$9 to extract proposal centers and produce the initial proposal set $d(u,v)$0 (Palladin et al., 22 Aug 2025). This proposal design is one of the method’s defining features: proposals are explicitly multimodal, distance-adaptive, and weather-aware.

4. Transformer decoder and optimization

The detector uses a transformer decoder with four stacked layers. The decoder starts from the multimodal proposal set $d(u,v)$1, refines these proposals with BEV features, and predicts final bounding boxes and classes. It is described as inspired by DETR-style training with Hungarian matching and as using multimodal predictive interaction layers from DeepInteraction (Palladin et al., 22 Aug 2025).

The training objective follows a DETR/TransFusion-style structure:

$d(u,v)$2

where $d(u,v)$3 is classification cross-entropy loss, $d(u,v)$4 is bounding-box regression loss, and $d(u,v)$5 is IoU-related loss (Palladin et al., 22 Aug 2025). Hungarian matching is used to assign predictions to ground truth, enforcing one-to-one matching and reducing duplicate detections.

The decoder design is important because the method’s robustness claim does not rest solely on early fusion. The paper explicitly states that the decoder weighs modalities according to distance and visibility, so the final detection process continues to model modality reliability after proposal formation (Palladin et al., 22 Aug 2025). This suggests that SAMFusion’s “sensor-adaptive” property is distributed across the pipeline rather than localized to one attention block or one fusion layer.

5. Experimental evaluation and ablation evidence

SAMFusion is evaluated primarily on SeeingThroughFog, which contains 12,997 annotated samples with day, night, fog, and snow scenes, split into 10,046 training, 1,000 validation, and 1,941 test samples. The method also reports supplemental evaluation on nuScenes. Metrics are KITTI-style 3D AP and BEV AP for passenger car and pedestrian, with IoU thresholds of 0.2 for car and 0.1 for pedestrian, computed with 40 recall positions and reported over distance bins 0–30 m, 30–50 m, and 50–80 m (Palladin et al., 22 Aug 2025).

The most prominent reported gains concern long-range pedestrian detection. On the SeeingThroughFog test set, pedestrian AP reaches 80.09 / 70.97 / 40.16 for day and 75.49 / 67.59 / 27.14 for night at 0–30 / 30–50 / 50–80 m (Palladin et al., 22 Aug 2025). The paper emphasizes the 50–80 m regime, reporting up to +34.85% improvement during the day and up to +17.03% improvement at night for pedestrian AP, and highlighting +17.2 AP over the next best method for vulnerable pedestrians in dense fog or challenging foggy scenes (Palladin et al., 22 Aug 2025). In snow, pedestrian AP is reported as 87.44 / 80.51 / 41.45, with improvements over the best baseline of +11.2 AP, +13.6 AP, and +15.62 AP across the three distance bins (Palladin et al., 22 Aug 2025).

Car detection also improves, though gains are smaller. On the overall SeeingThroughFog test set, car AP reaches 97.25 / 89.50 / 50.68 for day and 98.77 / 88.91 / 44.40 for night (Palladin et al., 22 Aug 2025). The paper attributes more limited car-class gains partly to the class being easier and to dataset annotation conventions.

Ablation studies isolate the contribution of sensing modalities and architectural choices. Input-modality ablations compare CL, GL, CLR, GLR, and CGLR settings, with the reported conclusions that RGB helps in daylight, gated camera helps more at night, radar helps especially at longer distances, and all four modalities together are best (Palladin et al., 22 Aug 2025). Component ablations indicate that stacking modalities alone is insufficient: depth-based transformation, multimodal proposals, and learned $d(u,v)$6 weighting materially affect performance, with multimodal proposals improving night pedestrian detection by 15.2% and the learned $d(u,v)$7 weighting boosting detection by up to 20.7% (Palladin et al., 22 Aug 2025). The paper also reports that gated-based proposals are more helpful than color-based proposals because gated images encode distance more explicitly.

6. Position within SAM-guided and multimodal fusion research

The exact title “SAMFusion” refers to the adverse-weather 3D detector described above (Palladin et al., 22 Aug 2025). At the same time, adjacent literature documents a broader movement toward using SAM-derived priors, SAM-adapted feature extractors, or fusion-guided prompting in multimodal vision. These related formulations are not identical to SAMFusion, but they clarify the wider technical context in which the method appears.

In multimodal salient object detection, Sammese adapts SAM to RGB-D and RGB-T inputs by extracting a multi-modal semantic feature with a Multi-modal Complementary Fusion Module, injecting it into the SAM image encoder via a multi-modal adapter, and generating semantic-geometric prompts for the SAM mask decoder, while keeping all SAM parameters frozen (Wang et al., 2024). In infrared-visible image fusion, SGDFuse uses SAM-generated semantic masks as explicit priors within a conditional diffusion framework, concatenating a preliminary fused image and two semantic masks into a five-channel condition for DDPM-style refinement (Zhang et al., 7 Aug 2025). SAGE similarly uses SAM semantic priors for infrared-visible fusion but introduces a Semantic Persistent Attention module and a bi-level distillation mechanism so that SAM is not required during inference (Wu et al., 3 Mar 2025). FusionSAM, in autonomous-driving segmentation, turns multimodal latent fusion into prompt generation for SAM through Latent Space Token Generation and Fusion Mask Prompting, reporting 63.0 mIoU on MFNet and 61.8 mIoU on FMB (Li et al., 2024). RoboFusion, in robust multi-modal 3D detection, uses SAM-derived image features rather than segmentation masks, adapting SAM into SAM-AD and combining AD-FPN, depth-guided wavelet attention, and adaptive fusion for corruption robustness (Song et al., 2024).

These works differ in modality sets, objectives, and training regimes, but they share a recurring principle: pretrained segmentation or foundation-model structure can be repurposed as semantic guidance, robust representation, or promptable control within multimodal fusion pipelines. This suggests that SAMFusion belongs to a larger research trajectory in which multimodal fusion is no longer treated as a purely low-level aggregation problem. Instead, semantic priors, prompt pathways, or pretrained visual representations are used to determine where fusion should occur, how modality reliability should be weighted, and which downstream tasks the fused representation should support.

7. Limitations and implications

The paper identifies several practical constraints. Generalization beyond the specific adverse-weather datasets is not fully established. The model is more complex than single-modality or simple fusion baselines. It relies on calibrated sensors and depth estimation for cross-modal alignment. Car-class gains are smaller than pedestrian gains, partly due to dataset annotation conventions and easier geometry (Palladin et al., 22 Aug 2025).

These limitations delimit the scope of the reported results. The strongest evidence concerns adverse-weather, multi-sensor autonomous-driving perception, especially long-range vulnerable road users. Within that scope, the framework’s main implication is methodological rather than merely incremental: it argues that adverse-weather fusion should be sensor-adaptive, distance-aware, and proposal-aware, rather than relying on a fixed fusion rule or a single dominant sensor. The combination of RGB, gated NIR, LiDAR, and radar is therefore not presented as simple redundancy, but as a mechanism for learning when and how each modality should dominate under changing visibility and range conditions (Palladin et al., 22 Aug 2025).

In that sense, SAMFusion is best characterized as a four-sensor BEV detection system whose defining contribution is learned trust allocation across modalities. Its reported gains for far-range pedestrians in fog and snow make that claim concrete, while related SAM-guided fusion literature indicates that such reliability-aware multimodal modeling is part of a broader shift in multimodal vision research (Palladin et al., 22 Aug 2025).