Adaptive Multi-Sensor Fusion

• Adaptive Multi-Sensor Fusion (AMF) is a technique that dynamically merges diverse sensor streams by adjusting weighting based on sensor reliability, environmental conditions, and task requirements.
• AMF employs methods such as hypernetwork gating, attention routing, consistency metrics, and uncertainty-based weighting to optimize fusion at various levels while managing trade-offs between accuracy and latency.
• AMF is applied in domains like autonomous driving, aerial robotics, maritime surveillance, and smart homes to enhance robustness and real-time performance through selective sensor integration.

Searching arXiv for recent and foundational papers on adaptive multi-sensor fusion to ground the article in current literature. Adaptive Multi-Sensor Fusion (AMF) denotes a class of estimation and perception methods that combine heterogeneous sensing streams while adapting fusion behavior to changing sensor quality, environmental conditions, task demands, and computational budgets. In the literature summarized here, AMF appears in automated driving semantic segmentation and localization, aerial robotics, intelligent transportation systems, maritime situational awareness, distributed multi-target tracking, ambient assisted living, and real-time calibration and boresighting. Across these settings, the common objective is not merely to fuse more sensors, but to decide how, when, and to what extent each modality should influence the fused state, feature map, or decision output (Wang et al., 2022, Lin et al., 7 Mar 2025, Lanegger et al., 2023, Liu et al., 2024).

1. Problem scope and domain of use

AMF is motivated by the observation that sensor reliability is strongly context dependent. In autonomous driving, RGB, depth or LiDAR, radar, GNSS, IMU, and chassis signals provide complementary information, yet each modality degrades under specific conditions such as urban canyons, tunnels, dense canopy, fog, snow, rain, low light, or occlusion. This motivates adaptive fusion for semantic segmentation, localization, 3D detection, and odometry rather than fixed fusion rules (Wang et al., 2022, Lin et al., 7 Mar 2025, Palladin et al., 22 Aug 2025, Lai et al., 28 Feb 2025).

The same logic appears outside road vehicles. In maritime autonomy under haze, the architecture combines radar, EO visible, EO infrared, sonar, weather sensors, geo-sensors, AIS, and on-shore Vessel Traffic Surveillance, with Adaptive Multi-Sensor Management changing sensor weighting and controls according to weather assessment and range (Prasad et al., 2017). In aerial robots operating near structures, consistency-driven selection is used to choose which state-estimate sources to fuse or not to fuse during indoor–outdoor transitions, bridge inspection, and contact-rich flights (Lanegger et al., 2023). In intelligent transportation system localization, GNSS pseudorange, time-differenced carrier phase, IMU, and 4D radar are tightly integrated in a factor graph, with online noise estimation and outlier detection providing the adaptive element (Liu et al., 2024). In distributed multi-target tracking, the adaptive element is realized as Adaptive Complementary Fusion over LMB densities, coupled to distributed sensor control (Blair et al., 21 Apr 2026).

The term is also used in broader heterogeneous fusion settings. Bayesian score-level fusion in the SPHERE smart-home platform jointly models room-level location and activity using PIR, wearable RSSI and accelerometer, and RGBD-derived features, with modality utility learned probabilistically (Diethe et al., 2017). Real-time automotive boresighting uses IMU and a sensor-mounted accelerometer to estimate camera misalignment online, thereby improving downstream fusion quality through continuous extrinsic correction (0710.4833). A plausible implication is that AMF is best understood as an operational principle—adaptive weighting, gating, alignment, or model switching across modalities—rather than as a single algorithmic family.

The acronym is not semantically unique across adjacent literatures. In "Parametric Generalized Adaptive Moment Features" it denotes Adaptive Moment Features rather than Adaptive Multi-Sensor Fusion (Kumar, 24 Jun 2026), and in AMF-MedIT it refers to an Align–Modulation–Fusion module for image–tabular integration (Yu et al., 24 Jun 2025). This suggests that, in technical usage, AMF should always be interpreted relative to the surrounding measurement model and task.

2. Fusion levels and architectural patterns

The literature exhibits several recurrent fusion topologies, ranging from state-estimate fusion to feature-level fusion to score-level fusion. In automated driving perception, A3Fusion uses middle fusion with explicit cross-modal links called fuseLinks, each implemented by a convolutional fuseFilter with parameter tensor dimension $$c_{\text{input}} \times c_{\text{output}} \times w \times h$$, to reconcile channel mismatch between RGB and depth features and to enable cross-level as well as bidirectional feature transfer (Wang et al., 2022). In adverse-weather 3D detection, SAMFusion performs attentive, depth-based blending between RGB, LiDAR, gated camera, and radar, followed by BEV aggregation and transformer decoding (Palladin et al., 22 Aug 2025). In A2DO, LiDAR vertex and normal images, RGB features, and IMU embeddings are fused through a multi-layer, multi-scale encoder followed by temporal and spatial adaptive filters (Lai et al., 28 Feb 2025).

Other systems place the adaptive mechanism later in the pipeline. The end-to-end localization network of (Lin et al., 7 Mar 2025) performs mid-level feature fusion in FusionNet but applies late-stage residual gating to measurement innovations. ADM-Fusion separates translation and rotation branches and uses an adaptive sensor mixture-of-experts with content-aware routing over modality-specific expert outputs before task-specific regression (Moughnieh et al., 23 Jun 2026). By contrast, the aerial-robot framework of (Lanegger et al., 2023) fuses at the state-estimate level inside a loosely coupled, multi-graph optimization architecture, and the SPHERE framework fuses modality-specific classifier scores probabilistically at the output level (Diethe et al., 2017).

Work	Fusion level	Adaptive mechanism
A3Fusion (Wang et al., 2022)	Middle fusion with fuseLinks	Hypernetwork-driven gating and pruning
Maritime architecture (Prasad et al., 2017)	Feature-level EO and decision-level multi-source fusion	Weather- and range-dependent weighting
Learning-based AV localization (Lin et al., 7 Mar 2025)	Mid-level features and late residual gating	Recurrent per-state innovation weighting
Aerial robots (Lanegger et al., 2023)	State-estimate-level fusion	Consistency-based sensor selection
UniMSF (Liu et al., 2024)	Tight factor-graph integration	Outlier detection and online noise estimation

A recurring design tension is whether to fuse raw or near-raw measurements early, or to defer fusion until modality-specific structure has been extracted. The reported systems generally avoid raw early fusion. A3Fusion explicitly addresses spatial-feature-misalignment at intermediate CNN layers rather than fusing raw sensor tensors (Wang et al., 2022). The maritime architecture performs EO preprocessing and feature extraction before weighted feature fusion and decision-level superposition with AIS, radar, and sonar (Prasad et al., 2017). The SPHERE system also uses late score fusion rather than direct feature concatenation, with separate modality-specific feature maps and heavy-tailed priors over modality weights and features (Diethe et al., 2017). This suggests that AMF is often paired with modality-specific front ends, and that adaptivity is then imposed on the interactions among already structured latent representations.

3. Adaptation mechanisms and representative formulations

One major AMF family uses explicit gating or pruning. In A3Fusion, a learned hypernetwork consisting of one convolution layer, an average pooling layer, a flattening layer, and one fully connected layer outputs per-link importance scores $w_l$. A global threshold $th$ prunes fuseLinks at inference time according to the rule $w_l < th \Rightarrow$ prune the $l$-th fuseLink, thereby trading accuracy for latency on a per-input basis (Wang et al., 2022). In the learning-based localization system of (Lin et al., 7 Mar 2025), FusionNet outputs element-wise residual weights $\mathbf{w}_n = \sigma(\mathbf{o}_n)$, and the state update is written as

$$\mathbf{X}^{\mathtt{upd}}_{n} = \mathbf{X}^{\mathtt{pred}}_{n} + \mathbf{w}_n \odot \mathbf{r}_n .$$

Because the gating conditions on measurement quality features, recent motion features, and normalized innovations, degraded GNSS can be down-weighted without explicit covariance tuning.

A second family uses adaptive routing across experts or modalities. ADM-Fusion computes routing logits $\ell_s$ from attention-enhanced sensor tokens and converts them into softmax weights

$$w_s = \frac{\exp(\ell_s)}{\sum_{j=1}^{S} \exp(\ell_j)},$$

after which the fused feature is $\mathbf{z} = \sum_{s=1}^{S} w_s\,\mathbf{e}_s$. The same system adds a routing balance regularizer

$w_l$0

to prevent router collapse and preserve modality diversity (Moughnieh et al., 23 Jun 2026). A2DO implements a different variant: its Temporal Feature Filter uses multi-head attention and Gumbel-Softmax decisions to keep or discard modality frames, and its Spatial Feature Filter uses self-attention-driven channel masking to suppress degraded channels (Lai et al., 28 Feb 2025). Lvio-Fusion replaces attention routing with actor–critic reinforcement learning: a TD3 agent outputs adaptive weights for camera and LiDAR factors in a tightly coupled SLAM graph, using a reward based on reciprocal relative pose error (Jia et al., 2021).

A third family uses consistency or conflict as the adaptation signal. For aerial robots, pairwise Cramér–von Mises distances are computed between body-frame velocities inferred from different sensors over a recent 1 s window, arranged in a consistency matrix $w_l$1, and used to decide which sensors to fuse or exclude (Lanegger et al., 2023). In the conflict-measure framework, each sensor is represented by an interval $w_l$2, conflict is derived from interval overlap across all sensor combinations, and the final fusion weights are

$w_l$3

Sensors with higher conflict receive smaller weights, improving over simple averaging under additive impulse noise, DC offset, and Gaussian noise (Wei et al., 2018).

A fourth family makes adaptivity explicit in the uncertainty model. The multi-sensor Student’s $w_l$4 filter uses local Student’s $w_l$5 Kalman filters and a suboptimal arithmetic-average fusion of local $w_l$6 posteriors, with the robust factor

$w_l$7

scaling the posterior covariance and thereby adapting to residual magnitude (Li et al., 2022). UniMSF adapts pseudorange covariances online by fitting a Gaussian mixture model to recent residuals with EM, while TDCP outliers are rejected through Doppler-integral gating (Liu et al., 2024). In the MEMS attitude-estimation framework, time-varying gains are computed directly from mean-square error estimates; for each angle,

$w_l$8

yielding a complementary filter with faster transient response than a fixed-gain design (Nemec et al., 2017).

The maritime system occupies a hybrid position. Its Adaptive Multi-Sensor Management changes sensor polling frequency and imaging controls, while feature weights in combined EO interpretation satisfy

$w_l$9

and the weights are informed by haze, luminance, humidity, wind, and historical weather templates (Prasad et al., 2017). A plausible implication is that AMF mechanisms can be categorized by what is being adapted—connections, gains, uncertainties, or sensor participation—but these categories frequently coexist within a single system.

4. Alignment, uncertainty, and robustness

AMF depends on alignment at several levels. A3Fusion emphasizes that middle fusion cannot simply use element-wise addition or concatenation when intermediate features are spatial-feature-misaligned, and resolves this with fuseFilters that transform source channels into the exact channel count expected by the target layer (Wang et al., 2022). The maritime architecture aligns EO feeds to world coordinates and common physical units, while AIS geodetic coordinates and radar or sonar detections are transformed into a common local world frame before association (Prasad et al., 2017). In SAMFusion, camera pixels are lifted to 3D using

$th$0

then transformed into the LiDAR frame and projected to BEV for cross-modal attention and range-aware fusion (Palladin et al., 22 Aug 2025).

The literature also treats uncertainty in qualitatively different ways. In factor-graph systems such as UniMSF and the urban-delivery localization stack, uncertainty remains explicit through measurement covariances, gating, and residual models (Liu et al., 2024, Qingqing et al., 2021). In the Student’s $th$1 filter, heavy-tailed state and measurement models preserve robustness to outliers while avoiding explicit correlation modeling through arithmetic-average density fusion (Li et al., 2022). By contrast, the learned localization model of (Lin et al., 7 Mar 2025) explicitly replaces covariance tuning with learned higher-order neural features and recurrent residual gating, and ADM-Fusion similarly uses learned routing rather than calibrated uncertainty outputs (Moughnieh et al., 23 Jun 2026). One common misconception is that AMF necessarily yields explicit probabilistic confidence; the learned systems here often improve robustness precisely by abandoning hand-tuned Gaussian covariance models, but they may lose explicit uncertainty outputs as a consequence (Lin et al., 7 Mar 2025, Moughnieh et al., 23 Jun 2026).

Robustness frequently depends on calibration and synchronization even when the adaptive mechanism operates downstream. A3Fusion states that its alignment centers on feature compatibility and pipeline alignment rather than explicit camera–LiDAR calibration equations, yet also notes that RGB–Depth setups require reliable extrinsic and intrinsic calibration and time synchronization (Wang et al., 2022). The real-time automotive boresighting system addresses this directly by estimating roll, pitch, and yaw misalignment between a sensor and the vehicle frame using a Kalman filter driven by IMU and sensor-mounted accelerometer data, then correcting video via affine transformation in real time (0710.4833). The last-mile urban localization stack similarly assumes a pre-built map, known fixed extrinsics, and synchronized ROS time stamps, and introduces map-corruption detection and LOAM-based patch reconstruction when scan-to-map registration becomes unreliable (Qingqing et al., 2021). This suggests that AMF robustness is inseparable from the quality of the geometric and temporal interfaces among sensors.

5. System realization and real-time computation

Several works treat AMF as a system co-design problem rather than a purely algorithmic one. A3Fusion is explicitly described as a software–hardware system for adaptive, agile, and aligned fusion. Its FPGA accelerator splits a large PE array into two halves, one for RGB and one for depth, and connects them with a specialized fuseLink buffer so that fused partial sums can be exchanged without off-chip memory transfers. On a Xilinx XC7VX690T-2FFG1761C FPGA at 80 MHz, this architecture yields up to 17.97% segmentation latency reduction relative to a baseline accelerator, at the cost of LUT +20.84%, FF +27.43%, and storage +73.78% (Wang et al., 2022).

A much earlier automotive FPGA system already embodied a related principle. Using a Xilinx Virtex-II XC2V1000 FPGA and a custom 32-bit soft processor, it fused IMU and accelerometer measurements for boresight estimation while implementing a pipelined video affine transform in hardware. The affine rotation block had a 5-cycle latency and one-pixel-per-cycle initiation interval, and static tests estimated 1–2° misalignments with confidence values around 0.011–0.012° (0710.4833). Although the task was alignment rather than semantic perception, the system illustrates that adaptive fusion in safety-critical settings has long been tied to deterministic hardware execution.

End-to-end learned AMF systems also report real-time characteristics, but with different trade-offs. The recurrent residual-gating localization model runs with 20 predictions per 5 Hz update and has inference time 8.70 ms per update on GPU, compared with 4.30 ms for EKF on CPU and 89.77 ms for the windowed Agrobot baseline on GPU (Lin et al., 7 Mar 2025). ADM-Fusion, in its four-sensor configuration, contains approximately 34M parameters and runs at 60–70 FPS on an NVIDIA RTX 3050 Ti (Moughnieh et al., 23 Jun 2026). A2DO reports 40–50 FPS on an NVIDIA RTX 3060 Ti (Lai et al., 28 Feb 2025). These numbers support a more general observation already made in (Wang et al., 2022): adaptive fusion is often valuable only when its control logic, routing, and synchronization overheads are co-optimized with the execution platform.

6. Empirical patterns, trade-offs, and open issues

The reported results consistently argue against the idea that fusing every available measurement all the time is optimal. In aerial robots, consistency-driven selection is described as more robust and accurate than fusing all sensors all the time, especially during bridge inspection and indoor–outdoor transitions (Lanegger et al., 2023). In the Student’s $th$2 framework, augmented measurement fusion is optimal under linear Gaussian assumptions, but under outliers the arithmetic-average Student’s $th$3 fusion becomes superior; the paper reports that AA-StKF with suboptimal weights is best in position RMSE when $th$4 and best in velocity RMSE when $th$5 (Li et al., 2022). The conflict-measure work shows the same pattern at a simpler level: adaptive conflict-based weighting improves over simple averaging under impulse noise, DC offset, and Gaussian noise (Wei et al., 2018).

In autonomous driving perception, the trade-off is often between accuracy and latency or between robustness and specialization. A3Fusion reports that bidirectional fuseLinks with distance = 2 reach 37.66% segmentation accuracy, improving by 2.19% over FuseNet, while the accelerator reduces segmentation latency by up to 17.97% (Wang et al., 2022). SAMFusion reports large long-range pedestrian gains under adverse weather, including 34.31 AP in fog at 50–80 m, a +17.2 AP improvement over the next best method, and 41.45 AP in snow at 50–80 m, a +15.62 AP improvement (Palladin et al., 22 Aug 2025). A2DO-LVIO with pre-training reports mean KITTI drift of 1.36% translational error and 0.54° rotational error, outperforming both classical and learning-based baselines in the reported experiments (Lai et al., 28 Feb 2025). ADM-Fusion reports that adding radar to LVIO lowers translational RPE RMSE from 26.93 mm to 9.57 mm, a 64% reduction, while slightly increasing rotational RPE from 0.025° to 0.030° (Moughnieh et al., 23 Jun 2026). UniMSF reports that Radar-UniMSF improves RMSE by 19.37% relative to IPT and by 7.8% relative to Radar-VMSF (Liu et al., 2024).

The trade-offs, however, remain substantial. Learned systems that replace explicit uncertainty modeling may become less interpretable and more sensitive to training coverage, especially in long-tail conditions or domain shift (Lin et al., 7 Mar 2025, Moughnieh et al., 23 Jun 2026). Systems that rely on pre-built maps or precise calibration inherit failure modes from map corruption and extrinsic drift (Qingqing et al., 2021, 0710.4833). Resource overheads can be non-trivial in hardware-specialized pipelines (Wang et al., 2022). Threshold-based consensus methods are sensitive to parameter selection, particularly in two-sensor degeneracy or high-vibration regimes (Lanegger et al., 2023). Distributed adaptive fusion with globally coordinated control improves cardinality and OSPA performance, but the fully distributed coordinate-descent strategy incurs higher runtime than myopic control (Blair et al., 21 Apr 2026). The maritime architecture explicitly notes that implementation and fine-tuning require further research and development, and does not report quantitative experiments (Prasad et al., 2017).

A final misconception is that “more sensors” automatically implies “better fusion.” Several papers show that optimal performance depends on selective participation, not maximal participation. A3Fusion finds that more fuseLinks generally improve accuracy but also increase compute and synchronization cost (Wang et al., 2022). The aerial-robot framework excludes sensors with poor consistency (Lanegger et al., 2023). The distributed LMB controller selects only active sensors per label for complementary fusion (Blair et al., 21 Apr 2026). ADM-Fusion reports that adaptive routing offers limited benefits when sensor reliability is uniformly high, as in clean simulated LiDAR conditions (Moughnieh et al., 23 Jun 2026). AMF therefore denotes not sensor accumulation, but sensor discrimination.

Taken together, the literature portrays AMF as a unifying systems principle for heterogeneous perception and estimation: adapt the fusion graph, the measurement gain, the modality set, or the execution schedule according to sensed reliability, task structure, and resource limits. Whether instantiated through hypernetwork pruning, attention routing, Bayesian score fusion, consistency matrices, Student’s $th$6 posteriors, actor–critic weighting, or factor-graph covariance adaptation, the defining property is the same: fusion remains conditional rather than static.