Real-Time Sensor Fusion

• Real-time sensor fusion is the integration of multiple sensor signals in a streaming, low-latency fashion to provide robust and accurate estimations.
• It utilizes both model-based approaches, like Kalman filters, and data-driven methods, including CNNs and transformers, to fuse diverse sensor modalities.
• Applications range from robotics and autonomous vehicles to industrial monitoring, emphasizing real-time performance, calibration, and adaptive resource management.

Real-time sensor fusion is the computational integration of multiple sensor signals in a low-latency, streaming fashion to provide robust, timely, and accurate estimation or classification for autonomous agents, infrastructure, and industrial processes. The field spans state estimation, high-dimensional perception, anomaly detection, privacy-aware analytics, and multi-agent scenarios, with applications ranging from low-SWaP robotics to large-scale infrastructure systems. Key technical themes include model-based and data-driven fusion, temporal and spatial alignment, calibration, uncertainty propagation, dynamic reliability weighting, and strict resource constraints.

1. Foundational Principles and Fusion Architectures

Real-time sensor fusion methodologies can be grouped by abstraction level (signal/raw, feature/intermediate, or decision), computational paradigm (model-based, neural, probabilistic), and system requirements (feasibility under size/weight/power or privacy guarantees).

Signal and Feature-Level Fusion

• Model-based Kalman-type architectures: Linear or extended Kalman filters, often with intermittent or asynchronous updates, are standard for fusing IMU, GNSS, LiDAR, and camera/visual odometry streams for real-time state estimation and object tracking (Qingqing et al., 2021, Hadjiloizou et al., 2022, Saba et al., 29 Sep 2025, Hajri et al., 2018).
• CNN-based and transformer-based feature fusion: For perception, deep learning approaches extract feature maps from each sensor (e.g., camera, LiDAR, radar) and fuse them via concatenation, attention (transformers), or custom architectures (e.g., ResNet, LSTM, MobileNet) for object detection, segmentation, or odometry—often supporting uncalibrated, domain-adaptive, or low-payload deployments (Valente et al., 2019, Tan et al., 2024, Choi et al., 2023, Chen et al., 2023, Li et al., 29 Jan 2026).
• Specialized pipelines: Category-specific systems, such as glass detection using ToF and sonar, leverage bespoke gating, engineered morphological kernels, and real-time geometric reasoning running entirely on CPU (Hopkins et al., 7 Oct 2025).

High-Level and Late Fusion

• Track-level and decision-level fusion: High-level filtering and assignment strategies fuse object estimates or anomaly detections using global nearest neighbour methods, extended Kalman filters, or ensemble learners, handling asynchronous, missing, or degraded data streams (Hajri et al., 2018, Ahmad et al., 3 Aug 2025, Weng et al., 25 Feb 2026, Jahja et al., 2021).

Advanced Fusion Scenarios

• Privacy-aware fusion: Real-time optimization of fusion policy under Rényi DP constraints, with adaptive control of privacy budgets via constrained dynamic programming and RL (Weng et al., 25 Feb 2026).
• Adaptive and robust fusion: Switching state-space models with runtime sensor reliability inference, robust to failures or outliers, using particle filtering and RNN-learned dynamics (Turan et al., 2017).
• Continuous-time asynchronous fusion: Spline-based temporal registration for UWB/IMU fusion, enabling microsecond-scale alignment without time discretization (Li et al., 2023).

2. Mathematical Formulations and Algorithmic Strategies

Kalman-Filter Family

Let the state $\mathbf{x}_k$ evolve via

$$\mathbf{x}_k = \mathbf{F}\mathbf{x}_{k-1} + \mathbf{w}_{k-1}$$

and receive asynchronous measurement(s)

$$\mathbf{z}_k^{(i)} = \mathbf{H}^{(i)}\mathbf{x}_k + \mathbf{v}_k^{(i)}$$

where $\mathbf{w}$ and $\mathbf{v}^{(i)}$ are zero-mean process and measurement noise. The real-time update process executes the predict-update steps, gating each measurement by its timestamp and reliability, potentially adjusting measurement noise adaptively in response to degraded sensor confidence or data loss (Saba et al., 29 Sep 2025, Hajri et al., 2018, Qingqing et al., 2021, Hadjiloizou et al., 2022, Hopkins et al., 7 Oct 2025, 0710.4833):

• Prediction:

$$\hat{\mathbf{x}}_{k|k-1} = \mathbf{F} \hat{\mathbf{x}}_{k-1|k-1}; \quad \mathbf{P}_{k|k-1} = \mathbf{F} \mathbf{P}_{k-1|k-1} \mathbf{F}^\top + \mathbf{Q}$$

• Update (for each available sensor):

$$\mathbf{K}_k = \mathbf{P}_{k|k-1} \mathbf{H}^{(i)\top} \left(\mathbf{H}^{(i)} \mathbf{P}_{k|k-1} \mathbf{H}^{(i)\top} + \mathbf{R}^{(i)}\right)^{-1}$$

$$\hat{\mathbf{x}}_{k|k} = \hat{\mathbf{x}}_{k|k-1} + \mathbf{K}_k \left(\mathbf{z}_k^{(i)} - \mathbf{H}^{(i)} \hat{\mathbf{x}}_{k|k-1}\right)$$

$$\mathbf{P}_{k|k} = (\mathbf{I} - \mathbf{K}_k \mathbf{H}^{(i)}) \mathbf{P}_{k|k-1}$$

For feature-level (deep) approaches, sensor branches $f_{a}$, $$\mathbf{x}_k = \mathbf{F}\mathbf{x}_{k-1} + \mathbf{w}_{k-1}$$0, ... produce hidden embeddings, concatenated and further fused by shared heads (e.g., output head $$\mathbf{x}_k = \mathbf{F}\mathbf{x}_{k-1} + \mathbf{w}_{k-1}$$1; transformer block, attention module, or linear regressor). Late fusion in ensemble form is common: $$\mathbf{x}_k = \mathbf{F}\mathbf{x}_{k-1} + \mathbf{w}_{k-1}$$2 with $$\mathbf{x}_k = \mathbf{F}\mathbf{x}_{k-1} + \mathbf{w}_{k-1}$$3 the $$\mathbf{x}_k = \mathbf{F}\mathbf{x}_{k-1} + \mathbf{w}_{k-1}$$4th sensor prediction and $$\mathbf{x}_k = \mathbf{F}\mathbf{x}_{k-1} + \mathbf{w}_{k-1}$$5 learned or adaptively re-estimated (Jahja et al., 2021).

Specialized kernels, morphological filters, and gating functions are used in custom pipelines, e.g., for finding specular reflections in depth images (Hopkins et al., 7 Oct 2025).

3. Real-Time Implementation and System Constraints

Latency and Computational Efficiency

Pipelines are often structured to ensure that the worst-case fusion/update step completes within the shortest sampling interval among input sensors:

• Embedded and edge devices (ARM Cortex, Jetson, FPGA, commodity CPUs) execute all critical steps (prediction, fusion, update) fully on-device, without offloading to the cloud, maintaining end-to-end latencies $$\mathbf{x}_k = \mathbf{F}\mathbf{x}_{k-1} + \mathbf{w}_{k-1}$$620–200 ms depending on application (Hopkins et al., 7 Oct 2025, Li et al., 29 Jan 2026, Saba et al., 29 Sep 2025, Chen et al., 2023, 0710.4833).
• Memory and compute optimizations include event-driven asynchronous updates, minimizing state dimensions, fixed-point arithmetic, low-rank or diagonal covariance approximations, and pruned networks (Li et al., 29 Jan 2026).

Synchronization and Calibration

• Timestamp-based temporal alignment (nearest-neighbor matching, ring buffers) aligns asynchronous sensor streams (Hopkins et al., 7 Oct 2025, Saba et al., 29 Sep 2025).
• Intrinsic/extrinsic calibration includes a combination of factory, manual, target-free (SIFT+FPFH+RANSAC+G-ICP) registration, and continuous online drift correction (Kloeker et al., 2020, Ahmad et al., 3 Aug 2025, Li et al., 2023).
• Spatial alignment leverages homogeneous transforms and Procrustes/SVD routines to maintain coherence across local/global frames and ensure fusion feasibility even under moving or vibrating sensor platforms (Ahmad et al., 3 Aug 2025, Kloeker et al., 2020).

4. Applications and Empirical Performance

Robotics and Mobile Perception

• Aerial Robotics: The fusion of ToF and ultrasonic sensors with lightweight, kernel-based speckle detection achieves real-time (2–10 Hz), onboard detection and dense mapping of transparent obstacles with precision and recall surpassing deep RGB baselines under strict SWaP constraints (Hopkins et al., 7 Oct 2025).
• Autonomous Vehicles: Multi-sensor architectures (LiDAR, camera, radar, IMU, GNSS) yield robust localization in urban environments via UKF/NDT coupling and dynamic GNSS/LiDAR odometry fallback; position errors reach sub-0.1 m, yaw errors $$\mathbf{x}_k = \mathbf{F}\mathbf{x}_{k-1} + \mathbf{w}_{k-1}$$7 at full sensor rate (Qingqing et al., 2021).
• Semantic Scene Understanding: Real-time point cloud segmentation (LaCRange) fuses LiDAR/camera signals in a range-view format with distortion-compensating knowledge distillation and context fusion; $$\mathbf{x}_k = \mathbf{F}\mathbf{x}_{k-1} + \mathbf{w}_{k-1}$$8 mIoU on SemanticKITTI at 20 FPS, surpassing prior art by optimizing both fusion accuracy and system efficiency (Tan et al., 2024).
• SLAM and Mapping: Fusion of IMU, wheel encoder, and absolute positioning (ultrasonic IPS/UWB/Vicon) via EKF, with synchronized SLAM update rates, reduces absolute trajectory error by $$\mathbf{x}_k = \mathbf{F}\mathbf{x}_{k-1} + \mathbf{w}_{k-1}$$9–$$\mathbf{z}_k^{(i)} = \mathbf{H}^{(i)}\mathbf{x}_k + \mathbf{v}_k^{(i)}$$0 and more than doubles point-cloud density (Phan et al., 2023, Li et al., 2023).

Industrial and Infrastructure Sensing

• Manufacturing Monitoring: Mid-level fusion of acoustic and visual data via a hybrid CNN delivers defect detection accuracy $$\mathbf{z}_k^{(i)} = \mathbf{H}^{(i)}\mathbf{x}_k + \mathbf{v}_k^{(i)}$$1 at 10 Hz and $$\mathbf{z}_k^{(i)} = \mathbf{H}^{(i)}\mathbf{x}_k + \mathbf{v}_k^{(i)}$$2 ms latency, enabling closed-loop process control in L-DED (Chen et al., 2023).
• Work Zone Vehicle Tracking: Kalman-filter-based late fusion of camera, LiDAR, and radar achieves $$\mathbf{z}_k^{(i)} = \mathbf{H}^{(i)}\mathbf{x}_k + \mathbf{v}_k^{(i)}$$3 m lateral, $$\mathbf{z}_k^{(i)} = \mathbf{H}^{(i)}\mathbf{x}_k + \mathbf{v}_k^{(i)}$$4 m longitudinal RMSE against RTK-GPS, with $$\mathbf{z}_k^{(i)} = \mathbf{H}^{(i)}\mathbf{x}_k + \mathbf{v}_k^{(i)}$$520 Hz throughput on edge hardware in adverse, occluded, or degraded-sensor cases (Saba et al., 29 Sep 2025, Ahmad et al., 3 Aug 2025).

Edge and Embedded Systems

• Human Activity Recognition (HAR): Hierarchical feature fusion (FFT, wavelet, Gabor) with resource-aware branching, attention, and depthwise separable convolution yields $$\mathbf{z}_k^{(i)} = \mathbf{H}^{(i)}\mathbf{x}_k + \mathbf{v}_k^{(i)}$$6 accuracy ($$\mathbf{z}_k^{(i)} = \mathbf{H}^{(i)}\mathbf{x}_k + \mathbf{v}_k^{(i)}$$7 KiB RAM, $$\mathbf{z}_k^{(i)} = \mathbf{H}^{(i)}\mathbf{x}_k + \mathbf{v}_k^{(i)}$$8 M MACC) for microcontroller-class devices, with runtime interpretability (Li et al., 29 Jan 2026).

5. Robustness, Adaptivity, and Privacy Constraints

Handling Asynchrony, Intermittency, and Degradation

• Real-time fusion systems handle partial and missing updates via dynamic masking (block-diagonal matrices), reliability estimation, data-driven weight adaptation (sliding window, ridge/lasso, Kalman-gain), and on-the-fly inflation of noise covariances to discount unreliable streams (Hadjiloizou et al., 2022, Jahja et al., 2021, Saba et al., 29 Sep 2025, Nemec et al., 2017).
• Robust assignment/filtering (e.g., nearest-neighbor gating, Mahalanobis distance, global assignment algorithms) ensures low incidence of false associations (Hajri et al., 2018, Ahmad et al., 3 Aug 2025).

Online Self-Calibration, Learning, and Fault Adaptation

• Fully online calibration is achieved via residual-driven stochastic gradient updates (Adam), with parameters for bias, scale, alignment, and temperature drift, directly in the sensor's operational environment—without static-motion gating (Nemec et al., 2017).
• Learning-based fusion integrates explicit fault-detection (switching state models) and RNN/LSTM-learned dynamical priors; real-time particle filtering with per-step sensor reliability tracking increases robustness against sensor failures and outliers, achieving $$\mathbf{z}_k^{(i)} = \mathbf{H}^{(i)}\mathbf{x}_k + \mathbf{v}_k^{(i)}$$9 mm and $\mathbf{w}$0 pose error in medical robotics (Turan et al., 2017).
• Distributed or hierarchical architectures adapt the fusion policy to local data quality or privacy budgets (e.g., adaptive privacy allocation under Rényi DP) (Weng et al., 25 Feb 2026).

Privacy Guarantees and Differential Privacy

• Real-time fusion under Rényi Differential Privacy constraints employs constrained optimality via closed-form Bellman equations and structured Gaussian mechanisms, leveraging RL-based optimization (PPO) for tractable, adaptive closed-loop control of fusion signal leakage, without Monte Carlo overhead (Weng et al., 25 Feb 2026).

6. Quantitative Benchmarks and Trade-offs

Accuracy, Throughput, and Computational Load

Task / System	Accuracy/Metric	Throughput / Latency	Platform
Transparent glass mapping (Hopkins et al., 7 Oct 2025)	mIoU 48–82%, Prec. 82–96%	2–10 Hz, 100 ms/frame	ARM CPU, sub-300g drone
Semantic segmentation (Tan et al., 2024)	mIoU 64.1% (SemanticKITTI)	20 FPS, 49.5 ms/frame	A100 GPU
Work zone trajectory fusion (Saba et al., 29 Sep 2025)	RMSE 0.8 m lat., 1.2 m long.	20 Hz, <20 ms/fusion	Edge CPU (Jetson/i7)
HAR on MCU (Li et al., 29 Jan 2026)	96.7% accuracy	1K+ inferences/s	STM32 M4, 22 KiB RAM
Lidar/radar tracking (Hajri et al., 2018)	MSE$\mathbf{w}$1 0.04–0.06 m$\mathbf{w}$2, v$\mathbf{w}$3 0.02–0.05	25 Hz, $\mathbf{w}$415 μs/track	Automotive-grade CPU
Medical fusion (Turan et al., 2017)	$\mathbf{w}$53 mm, $\mathbf{w}$61.5$\mathbf{w}$7 RMSE	25–30 Hz	GPU; robust to failures

Beyond these, adaptive or learning-based fusers demonstrate rapid convergence (tens of ms), immunity to rapid maneuvers or drift, and sustained performance even during temporally correlated sensor dropouts. The computational footprint is bounded by design, with resource-aware model branching, selective module activation, and conditional updates (Li et al., 29 Jan 2026, Nemec et al., 2017, Hadjiloizou et al., 2022).

7. Open Directions and Limitations

Several challenges remain:

• Fusing highly sparse or noisy modalities: Gaps in point clouds or RGB coverage can reduce segmentation or mapping quality; research on inpainting, context propagation, or better attention for missing data is ongoing (Tan et al., 2024).
• Extending real-time fusion to privacy-constrained, multi-agent, or distributed settings: Adaptive, RL-guided, privacy-preserving real-time fusion is nascent (Weng et al., 25 Feb 2026).
• Operating in adverse or variable environmental conditions: Night, weather, occlusion, and calibration drift require dynamic adaptation and model tuning (Ahmad et al., 3 Aug 2025).
• Temporal consistency and multi-sweep fusion: Most real-time pipelines operate per-scan/frame; consistency across longer temporal horizons (e.g., for moving objects) remains an open avenue (Tan et al., 2024).
• Generalization and modularity: System architectures increasingly leverage modular, sensor-agnostic designs for future extensibility (e.g., plug-in encoders for new modalities; CFF and transformers), with a focus on maintaining real-time constraints without bespoke hardware or cloud dependencies (Choi et al., 2023).

In summary, real-time sensor fusion, across diverse system classes, has matured to deliver robust, accurate, and low-latency estimation and perception under physical, computational, and privacy constraints, with extensive empirical validation across mobile robotics, automotive, industrial, and edge domains (Hopkins et al., 7 Oct 2025, Tan et al., 2024, Saba et al., 29 Sep 2025, Hajri et al., 2018, Li et al., 29 Jan 2026, Chen et al., 2023, Turan et al., 2017, Jahja et al., 2021, Weng et al., 25 Feb 2026). Continued advances are expected in resource-aware design, sensor fault detection, representation learning under uncertainty, and robust deployment in complex real-world environments.