State estimations and noise identifications with intermittent corrupted observations via Bayesian variational inference

Abstract: This paper focuses on the state estimation problem in distributed sensor networks, where intermittent packet dropouts, corrupted observations, and unknown noise covariances coexist. To tackle this challenge, we formulate the joint estimation of system states, noise parameters, and network reliability as a Bayesian variational inference problem, and propose a novel variational Bayesian adaptive Kalman filter (VB-AKF) to approximate the joint posterior probability densities of the latent parameters. Unlike existing AKF that separately handle missing data and measurement outliers, the proposed VB-AKF adopts a dual-mask generative model with two independent Bernoulli random variables, explicitly characterizing both observable communication losses and latent data authenticity. Additionally, the VB-AKF integrates multiple concurrent multiple observations into the adaptive filtering framework, which significantly enhances statistical identifiability. Comprehensive numerical experiments verify the effectiveness and asymptotic optimality of the proposed method, showing that both parameter identification and state estimation asymptotically converge to the theoretical optimal lower bound with the increase in the number of sensors.

• The paper introduces a dual-mask model in a VB-AKF framework that jointly estimates states and unknown noise parameters under corrupted and intermittent observations.
• It utilizes structured mean-field variational inference with closed-form updates to adaptively estimate noise covariances, packet dropout, and corruption probabilities.
• Experimental results demonstrate asymptotic optimality and dynamic resilience under severe dual degradation, setting clear theoretical limits on statistical identifiability.

Variational Bayesian Adaptive Kalman Filtering for Joint State and Noise Estimation under Intermittent and Corrupted Observations

Problem Formulation and Dual-Mask Modeling

The paper addresses robust state estimation and noise identification in distributed sensor networks with both intermittent packet dropouts and corrupted observations. The generative model is constructed for a linear dynamical system, where sensors are affected by both observable communication failures and latent measurement corruption. The novelty resides in modeling these uncertainties with two independent Bernoulli random variables: $\gamma_{i,k}$ for packet loss (observable) and $z_{i,k}$ for measurement authenticity (latent). Both rates ($\rho_k$ for survival and $\beta_k$ for cleanliness) are unknown and inferred online. Observation and process noise covariances ($R_k$, $Q_k$) are unknown and jointly estimated. The framework assumes mixtures of clean and corrupted observations, with extra noise variance $E_k$ for corrupted data.

Variational Inference Architecture and Sequential Fusion

The paper advances a variational Bayesian adaptive Kalman filter (VB-AKF) tailored for this dual-failure scenario, employing structured mean-field variational inference and conjugate prior assignments (inverse-Wishart for covariances, Beta for rates). Each time step involves:

• Bayesian mixture inference for $z_{i,k}$ via responsibilities $\pi_{i,k}$ using softmax over the log-posterior of clean/corrupted likelihoods.
• Closed-form updates for $\rho_k$ and $z_{i,k}$0 exploiting the independence afforded by the Bernoulli modeling and Beta conjugacy.
• Adaptive estimation of noise covariances: $z_{i,k}$1 from observations modulated by the inferred responsibility, and $z_{i,k}$2 from state transitions.
• Sequential centralized fusion over sensors with a gated Kalman gain, where the gain is modulated by both packet dropout and corruption probabilities. This structure is resilient to missing or unreliable data and integrates information from multiple sensors efficiently.
• Iterative VI at each time step to mitigate prior bias, leading to rapid convergence.

Experimental Evaluation: Asymptotic Optimality and Robustness

The VB-AKF is numerically validated through four classes of experiments, emphasizing statistical optimality, robustness under abrupt nonstationary noise, resilience to severe network/data degradation, and identifiability boundaries under varying noise intensities.

Asymptotic Optimality

Joint state estimation and noise identification converge to the theoretical minimum error as the number of sensors $z_{i,k}$3 increases, demonstrating the Bayesian property that the fusion of increasing evidence mitigates prior uncertainty (Figure 1).

Figure 2: State estimation and noise variance inferences under non-stationary perturbations with $z_{i,k}$4 sensor nodes. Top: State trajectory tracking; Middle: Inference of observation noise variance $z_{i,k}$5; Bottom: Inference of process noise variance $z_{i,k}$6.

Dynamic Resilience under Abrupt Noise Shifts

VB-AKF adapts rapidly to sudden changes in noise covariances, exhibiting zero-delay state tracking and resilient variance inference. The system captures abrupt process and observation noise transitions, outperforming standard KF constrained by static assumptions. Asymmetric coupling is revealed: $z_{i,k}$7 variance changes affect $z_{i,k}$8 inference more than the converse, and large observation noise compromises state estimation more than process noise.

Extreme Dual Degradation

Under 60% packet dropouts and 60% data corruption, the method maintains robust parameter identification and state estimation, with the sequential fusion and gated gain architecture suppressing outlier-induced variance inflation. Asymmetric robustness is observed: process noise identification remains stable, whereas observation noise and corruption rate are more sensitive to outliers (Figure 3, Figure 4).

Figure 4: Performance of the proposed VB-AKF under severe data degradation (60% packet dropout and 60% data corruption). Top: Dropout rate inference; Second: Data clean rate identification; Third: Joint inference of process noise $z_{i,k}$9 and observation noise $\rho_k$0; Bottom: State estimation via centralized sequential fusion.

Identifiability and Sensitivity Analysis

Ablation studies delineate the operational envelope and theoretical limits of the inference architecture:

• Identifiability degrades when baseline observation noise $\rho_k$1 approaches anomaly intensity $\rho_k$2, masking outliers in noise and amplifying false positives at low $\rho_k$3.
• Error decays exponentially with anomaly intensity beyond a threshold, reflecting the statistical separation of clean and corrupted components.
• Intense process noise $\rho_k$4 confounds corruption inference, as predictive uncertainty becomes statistically indistinguishable from corruption.
• Optimal performance is achieved when signal-to-anomaly contrast is maximized, with robust inference boundaries tightly defined.

Figure 5: Sensitivity analysis and statistical identifiability of corruption rate inference. (a) Identifiability degradation under strong baseline observation noise. (b) Exponential error convergence beyond the anomaly intensity threshold. (c) Identifiability degradation under intense process noise variations.

Theoretical and Practical Implications

The dual-mask VB-AKF enables unified joint inference of states and noise parameters under extreme distributed sensor fusion scenarios. The proposed mean-field VI with sequential gated fusion achieves asymptotic optimality with increasing samples and robust adaptation to severe data anomalies. The architecture exposes theoretical boundaries in statistical identifiability, which are functions of the interplay between process noise, measurement noise, and anomaly intensity.

Practically, this advances robust state estimation in large-scale networked sensor environments—applicable to distributed tracking, sensor reliability, and adaptive filtering in unreliable communication and harsh measurement conditions. The model's fundamental resilience is relevant for autonomous systems, industrial monitoring, and control networks with catastrophic data loss or corruption.

Theoretically, the coupling between $\rho_k$5, $\rho_k$6, and $\rho_k$7 highlights intrinsic limitations on what can be inferred from corrupted observations, paralleling constraints in classical observability and controllability. Future work could refine missing data models, analyze deeper couplings, and extend to nonlinear or non-Gaussian domains.

Conclusion

This work establishes a variational Bayesian adaptive Kalman filter with dual-mask modeling for robust joint state and noise estimation under simultaneous packet dropout and corrupted observations in distributed sensor networks. The proposed approach achieves asymptotic optimality, high robustness, and explicit identifiability limits, substantiated by comprehensive numerical experiments and ablation studies. The framework sets new standards for adaptive inference under complex data degradation regimes, with practical and theoretical ramifications for high-dimensional filtering, sensor fusion, and robust control.