--- title: Uncertainty-Based Filtering url: https://www.emergentmind.com/topics/uncertainty-based-filtering type: topic --- # Uncertainty-Based Filtering Searching arXiv for recent papers on uncertainty-based filtering and closely related filtering frameworks. Uncertainty-based filtering denotes a family of methods in which an explicitly computed uncertainty quantity is used to govern inference, selection, or control. In the cited literature, the operative uncertainty may be a posterior covariance, a KL- or Wasserstein-based variational objective, a convex-penalty or sublinear expectation over model classes, an entropy or disagreement score for pseudo-labels, a factor of safety for manipulation, or a kernel-derived uncertainty map for predictive image filtering. The common structure is that uncertainty is propagated or estimated and then used to determine how strongly measurements are trusted, which candidate states or samples are retained, and how overconfidence is avoided in downstream decisions [1704.00102]. ## 1. Scope and meanings of the term Across recent work, “uncertainty-based filtering” is not a single formalism but a recurring design pattern. In state estimation, it appears as Bayesian or robust filtering in which uncertainty is part of the recursion itself: examples include Kalman–Bucy variants under drift uncertainty, outer-measure filtering, penalty-based hidden Markov model filtering, Koopman-based filtering, and adaptive Kalman or UKF updates [2004.09202]. In machine learning pipelines, the term is used for selecting or rejecting pseudo-labels, reasoning traces, or recommendations on the basis of uncertainty estimates such as entropy, perplexity, self-consistency, or model disagreement [2403.11256]. In control and perception, it appears in safety filtering for manipulation and in uncertainty-aware predictive filtering for deraining, where uncertainty modulates whether a candidate action is rejected or whether a pixel is refined [2509.12674]. | Setting | Uncertainty quantity | Filtering action | |---|---|---| | State estimation | posterior covariance, penalty, risk measure, outer measure | update, propagate, or robustify the state estimate | | Data selection | entropy, disagreement, perplexity, predictive multiplicity | retain or reject samples or recommendations | | Safety and perception | factor of safety, uncertainty map | reject unsafe actions or trigger refinement | This diversity is substantive rather than merely terminological. A plausible implication is that the phrase names a role played by uncertainty, not a unique algorithmic family. Some papers use “filtering” in the strict signal-processing sense of recursively estimating hidden states; others use it in the data-curation sense of discarding uncertain items before training or deployment. ## 2. Variational and robust foundations One important line of work formulates filtering and uncertainty propagation as optimization in spaces of distributions. The Jordan–Kinderlehrer–Otto scheme writes uncertainty propagation for the Fokker–Planck equation as the proximal recursion $$ \rho_k = \arg\min_{\rho\in\mathcal{P}_2(\mathbb{R}^n)} \left\{ \tfrac12 W_2^2(\rho,\rho_{k-1}) + h F(\rho) \right\}, $$ while the Laugesen–Mehta–Meyn–Raginsky measurement update is a KL-proximal step, $$ \rho_k^+ = \arg\min_{\rho\in\mathcal{D}_2} \left\{ D_{KL}(\rho\|\rho_k^-) + h\,\Phi(\rho) \right\}. $$ In the linear Gaussian case, the small-step limits recover the Fokker–Planck propagation and the Kalman–Bucy filter, so propagation and measurement update become alternating proximal operators in Wasserstein and KL geometries [1704.00102]. A second foundation replaces a single prior by a family of priors indexed by uncertainty in model parameters or drifts. In the generalized Kalman–Bucy setting with drift uncertainty, the estimation problem is posed under a convex operator, with a dual representation over equivalent measures \(P^\theta\); under \(g\)-expectation there exists a worst-case prior \(P^*\), and the optimal estimator becomes the classical conditional expectation under that worst-case prior [2004.09202]. Closely related work on uncertain observation models expresses the robust problem as a minimum mean square estimator under a sublinear operator, \[ \inf_{\hat h_t}\sup_{\theta} E^{P^\theta}[\|x_t-\hat h_t\|^2], \] and again obtains a worst-case measure \(P^{\theta^*}\) under which the robust Kalman–Bucy recursion is run [1907.01550]. For hidden Markov models, uncertainty can be propagated forward as a penalty rather than as a single posterior law. In discrete time, a nonlinear expectation is defined by \[ \mathcal{E}_t(\xi)=\sup_{Q\in\mathcal{Q}}\left\{\mathbb{E}_Q[\xi\mid Y_t]-\left(\tfrac1k\alpha_t(Q)\right)^{k'}\right\}, \] and the object that is recursively propagated is the penalty \(\kappa_t(p)\) over possible filter states \(p\), not a single Bayes posterior [1606.00229]. In continuous-time finite-state HMMs with unknown time-dependent parameters and no prior or stochastic model for the parameter path, the same robust perspective leads to a pathwise rough differential equation for the filter and a value function \(\kappa(t,x,a)\) solving a rough Hamilton–Jacobi–Bellman equation; the filter simultaneously provides robust state estimates and parameter learning by penalizing “unreasonable” parameter paths [2005.04982]. A more radical generalization represents uncertainty by outer measures. In that framework, probability density functions are replaced by positive functions with supremum equal to one, and integrals in filtering and smoothing equations are replaced by supremums. Under linear-Gaussian assumptions, the Kalman filter recursion is recovered from weaker assumptions on the available information than in the classical probabilistic setting [1704.01233]. This suggests that uncertainty-based filtering can be formulated even when additive probabilistic structure is intentionally weakened. ## 3. Adaptive uncertainty in dynamical-state estimation A central contemporary use of uncertainty-based filtering is adaptive measurement weighting. In real-time betatron tune measurement from Schottky spectra, the latent tune is modeled as a scalar random walk, \[ x_k = F x_{k-1} + w_k,\qquad F=1,\qquad w_k\sim\mathcal{N}(0,Q), \] with direct observations \(z_k=H x_k+v_k\), \(H=1\), and measurement noise variance chosen from the neural network’s predicted uncertainty, \[ R_k=\max\!\bigl(\sigma_{\mathrm{nn},k}^2,R_{\min}\bigr),\qquad R_{\min}=10^{-12}. \] The CNN is trained with a Laplace negative log-likelihood, \[ \mathcal{L}_{\mathrm{NLL}}=\frac1N\sum_{i=1}^N\left(\frac{|e_i|}{\sigma_i}+\ln\sigma_i\right), \] so that \(\sigma\) tracks instantaneous error. On a static test set of 50 000 simulated spectra, the Spearman rank correlation between \(|e|\) and \(\sigma\) is \(\rho=0.501\) globally and \(\rho=0.534\) in the \(-20\) to \(-15\) dB regime. In temporal tracking at \(-20\) dB, MAE decreases from \(22.24\times10^{-4}\) for the pure CNN and \(20.66\times10^{-4}\) for a fixed-\(R\) Kalman filter to \(6.40\times10^{-4}\) for the adaptive-\(R\) filter; the 95th percentile error drops to \(15.13\times10^{-4}\) [2512.09302]. Here uncertainty is not an output to be reported after the fact; it directly controls the gain. An analogous principle appears in UKF-based long-horizon forecasting. In PDYffusion, the forecaster is written as \[ x_{t+1}=f(x_t)+w_t,\qquad y_t=h(x_t)+v_t, \] and sigma-point propagation yields a predictive covariance that is trained through the UKF negative log-likelihood \[ L_{\mathrm{UKF}}=\tfrac12\left(d\log 2\pi+\log|P_{t+h}|+(x_{t+h}-\hat x_{t+h})^\top P_{t+h}^{-1}(x_{t+h}-\hat x_{t+h})\right). \] The empirical evaluation uses CRPS, MSE, and the spread–skill ratio SSR, with SSR near 1 indicating that predicted spread matches realized error. Section 5.4 explicitly reports a trade-off in which adding noise raises MSE but reduces \(|1-\mathrm{SSR}|\), and the method is described as sitting near the Pareto-front of that trade-off [2604.09058]. In data-driven Kalman filtering, KalmanNet learns the gain rather than the covariances, yet its internal gain can be converted into an implied posterior covariance when \(H\) has full column rank. With \(R=I_n\), \[ P_k=(I_m-K_k H)\,H^\top\,[I_n-HK_k]^{-1}K_k. \] Under model mismatch, the extracted uncertainty more closely matches empirical error than the uncertainty produced by a mismatched classical Kalman filter [2110.04738]. This is a different form of uncertainty-based filtering: the update remains Kalman-like, but the uncertainty that drives the update is inferred from learned internal features. Koopman-based filtering in astrodynamics moves the same idea into an analytic polynomial setting. The Koopman Operator Filter expands both state and measurement observables in a basis of orthogonal polynomials, propagates moments of arbitrary order in closed form, and performs a Kalman-style measurement update with analytically computed \(P_{xy}\) and \(P_{yy}\). In the reported comparisons, EKF and IKF are “grossly over-confident,” UKF is also inconsistent, while the Koopman Operator Filter is “uniquely unbiased and self-consistent” in the sense that \(\sigma_{\mathrm{pred}}\approx\sigma_{\mathrm{eff}}\) [2407.20052]. Adaptive measurement trust also appears in flight-management uncertainty propagation. A six-dimensional Kalman filter uses a sigmoid-blended measurement covariance, \[ R_k = R_{\triangledown} + (R_{\triangle} - R_{\triangledown})\,\sigma(\Delta_{wp}), \] so that trust in flight-plan pseudo-measurements increases as progress toward a waypoint increases. On held-out ADS-B verification data, the method achieves 76% arrival-time prediction accuracy [2602.14948]. The same control-theoretic idea reappears in probabilistic numerical analysis: filtering-based probabilistic numerical ODE solvers quantify numerical uncertainty, but they “do not automatically solve this uncertainty propagation problem” when model parameters are uncertain. The proposed remedy is to combine ODE filters with numerical quadrature so that both parameter uncertainty and numerical solver uncertainty are marginalized; notably, the abstract reports that solver uncertainty can help prevent overconfidence, especially when using larger step sizes [2503.04684]. ## 4. Uncertainty-driven sample, pseudo-label, and recommendation selection In machine learning, uncertainty-based filtering often refers to selection rules that reject unreliable examples before self-training or supervised fine-tuning. In source-free unsupervised domain adaptation, Uncertainty-aware Pseudo-label-filtering Adaptation introduces Adaptive Pseudo-label Selection. For each target sample, predictions of its \(K\)-nearest neighbors are averaged, \[ \bar p_i=\frac1{|\mathcal{N}_i|}\sum_{j\in\mathcal{N}_i}p_j, \] and uncertainty is the entropy \[ u_i=H(\bar p_i)=-\sum_{c=1}^C [\bar p_i]_c \log [\bar p_i]_c. \] Only samples with \(u_i\le \theta_t\) are kept, often by percentile-based filtering per class. On VisDA-C, “No filtering” gives 82.0%, “Heuristic top 80% by Prob” gives 84.1%, and “APS (neighbor-entropy) top 80%” gives 86.3% [2403.11256]. A common misconception is that pointwise softmax confidence is sufficient; this comparison shows that local neighbor aggregation can be materially better. Speech pseudo-label filtering in DUST uses a different uncertainty signal: disagreement among Monte Carlo dropout decodes. For an unlabeled utterance \(x\), \[ U(x)=\max_{t=1,\dots,T}\frac{ED(\hat y^{ref},\hat y^t)}{|\hat y^{ref}|}, \] and pseudo-labels are accepted only when \(U(x)\le \tau\) [2211.07795]. The paper further evaluates calibration using ECE, RCE, and MCE, and shows that DUST filtering may fail under severe source and target domain mismatch because the teacher can be over-confident on wrong transcriptions. Increasing the number of stochastic passes, moving to character-level disagreement, using a more robust teacher, and increasing source-data diversity all improve the quality of the retained pseudo-labels [2211.07795]. A label-free variant appears in synthetic dataset creation for biological reasoning. There, uncertainty is estimated from self-consistency, semantic consistency, and predictive perplexity. If \(k\) sampled traces and one greedy trace are generated, the empirical answer distribution \(\hat p_j\) yields entropy-based uncertainty, \[ U_{\mathrm{ent}}(x)=-\sum_j \hat p_j \log \hat p_j, \] semantic disagreement is \[ U_{\mathrm{cons}}(x)=\frac{2}{k}\sum_{i=1}^k (1-\mathrm{sim}(r_0,r_i)), \] and greedy-trace perplexity is \[ U_{\mathrm{ppl}}(x)=\exp\!\Bigl(-\tfrac{1}{N}\sum_{n=1}^N \log P(t_n\mid t_{\epsilon)\) under the original uncertainty and is therefore rejected; after safe probing and posterior updating, both critical events satisfy \(S_c<\epsilon\) and the action is accepted [2509.12674]. The filtering step is thus explicitly a reject/accept decision driven by propagated parametric uncertainty. In image restoration, uncertainty-aware cascaded predictive filtering uses an uncertainty map computed directly from predicted kernels. For a rainy image \(I\), the first-stage spatially variant predictive filter estimates per-pixel kernels \(K\) and outputs \[ \hat I(x,y)=\sum_{(u,v)\in\Omega} K_{x,y}(u,v)\,I(x+u,y+v). \] The uncertainty map is then \[ U(x,y)=\frac{1}{(2r+1)^2}\sum_{(u,v)\in\Omega} K_{x,y}(u,v), \] and a second predictive network \(\phi_2\) receives \((\hat I_1,U)\) to produce a refinement stage [2201.02366]. On Rain100H, single SPFilt gives 31.39 dB / 0.9199, adding UC-PFilt yields 32.51 dB / 0.9303, and the full EfDeRain+ reaches 34.57 dB / 0.9513 [2201.02366]. Here uncertainty is not epistemic in the Bayesian sense; it is a learned difficulty signal extracted from the filtering kernels themselves. These examples show that uncertainty-based filtering can be conservative or selective without being probabilistic in a narrow sense. In manipulation, it is a safety margin aggregated over uncertain world parameters. In deraining, it is a per-pixel difficulty map guiding cascaded refinement. The mathematical forms differ, but in both cases the filter is conditioned on where the model is least trustworthy. ## 6. Calibration, overconfidence, and recurrent limitations A recurrent theme is that probabilistic structure alone does not guarantee correct uncertainty propagation. Filtering-based probabilistic numerical ODE solvers are explicitly described as efficient for quantifying numerical uncertainty, yet they “do not automatically solve this uncertainty propagation problem” when the dynamical system contains uncertain parameters [2503.04684]. DUST shows an analogous failure mode in self-training: disagreement-based filtering can break under severe domain mismatch because the teacher is miscalibrated [2211.07795]. The label-free reasoning work similarly reports that hybrid uncertainty metrics are more predictive than either consistency or perplexity alone, and that per-class filtering is necessary because absolute uncertainty scales differ across classes [2510.05871]. Calibration is therefore not ancillary. In the betatron-tune tracker, the network’s \(\sigma\) estimates are useful precisely because they are “strongly correlated with the actual instantaneous errors,” enabling adaptive \(R_k\) selection [2512.09302]. In PDYffusion, SSR is used to check whether forecast spread matches realized error, with SSR \(<1\) indicating under-dispersion and SSR \(>1\) indicating over-dispersion [2604.09058]. In Koopman filtering, unbiasedness and self-consistency are evaluated by comparing predicted and empirical uncertainty envelopes [2407.20052]. Across these examples, uncertainty-based filtering is effective only when the uncertainty proxy is itself reliable. Another recurring issue concerns what is being filtered. In robust HMM and Kalman–Bucy formulations, the filter acts on distributions, penalties, or worst-case priors [1606.00229]. In pseudo-label and recommendation settings, it acts on discrete candidate items [2008.10117]. In safety filtering, it acts on candidate control episodes [2509.12674]. This suggests that “filtering” should be read at the level of the decision variable: state, sample, action, or pixel. The literature also repeatedly identifies overconfidence as the principal pathology. Numerical uncertainty can “help prevent overconfidence” in propagated uncertainty estimates for ODE solvers [2503.04684]; adaptive-\(R\) Kalman filtering suppresses transient outliers by lowering the Kalman gain when \(\sigma_k\) is high [2512.09302]; and rough-path or convex-risk frameworks enlarge the admissible model class precisely so that filtering does not collapse onto a falsely precise single model [2005.04982]. A plausible synthesis is that uncertainty-based filtering is best understood as an anti-overconfidence mechanism: it uses uncertainty estimates not merely to summarize ignorance, but to alter the filter’s operational behavior in a way that preserves robustness under model mismatch, data noise, or environmental ambiguity. Source: https://www.emergentmind.com/topics/uncertainty-based-filtering