Kalman World Models: Recursive Bayesian Filtering

• Kalman World Models are state-space architectures that combine latent dynamics with recursive Bayesian filtering to propagate uncertainty and update beliefs.
• They leverage Kalman gains and Riccati geometry to replace conventional gradient descent with innovation-driven, online learning and adaptation.
• Applied in sequence tasks and autonomous driving, KWM architectures demonstrate robust performance improvements under distribution shifts.

Kalman World Models (KWM) denote a family of state-space approaches that combine latent dynamics, observation models, recursive inference, and uncertainty propagation in a Kalman-style framework. In a narrow and explicit sense, KWM refers to learned state-space models trained by recursive Bayesian filtering rather than reverse-mode automatic differentiation, with error signals interpreted as innovations and parameter updates implemented through Kalman-style gains (Kiruluta, 13 Mar 2026). In a broader research sense, the term covers latent world-model architectures that preserve key Kalman ingredients—latent belief state, prediction-correction recursion, covariance or covariance-like adaptation, and sometimes action-conditioned rollouts—while using neural parameterizations for transitions, emissions, or gain computation (Krishnan et al., 2015). The surrounding literature also includes Kalman-augmented models that improve latent dynamics or policy learning without implementing a full latent Bayesian filter, and the distinction between those categories is central to the topic (Manjunatha et al., 2023).

1. Conceptual scope and lineage

The modern KWM literature emerges from several partially overlapping lines of work. A first line is the deep latent state-space model tradition, exemplified by Deep Kalman Filters, which introduced a stochastic latent world model with latent state $z_t$, observation $x_t$, action $u_t$, action-conditioned transitions, probabilistic emissions, and amortized variational inference (Krishnan et al., 2015). In that formulation, the essential ingredients of a world model are already present: latent Markov dynamics, uncertainty-aware transitions, and rollout under interventions.

A second line is differentiable Kalman-style filtering in learned feature spaces. Recurrent Kalman Networks maintain an explicit latent Gaussian belief, propagate uncertainty through locally linear latent dynamics, and perform Kalman-style updates in a high-dimensional factorized latent representation learned from complex observations such as images (Becker et al., 2019). KalmanNet, by contrast, assumes known or approximate transition and observation maps and learns only the Kalman gain with a recurrent network, thereby treating belief update rather than full dynamics learning as the central learned component (Revach et al., 2021).

A third line concerns architectures that are Kalman-inspired or Kalman-augmented without constituting a full KWM in the strict sense. KARNet is best characterized as a Kalman-augmented latent dynamics model for autonomous driving: it injects an external error-state extended Kalman filter estimate of ego-motion into a recurrent latent video-prediction model, but the Kalman filter does not infer the neural latent state itself (Manjunatha et al., 2023). KOSS similarly provides a Kalman-inspired closed-loop selective state-space backbone for long-sequence modeling, but it does not define action inputs, rewards, probabilistic generative decoding, or planning (Wang et al., 18 Dec 2025).

The term “Kalman World Models” becomes explicit and programmatic in “From Gradients to Riccati Geometry,” which proposes KWM as a class of learned state-space models trained by recursive Bayesian filtering rather than backpropagation (Kiruluta, 13 Mar 2026). That paper sharpens the field’s vocabulary: KWM is not merely “Kalman-flavored” sequence modeling, but a reformulation of learning itself as online state and parameter estimation.

2. State-space structure and mathematical core

At the center of KWM is a state-space model of the form

$$x_{t+1} = f_\theta(x_t, u_t) + w_t,\qquad y_t = h_\theta(x_t) + v_t,$$

where $x_t$ is a latent state, $u_t$ an input or control, $y_t$ an observation or target, and $w_t,v_t$ process and observation noise (Kiruluta, 13 Mar 2026). In the strict KWM formulation, parameters may themselves be treated as latent dynamical states by augmenting the state vector as

$$z_t = \begin{bmatrix} x_t \ \theta_t \end{bmatrix},$$

so that learning becomes joint state-parameter estimation rather than batch optimization over static weights (Kiruluta, 13 Mar 2026).

The recursive update follows the classical prediction-correction pattern. For parameter filtering, the predictive step is

$$\theta_{t|t-1} = \theta_{t-1},\qquad P_{t|t-1} = P_{t-1}+Q,$$

with innovation

$x_t$0

Jacobian

$x_t$1

Kalman gain

$x_t$2

and posterior correction

$x_t$3

The covariance recursion can also be written in Riccati form, which supplies the geometry and stability properties emphasized by the KWM literature (Kiruluta, 13 Mar 2026).

The broader world-model perspective adds latent rollout structure. Deep Kalman Filters factorize the joint model as latent prior, action-conditioned latent transition, and latent-to-observation emission, which is precisely the decomposition expected of a stochastic latent world model (Krishnan et al., 2015). In that sense, KWM inherits from both control-theoretic filtering and neural latent sequence modeling: the former supplies recursive belief updates and uncertainty algebra, while the latter supplies expressive nonlinear transitions and observation mappings.

3. Learning as filtering and “Riccati geometry”

The defining claim of strict KWM is that training can be recast as online filtering. Instead of updating parameters with gradient descent, KWM uses innovation-driven posterior correction. The paper formalizing the term states the correspondence directly: training becomes online filtering, error signals become innovations, learning rates become gains computed from uncertainty, and optimizer state becomes posterior covariance (Kiruluta, 13 Mar 2026).

Under a locally linear Gaussian observation model,

$x_t$4

the Fisher information is

$x_t$5

and the Kalman update coincides with a natural-gradient step when the parameter covariance matches the inverse Fisher information, becoming exact in the small-noise limit (Kiruluta, 13 Mar 2026). This is the source of the phrase “Riccati geometry”: curvature adaptation is supplied by the covariance recursion rather than by a separately designed optimizer.

Not all KWM-relevant training regimes use this formulation. Deep Kalman Filters train by maximizing a variational lower bound with a structured approximate posterior and are strongest in a smoothing-oriented regime, particularly with the bidirectional recurrent recognition model $x_t$6-BRNN (Krishnan et al., 2015). Recurrent Kalman Networks avoid variational trajectory inference entirely and train end-to-end by backpropagation through a factorized filtering architecture whose Kalman updates reduce to elementwise operations (Becker et al., 2019). KalmanNet likewise retains ordinary supervised training, but shifts the learned burden from full latent dynamics learning to recurrent gain estimation (Revach et al., 2021).

A common misconception is that “gradient-free” in KWM implies the total absence of derivatives. The 2026 KWM formulation still uses Jacobians in its EKF-style nonlinear variants, while reserving its strongest “gradient-free” interpretation for replacing gradient-descent-based optimization as the main learning rule (Kiruluta, 13 Mar 2026). The underlying distinction is therefore between reverse-mode optimization and recursive Bayesian correction, not between all derivatives and no derivatives.

4. Architectural patterns and representative variants

Several recurrent architectural motifs recur across the KWM literature. One is the stochastic latent world model: Deep Kalman Filters maintain Gaussian latent transitions with learned mean and covariance, action-conditioned evolution, probabilistic emissions, and amortized inference over hidden state (Krishnan et al., 2015). This is the clearest early precursor to KWM as a probabilistic latent simulator.

A second motif is the explicit latent belief state with scalable uncertainty structure. Recurrent Kalman Networks split latent state into an observable “upper” part and a memory-like “lower” part, retain only upper, lower, and paired cross-covariances, and thereby turn Kalman updates into scalar operations while preserving uncertainty-aware belief correction in high-dimensional learned feature spaces (Becker et al., 2019).

A third motif is the learned gain or update operator. KalmanNet learns the Kalman gain from innovation-like features and difference features while keeping the model-based prior prediction and posterior correction form intact (Revach et al., 2021). Spike-Kal learns the gain with a spiking neural network from residual features $x_t$7, replacing analytic gain computation while keeping a model-based state-transition prediction step (Xiao et al., 17 Apr 2025). For 1-bit observations, Bussgang-aided KalmanNet uses dithered quantized residuals and GRUs to learn gain-related covariance surrogates under severe quantization and model mismatch (Jung et al., 23 Jul 2025).

A fourth motif is the closed-loop innovation-driven sequence backbone. KOSS treats selection as latent uncertainty minimization and uses a gain-like mechanism driven by innovation, yielding a closed-loop state/input fusion mechanism that is more filter-like than open-loop input gating; however, it lacks action conditioning, reward modeling, probabilistic decoding, and belief-state planning, so it is best regarded as a strong filtering or dynamics backbone rather than a complete KWM (Wang et al., 18 Dec 2025).

A fifth motif is Kalman augmentation without latent Kalman filtering. KARNet injects corrected and predicted ego state from an external error-state EKF into a recurrent latent visual dynamics model and downstream waypoint controller, improving learned driving representations and closed-loop behavior without making the Kalman filter the inference engine of the latent world model itself (Manjunatha et al., 2023).

5. Empirical performance and application domains

The empirical range of KWM-related methods is unusually broad. In the strict KWM formulation trained by filtering, reported results include 97.8% sequential MNIST accuracy, 88.4 WikiText-2 perplexity, 98.2 shifted perplexity, and 0.17 average forgetting, with the main advantage appearing under distribution shift and continual adaptation rather than only under stationary benchmarks (Kiruluta, 13 Mar 2026).

In autonomous driving, KARNet evaluates its world-model features in closed loop. In imitation learning on CARLA, the no-EEKF baseline reaches 42.26 route completion and 40.36 infractions, whereas early fusion of corrected and predicted ego state reaches 55.75 route completion and 11.2 infractions; on the Udacity real-data classification task, adding IMU/GNSS raises accuracy from 45.92 $x_t$8 1.66\% to 77.24 $x_t$9 1.24\% (Manjunatha et al., 2023). These results support the narrower claim that explicit ego-motion signals materially strengthen latent visual dynamics models.

KOSS reports strong gains on long-term sequence tasks, including 79.2% accuracy on the 50% interference selective-copying task versus 13.5% for Mamba, and MSE reductions of 2.92–36.23\% across nine long-term forecasting benchmarks (Wang et al., 18 Dec 2025). The relevance to KWM lies less in reward-free sequence modeling per se than in its innovation-driven, context-aware latent update mechanism.

Quantized sensing provides a specialized but revealing testbed. Under 1-bit observations on the Lorenz system, EKF and KalmanNet degrade to 17.85 dB and 12.95 dB, whereas BKF and BKNet reach -17.38 dB and -17.31 dB respectively; on the NCLT dataset, BKNet with 1-bit observations achieves 18.62, slightly better than ideal-observation KalmanNet at 19.15 (Jung et al., 23 Jul 2025). This suggests that observation-model fidelity can be as consequential for KWM performance as latent dynamics fidelity.

Tracking and sensing benchmarks also motivate structure-aware update discovery. Kalman Evolve reports up to 12\% RMSE reduction over the Optimized Kalman Filter on Doppler radar, LiDAR-based localization, and pedestrian tracking by discovering small non-affine modifications to the classical update while preserving recursive form (Saketos et al., 26 May 2026). A related line based on CGP and LLM-assisted evolution shows that symbolic search can rediscover near-optimal Kalman-like filters under ideal assumptions and produce interpretable alternatives that outperform the standard Kalman filter when those assumptions fail (Saketos et al., 13 Aug 2025).

6. Boundaries, misconceptions, and research directions

The literature repeatedly distinguishes full KWM from Kalman-adjacent architectures. KARNet is explicit on this point: it is a Kalman-augmented latent dynamics model, not a “true” latent Kalman filter over the scene, because the external EKF estimates physical ego state rather than latent belief over the learned world model (Manjunatha et al., 2023). KOSS makes a parallel distinction from the opposite direction: it provides a strong innovation-driven state-dynamics module, but not a complete world model because it lacks control inputs, reward prediction, probabilistic generative decoding, and planning (Wang et al., 18 Dec 2025). KalmanNet, Spike-Kal, and BKNet are likewise best understood as learned filtering or gain-estimation modules rather than full generative world models (Revach et al., 2021).

The strict 2026 KWM formulation also has clear limitations. Full covariance is expensive; scalable use depends on low-rank, block-diagonal, or Kronecker-factored approximations, which introduce approximation bias (Kiruluta, 13 Mar 2026). The strongest theory relies on local Gaussian assumptions, smoothness, observability, and, for global convergence statements, strong convexity. Frontier-scale validation on multi-billion-parameter models is explicitly absent (Kiruluta, 13 Mar 2026).

A plausible synthesis of the field is that KWM research now has two complementary objectives. One is better belief-update machinery: innovation-driven gains, factorized covariances, quantized-observation updates, spiking or neural gain approximators, and interpretable discovered corrections (Xiao et al., 17 Apr 2025). The other is full latent world modeling: action-conditioned dynamics, probabilistic latent belief, observation decoding, and planning in a unified model. KARNet states that what remains missing for a fully general KWM framework is “a jointly learned latent state-space model in which Kalman-style filtering or belief updating occurs inside the latent world model, with uncertainty, action-conditioned dynamics, and planning all treated in a unified way” (Manjunatha et al., 2023).

In that sense, KWM is not a single architecture but a research program. Its unifying proposition is that latent world modeling benefits when belief state, uncertainty, and recursive correction are treated as first-class objects rather than as incidental by-products of end-to-end sequence prediction (Kiruluta, 13 Mar 2026).