Training neural sequence models to approximate Bayesian prediction from Occam’s prior

Develop training methodologies for neural sequence models that robustly approximate Bayesian sequence prediction starting from an Occam’s razor prior, so they can produce accurate prospective forecasts in dynamically changing, non-stationary environments.

Background

A central practical gap highlighted by the authors is bridging the idealized Bayesian predictors in MUPI with real neural sequence models trained via stochastic gradient descent on episodic datasets.

Although some theory suggests SGD can approximate sampling-based Bayesian inference, the authors note that obtaining robust Bayesian predictors from an Occam’s prior—sufficient to support prospective prediction in non-stationary worlds—remains unresolved and constitutes a key step for practical deployment.

References

However, training neural sequence models to robustly approximate Bayesian prediction starting from an Occam's razor prior—thereby ensuring they can make accurate, prospective forecasts in dynamically changing worlds—remains an open problem \citep{bornschein2024transformers,de2024prospective}.

— Embedded Universal Predictive Intelligence: a coherent framework for multi-agent learning  (2511.22226 - Meulemans et al., 27 Nov 2025) in Section 6 (Discussion) — The connection between MUPI and current foundation models

The meta-learning experiments of \citet{graumoya2024universal} are the same measurement for a different target, the context-tree-weighting mixture and, aspirationally, Solomonoff's mixture, and their finding that transformers and LSTMs reach the same loss says that both architectures place adequate prior mass near that target at the scale they used; which of the two does so with the smaller complexity, and hence with fewer sequences, is the question the framework poses and their length-generalization results begin to answer.

— Learning Source Structure: Bayesian Prediction with Transformers  (2610.18725 - Feder et al., 13 Oct 2026) in Appendix B, Section “Two universalities: the source family and the architecture” (discussion following Appendix \ref{sec:twolevels})