---
title: 'Infoprop-Dyna: Uncertainty-Aware MBRL'
url: https://www.emergentmind.com/topics/infoprop-dyna
type: topic
---

# Infoprop-Dyna: Uncertainty-Aware MBRL

to=arxiv_search.search  菲律宾申博json
{"query":"2605.01096 OR \"Learning to Race in Minutes: Infoprop Dyna on the Mini Wheelbot\""}
to=arxiv_search.search  大发官网json
{"query":"\"Infoprop\" Dyna model-based reinforcement learning 2501.16918"}
to=arxiv_search.search  大发快三的json
{"query":"\"Infoprop-Dyna\" OR \"Infoprop Dyna\""}
Infoprop-Dyna is an uncertainty-aware model-based reinforcement learning framework in the Dyna tradition that learns a world model from real interaction data and uses that learned model, rather than a hand-built physics simulator, to generate large-scale parallel synthetic rollouts for policy optimization. In the robotics instantiation reported in "Learning to Race in Minutes: Infoprop Dyna on the Mini Wheelbot" [2605.01096], the framework is deployed directly on an underactuated unicycle robot and is reported to enable racing around a fixed track within 11 minutes of real-world experience.

## 1. Definition and algorithmic position

Infoprop-Dyna occupies the model-based branch of RL. Its central workflow is Dyna-style: real trajectories are collected from the environment, a dynamics model is fit to those data, synthetic trajectories are then generated from that learned model, and the synthetic data are used to accelerate policy learning. In the original formulation, Infoprop-Dyna is built around the rollout mechanism Infoprop, which is introduced as a replacement for standard trajectory sampling and is reported to achieve state-of-the-art performance in Dyna-style MBRL on common MuJoCo benchmark tasks while substantially increasing rollout length and data quality [2501.16918].

The defining distinction is its treatment of model uncertainty. Rather than using model rollouts naively, Infoprop-Dyna explicitly separates aleatoric uncertainty from epistemic uncertainty, treats model predictions as noisy observations of an underlying ground-truth signal, tracks accumulated corruption over rollout time, and terminates rollouts before synthetic data become too distorted for reliable policy learning. In the Mini Wheelbot paper, this same principle is described conceptually as the basis for mitigating model exploitation and enabling “reliable rollouts over longer horizons than any prior method,” while removing the need for carefully engineered simulators and domain randomization [2605.01096].

This positioning matters most for fast, nonlinear, and unstable systems. The Mini Wheelbot study frames such systems as especially difficult for simulator-centric pipelines because contact-induced nonlinearities, slip effects, and underactuated yaw dynamics make longer-horizon prediction essential while simultaneously making fixed simulators difficult to engineer faithfully.

## 2. Uncertainty separation, information propagation, and rollout termination

In the original Infoprop formulation, the learned environment dynamics are modeled as a Gaussian transition kernel,
$$
p_S(s_{t+1}\mid s_t,a_t)=\mathcal{N}\!\left(\mu_S(s_t,a_t),\Sigma_S(s_t,a_t)\right),
$$
implemented by a predictive AES model in which each ensemble member outputs
$$
p_{\theta_e}(s_{t+1}\mid s_t,a_t)=\mathcal{N}\!\left(\mu_{\theta_e}(s_t,a_t),\Sigma_{\theta_e}(s_t,a_t)\right).
$$
The ensemble members are trained on real data with Gaussian negative log-likelihood [2501.16918].

The uncertainty decomposition follows the law of total variance:
$$
\mathrm{Var}(s_{t+1})=\mathbb{E}_{\theta}\!\left[\mathrm{Var}(s_{t+1}\mid s_t,a_t,\theta)\right]+\mathrm{Var}_{\theta}\!\left(\mathbb{E}[s_{t+1}\mid s_t,a_t,\theta]\right).
$$
The first term is treated as aleatoric variance and the second as epistemic variance. To estimate the aleatoric component from the ensemble, Infoprop uses covariance intersection with uniform weights:
$$
\bar{\Sigma}(s_t,a_t)=\left(\frac{1}{E}\sum_{e=1}^E (\hat{\Sigma}^e)^{-1}\right)^{-1},
\qquad
\bar{\mu}(s_t,a_t)=\bar{\Sigma}(s_t,a_t)\left(\frac{1}{E}\sum_{e=1}^E (\hat{\Sigma}^e)^{-1}\hat{\mu}^e\right),
$$
while epistemic uncertainty is estimated as the sample covariance of ensemble means,
$$
\Sigma_{\mathrm{epi}}(s_t,a_t)=\frac{1}{E}\sum_{e=1}^E(\hat{\mu}^e-\bar{\mu})(\hat{\mu}^e-\bar{\mu})^\top.
$$

Infoprop then interprets the model rollout as a noisy channel,
$$
\hat{S}_{t+1}=\bar{S}_{t+1}+\varepsilon_{t+1},
$$
and constructs an “Infoprop state” by conditioning on the noisy model prediction through a Kalman-style update:
$$
K=\bar{\Sigma}(\bar{\Sigma}+\Sigma_{\mathrm{epi}})^{-1},
$$
$$
\mu_{\mathrm{IP}}=\bar{\mu}+K(\hat{s}_{t+1}-\bar{\mu}),
\qquad
\Sigma_{\mathrm{IP}}=(I-K)\bar{\Sigma}.
$$
This conditioning is intended to suppress the epistemic component while correctly propagating aleatoric uncertainty, and the induced distribution of Infoprop states is stated to equal the estimated environment distribution [2501.16918].

Rollout termination is based on an information-theoretic corruption measure. For Gaussian $Z$ with covariance $\Sigma_Z$ and per-dimension discretization $\Delta z^{(k)}$, the quantized entropy is
$$
H(Z)=\frac{1}{2}\log_2\!\big((2\pi e)^{n_Z}|\Sigma_Z|\big)-\sum_{k=1}^{n_Z}\log_2\!\big(\Delta z^{(k)}\big).
$$
The cumulative error accumulator is
$$
E_t:=\sum_{t'=0}^{t}H(\tilde{S}_{t'+1}),
$$
and rollouts terminate either when the single-step entropy exceeds $\lambda_1$ or when the cumulative entropy exceeds $\lambda_2$. In the MuJoCo experiments, these thresholds are scheduled from empirical quantiles with $\zeta_1=0.99$, $\zeta_2=0.01$, and $\xi=100$ [2501.16918].

The Mini Wheelbot hardware paper invokes these same principles conceptually but does not provide explicit formulas for the propagation mechanism, moment matching, linearization, sigma-point methods, variance aggregation, mutual information terms, KL terms, model loss functions, or policy objectives. It states only that the model is uncertainty-aware and that Infoprop-Dyna tracks uncertainty accumulation along rollouts [2605.01096].

## 3. World model, observations, and distributed training on the Mini Wheelbot

On the Mini Wheelbot, the learned transition model is defined over the robot’s physics state. The local track representation is not modeled directly; instead, it is deterministically reconstructed from the predicted physics state and known track geometry. Because the platform provides estimated states from an onboard estimator rather than fully observed, memoryless states, the learned model is conditioned on a short history of estimated physics states and control inputs and is trained to predict the next estimated state. This is described as implicitly capturing both the robot’s physical dynamics and the estimator’s dynamics [2605.01096].

The observation space combines the estimated physics state with a local track representation consisting of the relative positions of the next 30 track points expressed in polar coordinates. The action space is direct motor torque commands. The reward is specified qualitatively: it encourages higher velocity along the track, penalizes deviation from the centerline, and applies a terminal penalty for crashes or leaving the track.

The data and training loop is explicitly distributed. Real trajectories are collected on the physical robot by executing the current policy. Logs are sent via SSH to a local workstation for bookkeeping and then forwarded to a high-performance computing cluster. The training stack is JAX-based, built on BRAX, and uses massive parallelization to fit the world model and optimize the policy from synthetic rollouts generated directly by the learned model. Updated policy parameters are periodically saved, pulled back to the workstation, and deployed on the robot for the next data-collection round. A key design choice is that no physics simulator is used at any stage; the learned dynamics model itself replaces the simulator in the Dyna loop [2605.01096].

The initial dataset is warm-started with approximately one minute of safe trajectories collected by a joystick-driven approximate MPC controller and a human operator. The paper does not disclose batch sizes, world-model architecture, ensemble size, rollout length, learning rates, control frequency, or the number of real steps. It also does not specify rollout horizon, real/synthetic data mixing ratios, uncertainty-based acceptance weights, or whether policy optimization uses MPC, CEM, MPPI, or a learned actor-critic.

## 4. Robotic system and task characteristics

The Mini Wheelbot is described as an underactuated unicycle robot. Its yaw degree of freedom is highly nonlinear, indirectly influenced by wheel actuation through gyroscopic effects and geometric turning, and strongly dependent on ground-contact interactions. The platform’s small form factor implies small time constants and sensitivity to actuation differences, while high-speed motion introduces contact-induced nonlinearities and slip effects that are difficult to model. The study presents these properties as the reason longer-horizon planning is particularly important on this platform [2605.01096].

The task is racing around a fixed track. Terminal penalties are used for safety, and the learning process is entirely real-world after the approximately one-minute warm start. This is the main experimental claim of the work: the framework is presented as bypassing carefully designed simulators and domain randomization by learning directly from real robot interactions.

A central qualitative outcome is the emergence of controlled slipping for fast cornering. The paper states that the learned policy discovers controlled slipping as a strategy for handling fast corners, and characterizes this behavior as “non-trivial to capture using purely physics-based simulation.” This suggests that direct world-model learning on the true hardware can expose behaviors that a simulator-centric workflow might miss or discourage, especially when contact and slip dominate performance.

## 5. Empirical performance on the Mini Wheelbot

The reported learning timeline is short in wall-clock interaction time. During the first 5 minutes, the agent learns turning and cornering. At approximately 6 minutes, it completes its first lap. By 7 to 8 minutes, it performs multiple laps, though with some conservative turning and residual wobble in tight segments. From 9 to 11 minutes, it is described as mastering the track, with peak performance at approximately 11 minutes of real-world experience [2605.01096].

The hardware results are summarized against an AMPC baseline.

| Controller | Speed | Lap outcome |
|---|---|---|
| AMPC baseline | Average $0.15\,\mathrm{m/s}$; maximum $0.33\,\mathrm{m/s}$ | Fewer laps |
| Final RL racing agent | Average $0.5\,\mathrm{m/s}$; peak $0.97\,\mathrm{m/s}$ | More than $3\times$ the laps over the same period |

The reported improvement is therefore both in velocity and in sustained lap completion. The final RL racing agent reaches an average speed of $0.5\,\mathrm{m/s}$ and a peak speed of $0.97\,\mathrm{m/s}$, while completing more than three times the laps of the AMPC baseline over the same period [2605.01096].

The evaluation is narrow by design. The paper compares only against AMPC, qualitatively and quantitatively, and does not report comparisons to other MBRL or model-free RL baselines such as PETS, MBPO, Dreamer, SAC, or related methods. It also does not include ablation studies for uncertainty propagation, Dyna versus model-free learning, or rollout filtering, and it does not report calibration metrics such as NLL, RMSE, or ECE.

## 6. Limitations, scope, and terminological extensions

The principal limitations are methodological under-specification and narrow evaluation scope. For the Mini Wheelbot study, explicit mathematical details of the uncertainty parameterization, information-propagation equations, rollout horizon, mixing ratios, model losses, and policy objectives are absent. Broader generalization across tasks and layouts is left for future work, with the authors indicating plans to scale to larger arenas and arbitrary track layouts and to build a more modular JAX-based distributed Infoprop-Dyna interface applicable across robotics platforms [2605.01096].

In the broader MBRL literature, the original Infoprop-Dyna formulation also identifies dependence on well-calibrated uncertainty, the conservatism of covariance intersection under unknown cross-correlations, and instability arising from critic overfitting when long, high-quality synthetic rollouts change the replay distribution rapidly. On humanoid, the reported late-training instabilities are associated with spikes in critic loss, and regularization such as layer normalization is presented as a partial mitigation [2501.16918].

The label has also appeared in other technical contexts. In one usage, Infoprop-Dyna denotes informative input design within DMDc for system identification, where future control inputs are selected to minimize $\operatorname{tr}(\Gamma)$ or convex surrogates derived from $W=\Gamma^{-1}=\sigma^{-2}ZZ^\top$, with sequential identify–plan–execute–update operation and reduced-order SVD truncations for high-dimensional systems [2409.13088]. In another, the phrase is used descriptively for information propagation in dynamic complexity, where local automaton-state and substring-equality facts are maintained under updates: regular spanners are shown maintainable in DynPROP, core spanners in DynCQ, and generalized core spanners in DynFO [1909.10869].

A further extension appears in benchmarking-oriented materials literature. A Dyna-Mat-based synthesis frames “Infoprop-Dyna” as a dynamic evaluation paradigm for foundation ML interatomic potentials, emphasizing matched finite-temperature trajectories, observable-level metrics such as $U_{\mathrm{RMSE}}$, $F_{\mathrm{RMSE}}$, $E_{\mathrm{RDF}}$, $E_{\mathrm{VDOS}}$, and $\Delta P$, and accuracy–cost Pareto analysis under standardized MD protocols [2607.03433].

Taken together, these usages share an emphasis on propagating uncertainty or information through sequential dynamics. In the sense established by the RL papers, however, Infoprop-Dyna denotes a model-based RL framework in which uncertainty-aware information propagation makes long synthetic rollouts usable for learning, enabling direct real-world training without a physics simulator and, on the Mini Wheelbot, rapid acquisition of racing behavior from minutes of real experience.

Source: https://www.emergentmind.com/topics/infoprop-dyna