---
title: Reservoir Predictive Path Integral (RPPI)
url: https://www.emergentmind.com/topics/reservoir-predictive-path-integral-rppi
type: topic
---

# Reservoir Predictive Path Integral (RPPI)

Searching arXiv for the cited RPPI paper and related reservoir/path-integral work.
Using arXiv search for `2509.03839` and related queries.
Reservoir Predictive Path Integral (RPPI) is a control framework for unknown nonlinear dynamical systems that combines echo-state networks (ESNs) for fast online system identification with model predictive path integral (MPPI) control for sampling-based nonlinear optimal control. In the formulation introduced in "Reservoir Predictive Path Integral Control for Unknown Nonlinear Dynamics" [2509.03839], the ESN learns nonlinear dynamics online through recursive least squares (RLS), while MPPI directly propagates the learned nonlinear reservoir model during rollout evaluation without linearization approximations. The framework is extended to uncertainty-aware RPPI (URPPI), which uses readout-weight uncertainty from RLS to balance exploration and exploitation; experiments on the Duffing oscillator and four-tank systems report control-cost reductions of up to 60% relative to traditional quadratic programming-based model predictive control methods [2509.03839].

## 1. Conceptual definition and distinguishing features

RPPI addresses two coupled problems in online control of unknown nonlinear systems: fast learning of dynamics and real-time computation of optimal controls without linearization. Its central synthesis is between ESNs, described as reservoir computing models with fixed recurrent weights and fast readout adaptation, and MPPI, described as a sampling-based variant of model predictive control that optimizes over noisy control sequences and can handle arbitrary nonlinearities and cost forms [2509.03839].

Relative to linearization-based ESN-MPC schemes such as ESN-QPMPC, RPPI and URPPI are distinguished by three properties. First, they never linearize the ESN; nonlinear reservoir dynamics are propagated directly during sampling, preserving expressivity even under strong nonlinearity. Second, the cost function is not restricted to quadratic form, because arbitrary stage and terminal costs can be incorporated into the MPPI weighting. Third, URPPI uses ESN readout-weight uncertainty obtained from RLS to modulate exploration and exploitation by sampling perturbed output layers, rather than relying on ad hoc excitation signals or repeated re-linearization [2509.03839].

The paper situates RPPI against a broader trade-off in learning-based MPC. Neural-network- and Gaussian-process-based predictive controllers can be computationally heavy online, whereas ESNs retain a fixed reservoir and update only the readout. MPPI then provides a scalable sampling-based optimizer that can operate directly on the learned nonlinear model. A plausible implication is that RPPI is designed to preserve nonlinear model structure while keeping online adaptation lightweight.

## 2. ESN model and online identification mechanism

The ESN used in RPPI is a leaky integrator reservoir with linear readout. With reservoir state $x_t \in \mathbb{R}^{\hat N}$, input $u_t \in \mathbb{R}^{M}$, output $y_t \in \mathbb{R}^{L}$, and activation function $f$, the model is

$$
x_{t+1} = (1-\alpha)\,x_t + \alpha\, f\!\big(W_{\mathrm{res}} x_t + W_{\mathrm{in}} u_t\big), 
\qquad
\hat y_t = W_{\mathrm{out},t}^\top x_t.
$$

Here $W_{\mathrm{res}} \in \mathbb{R}^{\hat N \times \hat N}$ is a sparse fixed reservoir with spectral radius $< 1$, $W_{\mathrm{in}} \in \mathbb{R}^{\hat N \times M}$ is fixed input weights, and $W_{\mathrm{out},t} \in \mathbb{R}^{\hat N \times L}$ is the time-varying readout trained online. The paper does not include an explicit bias term or concatenated $[x_t;u_t;1]$ readout, although it notes that such an extension can be added in other implementations if needed [2509.03839].

Online training minimizes an exponentially discounted squared error,

$$
J_{\mathrm{id}}(W_{\mathrm{out}}) = \sum_{\tau=0}^{t} \gamma^{\,t-\tau}\,\|y_\tau - W_{\mathrm{out}}^\top x_\tau\|_2^2,
$$

with RLS updates

$$
W_{\mathrm{out},t} = W_{\mathrm{out},t-1} +
\frac{P_{t-1}x_t}{\gamma + x_t^\top P_{t-1}x_t}
\big(y_t - W_{\mathrm{out},t-1}^\top x_t\big)^\top,
$$

$$
P_t = \frac{1}{\gamma}\Big(
P_{t-1} -
\frac{P_{t-1}x_t x_t^\top P_{t-1}}
{\gamma + x_t^\top P_{t-1}x_t}
\Big).
$$

Under the assumption of a true linear readout
$y_t = \bar W_{\mathrm{out}}^\top x_t + \varepsilon_t$
with independent Gaussian noise
$\varepsilon_t \sim \mathcal{N}(0,\tilde\Sigma)$,
the parameter covariance for $\gamma = 1$ is

$$
\mathrm{Cov}\big(\mathrm{vec}(W_{\mathrm{out},t})\big) = \tilde\Sigma \otimes P_t.
$$

This covariance is the basis of URPPI’s uncertainty-aware sampling [2509.03839].

The reported implementation uses $\hat N = 400$ for both tasks, $f=\tanh$, $\gamma = 1.0$, $W_{\mathrm{out},-1}$ initialized uniformly in $[-0.1,0.1]$, and $P_{-1}=I_{\hat N}$. Reservoir sparsity, spectral radius, input scaling, and leak rate are task-specific: for the Duffing system, sparsity $0.48$, spectral radius $0.89$, input scaling in $[-2.0,2.0]$, and $\alpha=0.59$; for the four-tank system, sparsity $0.17$, spectral radius $0.78$, input scaling in $[-1.5,1.5]$, and $\alpha=0.078$. Washout length and additional preprocessing are not specified, and the readout is updated online at every control step [2509.03839].

## 3. MPPI integration and control-law construction

RPPI embeds the learned ESN directly inside MPPI rollout propagation. For rollout $i$ and time $t$,

$$
x_{t+1}^{(i)} =
f_{\mathrm{ESN}}\!\big(x_t^{(i)},u_t+\epsilon_t^{(i)}\big)
=
(1-\alpha)\,x_t^{(i)} +
\alpha\, f\!\big(W_{\mathrm{res}}x_t^{(i)} + W_{\mathrm{in}}(u_t+\epsilon_t^{(i)})\big),
$$

$$
\hat y_t^{(i)} = W_{\mathrm{out}}^{(i)\top} x_t^{(i)},
$$

where $W_{\mathrm{out}}^{(i)}$ is either the nominal readout, in standard RPPI/MPPI, or a perturbed sample, in URPPI. The trajectory cost for rollout $i$ is

$$
S^{(i)} =
\phi\big(x_T^{(i)}\big) +
\sum_{t=0}^{T-1}
\Big(
\ell\big(x_t^{(i)},u_t+\epsilon_t^{(i)}\big)
+\tfrac{1}{2}\epsilon_t^{(i)\top}R\epsilon_t^{(i)}
\Big).
$$

The MPPI weighting and control update are

$$
w^{(i)} = \exp\!\Big(-\frac{1}{\lambda}S^{(i)}\Big),
\qquad
u_t \leftarrow
u_t +
\frac{\sum_{i=1}^K w^{(i)}\epsilon_t^{(i)}}
{\sum_{i=1}^K w^{(i)}}.
$$

Control perturbations are sampled as
$\epsilon_t^{(i)} \sim \mathcal{N}(0,\Sigma)$,
with $\Sigma$ typically set to $\lambda R^{-1}$; the horizon $T$ (denoted $H$ in the paper), rollout count $K$, and temperature $\lambda>0$ are user-chosen [2509.03839].

The control loop is specified per control step. First, plant output $y_t$ is measured and the previous control is applied to the plant and ESN reservoir. Second, the ESN readout and precision matrix are updated by RLS. Third, uncertainty information $P_t$ is read, $\tilde\Sigma$ is chosen, and the readout sampling distribution
$\mathcal{N}(\mathrm{vec}(W_{\mathrm{out},t}),\tilde\Sigma\otimes P_t)$
is defined. Fourth, the nominal input sequence is initialized from the previous iteration, $K$ input-noise sequences and $\tilde K$ readout-weight copies are sampled, rollouts are simulated across the horizon, and costs are accumulated. Finally, the first element of the updated input sequence is applied, the horizon is shifted forward, and the procedure repeats [2509.03839].

The generic control objective used in RPPI is

$$
J_{\mathrm{ctrl}}(U_t)=
\sum_{\tau=t}^{t+H-1}\ell(y_\tau,u_\tau)
+
\phi(y_{t+H})
+
\sum_{\tau=t}^{t+H-1}
\tfrac{1}{2}(u_\tau-u^{\mathrm{ref}})^\top R (u_\tau-u^{\mathrm{ref}}).
$$

For the Duffing oscillator,
$\ell(y,u)=\tfrac{1}{2}Q(y-y^{\mathrm{ref}})^2+\tfrac{1}{2}Ru^2$,
$\phi(y)=\tfrac{1}{2}Q(y-y^{\mathrm{ref}})^2$,
with input constraint $u\in[-10,10]$.
For the four-tank system,
$\ell(y,u)=\tfrac{1}{2}(y-y^{\mathrm{ref}})^\top Q(y-y^{\mathrm{ref}})+\tfrac{1}{2}(u-u^{\mathrm{ref}})^\top R(u-u^{\mathrm{ref}})$,
$\phi(y)=\tfrac{1}{2}(y-y^{\mathrm{ref}})^\top Q(y-y^{\mathrm{ref}})$,
with box constraints $u\in[8,12]^2$ [2509.03839].

## 4. URPPI and the exploration–exploitation mechanism

URPPI extends RPPI by explicitly incorporating uncertainty in the ESN readout. Its mechanism uses the RLS precision matrix $P_t$ together with $\tilde\Sigma$ to sample multiple readouts,

$$
\mathrm{vec}\big(W_{\mathrm{out},\tau}^{(\tilde i)}\big)
\sim
\mathcal{N}\big(
\mathrm{vec}(W_{\mathrm{out},t}),
\tilde\Sigma \otimes P_t
\big),
\qquad
\tilde i = 1,\dots,\tilde K.
$$

In URPPI, readout perturbations are sampled at each prediction step, and the weighting integrates over both input-noise samples and output-weight samples by Monte Carlo [2509.03839].

The intended behavioral effect is explicit in the formulation. Early in learning, $P_t$ is large, indicating high uncertainty; sampled readout variations then induce broader variability in predicted outputs and in the weighted selection of control sequences, which favors exploration. As learning converges, $P_t$ shrinks, readout samples cluster, and the weights concentrate on low-cost trajectories, producing a transition toward exploitation. The paper states that the uncertainty modulation is primarily through readout sampling: $\Sigma$ is fixed per experiment, and no explicit scheduling of $\Sigma_t$ or $\lambda_t$ is provided [2509.03839].

Several practical heuristics are described as compatible with URPPI but not part of the paper’s implementation. These include exploration scaling via ESN uncertainty, such as
$\Sigma_t=\Sigma_0+\gamma_\Sigma \,\mathrm{diag}(\mathrm{Var}[\hat y_t])$
or
$\Sigma_t=\Sigma_0+\gamma_\Sigma \,\mathrm{tr}(P_t)I$,
and temperature scheduling
$\lambda_t=\lambda_0+\gamma_\lambda \,\mathrm{tr}(P_t)$.
Because these heuristics are explicitly identified as outside the reported implementation, they should not be conflated with the empirical results [2509.03839].

A common misconception is to treat URPPI as a generic Bayesian ensemble controller. The reported method does not use an ensemble or bootstrap, and it has no explicit calibration procedure beyond setting $\tilde\Sigma$. Its uncertainty model is the analytical covariance $\tilde\Sigma \otimes P_t$ inherited from the RLS readout estimate [2509.03839].

## 5. Benchmarks and reported empirical behavior

The first benchmark is the Duffing oscillator,

$$
\dot z_1(s)=z_2(s), \qquad
\dot z_2(s)=
-\frac{k}{m}z_1(s)-\frac{c}{m}z_2(s)-\frac{k_{\mathrm{nl}}}{m}z_1^3(s)+\frac{1}{m}u(s),
\qquad
y(s)=z_1(s),
$$

discretized by explicit Euler with $\Delta s=0.1\,\mathrm{s}$. The main experiment uses $k=1.1$, $k_{\mathrm{nl}}=0.9$, $m=1.1$, and $c=0.9$, distinct from a tuning set with all four parameters equal to $1.0$. The control weights are $Q=100$ and $R=1$; the reference switches between $\pm 1$ every $20\,\mathrm{s}$; and the horizon is $H=10$. MPPI uses $K=1000$ rollouts, $\lambda=1.9$, and scalar $\Sigma=1.9$. URPPI uses $\tilde K=10$ output-weight samples and $\tilde\Sigma=1.0$ [2509.03839].

For the Duffing system, the reported results over 20 random seeds on a CPU multi-threaded implementation are that MPPI and URPPI achieve asymptotic tracking, whereas QPMPC does not converge to the reference. Over $T=400$ steps, URPPI yields a time-integrated cost approximately $63\%$ lower than QPMPC and approximately $7\%$ lower than MPPI. The average runtime per control step is $0.034\,\mathrm{s}$ for MPPI and $0.046\,\mathrm{s}$ for URPPI. Spectral analysis reports that MPPI and URPPI produce broader frequency content than QPMPC, and that URPPI exhibits the widest spread early, for $t<3\,\mathrm{s}$, which the paper interprets as effective exploration guided by uncertainty [2509.03839].

The second benchmark is the four-tank process,

$$
\dot z_1 =
-\frac{a_1}{A_1}\sqrt{2gz_1}
+\frac{a_3}{A_1}\sqrt{2gz_3}
+\frac{b_1}{A_1}u_1,
$$

$$
\dot z_2 =
-\frac{a_2}{A_2}\sqrt{2gz_2}
+\frac{a_4}{A_2}\sqrt{2gz_4}
+\frac{b_2}{A_2}u_2,
$$

$$
\dot z_3 =
-\frac{a_3}{A_3}\sqrt{2gz_3}
+\frac{1-b_2}{A_3}u_2,
\qquad
\dot z_4 =
-\frac{a_4}{A_4}\sqrt{2gz_4}
+\frac{1-b_1}{A_4}u_1,
\qquad
y=[z_1\;z_2]^\top,
$$

with $\Delta s=1.0\,\mathrm{s}$. The parameters are
$A_1=A_3=28$,
$A_2=A_4=32$,
$a_1=a_3=0.071$,
$a_2=a_4=0.057$,
standard $g$,
and volume splits
$b_1=0.693$,
$b_2=0.606$,
distinct from a tuning choice
$b_1=0.7$,
$b_2=0.6$.
The cost uses
$Q=I_2$,
$R=I_2$,
$u^{\mathrm{ref}}=[10,10]^\top$,
reference alternation between $[13,13]^\top$ and $[11,11]^\top$ every $150\,\mathrm{s}$,
horizon $H=10$,
and control constraint $u\in[8,12]^2$.
MPPI uses
$K=1000$,
$\lambda=0.71$,
$\Sigma=0.71I_2$;
URPPI uses
$\tilde K=10$
and
$\tilde\Sigma=0.1I_2$ [2509.03839].

For the four-tank system, MPPI and URPPI track references reliably, while QPMPC deviates. Over $T=1000$ steps, URPPI reports a time-integrated cost approximately $33.1\%$ lower than QPMPC and approximately $3.9\%$ lower than MPPI. Runtime per control step is $0.037\,\mathrm{s}$ for MPPI and $0.049\,\mathrm{s}$ for URPPI. Spectral analysis again indicates broader frequency content for MPPI and URPPI than for QPMPC, with the widest URPPI spread around $t\approx 40\,\mathrm{s}$ [2509.03839].

## 6. Computational profile, limitations, and relation to adjacent research

The reported per-step computational complexity is

$$
C_{\mathrm{total}}=
O\big(KH(\hat N^2 + M\hat N + L\hat N + M^2 + L^2)\big)
+
O(\hat N^3 + L\hat N^2).
$$

More specifically, ESN-RLS updates cost
$O(\hat N^2 + L\hat N)$;
sampling $K$ input sequences costs
$O(KHM)$;
sampling $\tilde K$ readout copies after a Cholesky factorization of $P_t$ costs
$O(\hat N^3 + L\hat N^2)$;
trajectory simulation across $K\times H$ steps costs
$O(KH(\hat N^2 + M\hat N + L\hat N))$;
and cost evaluation and weighted averaging add
$O(KH(M^2 + L^2) + KHM)$.
The experiments use CPU multithreading with 16 threads; Monte Carlo sampling of inputs and readout layers is described as embarrassingly parallel; GPU parallelization is compatible but was not used. Memory footprint is not reported [2509.03839].

The framework’s theoretical guarantees are deliberately limited. The assumptions are output-only measurements, ESN approximation of the input-output map through the reservoir state, and RLS stability conditions requiring persistent excitation and invertibility of $\sum_\tau x_\tau x_\tau^\top$, with $\gamma=1$ in the experiments. No formal observability or identifiability analysis is given, and there is no formal closed-loop stability proof. The paper further notes that MPPI’s weighting is sensitive to $\lambda$: too small a temperature may cause sample degeneracy, whereas too large a value may slow convergence. Performance is reported to depend on reservoir hyperparameters, MPPI parameters, and URPPI uncertainty scaling through $\tilde\Sigma$ [2509.03839].

Several implementation recommendations are presented as practical guidance rather than benchmark-defining requirements. These include spectral radius below $1$, leak rates matched to plant speed, reservoir sizes of a few hundred, input scaling that avoids activation saturation, short horizons for real-time feasibility, and a few hundred to a thousand rollouts. The paper also notes that a brief washout is common when initializing ESNs offline, although washout is not specified in the RPPI experiments. These points matter because they distinguish the reported method from a misconception that RPPI is parameter-free; its performance is explicitly described as sensitive to both reservoir and MPPI design choices [2509.03839].

A related theoretical line of work analyzes reservoir computers with path-integral methods in the thermodynamic limit. "Path integral approach to universal dynamics of reservoir computers" derives a generating-functional description, classifies random reservoirs into universality classes through the large-$N$ cumulant-generating function $K(q)$, links these classes to eigenvalue spectra, and reports that computational performance in inference and forecasting tasks is often strongest near phase boundaries, including non-chaotic ones [2112.01886]. This suggests a theoretical backdrop for the broader conjunction of reservoir models and path-integral formalisms. However, that work addresses random-network dynamics, synchronization, inference, and forecasting rather than the online nonlinear control architecture introduced as RPPI [2112.01886].

Source: https://www.emergentmind.com/topics/reservoir-predictive-path-integral-rppi