Reservoir Predictive Path Integral (RPPI)

• The paper introduces RPPI, a framework that combines echo-state networks with model predictive path integral control to enable rapid online learning of complex nonlinear dynamics.
• It employs recursive least squares for real-time update of the ESN’s readout, allowing direct propagation of nonlinear dynamics without relying on linear approximations.
• URPPI extends the method by leveraging uncertainty in the ESN readout to balance exploration and exploitation, achieving control cost reductions of up to 60% in benchmark tests.

Searching arXiv for the cited RPPI paper and related reservoir/path-integral work. Using arXiv search for ([2509.03839](/papers/2509.03839)) and related queries. Reservoir Predictive Path Integral (RPPI) is a control framework for unknown nonlinear dynamical systems that combines echo-state networks (ESNs) for fast online system identification with model predictive path integral (MPPI) control for sampling-based nonlinear optimal control. In the formulation introduced in "Reservoir Predictive Path Integral Control for Unknown Nonlinear Dynamics" (Inoue et al., 4 Sep 2025), the ESN learns nonlinear dynamics online through recursive least squares (RLS), while MPPI directly propagates the learned nonlinear reservoir model during rollout evaluation without linearization approximations. The framework is extended to uncertainty-aware RPPI (URPPI), which uses readout-weight uncertainty from RLS to balance exploration and exploitation; experiments on the Duffing oscillator and four-tank systems report control-cost reductions of up to 60% relative to traditional quadratic programming-based model predictive control methods (Inoue et al., 4 Sep 2025).

1. Conceptual definition and distinguishing features

RPPI addresses two coupled problems in online control of unknown nonlinear systems: fast learning of dynamics and real-time computation of optimal controls without linearization. Its central synthesis is between ESNs, described as reservoir computing models with fixed recurrent weights and fast readout adaptation, and MPPI, described as a sampling-based variant of model predictive control that optimizes over noisy control sequences and can handle arbitrary nonlinearities and cost forms (Inoue et al., 4 Sep 2025).

Relative to linearization-based ESN-MPC schemes such as ESN-QPMPC, RPPI and URPPI are distinguished by three properties. First, they never linearize the ESN; nonlinear reservoir dynamics are propagated directly during sampling, preserving expressivity even under strong nonlinearity. Second, the cost function is not restricted to quadratic form, because arbitrary stage and terminal costs can be incorporated into the MPPI weighting. Third, URPPI uses ESN readout-weight uncertainty obtained from RLS to modulate exploration and exploitation by sampling perturbed output layers, rather than relying on ad hoc excitation signals or repeated re-linearization (Inoue et al., 4 Sep 2025).

The paper situates RPPI against a broader trade-off in learning-based MPC. Neural-network- and Gaussian-process-based predictive controllers can be computationally heavy online, whereas ESNs retain a fixed reservoir and update only the readout. MPPI then provides a scalable sampling-based optimizer that can operate directly on the learned nonlinear model. A plausible implication is that RPPI is designed to preserve nonlinear model structure while keeping online adaptation lightweight.

2. ESN model and online identification mechanism

The ESN used in RPPI is a leaky integrator reservoir with linear readout. With reservoir state $x_t \in \mathbb{R}^{\hat N}$, input $u_t \in \mathbb{R}^{M}$, output $y_t \in \mathbb{R}^{L}$, and activation function $f$, the model is

$$x_{t+1} = (1-\alpha)\,x_t + \alpha\, f\!\big(W_{\mathrm{res}} x_t + W_{\mathrm{in}} u_t\big), \qquad \hat y_t = W_{\mathrm{out},t}^\top x_t.$$

Here $$W_{\mathrm{res}} \in \mathbb{R}^{\hat N \times \hat N}$$ is a sparse fixed reservoir with spectral radius $< 1$, $W_{\mathrm{in}} \in \mathbb{R}^{\hat N \times M}$ is fixed input weights, and $$W_{\mathrm{out},t} \in \mathbb{R}^{\hat N \times L}$$ is the time-varying readout trained online. The paper does not include an explicit bias term or concatenated $[x_t;u_t;1]$ readout, although it notes that such an extension can be added in other implementations if needed (Inoue et al., 4 Sep 2025).

Online training minimizes an exponentially discounted squared error,

$u_t \in \mathbb{R}^{M}$0

with RLS updates

$u_t \in \mathbb{R}^{M}$1

$u_t \in \mathbb{R}^{M}$2

Under the assumption of a true linear readout

$u_t \in \mathbb{R}^{M}$3

with independent Gaussian noise $u_t \in \mathbb{R}^{M}$4, the parameter covariance for $u_t \in \mathbb{R}^{M}$5 is

$u_t \in \mathbb{R}^{M}$6

This covariance is the basis of URPPI’s uncertainty-aware sampling (Inoue et al., 4 Sep 2025).

The reported implementation uses $u_t \in \mathbb{R}^{M}$7 for both tasks, $u_t \in \mathbb{R}^{M}$8, $u_t \in \mathbb{R}^{M}$9, $y_t \in \mathbb{R}^{L}$0 initialized uniformly in $y_t \in \mathbb{R}^{L}$1, and $y_t \in \mathbb{R}^{L}$2. Reservoir sparsity, spectral radius, input scaling, and leak rate are task-specific: for the Duffing system, sparsity $y_t \in \mathbb{R}^{L}$3, spectral radius $y_t \in \mathbb{R}^{L}$4, input scaling in $y_t \in \mathbb{R}^{L}$5, and $y_t \in \mathbb{R}^{L}$6; for the four-tank system, sparsity $y_t \in \mathbb{R}^{L}$7, spectral radius $y_t \in \mathbb{R}^{L}$8, input scaling in $y_t \in \mathbb{R}^{L}$9, and $f$0. Washout length and additional preprocessing are not specified, and the readout is updated online at every control step (Inoue et al., 4 Sep 2025).

3. MPPI integration and control-law construction

RPPI embeds the learned ESN directly inside MPPI rollout propagation. For rollout $f$1 and time $f$2,

$f$3

$f$4

where $f$5 is either the nominal readout, in standard RPPI/MPPI, or a perturbed sample, in URPPI. The trajectory cost for rollout $f$6 is

$f$7

The MPPI weighting and control update are

$f$8

Control perturbations are sampled as $f$9, with $$x_{t+1} = (1-\alpha)\,x_t + \alpha\, f\!\big(W_{\mathrm{res}} x_t + W_{\mathrm{in}} u_t\big), \qquad \hat y_t = W_{\mathrm{out},t}^\top x_t.$$0 typically set to $$x_{t+1} = (1-\alpha)\,x_t + \alpha\, f\!\big(W_{\mathrm{res}} x_t + W_{\mathrm{in}} u_t\big), \qquad \hat y_t = W_{\mathrm{out},t}^\top x_t.$$1; the horizon $$x_{t+1} = (1-\alpha)\,x_t + \alpha\, f\!\big(W_{\mathrm{res}} x_t + W_{\mathrm{in}} u_t\big), \qquad \hat y_t = W_{\mathrm{out},t}^\top x_t.$$2 (denoted $$x_{t+1} = (1-\alpha)\,x_t + \alpha\, f\!\big(W_{\mathrm{res}} x_t + W_{\mathrm{in}} u_t\big), \qquad \hat y_t = W_{\mathrm{out},t}^\top x_t.$$3 in the paper), rollout count $$x_{t+1} = (1-\alpha)\,x_t + \alpha\, f\!\big(W_{\mathrm{res}} x_t + W_{\mathrm{in}} u_t\big), \qquad \hat y_t = W_{\mathrm{out},t}^\top x_t.$$4, and temperature $$x_{t+1} = (1-\alpha)\,x_t + \alpha\, f\!\big(W_{\mathrm{res}} x_t + W_{\mathrm{in}} u_t\big), \qquad \hat y_t = W_{\mathrm{out},t}^\top x_t.$$5 are user-chosen (Inoue et al., 4 Sep 2025).

The control loop is specified per control step. First, plant output $$x_{t+1} = (1-\alpha)\,x_t + \alpha\, f\!\big(W_{\mathrm{res}} x_t + W_{\mathrm{in}} u_t\big), \qquad \hat y_t = W_{\mathrm{out},t}^\top x_t.$$6 is measured and the previous control is applied to the plant and ESN reservoir. Second, the ESN readout and precision matrix are updated by RLS. Third, uncertainty information $$x_{t+1} = (1-\alpha)\,x_t + \alpha\, f\!\big(W_{\mathrm{res}} x_t + W_{\mathrm{in}} u_t\big), \qquad \hat y_t = W_{\mathrm{out},t}^\top x_t.$$7 is read, $$x_{t+1} = (1-\alpha)\,x_t + \alpha\, f\!\big(W_{\mathrm{res}} x_t + W_{\mathrm{in}} u_t\big), \qquad \hat y_t = W_{\mathrm{out},t}^\top x_t.$$8 is chosen, and the readout sampling distribution

$$x_{t+1} = (1-\alpha)\,x_t + \alpha\, f\!\big(W_{\mathrm{res}} x_t + W_{\mathrm{in}} u_t\big), \qquad \hat y_t = W_{\mathrm{out},t}^\top x_t.$$9

is defined. Fourth, the nominal input sequence is initialized from the previous iteration, $$W_{\mathrm{res}} \in \mathbb{R}^{\hat N \times \hat N}$$0 input-noise sequences and $$W_{\mathrm{res}} \in \mathbb{R}^{\hat N \times \hat N}$$1 readout-weight copies are sampled, rollouts are simulated across the horizon, and costs are accumulated. Finally, the first element of the updated input sequence is applied, the horizon is shifted forward, and the procedure repeats (Inoue et al., 4 Sep 2025).

The generic control objective used in RPPI is

$$W_{\mathrm{res}} \in \mathbb{R}^{\hat N \times \hat N}$$2

For the Duffing oscillator, $$W_{\mathrm{res}} \in \mathbb{R}^{\hat N \times \hat N}$$3, $$W_{\mathrm{res}} \in \mathbb{R}^{\hat N \times \hat N}$$4, with input constraint $$W_{\mathrm{res}} \in \mathbb{R}^{\hat N \times \hat N}$$5. For the four-tank system, $$W_{\mathrm{res}} \in \mathbb{R}^{\hat N \times \hat N}$$6, $$W_{\mathrm{res}} \in \mathbb{R}^{\hat N \times \hat N}$$7, with box constraints $$W_{\mathrm{res}} \in \mathbb{R}^{\hat N \times \hat N}$$8 (Inoue et al., 4 Sep 2025).

4. URPPI and the exploration–exploitation mechanism

URPPI extends RPPI by explicitly incorporating uncertainty in the ESN readout. Its mechanism uses the RLS precision matrix $$W_{\mathrm{res}} \in \mathbb{R}^{\hat N \times \hat N}$$9 together with $< 1$0 to sample multiple readouts,

$< 1$1

In URPPI, readout perturbations are sampled at each prediction step, and the weighting integrates over both input-noise samples and output-weight samples by Monte Carlo (Inoue et al., 4 Sep 2025).

The intended behavioral effect is explicit in the formulation. Early in learning, $< 1$2 is large, indicating high uncertainty; sampled readout variations then induce broader variability in predicted outputs and in the weighted selection of control sequences, which favors exploration. As learning converges, $< 1$3 shrinks, readout samples cluster, and the weights concentrate on low-cost trajectories, producing a transition toward exploitation. The paper states that the uncertainty modulation is primarily through readout sampling: $< 1$4 is fixed per experiment, and no explicit scheduling of $< 1$5 or $< 1$6 is provided (Inoue et al., 4 Sep 2025).

Several practical heuristics are described as compatible with URPPI but not part of the paper’s implementation. These include exploration scaling via ESN uncertainty, such as

$< 1$7

or $< 1$8, and temperature scheduling $< 1$9. Because these heuristics are explicitly identified as outside the reported implementation, they should not be conflated with the empirical results (Inoue et al., 4 Sep 2025).

A common misconception is to treat URPPI as a generic Bayesian ensemble controller. The reported method does not use an ensemble or bootstrap, and it has no explicit calibration procedure beyond setting $W_{\mathrm{in}} \in \mathbb{R}^{\hat N \times M}$0. Its uncertainty model is the analytical covariance $W_{\mathrm{in}} \in \mathbb{R}^{\hat N \times M}$1 inherited from the RLS readout estimate (Inoue et al., 4 Sep 2025).

5. Benchmarks and reported empirical behavior

The first benchmark is the Duffing oscillator,

$W_{\mathrm{in}} \in \mathbb{R}^{\hat N \times M}$2

discretized by explicit Euler with $W_{\mathrm{in}} \in \mathbb{R}^{\hat N \times M}$3. The main experiment uses $W_{\mathrm{in}} \in \mathbb{R}^{\hat N \times M}$4, $W_{\mathrm{in}} \in \mathbb{R}^{\hat N \times M}$5, $W_{\mathrm{in}} \in \mathbb{R}^{\hat N \times M}$6, and $W_{\mathrm{in}} \in \mathbb{R}^{\hat N \times M}$7, distinct from a tuning set with all four parameters equal to $W_{\mathrm{in}} \in \mathbb{R}^{\hat N \times M}$8. The control weights are $W_{\mathrm{in}} \in \mathbb{R}^{\hat N \times M}$9 and $$W_{\mathrm{out},t} \in \mathbb{R}^{\hat N \times L}$$0; the reference switches between $$W_{\mathrm{out},t} \in \mathbb{R}^{\hat N \times L}$$1 every $$W_{\mathrm{out},t} \in \mathbb{R}^{\hat N \times L}$$2; and the horizon is $$W_{\mathrm{out},t} \in \mathbb{R}^{\hat N \times L}$$3. MPPI uses $$W_{\mathrm{out},t} \in \mathbb{R}^{\hat N \times L}$$4 rollouts, $$W_{\mathrm{out},t} \in \mathbb{R}^{\hat N \times L}$$5, and scalar $$W_{\mathrm{out},t} \in \mathbb{R}^{\hat N \times L}$$6. URPPI uses $$W_{\mathrm{out},t} \in \mathbb{R}^{\hat N \times L}$$7 output-weight samples and $$W_{\mathrm{out},t} \in \mathbb{R}^{\hat N \times L}$$8 (Inoue et al., 4 Sep 2025).

For the Duffing system, the reported results over 20 random seeds on a CPU multi-threaded implementation are that MPPI and URPPI achieve asymptotic tracking, whereas QPMPC does not converge to the reference. Over $$W_{\mathrm{out},t} \in \mathbb{R}^{\hat N \times L}$$9 steps, URPPI yields a time-integrated cost approximately $[x_t;u_t;1]$0 lower than QPMPC and approximately $[x_t;u_t;1]$1 lower than MPPI. The average runtime per control step is $[x_t;u_t;1]$2 for MPPI and $[x_t;u_t;1]$3 for URPPI. Spectral analysis reports that MPPI and URPPI produce broader frequency content than QPMPC, and that URPPI exhibits the widest spread early, for $[x_t;u_t;1]$4, which the paper interprets as effective exploration guided by uncertainty (Inoue et al., 4 Sep 2025).

The second benchmark is the four-tank process,

$[x_t;u_t;1]$5

$[x_t;u_t;1]$6

$[x_t;u_t;1]$7

with $[x_t;u_t;1]$8. The parameters are $[x_t;u_t;1]$9, $u_t \in \mathbb{R}^{M}$00, $u_t \in \mathbb{R}^{M}$01, $u_t \in \mathbb{R}^{M}$02, standard $u_t \in \mathbb{R}^{M}$03, and volume splits $u_t \in \mathbb{R}^{M}$04, $u_t \in \mathbb{R}^{M}$05, distinct from a tuning choice $u_t \in \mathbb{R}^{M}$06, $u_t \in \mathbb{R}^{M}$07. The cost uses $u_t \in \mathbb{R}^{M}$08, $u_t \in \mathbb{R}^{M}$09, $u_t \in \mathbb{R}^{M}$10, reference alternation between $u_t \in \mathbb{R}^{M}$11 and $u_t \in \mathbb{R}^{M}$12 every $u_t \in \mathbb{R}^{M}$13, horizon $u_t \in \mathbb{R}^{M}$14, and control constraint $u_t \in \mathbb{R}^{M}$15. MPPI uses $u_t \in \mathbb{R}^{M}$16, $u_t \in \mathbb{R}^{M}$17, $u_t \in \mathbb{R}^{M}$18; URPPI uses

$u_t \in \mathbb{R}^{M}$19

and $u_t \in \mathbb{R}^{M}$20 (Inoue et al., 4 Sep 2025).

For the four-tank system, MPPI and URPPI track references reliably, while QPMPC deviates. Over $u_t \in \mathbb{R}^{M}$21 steps, URPPI reports a time-integrated cost approximately $u_t \in \mathbb{R}^{M}$22 lower than QPMPC and approximately $u_t \in \mathbb{R}^{M}$23 lower than MPPI. Runtime per control step is $u_t \in \mathbb{R}^{M}$24 for MPPI and $u_t \in \mathbb{R}^{M}$25 for URPPI. Spectral analysis again indicates broader frequency content for MPPI and URPPI than for QPMPC, with the widest URPPI spread around $u_t \in \mathbb{R}^{M}$26 (Inoue et al., 4 Sep 2025).

6. Computational profile, limitations, and relation to adjacent research

The reported per-step computational complexity is

$u_t \in \mathbb{R}^{M}$27

More specifically, ESN-RLS updates cost $u_t \in \mathbb{R}^{M}$28; sampling $u_t \in \mathbb{R}^{M}$29 input sequences costs $u_t \in \mathbb{R}^{M}$30; sampling $u_t \in \mathbb{R}^{M}$31 readout copies after a Cholesky factorization of $u_t \in \mathbb{R}^{M}$32 costs $u_t \in \mathbb{R}^{M}$33; trajectory simulation across $u_t \in \mathbb{R}^{M}$34 steps costs $u_t \in \mathbb{R}^{M}$35; and cost evaluation and weighted averaging add $u_t \in \mathbb{R}^{M}$36. The experiments use CPU multithreading with 16 threads; Monte Carlo sampling of inputs and readout layers is described as embarrassingly parallel; GPU parallelization is compatible but was not used. Memory footprint is not reported (Inoue et al., 4 Sep 2025).

The framework’s theoretical guarantees are deliberately limited. The assumptions are output-only measurements, ESN approximation of the input-output map through the reservoir state, and RLS stability conditions requiring persistent excitation and invertibility of $u_t \in \mathbb{R}^{M}$37, with $u_t \in \mathbb{R}^{M}$38 in the experiments. No formal observability or identifiability analysis is given, and there is no formal closed-loop stability proof. The paper further notes that MPPI’s weighting is sensitive to $u_t \in \mathbb{R}^{M}$39: too small a temperature may cause sample degeneracy, whereas too large a value may slow convergence. Performance is reported to depend on reservoir hyperparameters, MPPI parameters, and URPPI uncertainty scaling through $u_t \in \mathbb{R}^{M}$40 (Inoue et al., 4 Sep 2025).

Several implementation recommendations are presented as practical guidance rather than benchmark-defining requirements. These include spectral radius below $u_t \in \mathbb{R}^{M}$41, leak rates matched to plant speed, reservoir sizes of a few hundred, input scaling that avoids activation saturation, short horizons for real-time feasibility, and a few hundred to a thousand rollouts. The paper also notes that a brief washout is common when initializing ESNs offline, although washout is not specified in the RPPI experiments. These points matter because they distinguish the reported method from a misconception that RPPI is parameter-free; its performance is explicitly described as sensitive to both reservoir and MPPI design choices (Inoue et al., 4 Sep 2025).

A related theoretical line of work analyzes reservoir computers with path-integral methods in the thermodynamic limit. "Path integral approach to universal dynamics of reservoir computers" derives a generating-functional description, classifies random reservoirs into universality classes through the large-$u_t \in \mathbb{R}^{M}$42 cumulant-generating function $u_t \in \mathbb{R}^{M}$43, links these classes to eigenvalue spectra, and reports that computational performance in inference and forecasting tasks is often strongest near phase boundaries, including non-chaotic ones (Haruna et al., 2021). This suggests a theoretical backdrop for the broader conjunction of reservoir models and path-integral formalisms. However, that work addresses random-network dynamics, synchronization, inference, and forecasting rather than the online nonlinear control architecture introduced as RPPI (Haruna et al., 2021).