PPAC Framework: RL and Optimal Control

• The PPAC framework is a modular reinforcement learning architecture that combines trajectory proposal, planning, policy optimization, and value estimation.
• It leverages Bayesian MPC with constrained Stein Variational Gradient Descent to refine control trajectories, ensuring safety and stability.
• The integration with soft actor-critic methods enhances sample efficiency and sim-to-real transfer in complex continuous control tasks.

The Proposer–Predictor–Actor–Critic (PPAC) framework is a modular reinforcement learning (RL) architecture that integrates trajectory proposal, planning, policy optimization, and value estimation into a unified structure. In the context of Q-STAC (Q-guided STein variational model predictive Actor-Critic), the PPAC approach synergizes Bayesian Model Predictive Control (MPC) with actor-critic RL via constrained Stein Variational Gradient Descent (SVGD), enabling principled optimal control under explicit safety and distributional constraints, guided directly by soft Q-value objectives rather than handcrafted reward shaping or cost design (Cai et al., 9 Jul 2025).

1. Modular Structure of PPAC in Q-STAC

The PPAC framework in Q-STAC decomposes the RL and planning pipeline into four synergistic modules:

1. Proposer: Generates candidate horizon-$H$ control trajectories as "particles" leveraging a parameterized Gaussian policy (MLP-parameterized $\pi_\phi$), subsequently refined by SVGD.
2. Predictor: Guides proposal refinement by evaluating candidate trajectories using soft Q-value accumulation, formalized as a log-likelihood under an optimality posterior.
3. Actor: Encodes the state-conditional prior over control sequences with a sequence-generating Gaussian MLP, and is trained via maximum-entropy KL minimization (soft actor-critic objective).
4. Critic: Estimates action-value functions using soft-Q learning, employing a Bellman mean-squared error objective and target networks for stability.

Each component functions as a semi-autonomous sub-system, interfacing via shared value networks, distributions over sequences, and explicit constraints.

2. Proposer: Stein Variational Gradient Descent with Constraints

At every time step $t$, the proposer samples $m$ control sequence particles $U^i = (u^i_1, \ldots, u^i_H)$ from the Gaussian prior output by $\pi_\phi$:

• For $h=1\ldots H$:
  • $\mu_h, \sigma_h \leftarrow \text{MLP}_\phi(x_t)$
  • $U^i_h \sim \mathcal{N}(\mu_h, \sigma_h)$ for $i=1\ldots m$

The particle set undergoes $\pi_\phi$0 iterations of SVGD to approximate the posterior over control sequences conditioned on maximizing expected Q-values. In unconstrained SVGD, the update

$\pi_\phi$1

uses

$\pi_\phi$2

Q-STAC extends this by introducing an augmented Lagrangian bound constraint:

• $\pi_\phi$3
• $\pi_\phi$4

The SVGD gradient uses $\pi_\phi$5 instead of $\pi_\phi$6, and particles are projected within the prior's $\pi_\phi$7 bounds, ensuring numerical stability and safety.

3. Predictor: Q-Guided Planning Objective

The predictor module quantifies the "likelihood" of a trajectory $\pi_\phi$8 under the optimality event $\pi_\phi$9 as

$t$0

Thus, the log-likelihood is

$t$1

where $t$2, and $t$3 is the Gaussian prior. The planning objective for SVGD becomes the maximization of $t$4 regularized by prior proximity and the aforementioned constraints.

4. Actor: State-Conditional Policy and Soft-Actor-Critic Loss

The actor is parameterized as an MLP mapping state $t$5 to $t$6, defining the Gaussian prior $t$7. After SVGD refinement, the resulting action distribution entropy is estimated in closed form as per S$t$8AC. The actor parameters are optimized to minimize the soft actor-critic (SAC) policy objective:

$t$9

This expresses an implicit maximum-entropy ("soft") policy improvement step, ensuring expressive but Q-aligned policy distributions.

5. Critic: Soft-Q Learning and Target Network Stabilization

The critic maintains a soft-Q network $m$0 and target $m$1. Its update minimizes the Bellman mean squared error:

$m$2

where

$m$3

The target network parameters $m$4 are updated as an exponential moving average of $m$5, providing training stability.

6. Integrated Algorithm and Empirical Performance

The Q-STAC algorithm alternates between trajectory proposal/refinement, evaluation, policy improvement, and critic updates, detailed in the following process:

Step	Description	Role
1	Observe $m$6	State observation
2	Actor (Proposer): Compute $m$7 MLP$m$8; sample $m$9 sequences	Action priorization
3	Predictor (SVGD loop): For $U^i = (u^i_1, \ldots, u^i_H)$0 steps, roll out $U^i = (u^i_1, \ldots, u^i_H)$1 using $U^i = (u^i_1, \ldots, u^i_H)$2, evaluate $U^i = (u^i_1, \ldots, u^i_H)$3, update via SVGD + constraints, dual ascent on $U^i = (u^i_1, \ldots, u^i_H)$4	Planning/refinement
4	Select one trajectory $U^i = (u^i_1, \ldots, u^i_H)$5: random for exploration, or highest $U^i = (u^i_1, \ldots, u^i_H)$6 at test	Selection/execution
5	Apply $U^i = (u^i_1, \ldots, u^i_H)$7, observe $U^i = (u^i_1, \ldots, u^i_H)$8, store transition in buffer	Environment step
6	Critic & actor updates (SAC-style) over minibatch	Policy/value update

Empirical comparisons show that Q-STAC, governed by the PPAC architecture, achieves higher sample efficiency (requiring 30–70% fewer environment steps for 80% optimal return), improved safety (by enforcing the $U^i = (u^i_1, \ldots, u^i_H)$9 prior constraint), and robust sim-to-real transfer performance without additional fine-tuning—demonstrating ≈93% obstacle-avoidance and 80% pick-and-reach success on Kinova arms (Cai et al., 9 Jul 2025).

7. Context and Implications

The PPAC structure in Q-STAC exemplifies the trend toward integrating probabilistic planning (MPC), variational inference methods (SVGD), and expressive RL architectures (SAC-style actor-critic) for complex continuous control. By eliminating explicit cost function engineering in favor of Q-guided objectives and enforcing safety via constrained inference, this approach addresses common RL limitations such as data inefficiency, unsafe exploration, and poor long-horizon planning. A plausible implication is that modular, constrained PPAC-like systems enable high-performance RL in domains requiring both inductive generalization and rigorous safety guarantees.