PPAC Framework: RL and Optimal Control

• The PPAC framework is a modular reinforcement learning architecture that combines trajectory proposal, planning, policy optimization, and value estimation.
• It leverages Bayesian MPC with constrained Stein Variational Gradient Descent to refine control trajectories, ensuring safety and stability.
• The integration with soft actor-critic methods enhances sample efficiency and sim-to-real transfer in complex continuous control tasks.

The Proposer–Predictor–Actor–Critic (PPAC) framework is a modular reinforcement learning (RL) architecture that integrates trajectory proposal, planning, policy optimization, and value estimation into a unified structure. In the context of Q-STAC (Q-guided STein variational model predictive Actor-Critic), the PPAC approach synergizes Bayesian Model Predictive Control (MPC) with actor-critic RL via constrained Stein Variational Gradient Descent (SVGD), enabling principled optimal control under explicit safety and distributional constraints, guided directly by soft Q-value objectives rather than handcrafted reward shaping or cost design (Cai et al., 9 Jul 2025).

1. Modular Structure of PPAC in Q-STAC

The PPAC framework in Q-STAC decomposes the RL and planning pipeline into four synergistic modules:

1. Proposer: Generates candidate horizon-$H$ control trajectories as "particles" leveraging a parameterized Gaussian policy (MLP-parameterized $\pi_\phi$), subsequently refined by SVGD.
2. Predictor: Guides proposal refinement by evaluating candidate trajectories using soft Q-value accumulation, formalized as a log-likelihood under an optimality posterior.
3. Actor: Encodes the state-conditional prior over control sequences with a sequence-generating Gaussian MLP, and is trained via maximum-entropy KL minimization (soft actor-critic objective).
4. Critic: Estimates action-value functions using soft-Q learning, employing a Bellman mean-squared error objective and target networks for stability.

Each component functions as a semi-autonomous sub-system, interfacing via shared value networks, distributions over sequences, and explicit constraints.

2. Proposer: Stein Variational Gradient Descent with Constraints

At every time step $t$, the proposer samples $m$ control sequence particles $U^i = (u^i_1, \ldots, u^i_H)$ from the Gaussian prior output by $\pi_\phi$:

• For $h=1\ldots H$:
  • $\mu_h, \sigma_h \leftarrow \text{MLP}_\phi(x_t)$
  • $U^i_h \sim \mathcal{N}(\mu_h, \sigma_h)$ for $i=1\ldots m$

The particle set undergoes $K$ iterations of SVGD to approximate the posterior over control sequences conditioned on maximizing expected Q-values. In unconstrained SVGD, the update

$U^i \leftarrow U^i + \epsilon\,\hat{\phi}^*(U^i)$

uses

$$\hat{\phi}^*(U^i) = \frac{1}{m} \sum_{j=1}^m \left[ k(U^j, U^i) \nabla_{U^j} \log p(U^j) + \nabla_{U^j} k(U^j, U^i) \right]$$

Q-STAC extends this by introducing an augmented Lagrangian bound constraint:

• $$g(U) = \mathrm{clamp}(U, \mu-3\sigma, \mu+3\sigma) - U$$
• $$\mathcal{L}(U, \lambda) = \log p(U \mid O_\tau; f, x) - \lambda^\top g(U) - \frac{c}{2} \|g(U)\|^2$$

The SVGD gradient uses $\nabla_U \mathcal{L}$ instead of $\nabla_U \log p(U)$, and particles are projected within the prior's $3\sigma$ bounds, ensuring numerical stability and safety.

3. Predictor: Q-Guided Planning Objective

The predictor module quantifies the "likelihood" of a trajectory $\tau^i = (X^i, U^i)$ under the optimality event $O_\tau$ as

$$p(O_\tau \mid \tau^i) \propto \exp\left( \sum_{h=0}^H Q_{\text{soft}}(x^i_{t+h}, u^i_{t+h}) \right)$$

Thus, the log-likelihood is

$$\log p(U^i \mid O_\tau; f, x) = Q_{\text{soft}}[\tau^i] + \log q_\phi^0(U^i; x)$$

where $$Q_{\text{soft}}[\tau^i] = \sum_{h=0}^H Q_{\text{soft}}(x_{t+h}^i, u_{t+h}^i)$$, and $q_\phi^0$ is the Gaussian prior. The planning objective for SVGD becomes the maximization of $Q_{\text{soft}}[\tau]$ regularized by prior proximity and the aforementioned constraints.

4. Actor: State-Conditional Policy and Soft-Actor-Critic Loss

The actor is parameterized as an MLP mapping state $x_t$ to $\{\mu_h, \sigma_h\}_{h=1}^H$, defining the Gaussian prior $$q_\phi^0(U; x) = \prod_{h=1}^H \mathcal{N}(u_h; \mu_h, \sigma_h)$$. After SVGD refinement, the resulting action distribution entropy is estimated in closed form as per S$^2$AC. The actor parameters are optimized to minimize the soft actor-critic (SAC) policy objective:

$$J_\pi(\phi) = \mathbb{E}_{s \sim D} \left[ D_{\text{KL}}\left( \pi_\phi(\cdot \mid s) \bigg\| \frac{\exp Q_\theta(s, \cdot)}{Z} \right) \right]$$

This expresses an implicit maximum-entropy ("soft") policy improvement step, ensuring expressive but Q-aligned policy distributions.

5. Critic: Soft-Q Learning and Target Network Stabilization

The critic maintains a soft-Q network $Q_\theta(s, a)$ and target $Q_{\bar{\theta}}(s, a)$. Its update minimizes the Bellman mean squared error:

$$J_Q(\theta) = \mathbb{E}_{(s,a,r,s') \sim D} \left[ \frac{1}{2} \left( Q_\theta(s, a) - (r + \gamma V_{\bar{\theta}}(s')) \right)^2 \right]$$

where

$$V_{\bar{\theta}}(s') = \mathbb{E}_{a' \sim \pi} [Q_{\bar{\theta}}(s', a') - \alpha \log \pi(a' \mid s')]$$

The target network parameters $\bar{\theta}$ are updated as an exponential moving average of $\theta$, providing training stability.

6. Integrated Algorithm and Empirical Performance

The Q-STAC algorithm alternates between trajectory proposal/refinement, evaluation, policy improvement, and critic updates, detailed in the following process:

Step	Description	Role
1	Observe $x_t$	State observation
2	Actor (Proposer): Compute $\mu_{1:H}, \sigma_{1:H} \leftarrow$ MLP$_\phi(x_t)$; sample $m$ sequences	Action priorization
3	Predictor (SVGD loop): For $K$ steps, roll out $U^i$ using $f$, evaluate $Q_\text{soft}$, update via SVGD + constraints, dual ascent on $\lambda$	Planning/refinement
4	Select one trajectory $j$: random for exploration, or highest $Q$ at test	Selection/execution
5	Apply $u_t=U^j_1$, observe $(x_{t+1}, r_t)$, store transition in buffer	Environment step
6	Critic & actor updates (SAC-style) over minibatch	Policy/value update

Empirical comparisons show that Q-STAC, governed by the PPAC architecture, achieves higher sample efficiency (requiring 30–70% fewer environment steps for 80% optimal return), improved safety (by enforcing the $3\sigma$ prior constraint), and robust sim-to-real transfer performance without additional fine-tuning—demonstrating ≈93% obstacle-avoidance and 80% pick-and-reach success on Kinova arms (Cai et al., 9 Jul 2025).

7. Context and Implications

The PPAC structure in Q-STAC exemplifies the trend toward integrating probabilistic planning (MPC), variational inference methods (SVGD), and expressive RL architectures (SAC-style actor-critic) for complex continuous control. By eliminating explicit cost function engineering in favor of Q-guided objectives and enforcing safety via constrained inference, this approach addresses common RL limitations such as data inefficiency, unsafe exploration, and poor long-horizon planning. A plausible implication is that modular, constrained PPAC-like systems enable high-performance RL in domains requiring both inductive generalization and rigorous safety guarantees.