---
title: Planner–Actor–Critic (PAC) Architectures
url: https://www.emergentmind.com/topics/planner-actor-critic-pac
type: topic
---

# Planner–Actor–Critic (PAC) Architectures

Planner–Actor–Critic (PAC) denotes a class of architectures that augment actor–critic learning with an explicit planning, coordination, or reflective module. In the cited literature, the term does not refer to a single canonical algorithm; rather, it has been used for several related decompositions in which a planner generates a high-level plan, reference, or intermediate representation, an actor executes or decodes that guidance into actions, and a critic evaluates plans, values, or outputs to improve subsequent behavior. Representative instantiations include human-centered symbolic planning and learning in PACMAN, stochastic step-wise deformable image registration in SPAC, layered planning–tracking control with a learned dual network, and multi-agent 3D modeling with iterative critique [1906.07268, 1909.09209, 2112.07415, 2408.01639, 2601.05016].

## 1. Historical emergence and scope of the term

The earliest formulation in the supplied corpus is PACMAN, introduced in 2019 as a human-centered architecture that combines symbolic planning, Actor–Critic reinforcement learning, and human teaching for goal-directed decision-making. In that setting, the planner uses prior, high-level, deterministic symbolic knowledge, the actor–critic pair fine-tunes behavior from environmental rewards, and human feedback can directly shape policy updates [1906.07268, 1909.09209].

In 2021, the term reappeared in medical image analysis through Stochastic Planner-Actor-Critic (SPAC), where the planner was no longer symbolic but instead a stochastic neural component producing a low-dimensional plan to make high-dimensional continuous deformable image registration tractable. The planner mediates between image state and deformation action, while the critic evaluates the plan rather than the full action [2112.07415].

A 2024 layered control formulation further broadened the pattern by jointly training a trajectory planner and a tracking controller via actor–critic learning, with an explicit dual network that coordinates the two layers. Here the planner produces reference trajectories, the lower layer tracks them, and the learned dual variable perturbs the reference to reach effective consensus between planning and execution [2408.01639].

By 2026, a Planner–Actor–Critic framework was applied to agent-augmented 3D modeling. In that usage, the planner decomposes user goals into a step-by-step todo list, the actor executes modeling commands via Blender-MCP, and the critic analyzes screenshots and execution results to determine whether further iteration is warranted [2601.05016].

This trajectory suggests that PAC is best understood as a reusable architectural motif rather than a domain-specific algorithm.

## 2. Recurrent structural pattern

Across the cited formulations, PAC separates high-level deliberation from low-level execution and evaluation. The precise semantics of “planner,” “actor,” and “critic” vary by domain, but the decomposition is consistent in intent: reduce search complexity, improve coordination, or enable iterative refinement beyond a direct state-to-action actor–critic mapping.

| Formulation | Planner role | Actor and critic roles |
|---|---|---|
| PACMAN [1906.07268, 1909.09209] | Generates goal-directed symbolic plans using prior knowledge and the current stochastic policy | Actor updates $\pi_\theta(a \mid s)$; critic estimates $V_x(s)$; human feedback may replace TD error |
| SPAC [2112.07415] | Produces low-dimensional stochastic plan $p_t$ from state $s_t$ | Actor decodes plan into high-dimensional deformation field; critic evaluates $(s_t,p_t)$ |
| Layered control [2408.01639] | Optimizes reference trajectory $r_{0:T}$ | Tracking controller executes controls; tracking value function predicts tracking cost; dual network coordinates planner and tracker |
| 3D modeling PAC [2601.05016] | Creates and updates a step-by-step modeling plan | Actor executes plan steps and scripts; critic inspects scene state and screenshots and suggests next steps |

In standard actor–critic, the policy and critic typically operate directly on action selection and value estimation. Several PAC variants explicitly depart from that arrangement. SPAC states the contrast most directly: classic actor–critic maps state to action, whereas SPAC inserts an intermediate low-dimensional plan so that the critic operates in plan space rather than full action space [2112.07415]. In PACMAN, the planner constrains exploration through symbolic feasibility, and in layered control the planner is not simply a policy head but an optimization layer informed by a learned tracking value [1909.09209, 2408.01639].

## 3. Mathematical and algorithmic formulations

SPAC formalizes planning as a stochastic latent layer. Given state $s_t=(I_F, I_{M_t})$, the planner samples a plan
$$
p_t \sim \kappa_\psi(p_t \mid s_t),
$$
and the actor generates the deformation action
$$
a_t = \pi_\phi(a_t \mid p_t).
$$
Registration proceeds step-wise through an accumulated deformation field
$$
\Omega^t_{\psi,\phi} =
\begin{cases}
0 & t=0 \\
{\cal C}(a_t,\Omega^{t-1}_{\psi,\phi}) & t>0,
\end{cases}
$$
with composition
$$
{\cal C}(a_t,\Omega^{t-1}_{\psi,\phi}) = \Omega^{t-1}_{\psi,\phi} + \left(a_t \circ \Omega^{t-1}_{\psi,\phi}\right).
$$
The critic evaluates the soft Q-value $Q_\theta(s_t,p_t)$, and the framework combines entropy-regularized reinforcement learning with unsupervised registration terms based on local normalized cross-correlation and total variation regularization [2112.07415].

PACMAN uses a different but equally explicit update loop. After sample-based symbolic planning, the critic computes the temporal-difference error
$$
\delta_i = r_i + \gamma V_{x_i}(s_{i+1}) - V_{x_i}(s_i),
$$
updates the value function as
$$
x_{i+1} = x_i + \alpha \, \delta_i \nabla V_{x_i}(s_i),
$$
and updates the actor by
$$
\theta_{i+1} = \theta_i + \beta \, \delta_i \nabla \log \pi_{\theta_i}(a_i\mid s_i).
$$
If human feedback $f_i$ is present, $\delta_i$ is replaced by $f_i$, following the interpretation of feedback as a policy-dependent advantage signal [1909.09209].

The layered control formulation rewrites the optimal control problem using a redundant variable and a partial augmented Lagrangian, yielding a nested decomposition into planning and tracking. Its planner is parameterized by the learned tracking value and the dual network:
$$
\pi^\mathrm{plan}_{\theta,\psi}(\xi)
=
\arg\min_{r\in\mathcal{X}}
\mathcal{C}(r) + p_\psi^{\pi^\mathrm{track}}(r + v_\theta(\xi); \xi).
$$
The dual network is updated by gradient ascent in the direction of the tracking mismatch:
$$
\theta^+ \leftarrow \theta + \eta \nabla_\theta \left[ \frac{1}{B} \sum_{i=1}^B (r_i - x_i)^\top v_\theta(\xi_i) \right].
$$
The gradient is taken only on $v_\theta$, not through trajectory generation or tracking, to match the spirit of dual ascent [2408.01639].

## 4. Sources of guidance: prior knowledge, unsupervised objectives, and human input

One of the main distinctions among PAC formulations is the source of supervisory structure.

In PACMAN, guidance comes from symbolic prior knowledge represented in action language $\mathcal{BC}$ and solved by ASP tools such as Clingo. The symbolic planner uses the current stochastic policy as a sampling oracle to generate possible plans from initial state $I$ to goal state $G$. This creates a bidirectional loop: planning is biased by the learned policy, while learning is constrained by symbolic structure. Human feedback is integrated directly into policy optimization and can be helpful, infrequent, inconsistent, misleading, or combinations thereof [1906.07268, 1909.09209].

SPAC, by contrast, is explicitly unsupervised. It does not require ground-truth deformation fields, and its registration objective uses similarity and regularization terms:
$$
G(I_F, I_M \circ \Omega^t_{\psi,\phi}) = NCC(I_F, I_M \circ \Omega^t_{\psi,\phi}),
$$
$$
R(\Omega^t_{\psi,\phi}) = \|\nabla \Omega^t_{\psi,\phi}\|_2^2.
$$
Its registration environment computes rewards as Dice score improvement using unsupervised segmentation maps obtained by K-means clustering [2112.07415].

The layered control framework assumes only a simulator with black-box dynamics. It jointly learns a trajectory planner, a tracking controller, and a dual network that preemptively corrects the reference signal handed to the tracker. The coordination signal is therefore neither symbolic nor purely reward-based; it is an explicit consensus mechanism derived from the augmented-Lagrangian reformulation [2408.01639].

In the 3D modeling formulation, supervision is human-in-the-loop and operational rather than scalar-reward-centric. The planner incorporates user and critic input, the actor executes modeling steps through discovered MCP tools, and the critic analyzes viewport screenshots and execution results. The loop is Planner $\rightarrow$ Actor $\rightarrow$ Critic, with the human supervisor able to review, intervene, redirect, or modify plans at any stage [2601.05016].

## 5. Representative applications and reported empirical behavior

PACMAN was evaluated in Four Rooms and Taxi. The reported human feedback settings were Ideal, Infrequent, Inconsistent, and Infrequent + Inconsistent. Compared methods included PACMAN, PACMAN without the symbolic planner, TAMER+RL reward shaping, and BQL Reward Shaping. The reported findings were a significant jump-start at the early stage of learning, rapid convergence with small variance, and robustness to inconsistent, infrequent, and misleading feedback. The ablation without the symbolic planner reduced both jump-start and robustness [1906.07268, 1909.09209].

SPAC was evaluated on several 2D and 3D medical image datasets, including MNIST, 2D MRI (LPBA), and 3D CT (Liver, including LSPIG). The reported metrics were Dice score, runtime, and parameter count. The paper reports that SPAC outperforms SyN, Elastix, LDDMM, VoxelMorph/VM-diff, R2N2, RCN, and SYMNet; that performance increases steadily with the number of steps; and that removing the planner or reverting critic evaluation to the actor’s action leads to worse performance. The strongest improvements are described for datasets with large deformations [2112.07415].

The layered control method was validated in unconstrained LQR, constrained LQR, and a nonlinear unicycle model. For unconstrained LQR, the reported performance was near-optimal cost within 1–3% of true optimum and almost perfect tracking over varying system sizes, with tracking error reductions over an order of magnitude relative to a heuristic no-dual approach. For nonlinear unicycle control, the reported cost was comparable to iLQR, but without access to the system model, and the learned dual network improved tracking performance even when cost improvement was subtler [2408.01639].

The 3D modeling PAC framework evaluated primitive tasks such as a birthday cake with layers and candles, a square dining table with four legs, and a classic car using only primitives. The reported evaluation used geometry count, vertex count, similarity between iterations, and human Likert-scale ratings for task alignment and aesthetics. The findings were progressive refinement over 1, 3, and 5 PAC iterations, increased geometric complexity, improvements in geometric accuracy, aesthetic quality, and task completion rates relative to single-prompt modeling, and a tendency for aesthetic and task-alignment scores to increase from the 1st to the 3rd iteration before plateauing [2601.05016].

## 6. Interpretation, ambiguities, and limitations

A common misconception is that Planner–Actor–Critic denotes one settled formalism. The cited literature indicates otherwise. In PACMAN, the planner is a symbolic reasoning system; in SPAC it is a stochastic low-dimensional latent plan; in layered control it is a trajectory optimizer coupled to a learned dual network; and in 3D modeling it is a task-decomposition agent operating over tools and screenshots. This suggests a shared decomposition principle rather than a uniform algorithmic template [1909.09209, 2112.07415, 2408.01639, 2601.05016].

A second misconception is that the critic must always evaluate the executed action. SPAC explicitly evaluates plans in the joint space of state and plan, not state and action, to make value function approximation more tractable. In layered control, the critic role is partly played by a tracking value function that predicts total tracking cost for a reference trajectory. In the 3D modeling system, the critic is an analysis agent that provides structured feedback on scene quality and whether iteration should continue [2112.07415, 2408.01639, 2601.05016].

Reported limitations are likewise domain-specific. In 3D modeling, the paper notes script reliability issues, incomplete incorporation of feedback in later iterations, toolset limitations that currently favor primitive modeling, and variable impact of human feedback. In layered control, optimal consensus $r^* = x^*$ occurs if and only if the dual variable $\nu^*$ attains its local optimum, and under suboptimal tracking the dual learning converges to a ball whose radius is determined by the tracking error size. In SPAC, the planner is motivated by the difficulty of high-dimensional continuous state and action spaces in conventional reinforcement learning, indicating that tractability remains a central design concern [2601.05016, 2408.01639, 2112.07415].

The acronym “PAC” is also ambiguous outside Planner–Actor–Critic. “PAC-Bayesian Soft Actor-Critic Learning” uses PAC in the Probably Approximately Correct sense, not as Planner–Actor–Critic. That paper nevertheless notes a conceptual link: its multiple-shooting action selection, guided by a probabilistic critic, is described as reminiscent of planner–actor–critic designs because a planner-like search operates over actor and critic samples before action commitment [2301.12776].

Source: https://www.emergentmind.com/topics/planner-actor-critic-pac