Planner–Actor–Critic (PAC) Architectures

• Planner–Actor–Critic (PAC) is a modular framework that separates high-level planning, low-level action execution, and critique evaluation to improve learning efficiency and decision-making.
• Its design supports diverse implementations including symbolic planning in PACMAN, stochastic latent plans in SPAC, dual network coordination in layered control, and iterative refinement in 3D modeling.
• Empirical evaluations in navigation, medical image registration, control systems, and 3D modeling demonstrate PAC's capability to boost convergence speed, robustness, and overall performance.

Planner–Actor–Critic (PAC) denotes a class of architectures that augment actor–critic learning with an explicit planning, coordination, or reflective module. In the cited literature, the term does not refer to a single canonical algorithm; rather, it has been used for several related decompositions in which a planner generates a high-level plan, reference, or intermediate representation, an actor executes or decodes that guidance into actions, and a critic evaluates plans, values, or outputs to improve subsequent behavior. Representative instantiations include human-centered symbolic planning and learning in PACMAN, stochastic step-wise deformable image registration in SPAC, layered planning–tracking control with a learned dual network, and multi-agent 3D modeling with iterative critique (Lyu et al., 2019, Lyu et al., 2019, Luo et al., 2021, Yang et al., 2024, Gao et al., 8 Jan 2026).

1. Historical emergence and scope of the term

The earliest formulation in the supplied corpus is PACMAN, introduced in 2019 as a human-centered architecture that combines symbolic planning, Actor–Critic reinforcement learning, and human teaching for goal-directed decision-making. In that setting, the planner uses prior, high-level, deterministic symbolic knowledge, the actor–critic pair fine-tunes behavior from environmental rewards, and human feedback can directly shape policy updates (Lyu et al., 2019, Lyu et al., 2019).

In 2021, the term reappeared in medical image analysis through Stochastic Planner-Actor-Critic (SPAC), where the planner was no longer symbolic but instead a stochastic neural component producing a low-dimensional plan to make high-dimensional continuous deformable image registration tractable. The planner mediates between image state and deformation action, while the critic evaluates the plan rather than the full action (Luo et al., 2021).

A 2024 layered control formulation further broadened the pattern by jointly training a trajectory planner and a tracking controller via actor–critic learning, with an explicit dual network that coordinates the two layers. Here the planner produces reference trajectories, the lower layer tracks them, and the learned dual variable perturbs the reference to reach effective consensus between planning and execution (Yang et al., 2024).

By 2026, a Planner–Actor–Critic framework was applied to agent-augmented 3D modeling. In that usage, the planner decomposes user goals into a step-by-step todo list, the actor executes modeling commands via Blender-MCP, and the critic analyzes screenshots and execution results to determine whether further iteration is warranted (Gao et al., 8 Jan 2026).

This trajectory suggests that PAC is best understood as a reusable architectural motif rather than a domain-specific algorithm.

2. Recurrent structural pattern

Across the cited formulations, PAC separates high-level deliberation from low-level execution and evaluation. The precise semantics of “planner,” “actor,” and “critic” vary by domain, but the decomposition is consistent in intent: reduce search complexity, improve coordination, or enable iterative refinement beyond a direct state-to-action actor–critic mapping.

Formulation	Planner role	Actor and critic roles
PACMAN (Lyu et al., 2019, Lyu et al., 2019)	Generates goal-directed symbolic plans using prior knowledge and the current stochastic policy	Actor updates $\pi_\theta(a \mid s)$; critic estimates $V_x(s)$; human feedback may replace TD error
SPAC (Luo et al., 2021)	Produces low-dimensional stochastic plan $p_t$ from state $s_t$	Actor decodes plan into high-dimensional deformation field; critic evaluates $(s_t,p_t)$
Layered control (Yang et al., 2024)	Optimizes reference trajectory $r_{0:T}$	Tracking controller executes controls; tracking value function predicts tracking cost; dual network coordinates planner and tracker
3D modeling PAC (Gao et al., 8 Jan 2026)	Creates and updates a step-by-step modeling plan	Actor executes plan steps and scripts; critic inspects scene state and screenshots and suggests next steps

In standard actor–critic, the policy and critic typically operate directly on action selection and value estimation. Several PAC variants explicitly depart from that arrangement. SPAC states the contrast most directly: classic actor–critic maps state to action, whereas SPAC inserts an intermediate low-dimensional plan so that the critic operates in plan space rather than full action space (Luo et al., 2021). In PACMAN, the planner constrains exploration through symbolic feasibility, and in layered control the planner is not simply a policy head but an optimization layer informed by a learned tracking value (Lyu et al., 2019, Yang et al., 2024).

3. Mathematical and algorithmic formulations

SPAC formalizes planning as a stochastic latent layer. Given state $s_t=(I_F, I_{M_t})$, the planner samples a plan

$p_t \sim \kappa_\psi(p_t \mid s_t),$

and the actor generates the deformation action

$a_t = \pi_\phi(a_t \mid p_t).$

Registration proceeds step-wise through an accumulated deformation field

$$\Omega^t_{\psi,\phi} = \begin{cases} 0 & t=0 \ {\cal C}(a_t,\Omega^{t-1}_{\psi,\phi}) & t>0, \end{cases}$$

with composition

$V_x(s)$0

The critic evaluates the soft Q-value $V_x(s)$1, and the framework combines entropy-regularized reinforcement learning with unsupervised registration terms based on local normalized cross-correlation and total variation regularization (Luo et al., 2021).

PACMAN uses a different but equally explicit update loop. After sample-based symbolic planning, the critic computes the temporal-difference error

$V_x(s)$2

updates the value function as

$V_x(s)$3

and updates the actor by

$V_x(s)$4

If human feedback $V_x(s)$5 is present, $V_x(s)$6 is replaced by $V_x(s)$7, following the interpretation of feedback as a policy-dependent advantage signal (Lyu et al., 2019).

The layered control formulation rewrites the optimal control problem using a redundant variable and a partial augmented Lagrangian, yielding a nested decomposition into planning and tracking. Its planner is parameterized by the learned tracking value and the dual network:

$V_x(s)$8

The dual network is updated by gradient ascent in the direction of the tracking mismatch:

$V_x(s)$9

The gradient is taken only on $p_t$0, not through trajectory generation or tracking, to match the spirit of dual ascent (Yang et al., 2024).

4. Sources of guidance: prior knowledge, unsupervised objectives, and human input

One of the main distinctions among PAC formulations is the source of supervisory structure.

In PACMAN, guidance comes from symbolic prior knowledge represented in action language $p_t$1 and solved by ASP tools such as Clingo. The symbolic planner uses the current stochastic policy as a sampling oracle to generate possible plans from initial state $p_t$2 to goal state $p_t$3. This creates a bidirectional loop: planning is biased by the learned policy, while learning is constrained by symbolic structure. Human feedback is integrated directly into policy optimization and can be helpful, infrequent, inconsistent, misleading, or combinations thereof (Lyu et al., 2019, Lyu et al., 2019).

SPAC, by contrast, is explicitly unsupervised. It does not require ground-truth deformation fields, and its registration objective uses similarity and regularization terms:

$p_t$4

$p_t$5

Its registration environment computes rewards as Dice score improvement using unsupervised segmentation maps obtained by K-means clustering (Luo et al., 2021).

The layered control framework assumes only a simulator with black-box dynamics. It jointly learns a trajectory planner, a tracking controller, and a dual network that preemptively corrects the reference signal handed to the tracker. The coordination signal is therefore neither symbolic nor purely reward-based; it is an explicit consensus mechanism derived from the augmented-Lagrangian reformulation (Yang et al., 2024).

In the 3D modeling formulation, supervision is human-in-the-loop and operational rather than scalar-reward-centric. The planner incorporates user and critic input, the actor executes modeling steps through discovered MCP tools, and the critic analyzes viewport screenshots and execution results. The loop is Planner $p_t$6 Actor $p_t$7 Critic, with the human supervisor able to review, intervene, redirect, or modify plans at any stage (Gao et al., 8 Jan 2026).

5. Representative applications and reported empirical behavior

PACMAN was evaluated in Four Rooms and Taxi. The reported human feedback settings were Ideal, Infrequent, Inconsistent, and Infrequent + Inconsistent. Compared methods included PACMAN, PACMAN without the symbolic planner, TAMER+RL reward shaping, and BQL Reward Shaping. The reported findings were a significant jump-start at the early stage of learning, rapid convergence with small variance, and robustness to inconsistent, infrequent, and misleading feedback. The ablation without the symbolic planner reduced both jump-start and robustness (Lyu et al., 2019, Lyu et al., 2019).

SPAC was evaluated on several 2D and 3D medical image datasets, including MNIST, 2D MRI (LPBA), and 3D CT (Liver, including LSPIG). The reported metrics were Dice score, runtime, and parameter count. The paper reports that SPAC outperforms SyN, Elastix, LDDMM, VoxelMorph/VM-diff, R2N2, RCN, and SYMNet; that performance increases steadily with the number of steps; and that removing the planner or reverting critic evaluation to the actor’s action leads to worse performance. The strongest improvements are described for datasets with large deformations (Luo et al., 2021).

The layered control method was validated in unconstrained LQR, constrained LQR, and a nonlinear unicycle model. For unconstrained LQR, the reported performance was near-optimal cost within 1–3% of true optimum and almost perfect tracking over varying system sizes, with tracking error reductions over an order of magnitude relative to a heuristic no-dual approach. For nonlinear unicycle control, the reported cost was comparable to iLQR, but without access to the system model, and the learned dual network improved tracking performance even when cost improvement was subtler (Yang et al., 2024).

The 3D modeling PAC framework evaluated primitive tasks such as a birthday cake with layers and candles, a square dining table with four legs, and a classic car using only primitives. The reported evaluation used geometry count, vertex count, similarity between iterations, and human Likert-scale ratings for task alignment and aesthetics. The findings were progressive refinement over 1, 3, and 5 PAC iterations, increased geometric complexity, improvements in geometric accuracy, aesthetic quality, and task completion rates relative to single-prompt modeling, and a tendency for aesthetic and task-alignment scores to increase from the 1st to the 3rd iteration before plateauing (Gao et al., 8 Jan 2026).

6. Interpretation, ambiguities, and limitations

A common misconception is that Planner–Actor–Critic denotes one settled formalism. The cited literature indicates otherwise. In PACMAN, the planner is a symbolic reasoning system; in SPAC it is a stochastic low-dimensional latent plan; in layered control it is a trajectory optimizer coupled to a learned dual network; and in 3D modeling it is a task-decomposition agent operating over tools and screenshots. This suggests a shared decomposition principle rather than a uniform algorithmic template (Lyu et al., 2019, Luo et al., 2021, Yang et al., 2024, Gao et al., 8 Jan 2026).

A second misconception is that the critic must always evaluate the executed action. SPAC explicitly evaluates plans in the joint space of state and plan, not state and action, to make value function approximation more tractable. In layered control, the critic role is partly played by a tracking value function that predicts total tracking cost for a reference trajectory. In the 3D modeling system, the critic is an analysis agent that provides structured feedback on scene quality and whether iteration should continue (Luo et al., 2021, Yang et al., 2024, Gao et al., 8 Jan 2026).

Reported limitations are likewise domain-specific. In 3D modeling, the paper notes script reliability issues, incomplete incorporation of feedback in later iterations, toolset limitations that currently favor primitive modeling, and variable impact of human feedback. In layered control, optimal consensus $p_t$8 occurs if and only if the dual variable $p_t$9 attains its local optimum, and under suboptimal tracking the dual learning converges to a ball whose radius is determined by the tracking error size. In SPAC, the planner is motivated by the difficulty of high-dimensional continuous state and action spaces in conventional reinforcement learning, indicating that tractability remains a central design concern (Gao et al., 8 Jan 2026, Yang et al., 2024, Luo et al., 2021).

The acronym “PAC” is also ambiguous outside Planner–Actor–Critic. “PAC-Bayesian Soft Actor-Critic Learning” uses PAC in the Probably Approximately Correct sense, not as Planner–Actor–Critic. That paper nevertheless notes a conceptual link: its multiple-shooting action selection, guided by a probabilistic critic, is described as reminiscent of planner–actor–critic designs because a planner-like search operates over actor and critic samples before action commitment (Tasdighi et al., 2023).