---
title: Planner–Critic Process in Modular AI
url: https://www.emergentmind.com/topics/planner-critic-process
type: topic
---

# Planner–Critic Process in Modular AI

A Planner–Critic Process is a computational paradigm in which high-level planning and low-level evaluation are decoupled: a planner (or planning program) proposes candidate plans or subgoals, which are then evaluated by a critic—an explicit model or module trained (or engineered) to estimate the desirability, rationality, or long-term value of the planner's suggestion. This process is central to a wide class of recent advances in multi-agent systems, autonomous control, task-oriented agents, and reasoning-augmented large language model (LLM) pipelines. Architectures variably referred to as Planner–Critic, Planner–Actor–Critic, or (in cognitive neuroscience) Proposer–Predictor–Critic, all exploit the computational benefits of modular decomposition: planning modules isolate hierarchical, compositional reasoning, while critic modules ensure that plans align with task constraints, safety specifications, or long-range objectives.

## 1. Modular Architectures and Planner–Critic Loop

Planner–Critic systems are characterized by explicit separation between the high-level generator of candidate actions (planner) and the evaluator (critic), with distinct information flows and learning objectives for each. In advanced multi-agent frameworks for sequential and long-horizon tasks, the most prominent instantiations are:

- **Three-agent pattern**: Planner generates subgoals from global state, instruction, and memory snapshot; Actor executes, Memory Manager maintains contextual embeddings; Critic (often a VLM or separate LLM) assigns episodic trajectory-level scalar feedback to every Planner output [2605.02168].
- **Hierarchical planning and beam search**: A Planner LLM decomposes instructions into subgoals and sequences; an Actor proposes concrete actions; the Critic uses look-ahead (rollouts in simulated or memory-augmented space) to select which subgoal or sequence is most promising [2510.22732].
- **Program-mixture and domain-adaptive critic**: Planner emits a mixture-of-experts (MoE) of code-generated plans; a learned light-weight critic (fine-tuned on past experience) scores all candidates for long-horizon rewards; only the highest-rated plan is executed [2509.19077].

A core feature is iterative refinement: the Planner proposes, the Critic evaluates, and the process repeats—either until the critic's score exceeds a threshold, the plan converges, or a budget is exhausted [2410.01428], [2503.10049].

## 2. Mathematical Formulation and Optimization Objectives

Almost all Planner–Critic schemes are formalized as Markov Decision Processes (MDPs) or Partially Observable MDPs (POMDPs):

- **Planning as stochastic policy**: Given state $s$, task instruction $Q$, current memory $M$, and observation $O_t$, the Planner emits a high-level plan $P_t = \mathrm{Planner}_{\theta_p}(Q, O_t, M_t)$ [2605.02168].
- **Critic as value/rationality estimator**: The Critic computes scalar $V_{\text{critic}}(s, P)$ or (episodically) $r(\tau)$: a judgment over entire trajectories or subgoals [2503.10049], [2605.02168].
- **Reinforcement Learning for Planner**: Planners are typically optimized by policy-gradient objectives (e.g., Group Relative Policy Optimization, GRPO), where only the Planner parameters are updated, using critic- or reward-model estimates as return signals. For trajectory-level reward $R(\tau)$, advantage normalization and KL-regularization against a reference policy are standard:

  \[
  J(\theta_p) = \mathbb{E}_t \left[ \rho_t A_t - \beta \cdot D_{\text{KL}}\left(\pi_{\theta_p}(\cdot|\cdot) \| \pi_{\text{ref}}(\cdot|\cdot)\right) \right]
  \]
  \[
  \mathcal{L}(\theta_p) = -J(\theta_p)
  \]
  [2605.02168]

- **Critic training**: Critics may use temporal-difference (TD) losses, pairwise ranking (for plan rationality/discriminators), or fitted value regression. For domain-adaptive critics, PPO with only LoRA/MLP adapters is used for efficiency [2509.19077].

This modularity allows for freezing non-planner components during RL, reducing sample complexity and "role confusion" [2605.02168].

## 3. Compute Allocation and Scaling Behavior

A distinctive insight from recent Planner–Critic studies is the observed unbalanced returns on compute and model capacity when partitioned across modules:

| Component          | α     | β    | R²   |
|--------------------|-------|------|------|
| All Modules        | 15.6  | 18.1 | 0.58 |
| Planner            | 16.0  | 12.7 | 0.82 |
| Actor              | 12.0  | 13.0 | 0.76 |
| Memory Manager     |  5.6  | 12.7 | 0.89 |

With overall success rates $y$ regressed against $\log$ model size $x$ ($y = \alpha \cdot \log(x) + \beta$), almost all available performance gains accrue to increasing the Planner's size, justifying a "planner-centric" design [2605.02168]. Actor and Memory Manager display considerably lower scaling coefficients ($\alpha$), while their capacity can be kept minimal.

## 4. Implementations of Critic Modules

Planner–Critic instantiations vary according to domain and objectives:

- **Vision-Language Model (VLM) as Critic**: In long-horizon automation (web, OS, tool use), a frozen large VLM (Qwen2.5-VL-32B) serves as judge, reading the full interaction transcript and scoring coarse scalar outcomes by rubric (1/3/5 points for correctness, reasoning coherence, interaction efficiency); human-VLM agreement is 88–97% [2605.02168].
- **Domain-Adaptive LLM Critics**: For code-driven planning, critics are smaller LMs with task-specific adapters, trained to discriminate among n candidate high-level planning programs; scoring is based on normalized token likelihood or ranking [2509.19077].
- **Transformer-based Rationality Critics**: In multi-agent collaboration, the critic LLM encodes the environmental state and sequence of subtasks, outputs a rationality (feasibility) score, and can train by regression, ranking, or reward bootstrapping [2503.10049].
- **Multimodal and Structural Critics**: For creative domains (3D modeling), critics combine vision-language encoders (CLIP, 3DLLaVA) and analysis LLMs to score alignment with prompt and aesthetics, and supply structured feedback for iterative plan revision [2601.05016].

Critic outputs are consistently used to provide learning signals to the planner policy, either as final targets in RL, differentiable rewards, or to trigger further plan refinement.

## 5. Algorithmic Workflow and Iterative Loop

Planner–Critic systems operate in tightly coupled, multi-stage cycles. A typical generic loop appears as follows [2605.02168], [2601.05016], [2503.10049]:

1. **Initialization**: Reset environment, memory, and planner policy.
2. **Planning**: The Planner, given the current state, task, and context, emits a high-level plan or goal decomposition.
3. **Execution**: (If present) An Actor executes the Planner's output as low-level actions in the target environment.
4. **Memory Update**: Memory manager (if present) records outcomes to support non-myopic planning.
5. **Critic Evaluation**: After rollout, a critic (VLM, LLM, or specialized module) scores the trajectory, subgoal sequence, or entire plan.
6. **Update**: Only the Planner is fine-tuned to maximize critic-assigned returns, with the Actor and Memory Manager frozen (if RL is employed).
7. **Iteration**: Repeat for $N$ cycles or until critic score meets threshold.

Specific implementations may include correction loops (autoregressive resampling of unsafe action tokens until the critic predicts safety, as in autonomous driving [2603.15771]), replay and buffer-based training, as well as human-in-the-loop critique for high-fidelity or creative domains [2601.05016].

## 6. Empirical Outcomes and Comparative Analysis

Quantitative evaluations across automation, simulation, programming, and creative tasks consistently demonstrate the empirical benefits of Planner–Critic decomposition:

- Multi-agent frameworks yield substantial increases in long-horizon benchmark success rates: e.g., from 12.5% to 35.1% on WebVoyager via planner-only RL (+28% relative) [2605.02168].
- Planner-centric compute budgeting is empirically optimal; returns on parameters for non-planner modules saturate rapidly [2605.02168].
- Critic-based filtering and refinement (plan-selection from diverse code programs, retrieval-augmentation guided by LLM critics) drive substantial reductions in inference cost or LLM token budget (up to 91% reduction in query costs on tabular environments) and increase task success rates by 20–30+ points, versus stepwise greedy refinement or pure imitation baselines [2509.19077], [2410.01428].
- In collaborative or multi-agent RL, critic-guided plan acceptance rates and task completion frequencies are consistently higher, specifically when the critic can flag and force correction of locally infeasible or globally inconsistent subplans [2503.10049], [2601.05016].

Planner–Critic architectures also supply interpretability: the critic’s explicit scoring and feedback unlock post-hoc analysis of planner errors and corrective mechanisms.

## 7. Variants, Extensions, and Cognitive Parallels

Beyond engineering applications, the Planner–Critic principle has direct analogues in neuroscience and human decision modeling (Proposer–Predictor–Actor–Critic frameworks [1912.07660]). The modularity supports serial and hierarchical composition, trade-offs between speed and accuracy (immediate plan acceptance vs. optional forward simulation), and robustness to feedback noise or adversarial/human-in-the-loop oversight [1909.09209], [1912.07660], [2601.05016]. Moreover, combining Planner–Critic with advanced model-predictive control (MPCritic) or actor–critic reinforcement learning—often with learned or amortized value functions tightly coupled to the planner—reinforces or extends planning horizons and accelerates sample efficiency [2604.01477], [2011.06752], [2408.01639].

Planner–Critic research continues to expand in scope, targeting improved scaling, modular abstraction, and domain adaptation, while empirical evidence suggests the paradigm is broadly applicable to any domain where plan evaluation can be decoupled from proposal and subjected to targeted optimization.

Source: https://www.emergentmind.com/topics/planner-critic-process