---
title: Planner–Coder–Critic Pipeline
url: https://www.emergentmind.com/topics/planner-coder-critic-agentic-pipeline
type: topic
---

# Planner–Coder–Critic Pipeline

A Planner–Coder–Critic Agentic Pipeline is a unified architectural paradigm that fuses structured planning, code-based policy refinement, and critical feedback mechanisms into an interactive loop for sequential decision-making, reinforcement learning, and complex task automation. This design coordinates symbolic reasoning, experiential learning, and human or automated critique to accelerate learning, improve sample efficiency, and foster robustness in dynamic or uncertain environments. Building on rigorous foundations in both classical planning and reinforcement learning, it integrates modules that generate, execute, and iteratively improve plans or policies through structured knowledge, code-based modeling, and feedback incorporation.

## 1. Architecture and Core Components

A Planner–Coder–Critic pipeline decomposes the agent’s cognition into three synergistic modules:

- **Planner:** Generates initial, often high-level or symbolic, plans leveraging formal domain knowledge, LLMs, or combinatorial search over structured representations such as action languages.
    - Example: The PACMAN framework uses a symbolic planner over formal action descriptions (e.g., the action language ℬC) to produce goal-directed plans, employing sample-based planning augmented with stochastic policies ($\pi_\theta$) as oracles [1906.07268, 1909.09209].
    - In CoPiC, LLMs generate multiple planning programs—instances of code that produce or refine candidate action sequences conditioned on task and environmental context [2509.19077].
- **Coder:** Converts (or directly produces) executable representations, such as Python functions or code blocks, which are evaluated or executed in the environment, possibly iteratively refined using feedback or interaction histories.
    - In AdaPlanner, the coder is instantiated via code-style LLM prompt structures; the LLM generates “solution” functions in Python with assertions as checkpoints [2305.16653].
    - In CP-Agent, the agent employs a persistent IPython kernel to iteratively write, debug, and verify code for constraint programming, dynamically refining its model with execution feedback [2508.07468].
- **Critic:** Evaluates the quality and future utility of candidate policies or plans using value functions, learned reward models, or external human feedback, and provides corrective signals to the planner or coder.
    - PACMAN allows human feedback signals to override the TD error, effectively treating fᵢ as an alternative advantage estimate in actor–critic RL updates: $\delta_i \leftarrow f_i$ [1906.07268].
    - In CoPiC, a domain-adaptive critic (trained via RL with LoRA) scores LLM-generated planning programs for expected long-term reward alignment, selecting the optimal candidate for execution [2509.19077].
    - PlanCritic formalizes critique using an RLHF-trained reward model to iteratively mutate and optimize PDDL constraint sets via a genetic algorithm, maximizing adherence to human preferences [2412.00300].

This modular segregation facilitates a bidirectional, interactive learning loop: plans are synthesized, executed (or simulated), critiqued, and refined, with each module leveraging different modalities (symbolic, code, behavioral feedback).

## 2. Symbolic Planning and Sample-Based Augmentation

The planning component typically encodes prior knowledge or high-level abstractions to constrain search and improve exploration efficiency:

- **Formal Action Semantics:** Domains are encoded using fluent and action constants, with static and dynamic causal laws. PACMAN defines dynamic laws as $a \ A_0 \ A_1, \dots, A_m$ and augments these with predicates $p(s,a)$ that enable integration with probabilistic policy sampling [1906.07268].
- **Sample-Based Planning:** Rather than produce plans in isolation, the planner incorporates the current stochastic policy to bias action selection. This is formalized via:
  $$
  D^{l}_{\pi} = D_s \cup D_d \cup \bigcup_{t=1}^l P^{t}_{\pi}
  $$
  where $D_s$ and $D_d$ are static and dynamic laws, and $P^{t}_{\pi}$ is generated by sampling the current policy at each step. The composite program is translated into an answer set program via a mapping $\mathcal{M}$, producing executable, symbolically guided plan candidates.

- **Underlying Rationale:** By leveraging symbolic plans, the agent achieves “jump-started” exploration, biasing trajectories toward semantically meaningful regions of the state space and reducing sample complexity.

## 3. Integration with Reinforcement Learning and Code-Based Policy Refinement

Execution of planned policies is refined through standard or customized reinforcement learning techniques:

- **Actor–Critic Algorithms:** The agent learns through both environmental rewards and (optionally) human-provided feedback:
  - **Policy Update:**
    $$
    \theta_{i+1} = \theta_i + \beta \delta_i \nabla \log \pi_\theta(a_i | s_i)
    $$
  - **Value Function Update:**
    $$
    x_{i+1} = x_i + \alpha \delta_i \nabla V_x(s_i)
    $$
  - **Advantage/TD Error:**
    $$
    \delta_i = r_i + \gamma V_x(s_{i+1}) - V_x(s_i)
    $$
   Optionally replaced with $f_i$ per human feedback.

- **Code-Driven Policy Generation:** LLMs or agent frameworks synthesize executable code (e.g., Python plans) that interact with the environment, update beliefs, and process feedback iteratively. Candidate plans are refined based on execution logs, assertion failures, or summarized histories, exemplified by AdaPlanner’s code-based prompting and CoPiC’s planning-program evolution with history summarization [2509.19077, 2305.16653, 2508.07468].

- **Dual Network Coordination:** In layered control, a dual variable $v_\theta(\xi)$ is learned to ensure consensus between planner outputs and tracker execution ($r^* = x^*$), with convergence guarantees established in LQR; error updates follow
  $$
  \Theta^{+} = \Theta + \eta \sum_{i=1}^{B} (r_i - x_i) \xi_i^T / B
  $$
  yielding exponentially contracting tracking error [2408.01639].

## 4. Feedback, Critique, and Robustness Mechanisms

- **Human-Centered Feedback:** The PACMAN architecture incorporates human feedback directly as an alternative advantage estimate, supporting robustness to infrequent or inconsistent supervised signals without destabilizing policy updates [1906.07268, 1909.09209].

- **Automatic Plan Critique and RLHF:** In PlanCritic, plans generated by a symbolic planner are critiqued via an RLHF-trained reward model; a genetic algorithm mutates constraint sets, and the best candidates are evolved according to adherence to user preferences [2412.00300].

- **Domain-Adaptive Critics:** CoPiC employs a critic LLM that scores candidate planning programs normalized by length and interaction context, then softmax-selects plans most likely to yield high long-term reward. The critic is refined by RL based on end-of-episode environment returns, ensuring long-horizon efficacy even with sparse intermediate rewards [2509.19077].

- **Code-Embedded Verification:** Assertion-driven code (as in AdaPlanner) provides natural points for critique—failed assertions or error messages trigger plan refinement (either localized or global), minimizing silent failure and constraining hallucinations through structured code interfaces [2305.16653].

## 5. Empirical Results and Comparative Evaluation

A consistent empirical advantage is observed across diverse sequential decision-making and control benchmarks:

| Framework     | Key Benchmark / Domain                   | Performance Gains over Baselines                | Robustness / Sample Efficiency                             |
|---------------|------------------------------------------|------------------------------------------------|-----------------------------------------------------------|
| PACMAN        | Four Rooms, Taxi                         | Jump-starts, rapid convergence, low variance   | Robust to infrequent/inconsistent human feedback          |
| CoPiC         | ALFWorld, NetHack, StarCraft II          | 23.33% ↑ SR, 91.27% ↓ query cost vs AdaPlanner | High SR at substantially reduced token/query cost         |
| PlanCritic    | Waterway disaster recovery                | 75% SR vs 53% for LLM-only neurosymbolic       | RLHF + GA yields correction in 88% of initial plan errors |
| AdaPlanner    | ALFWorld, MiniWoB++                      | Outperforms baselines with 2x–600x fewer demos | Code-style interface dramatically reduces hallucinations   |
| Layered RL    | Linear, nonlinear (unicycle) control     | Near-iLQR cost, ≤0.02 mean tracking deviation  | Explicit dual ensures tracking–planning consensus         |

Key findings include:
- **Jump-Start and Efficiency:** Architectures leveraging symbolic or code-driven planning provide substantial jump-start at early learning stages, accelerate convergence, and decrease variance relative to canonical RL or naively reactive LLMs [1906.07268, 1909.09209, 2509.19077, 2305.16653].
- **Query/Compute Cost Minimization:** By selecting candidate plans using a learned critic, approaches such as CoPiC achieve high success rates with up to 80–90% reductions in token or query cost compared to iterative LLM feedback baselines [2509.19077].
- **Robustness to Feedback Quality:** Symbolic constraints and plan verification (e.g., assertion-based checks, hard-coded action preconditions) act as guardrails, preventing policy drift under misleading or sparse feedback [1906.07268, 2509.19077].
- **Correction and Adaptivity:** Genetic search and domain-adaptive critics enable dynamic correction of plan errors, surpassing static LLM-generated plan approaches [2412.00300].

## 6. Theoretical Guarantees and Implementation Considerations

- **Convergence:** PACMAN’s iterative planning/RL loop, and Layered RL with explicit dual variable learning, are proven to converge under suitable conditions (e.g., monotonicity in UPOM [2010.01909]; exponential contraction for dual variable in LQR [2408.01639]).
- **Mutual Reinforcement:** The planner steers exploration toward high-value or goal-relevant states, while RL-derived policy updates recursively improve subsequent plan proposals; human or critic feedback further aligns outputs with external goals or preferences.
- **Limitations:** Execution requires precise domain models (for symbolic planning), and mismatch between symbolic abstractions and low-level environment dynamics can impede exploration if constraints are too rigid. Integration of stochastic feedback and symbolic rules presents additional complexity, especially in high-dimensional or perception-heavy settings [1906.07268].
- **Generalizability:** Modular code-based and sample-driven planning schemas enable adaptive generalization to new tasks, provided sufficient coverage and robust critic adaptation.

## 7. Relationship to Broader Agentic Methodologies

The Planner–Coder–Critic pipeline represents a convergence of symbolic AI (planning, logic programming), sample-based RL, LLM-powered code generation, and feedback-driven or RLHF-based critique. It stands in contrast to strictly end-to-end or black-box RL approaches by explicitly leveraging structured priors and modular verification. The bidirectional loop architecture enables bidirectional transfer of information across planning, learned policy, and external feedback, bridging gap between classical AI and modern learning-based agentic systems.

This paradigm increasingly informs state-of-the-art agentic designs in sequential decision-making, constraint programming, robotic control, and language-guided planning, serving as an extensible scaffold for integrating reasoning, execution, and adaptive critique.

Source: https://www.emergentmind.com/topics/planner-coder-critic-agentic-pipeline