LLM Planner and Critic Model

• LLM Planner and Critic Model is a hybrid architecture that integrates stepwise planning with iterative critique to boost logical consistency and task adherence.
• It employs a generate-test-critique loop where a planner outputs candidate solutions that are scrutinized using chain-of-thought analysis and formal logic.
• Empirical results demonstrate that iterative refinement with detailed feedback significantly improves performance and reduces errors in domains like math and code synthesis.

A LLM Planner and Critic Model is a composite architecture that couples the generative planning capabilities of a LLM (“planner”) with the evaluative (“critic”) capabilities of a supervisory or verification model. The planner is responsible for step-by-step reasoning, action selection, or structured output generation, while the critic provides detailed feedback—either in the form of explicit error identification, logic-based rules, or actionable natural language critique—enabling iterative refinement and improved reliability. This dual-system paradigm is motivated by the need to ensure both competence (problem-solving ability) and correctness (adherence to task constraints or logical consistency) in LLM-based autonomous decision-making systems (Luo et al., 2023, Kalyanpur et al., 25 Jun 2024, Zheng et al., 29 Aug 2024, Yang et al., 20 Mar 2025, Gokhale et al., 4 Jul 2025).

1. Architectural Principles and Evaluation Frameworks

The prevailing design principle in LLM Planner and Critic Models is modularity: planners generate candidate outputs (e.g., reasoning chains, code, plans), which are then subject to systematic critique by one or more distinct modules. Critique ability is commonly assessed using structured evaluation frameworks involving:

• Chain-of-thought (CoT) analysis: The planner produces an explicit stepwise reasoning trace, which the critic scrutinizes for logical gaps, factual errors, or violations of specified constraints (Luo et al., 2023, Zheng et al., 29 Aug 2024).
• Judgment accuracy metrics: Given a set of generated solutions and their critiques, model performance is measured by the critic's ability to accurately declare outcomes as “correct” or “incorrect” with respect to ground truth.
• Certainty and uncertainty rates: For a query $q$, the frequency of different answers produced by sampling the planner is used to compute the uncertainty rate $\text{UR}_{\text{LM}}(q; k)$ and the certainty score $$\text{Certainty}_{\text{LM}}(q; k) = -\log(\text{UR}_{\text{LM}}(q; k))$$ (Luo et al., 2023).

Critique frameworks can target domains such as math problem solving, code synthesis, question answering, and formal planning, with benchmarks such as CriticBench explicitly designed for multi-domain evaluation (Luo et al., 2023).

2. Planning and Critique Loop Mechanisms

Modern LLM Planner and Critic architectures instantiate an iterative "Generate-Test-Critique" or "Actor-Critic" loop:

• Generation/Planning: The LLM planner outputs one or more candidate solutions or plan steps, often employing structured formats (e.g., logic programs, stepwise rationales, PDDL domains, action sequences).
• Critique/Verification: The critic evaluates each candidate, leveraging techniques such as:
  • Stepwise chain-of-thought review (Zheng et al., 29 Aug 2024, Yang et al., 1 May 2025)
  • Automated reasoning via formal solvers (e.g., Clingo for ASP code) (Kalyanpur et al., 25 Jun 2024)
  • Logic formula enforcement, e.g., via LTL/Büchi automata (Gokhale et al., 4 Jul 2025)
  • Empirical statistical validation (e.g., hypothesis testing for model-data discrepancy) (Li et al., 10 Nov 2024)
  • Reward-based selection using learned critic models (Li et al., 2 Oct 2024, Wang et al., 12 Mar 2025)
• Refinement/Improvement: Critic results are used to filter, revise, or prompt regeneration by the planner. In some frameworks, only those candidates passing specific critic filters are retained or voted upon for the final answer (e.g., self-check filtering as in (Luo et al., 2023)).

This iterative mechanism forms a closed feedback loop, enabling self-improvement and resilience against initial failures, redundant steps, or hallucinated outputs.

3. Critic Model Implementations and Methodologies

Critic models in this paradigm embody several key methodologies:

• Multi-step and Multi-perspective Critique: Rather than a superficial global judgment, advanced critics generate "deliberate" feedback for each reasoning step. For example, DeepCritic creates stepwise critiques augmented by secondary meta-critique and multi-perspective reasoning, consolidating these into a singular, final judgment per step (Yang et al., 1 May 2025).
• Formal Symbolic Logic Integration: The LTLCrit framework equips the critic with the ability to induce and enforce linear temporal logic (LTL) constraints, enabling safety and efficiency guarantees over sequences of planner actions. Symbolic rules such as $G(\varphi_s \to X(\varphi_a))$ explicitly prevent unsafe or redundant action patterns (Gokhale et al., 4 Jul 2025).
• Automated Reasoning via Dedicated Engines: LLM-ARC combines a natural language-to-logic planner with a critic implemented as an ASP solver, which executes candidate logic programs and tests, providing targeted feedback for error localization and correction (Kalyanpur et al., 25 Jun 2024).
• Natural Language and Statistical Critique: CriticAL leverages LLMs to propose summary statistics for scientific model evaluation, conducting hypothesis tests to ensure only statistically significant discrepancies are flagged, thus minimizing hallucinated or spurious critiques (Li et al., 10 Nov 2024).
• Reward-Driven or MCTS-trained Critics: CR-Planner deploys critic models for both sub-goal selection and execution evaluation, with critic training data collected via Monte Carlo Tree Search (MCTS) to explore long-term impacts of decisions (Li et al., 2 Oct 2024).

4. Empirical Results and Key Findings

Empirical studies across diverse domains demonstrate several commonalities:

• Emergent Ability with Scale: Non-trivial critique ability is observed primarily in larger LLMs; smaller models often perform near randomly when critiquing complex outputs (Luo et al., 2023).
• Iterative Critique Enhances Performance: Incorporating self-critique or critic filtering yields substantial gains. For instance, self-check filtering produces up to 9.5% error rate reductions in math reasoning tasks (Luo et al., 2023), while iterative refinement strategies in Critic-CoT raise GSM8K math accuracy from 89.6% (base) to 95.4% (majority voting with critic filter) (Zheng et al., 29 Aug 2024).
• Stepwise and Multi-perspective Critique Is More Effective: Critics that assess each reasoning step, rather than holistic judgments, enable more precise error localization and correction (Yang et al., 1 May 2025, Zheng et al., 29 Aug 2024, Kalyanpur et al., 25 Jun 2024).
• Model-Agnostic Wrapper Support: Architectures such as LTLCrit are compatible with a variety of LLM planners, layering logic-based safety and efficiency checks atop diverse generative models, including SayCan and InnerMonologue, achieving 100% task completion on Minecraft diamond-mining with increased efficiency and reduced failure rates (Gokhale et al., 4 Jul 2025).
• Natural Language Critique Outperforms Purely Numeric Rewards: Critique-Guided Improvement (CGI) demonstrates that actionable, fine-grained language feedback from a dedicated critic can surpass both reward-model-based methods and even large models (e.g., an 8B critic outperforming GPT-4 in feedback quality by 29.16%) in guiding robust exploration and performance (Yang et al., 20 Mar 2025).

5. Domain-Specific and Multimodal Critic Extensions

LLM planner and critic models have been specialized to address the requirements of specific domains:

• Formal Planning with Human Preferences: PlanCritic integrates an LLM to translate user-specified planning preferences into mid-level goals or constraints, optimizing plans with a genetic algorithm and RLHF-tuned reward model to align outcomes with human intent (Burns et al., 30 Nov 2024).
• Data Visualization Critique: VIS-Shepherd demonstrates the utility of MLLM-based critics for data visualization, offering actionable, domain-specific feedback that enables self-correction in LLM-generated charts, with fine-tuned 7B models rivaling much larger proprietary models (Pan et al., 16 Jun 2025).
• Multi-Agent Systems: Advanced frameworks (e.g., SAMALM, LGC-MARL) deploy planner–critic architectures for decentralized robot fleets, leveraging actor-critic mechanisms, graph-based dependency modeling, and entropy-based score fusion to balance local autonomy and global coordination (Wang et al., 12 Mar 2025, Jia et al., 13 Mar 2025).

6. Limitations, Open Challenges, and Future Directions

Despite substantial progress, several fundamental challenges remain:

• Self-Critique Remains Difficult: Even state-of-the-art LLMs underperform or exhibit systematic errors in self-critique settings, especially outside of mathematics (e.g., code completion, factual QA) (Luo et al., 2023).
• Trust, Human Alignment, and Transparency: User studies confirm that correctness is the primary determinant of human trust in LLM planners, with explanations and refinement offering improvements in perceived transparency but not always in real trust or objective performance (Chen et al., 27 Feb 2025). Integration of formal verification and explicit critic calibration is suggested as a remedial pathway.
• Unified Performance Criteria: There is a growing call for standardized multi-dimensional performance criteria to holistically evaluate both planners and critics, encompassing completeness, executability, optimality, representation, generalization, and efficiency (Wei et al., 16 Feb 2025).
• Learning and Generalization: Future research is expected to focus on automated abstraction (learning symbolic propositions), online adaptation, integration of multi-modal feedback (e.g., for vision or robotics), and extension of formal critic supervision to multi-agent and human-robot teams (Gokhale et al., 4 Jul 2025, Wang et al., 12 Mar 2025).
• Critic Model Robustness and Hallucination Prevention: Reliable identification and mitigation of hallucinated or spurious critiques remains an active area, with empirical validation, statistical calibration, and logic-based constraint enforcement representing promising directions (Li et al., 10 Nov 2024, Yang et al., 1 May 2025).

7. Schematic Summary Table

Component	Planning Role	Critic Role
LLM Planner	Stepwise reasoning, plan/code/logic gen.	Chain-of-thought or output generation
Critic Model	—	Stepwise review, formal logic/test, or feedback
Verifier/Wrapper	(optional) as controller or orchestrator	Enforce external/learned constraints

Example Implementations:

• LLM-ARC: ASP code generation by actor (planner), evaluated/fixed by ARC (critic) (Kalyanpur et al., 25 Jun 2024)
• Critic-CoT: Stepwise reasoning and self-critique/refinement (Zheng et al., 29 Aug 2024)
• PlanCritic: Formal plan generation, RLHF reward model, and GA-optimized constraint search (Burns et al., 30 Nov 2024)
• LTLCrit: LLM planner actions shielded by LTL-logic critic with formal verification (Gokhale et al., 4 Jul 2025)

---

The LLM Planner and Critic Model paradigm unifies language-based planning and rigorous, often symbolic, critique, yielding systems that approach autonomous self-improvement, robustness to error, and a degree of interpretability essential for trustworthy and adaptive deployment in real-world domains.