Critic PI2: Domain-Adaptive Plan Evaluator

• Critic PI2 is a domain-adaptive scoring mechanism that evaluates candidate plans from LLM-generated programmatic planners for long-horizon rewards.
• It uses a lightweight language model with LoRA adapters and an MLP head to compute plan likelihood and value estimates, fine-tuned via PPO.
• Empirical results demonstrate significant improvements in success rates and query efficiency across environments like ALFWorld, NetHack, and StarCraft II.

Critic PI2 refers to the core domain-adaptive plan-evaluation module within the Code Driven Planning with Domain-Adaptive Critic (CoPiC) framework, which addresses the challenge of robust sequential decision-making for agents in partially observed Markov Decision Processes, especially when leveraging LLMs for open-ended planning. The Critic PI2 acts as an environment-adaptive scoring function for candidate plans produced by LLM-generated programmatic planners, guiding execution toward long-horizon rewards and substantially reducing the frequency of costly LLM queries (Tian et al., 23 Sep 2025).

1. Formal Definition and Functional Role

Within the CoPiC system, the critic $C_\theta$ serves as a parametric function mapping an observation-plan pair $(o, p)$ (together with the task instruction $I$) to a scalar “plan score” reflecting the expected return of executing $p$ from $o$ under task $I$. Rather than simply relying on immediate feedback after each action, $C_\theta$ estimates the compatibility of the entire candidate plan with long-term environmental reward. This differentiates Critic PI2 from conventional stepwise feedback strategies in prior LLM-centric planning methods.

Formally, given the MDP $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, T, R, \gamma)$$, CoPiC evaluates $n$ high-level candidate plans $\{p_i\}_{i=1}^n$ generated by planning programs $\{\rho_i\}$. For each $p_i$, the critic computes

$$\mathrm{score}(p_i) = \frac{\exp(\mathrm{logit}(d_{p_i}))}{\sum_{j=1}^n \exp(\mathrm{logit}(d_{p_j}))}$$

where $d_{p_i}$ is a natural-language plan description and $\mathrm{logit}(d_{p_i})$ is the log-likelihood (length-normalized) of $d_{p_i}$ under $C_\theta$ conditioned on the task prompt and current observation.

2. Model Architecture and Adaptation Mechanism

The critic is implemented by initializing a lightweight LLM backbone (e.g., TinyLlama), extended with parameter-efficient LoRA adapters and an MLP head. Its architecture supports both plan-likelihood estimation and value function regression:

• The LoRA-adapted LLM provides a distribution $\pi_\phi(p \mid o)$ over natural-language plan descriptions.
• The MLP head computes a scalar value function $V_\phi(o,p)$, which approximates expected future return.

The plan-likelihood path conditions on a “critic prompt” $d_{cp}$ and autoregressively factors the probability for a candidate plan sequence $d_{p_i} = (w^1, ..., w^{N_i})$: $$\mathrm{prob}(d_{p_i} \mid d_{cp}) = \prod_{k=1}^{N_i} \mathrm{prob}(w^k \mid d_{cp}, w^{<k})$$ Enabling the critic to generalize beyond short-horizon feedback, model parameters are fine-tuned using Proximal Policy Optimization (PPO), ensuring the critic captures environment-specific reward structure.

3. Learning and Interaction Loop

Critic PI2 is updated in CoPiC’s interleaved planning and learning phases. During each interaction episode:

• All planners $\{\rho_i\}$ generate candidate plans given the current instruction $I$ and observation $o$.
• The critic $C_\theta$ scores each $p_i$; a plan $p^*$ is sampled (or selected) for execution according to the softmax over scores.
• The tuple $(o, \{p_i\}, p^*, r, o')$ is stored, capturing the environmental feedback.
• After episodic rollout, $C_\theta$ is fine-tuned with PPO, using the collected data as experience.

The loss consists of an actor-critic PPO formulation: $$\mathcal{L}(\phi) = \mathbb{E}_{(o,p,r,o') \sim D}\left[ -\min(r_t(\phi) A_t, \mathrm{clip}(r_t(\phi), 1-\epsilon, 1+\epsilon) A_t) + \frac{1}{2} (A_t)^2 \right]$$ where $A_t = r + \gamma V_\phi(o') - V_\phi(o)$ is the advantage, and $r_t(\phi)$ is the likelihood ratio.

4. Impact on Query Efficiency and Planning Performance

In contrast to iterative refinement paradigms such as Reflexion and AdaPlanner, which query the LLM at each environment step, Critic PI2 processes all candidate plans within a single decision cycle, requiring only $n$ LLM queries per cycle. Empirical results show an average query token cost reduction of approximately 79.4% against baselines (e.g., 0.16–0.33M tokens vs. 0.8–2.5M per task in ALFWorld) (Tian et al., 23 Sep 2025). Simultaneously, the critic-driven evaluation mechanism yields substantial gains in success rate: +20.4% in ALFWorld, +13.7% in NetHack, and +44.8% in StarCraft II Unit Building (Hard).

Environment	$\Delta$ Success Rate	$\Delta$ Query Cost
ALFWorld	+20.4%	-83.8%
NetHack	+13.7%	-71.0%
SC2 (Hard)	+44.8%	-87.9%

The critic is thus essential for both efficiency and reward-aligned plan selection. Ablation experiments reveal that random selection among planners reduces success by 20–60% under equal query budgets.

5. Design Choices, Ablations, and Limitations

Three primary factors influence Critic PI2’s efficacy:

• Presence of the Critic: Omission lowers success rates by up to 60%, empirically demonstrating the necessity of environment-adaptive scoring.
• Number of Planners ($n$): Empirical analysis indicates $n=3$ yields optimal coverage versus evaluation complexity.
• Program Evolution: Iterative fine-tuning and evolution of planner-programs, guided by feedback and critic-reward signals, allow the system to improve from 75.6% to 91.4% success rate in two rounds, with further convergence to 100% by the fourth iteration.

Key limitations include the requirement that the initial LLM must have strong code-generation capability to bootstrap viable programmatic planners and the potential expense of acquiring sufficient environment interaction data to fine-tune the critic in real-world domains.

6. Relationship to Prior Work and Extensions

Critic PI2 replaces stepwise LLM query architectures with plan-level evaluation, introducing a generalizable critic-based value estimator that is tunable to environmental rewards. The compatible design with PPO and LoRA enables efficient, scalable adaptation to new domains. By supporting high-level programmatic plan generation and selective execution, Critic PI2 positions CoPiC to outperform prior LLM-based planning methods both in efficiency and long-term task competence.

Proposed future extensions for Critic PI2 include scaling to complex, long-horizon environments (e.g., entire StarCraft II matches, realistic household tasks) and the development of multi-agent critic architectures for collaborative or adversarial task settings (Tian et al., 23 Sep 2025).