CoPiC: Domain-Adaptive Planning Framework

• CoPiC is a framework for efficient, domain-adaptive, long-horizon planning that integrates LLM-generated programs with a learned critic for robust decision-making.
• It decouples plan generation from evaluation, reducing token costs and mitigating hallucinated, infeasible plans through environment-specific grounding.
• Empirical results on ALFWorld, NetHack, and StarCraft II show significant improvements in success rates and query efficiency compared to leading LLM planners.

CoPiC (Code-Driven Planning with Domain-Adaptive Critic) is a framework for efficient, domain-adaptive, long-horizon planning in sequential decision-making environments using LLMs as program generators and a learned critic as a long-term reward estimator. CoPiC addresses two central limitations in prior LLM-based planning: the lack of grounding to environment specifics (leading to hallucinated or infeasible plans) and the high query cost incurred by per-step, short-term LLM invocation. By decoupling code generation from plan evaluation and leveraging domain-adaptive learning, CoPiC achieves substantial improvements in both solution quality and computational efficiency relative to leading LLM-based planners (Tian et al., 23 Sep 2025).

1. Motivation and Problem Landscape

State-of-the-art LLM planners such as ReAct, Reflexion, and AdaPlanner operate by repeatedly querying the LLM to generate or refine an action sequence at each decision point. This process is expensive due to high token requirements and tends to optimize for short-term, myopic rewards driven by recent environmental feedback. Critically, LLMs pretrained on general corpora are prone to hallucinate actions or interactions not supported in specific environments (e.g., referring to objects or modalities that do not exist), resulting in infeasible or suboptimal plans.

CoPiC is designed to:

1. Minimize LLM query frequency (reducing inference cost).
2. Improve long-horizon task completion rates by promoting global reward alignment.
3. Adapt plan generation and assessment to environment idiosyncrasies via learned feedback, as opposed to overfitting on transient, shallow feedback.

2. Methodology: Architecture and Pipeline

CoPiC operates in two recurring phases: planning and learning.

• Planning Phase:
  • Program Generation: An LLM, prompted with a domain- and task-specific “init prompt,” synthesizes $n$ high-level Python planning programs $\{\rho_i\}_{i=1}^n$, each accepting task instruction $I$ and observation $o$, and outputting a candidate plan $p_i$.
  • Plan Scoring: Each plan $p_i$ is scored by a compact LLM critic $C_\theta$, which computes log-likelihoods of natural language plan descriptions, applies length normalization, and softmaxes across candidates. The top-scoring plan is selected for execution.
• Learning Phase:
  • Program Evolution: The LLM refines the set $\{\rho_i\}$ using an “evolve prompt” that summarizes performance over the last $M$ episodes.
  • Critic Fine-Tuning: Transitions $(I, o, p, r, o', \text{done})$ are marshaled into a replay buffer. Proximal Policy Optimization (PPO) with Low-Rank Adaptation (LoRA) is applied to further train $\{\rho_i\}_{i=1}^n$0 so that its value estimates $\{\rho_i\}_{i=1}^n$1—where $\{\rho_i\}_{i=1}^n$2—better reflect true long-term returns.

The process alternates until episode-level success rates plateau, yielding a refined ensemble of planning programs and a specialized critic.

3. Formalism: Domain-Adaptive Critic

The critic component is implemented as a lightweight LLM with LoRA-augmented final layers, abstracted as learning a state-value mapping $\{\rho_i\}_{i=1}^n$3. Key elements:

• TD-Error Minimization:

$\{\rho_i\}_{i=1}^n$4

• Inference-Time Scoring:
  • Plan descriptions $\{\rho_i\}_{i=1}^n$5 are tokenized and their conditional likelihood under a critic prompt $\{\rho_i\}_{i=1}^n$6 is calculated:

  $\{\rho_i\}_{i=1}^n$7 - Length normalization:

  $\{\rho_i\}_{i=1}^n$8 - Scores are softmax-normalized over all $\{\rho_i\}_{i=1}^n$9 candidates:

  $I$0 - The plan $I$1 is executed.

4. Algorithmic Workflow

A concise view of the CoPiC process:

Step	Description
1. Initial Program Synthesis	LLM generates $I$2 candidate planning programs via init prompt
2. Candidate Plan Generation	Each $I$3 produces a plan $I$4 given current $I$5
3. Plan Selection	Critic $I$6 softmax-scores and chooses the best $I$7
4. Execution and Feedback	Execute $I$8 in environment, collect reward and feedback
5. Critic Fine-Tuning	PPO-based RL updates $I$9 using replayed experience
6. Program Evolution	If stabilized, evolve programs via LLM using an episode summary

Program evolution ensures continual adaptation to environment specifics, while critic fine-tuning enhances reward alignment.

5. Empirical Performance and Cost Analysis

Evaluated on ALFWorld, NetHack, and StarCraft II Unit Building, CoPiC establishes superiority over advanced LLM planning baselines (Reflexion, AdaPlanner):

• Achieves a mean 23.33% improvement in success rate and 91.27% lower query cost (tokens) compared to Reflexion on ALFWorld.
• Attains an average 79.39% reduction in total LLM query token count across all benchmarks.
• Program evolution is central: success rate in ALFWorld increases from ~75% to 100% after four refinement iterations.
• Removal of the critic (random candidate selection) reduces success rates by 20–60% at fixed token cost.
• $o$0 planning programs found optimal for success/cost trade-off.
• Open-source LLMs (e.g., DeepSeek, Qwen2.5-Coder) yield comparable gains, suggesting model-agnosticism.

Metric	CoPiC vs. Best Baseline	Highlighted Result
Success Rate	+20.29% average improvement	Up to 100% on SC2 Hard
LLM Token Cost	−79.39% reduction	Up to 96% reduction (NetHack)
ALFWorld (tokens)	0.74M → 0.16M	+20% absolute SR delta

6. Strengths, Limitations, and Prospects

Strengths

• Token Efficiency: Drastic savings versus step-wise querying planners.
• Long-Horizon Reward Focus: Critic-driven selection enables non-greedy, long-term optimal plans.
• Environment Adaptivity: Iteratively evolved programs capture environment-specific constraints absent from LLM pretraining.

Limitations

• Requires Initial Program Synthesis: Baseline LLM must yield a working program in the environment.
• RL Training Overhead: Critic adaptation incurs training cost and environment interaction demands.

Directions for Future Work

• Extension to richer, more dynamic domains (e.g., full StarCraft II, Civilization).
• Applications to real-world robotics/industrial process planning.
• Development of more expressive critic architectures, introduction of Q-function-based or multi-agent frameworks.

CoPiC establishes a general approach for amortized, domain-adaptive, and cost-efficient LLM-based planning in complex sequential environments, marking a significant advancement over immediate-feedback planners by closing both the cost and reward alignment gaps (Tian et al., 23 Sep 2025).