---
title: 'CoPiC: Domain-Adaptive Planning Framework'
url: https://www.emergentmind.com/topics/copic
type: topic
---

# CoPiC: Domain-Adaptive Planning Framework

CoPiC (Code-Driven Planning with Domain-Adaptive Critic) is a framework for efficient, domain-adaptive, long-horizon planning in sequential decision-making environments using large language models (LLMs) as program generators and a learned critic as a long-term reward estimator. CoPiC addresses two central limitations in prior LLM-based planning: the lack of grounding to environment specifics (leading to hallucinated or infeasible plans) and the high query cost incurred by per-step, short-term LLM invocation. By decoupling code generation from plan evaluation and leveraging domain-adaptive learning, CoPiC achieves substantial improvements in both solution quality and computational efficiency relative to leading LLM-based planners [2509.19077].

## 1. Motivation and Problem Landscape

State-of-the-art LLM planners such as ReAct, Reflexion, and AdaPlanner operate by repeatedly querying the LLM to generate or refine an action sequence at each decision point. This process is expensive due to high token requirements and tends to optimize for short-term, myopic rewards driven by recent environmental feedback. Critically, LLMs pretrained on general corpora are prone to hallucinate actions or interactions not supported in specific environments (e.g., referring to objects or modalities that do not exist), resulting in infeasible or suboptimal plans.

CoPiC is designed to:

1. Minimize LLM query frequency (reducing inference cost).
2. Improve long-horizon task completion rates by promoting global reward alignment.
3. Adapt plan generation and assessment to environment idiosyncrasies via learned feedback, as opposed to overfitting on transient, shallow feedback.

## 2. Methodology: Architecture and Pipeline

CoPiC operates in two recurring phases: planning and learning.

- **Planning Phase**:
  - **Program Generation**: An LLM, prompted with a domain- and task-specific “init prompt,” synthesizes $n$ high-level Python planning programs $\{\rho_i\}_{i=1}^n$, each accepting task instruction $I$ and observation $o$, and outputting a candidate plan $p_i$.
  - **Plan Scoring**: Each plan $p_i$ is scored by a compact language model critic $C_\theta$, which computes log-likelihoods of natural language plan descriptions, applies length normalization, and softmaxes across candidates. The top-scoring plan is selected for execution.

- **Learning Phase**:
  - **Program Evolution**: The LLM refines the set $\{\rho_i\}$ using an “evolve prompt” that summarizes performance over the last $M$ episodes.
  - **Critic Fine-Tuning**: Transitions $(I, o, p, r, o', \text{done})$ are marshaled into a replay buffer. Proximal Policy Optimization (PPO) with Low-Rank Adaptation (LoRA) is applied to further train $C_\theta$ so that its value estimates $V_\phi(s)$—where $s = (I,o,\{p_i\})$—better reflect true long-term returns.

The process alternates until episode-level success rates plateau, yielding a refined ensemble of planning programs and a specialized critic.

## 3. Formalism: Domain-Adaptive Critic 

The critic component is implemented as a lightweight LLM with LoRA-augmented final layers, abstracted as learning a state-value mapping $V_\phi(s)$. Key elements:

- **TD-Error Minimization**:
  \[
  \mathcal{L}(\phi)
  = \mathbb{E}_{(s,a,r,s')\sim D}\left[(r+\gamma V_\phi(s') - V_\phi(s))^2\right]
  \]
- **Inference-Time Scoring**:
  - Plan descriptions $d_{p_i}$ are tokenized and their conditional likelihood under a critic prompt $d_{cp}$ is calculated:
    \[
    \mathrm{prob}(d_{p_i}\mid d_{cp}) = \prod_{k=1}^{N_i} P(w_i^k \mid d_{cp}, w_i^{<k})
    \]
  - Length normalization:
    \[
    \mathrm{logit}(d_{p_i}\mid d_{cp}) = \frac{\log \mathrm{prob}(d_{p_i}\mid d_{cp})}{W_i}
    \]
  - Scores are softmax-normalized over all $n$ candidates:
    \[
    \mathrm{score}(p_i) = \frac{\exp(\mathrm{logit}(d_{p_i}\mid d_{cp}))}{\sum_{j=1}^n \exp(\mathrm{logit}(d_{p_j}\mid d_{cp}))}
    \]
  - The plan $p = \arg\max_i \mathrm{score}(p_i)$ is executed.

## 4. Algorithmic Workflow

A concise view of the CoPiC process:

| Step                      | Description                                                     |
|---------------------------|-----------------------------------------------------------------|
| 1. Initial Program Synthesis | LLM generates $n$ candidate planning programs via init prompt     |
| 2. Candidate Plan Generation  | Each $\rho_i$ produces a plan $p_i$ given current $(I, o)$        |
| 3. Plan Selection            | Critic $C_\theta$ softmax-scores and chooses the best $p_i$       |
| 4. Execution and Feedback    | Execute $p_i$ in environment, collect reward and feedback         |
| 5. Critic Fine-Tuning        | PPO-based RL updates $C_\theta$ using replayed experience         |
| 6. Program Evolution         | If stabilized, evolve programs via LLM using an episode summary   |

Program evolution ensures continual adaptation to environment specifics, while critic fine-tuning enhances reward alignment.

## 5. Empirical Performance and Cost Analysis

Evaluated on ALFWorld, NetHack, and StarCraft II Unit Building, CoPiC establishes superiority over advanced LLM planning baselines (Reflexion, AdaPlanner):

- Achieves a mean **23.33% improvement in success rate** and **91.27% lower query cost** (tokens) compared to Reflexion on ALFWorld.
- Attains an **average 79.39% reduction** in total LLM query token count across all benchmarks.
- **Program evolution** is central: success rate in ALFWorld increases from ~75% to 100% after four refinement iterations.
- Removal of the critic (random candidate selection) reduces success rates by 20–60% at fixed token cost.
- $n=3$ planning programs found optimal for success/cost trade-off.
- Open-source LLMs (e.g., DeepSeek, Qwen2.5-Coder) yield comparable gains, suggesting model-agnosticism.

| Metric                | CoPiC vs. Best Baseline        | Highlighted Result             |
|-----------------------|-------------------------------|-------------------------------|
| Success Rate          | +20.29% average improvement    | Up to 100% on SC2 Hard        |
| LLM Token Cost        | −79.39% reduction              | Up to 96% reduction (NetHack) |
| ALFWorld (tokens)     | 0.74M → 0.16M                  | +20% absolute SR delta        |

## 6. Strengths, Limitations, and Prospects

### Strengths

- **Token Efficiency**: Drastic savings versus step-wise querying planners.
- **Long-Horizon Reward Focus**: Critic-driven selection enables non-greedy, long-term optimal plans.
- **Environment Adaptivity**: Iteratively evolved programs capture environment-specific constraints absent from LLM pretraining.

### Limitations

- **Requires Initial Program Synthesis**: Baseline LLM must yield a working program in the environment.
- **RL Training Overhead**: Critic adaptation incurs training cost and environment interaction demands.

### Directions for Future Work

- Extension to richer, more dynamic domains (e.g., full StarCraft II, Civilization).
- Applications to real-world robotics/industrial process planning.
- Development of more expressive critic architectures, introduction of Q-function-based or multi-agent frameworks.

CoPiC establishes a general approach for amortized, domain-adaptive, and cost-efficient LLM-based planning in complex sequential environments, marking a significant advancement over immediate-feedback planners by closing both the cost and reward alignment gaps [2509.19077].

Source: https://www.emergentmind.com/topics/copic