---
title: Planner-Observer-Reflector Loops
url: https://www.emergentmind.com/topics/planner-observer-reflector-loops
type: topic
---

# Planner-Observer-Reflector Loops

Planner–Observer–Reflector (POR) loops are closed-loop architectures for sequential decision-making in which planning, observation, and reflection are instantiated as explicit, interacting modules. These loops enable embodied language model (LLM) agents to overcome the limitations of independent trial-and-error execution by integrating multi-level feedback—both internal and external—during deployment. POR loops serve as a general abstraction encompassing reflective test-time planning and adaptive planning from feedback, facilitating superior long-horizon performance, robust adaptation, and efficient credit assignment in complex environments [2602.21198][2305.16653].

## 1. Formal Structure and Component Mapping

In the POR loop abstraction, three interlocking modules operate in concert:

- **Planner:** Generates action proposals or full plans conditioned on the current task and state. In LLM-based agents, the planner may sample multiple candidate actions using a high-temperature decoding scheme or produce hierarchical code plans with subgoals [2602.21198][2305.16653].
- **Observer:** Measures the alignment of expected and observed outcomes, either by simulating and scoring candidate actions prior to execution or by validating asserted subgoals through environment feedback.
- **Reflector:** Revises, critiques, or trains the agent’s behavior based on both immediate and retrospective feedback, supporting mechanisms for in-situ reflection, post-execution diagnosis, and hindsight re-evaluation with long-horizon credit assignment.

### Mapping for Reflective Test-Time Planning

| Component  | Instantiation       | Primary Role                                                                                  |
|------------|--------------------|----------------------------------------------------------------------------------------------|
| Planner    | Policy $\pi_\theta$| High-temperature sampling of $N$ candidates; proposes next actions                           |
| Observer   | Internal LLM $V_i$ | “Reflection-in-action”: critiques and scores each candidate pre-execution                     |
| Reflector  | External LLM $V_e$ | “Reflection-on-action” (post-hoc); “Retrospective reflection” (hindsight for credit assignment)|

### Mapping for AdaPlanner

| Component  | Instantiation       | Primary Role                                                                              |
|------------|--------------------|------------------------------------------------------------------------------------------|
| Planner    | LLM as coder       | Generates Pythonic code plans with subgoal-based hierarchy                               |
| Observer   | Assertion + state  | Compares observed state to subgoal assertion outcomes                                    |
| Reflector  | LLM re-prompt      | Refines plans or actions in response to assertion failures or ambiguous environment feedback|

## 2. Operational Dynamics: Loop Execution and Reflection Modes

POR loops proceed by integrating multi-level feedback and enabling adaptive updates throughout the agent’s operation:

1. **Planning Phase:** The Planner generates multiple action candidates (sampling) or a structured, hierarchical plan (code generation), based on the current context and task goals.
2. **Observation and Internal Reflection:** Each candidate is mentally simulated (Reflective Test-Time Planning) or empirically checked against in-code assertions (AdaPlanner). Scores and critiques are produced, enabling best-of-N selection [2602.21198] or signaling whether local success/failure has occurred via assertion checks [2305.16653].
3. **Execution and External Reflection:** The selected action is executed in the environment. Post-execution, external reflection is triggered to evaluate outcome quality, provide immediate feedback, or prompt full or partial plan revision.
4. **Retrospective Reflection and Long-Horizon Credit:** After a fixed memory window or logical milestone, retrospective evaluation is performed with full-horizon context, assigning credit (or blame) to past decisions and supporting self-supervised update of both evaluation and policy components [2602.21198].
5. **Online Training and Refinement:** Both the internal reflection model and policy are updated with reflective feedback—through either supervised learning (for evaluative models) or REINFORCE-style policy gradients using rescaled reward signals from hindsight evaluations.

## 3. Key Mathematical Formulations and Learning Objectives

POR loops formalize both the selection and update phases through explicit mathematical formulations.

### Candidate Scoring (Reflective Test-Time Planning)

At each decision step $t$,
- Action candidates: $\hat{a}_{t, k} \sim \pi_\theta(\cdot|x_{\text{action}}; T)$ for $k=1,\dots, N$
- Observer critique and score: $(f_i^{t,k}, s_i^{t,k}) = V_i(x^{k}_{\text{int}})$, with $s_i^{t,k} \in [0, 100]$
- Best-of-N action selection: $a_t^* = \arg\max_k s_i^{t,k}$

### Training Objectives (Reflective Test-Time Planning)

- Internal reflection model: updated by supervised learning on hindsight-augmented reflection pairs
  $$
  \mathcal{L}_{\text{internal}}(\phi_i) = \mathbb{E}_{(a_j, f_r^j) \in D_{\text{train}}} [-\log p_{\phi_i}(f_r^j|x_{\text{int}}(a_j))]
  $$
- Policy gradient (REINFORCE) update:
  $$
  r_j = 2 \cdot (s_r^j/100) - 1
  $$
  $$
  \mathcal{L}_{\text{policy}}(\theta) = - \mathbb{E}_{(a_j, x_{\text{action}}(a_j)) \in D_{\text{train}}} [r_j \cdot \log \pi_\theta(a_j|x_{\text{action}}(a_j))]
  $$

### AdaPlanner Plan Generation and Adaptation

- Plan generation (Planner): $P_0 \sim \rho(P|g, o_1)$ where $P = (a_1, ..., a_T)$
- Plan refinement on deviation: $P_t \sim \rho(P|g, c_t, P_{t-1})$ if feedback deviates
- Action refinement (in-plan): $a_t' \sim \pi(a|g, c_t, P_{t-1})$ for local feedback

## 4. Practical Implementations and Pseudocode

Reflective Test-Time Planning [2602.21198] employs synchronous loops over time steps, combining candidate sampling, simulated evaluation, action execution, reflection, memory updates, and periodic test-time training:

```python
for t = 1..T_max:
    # Planner: Sample N candidate actions
    candidates = sample_actions(Planner, context, N)
    # Observer: Internal scoring
    scores, crits = zip(*(Observer(context, cand) for cand in candidates))
    a_star = candidates[argmax(scores)]
    # Execute action in environment
    outcome = env.step(a_star)
    # Reflector: Immediate reflection on actual outcome
    reflect = Reflector(context, a_star, outcome)
    store_experience(...)
    # Retrospective reflection and policy update (when triggered)
    if len(memory) == K or milestone_reached():
        hindsight_reflect()
        update_models()
        clear_memory()
```

AdaPlanner [2305.16653] operates on code-based plan execution with dynamic refinement:

1. Generate solution code with embedded subgoal assertions.
2. Execute actions; if assertions hold, proceed and extract variables by agent-invoked parsing. If an assertion fails, capture error, re-prompt LLM to generate revised code (resume with minimal loss), and continue execution.

## 5. Addressing Temporal Credit Assignment and Robust Adaptation

POR loops explicitly address the temporal credit assignment problem by leveraging retrospective, hindsight-based feedback. Retrospective reflection permits credit to flow from long-range outcomes back to temporally distant actions, obviating the need for explicit multi-step rollouts during action selection [2602.21198]. This approach ensures that actions with deleterious downstream effects are systematically penalized and model updates promote avoidance of strategies with hidden long-term risks.

The integration of in-plan and out-of-plan refinement mechanisms, as instantiated in AdaPlanner, enhances robustness to unexpected environment feedback and reduces the incidence of hallucination through constrained code-based prompts. The introduction of skill discovery further augments sample efficiency by incorporating successful solution exemplars as additional few-shot demonstrations, with empirical results demonstrating significant improvements in both ALFWorld and MiniWoB++ environments. AdaPlanner achieves +3.73% and +4.11% absolute increases in success rates over baselines, while using 2× and ≈600× fewer samples, respectively [2305.16653].

## 6. Applications, Benchmarks, and Empirical Highlights

POR loop methodologies have been evaluated extensively on both simulated and real-robot domains:

- **Long-Horizon Household and MuJoCo Cupboard Fitting:** Reflective Test-Time Planning demonstrates significant gains in long-horizon embodied tasks, with ablative studies validating the complementary nature of internal and external reflection [2602.21198].
- **ALFWorld and MiniWoB++:** AdaPlanner, employing the POR loop, outperforms prior LLM-agent baselines in success rate and sample efficiency, utilizing a code-constrained interface and memory-based few-shot skill discovery [2305.16653].

| System         | Benchmark         | Success Rate (%) | Sample Efficiency (Few-shot Examples) |
|----------------|------------------|------------------|---------------------------------------|
| AdaPlanner     | ALFWorld (GPT-3) | 91.79            | 6 (vs. 12 for ReAct/Reflexion)        |
| AdaPlanner     | MiniWoB++        | 92.87            | 59 (vs. 23,000+ in CC-Net)            |

A key result is the top-left positioning of AdaPlanner on the success-vs.-sample plots, indicating a desirable tradeoff between accuracy and sample cost.

## 7. Significance and Future Directions

POR loops constitute a principled architectural motif for adaptive, LLM-based sequential agents. By formalizing the interplay between planning, observation, and reflection—including both immediate and retrospective mechanisms—POR frameworks enable agents to accumulate deployment experience, perform efficient long-horizon credit assignment, and adapt continually to unmodeled feedback. The convergence of internal simulation (reflection-in-action), post-hoc diagnosis (reflection-on-action), and hindsight-based update (retrospective reflection) provides a robust foundation for autonomous agents in open-ended, dynamic environments [2602.21198][2305.16653].

Further research on POR loops is likely to explore richer credit assignment protocols, tighter integration with skill induction and transfer, and broader application in multi-modal and physically grounded domains. A plausible implication is that such architectures may generalize to a wide class of sequential high-level planning, reasoning, and learning problems beyond the current scope of embodied language agent benchmarks.

Source: https://www.emergentmind.com/topics/planner-observer-reflector-loops