Planner-Observer-Reflector Loops

• Planner-Observer-Reflector Loops are closed-loop architectures for sequential decision-making that explicitly coordinate planning, observation, and reflection modules.
• They enable LLM agents to overcome trial-and-error limitations by integrating both internal simulations and external feedback for refined action proposals.
• They incorporate immediate and retrospective feedback mechanisms to improve long-horizon credit assignment and robustly adapt to complex, dynamic environments.

Planner–Observer–Reflector (POR) loops are closed-loop architectures for sequential decision-making in which planning, observation, and reflection are instantiated as explicit, interacting modules. These loops enable embodied LLM agents to overcome the limitations of independent trial-and-error execution by integrating multi-level feedback—both internal and external—during deployment. POR loops serve as a general abstraction encompassing reflective test-time planning and adaptive planning from feedback, facilitating superior long-horizon performance, robust adaptation, and efficient credit assignment in complex environments (Hong et al., 24 Feb 2026, &&&1&&&).

1. Formal Structure and Component Mapping

In the POR loop abstraction, three interlocking modules operate in concert:

• Planner: Generates action proposals or full plans conditioned on the current task and state. In LLM-based agents, the planner may sample multiple candidate actions using a high-temperature decoding scheme or produce hierarchical code plans with subgoals (Hong et al., 24 Feb 2026, Sun et al., 2023).
• Observer: Measures the alignment of expected and observed outcomes, either by simulating and scoring candidate actions prior to execution or by validating asserted subgoals through environment feedback.
• Reflector: Revises, critiques, or trains the agent’s behavior based on both immediate and retrospective feedback, supporting mechanisms for in-situ reflection, post-execution diagnosis, and hindsight re-evaluation with long-horizon credit assignment.

Mapping for Reflective Test-Time Planning

Component	Instantiation	Primary Role
Planner	Policy $\pi_\theta$	High-temperature sampling of $N$ candidates; proposes next actions
Observer	Internal LLM $V_i$	“Reflection-in-action”: critiques and scores each candidate pre-execution
Reflector	External LLM $V_e$	“Reflection-on-action” (post-hoc); “Retrospective reflection” (hindsight for credit assignment)

Mapping for AdaPlanner

Component	Instantiation	Primary Role
Planner	LLM as coder	Generates Pythonic code plans with subgoal-based hierarchy
Observer	Assertion + state	Compares observed state to subgoal assertion outcomes
Reflector	LLM re-prompt	Refines plans or actions in response to assertion failures or ambiguous environment feedback

2. Operational Dynamics: Loop Execution and Reflection Modes

POR loops proceed by integrating multi-level feedback and enabling adaptive updates throughout the agent’s operation:

1. Planning Phase: The Planner generates multiple action candidates (sampling) or a structured, hierarchical plan (code generation), based on the current context and task goals.
2. Observation and Internal Reflection: Each candidate is mentally simulated (Reflective Test-Time Planning) or empirically checked against in-code assertions (AdaPlanner). Scores and critiques are produced, enabling best-of-N selection (Hong et al., 24 Feb 2026) or signaling whether local success/failure has occurred via assertion checks (Sun et al., 2023).
3. Execution and External Reflection: The selected action is executed in the environment. Post-execution, external reflection is triggered to evaluate outcome quality, provide immediate feedback, or prompt full or partial plan revision.
4. Retrospective Reflection and Long-Horizon Credit: After a fixed memory window or logical milestone, retrospective evaluation is performed with full-horizon context, assigning credit (or blame) to past decisions and supporting self-supervised update of both evaluation and policy components (Hong et al., 24 Feb 2026).
5. Online Training and Refinement: Both the internal reflection model and policy are updated with reflective feedback—through either supervised learning (for evaluative models) or REINFORCE-style policy gradients using rescaled reward signals from hindsight evaluations.

3. Key Mathematical Formulations and Learning Objectives

POR loops formalize both the selection and update phases through explicit mathematical formulations.

Candidate Scoring (Reflective Test-Time Planning)

At each decision step $t$,

• Action candidates: $$\hat{a}_{t, k} \sim \pi_\theta(\cdot|x_{\text{action}}; T)$$ for $k=1,\dots, N$
• Observer critique and score: $(f_i^{t,k}, s_i^{t,k}) = V_i(x^{k}_{\text{int}})$, with $s_i^{t,k} \in [0, 100]$
• Best-of-N action selection: $a_t^* = \arg\max_k s_i^{t,k}$

Training Objectives (Reflective Test-Time Planning)

• Internal reflection model: updated by supervised learning on hindsight-augmented reflection pairs

$$\mathcal{L}_{\text{internal}}(\phi_i) = \mathbb{E}_{(a_j, f_r^j) \in D_{\text{train}}} [-\log p_{\phi_i}(f_r^j|x_{\text{int}}(a_j))]$$

• Policy gradient (REINFORCE) update:

$r_j = 2 \cdot (s_r^j/100) - 1$

$$\mathcal{L}_{\text{policy}}(\theta) = - \mathbb{E}_{(a_j, x_{\text{action}}(a_j)) \in D_{\text{train}}} [r_j \cdot \log \pi_\theta(a_j|x_{\text{action}}(a_j))]$$

AdaPlanner Plan Generation and Adaptation

• Plan generation (Planner): $P_0 \sim \rho(P|g, o_1)$ where $P = (a_1, ..., a_T)$
• Plan refinement on deviation: $P_t \sim \rho(P|g, c_t, P_{t-1})$ if feedback deviates
• Action refinement (in-plan): $a_t' \sim \pi(a|g, c_t, P_{t-1})$ for local feedback

4. Practical Implementations and Pseudocode

Reflective Test-Time Planning (Hong et al., 24 Feb 2026) employs synchronous loops over time steps, combining candidate sampling, simulated evaluation, action execution, reflection, memory updates, and periodic test-time training:

for t = 1..T_max: # Planner: Sample N candidate actions candidates = sample_actions(Planner, context, N) # Observer: Internal scoring scores, crits = zip(*(Observer(context, cand) for cand in candidates)) a_star = candidates[argmax(scores)] # Execute action in environment outcome = env.step(a_star) # Reflector: Immediate reflection on actual outcome reflect = Reflector(context, a_star, outcome) store_experience(...) # Retrospective reflection and policy update (when triggered) if len(memory) == K or milestone_reached(): hindsight_reflect() update_models() clear_memory()

AdaPlanner (Sun et al., 2023) operates on code-based plan execution with dynamic refinement:

1. Generate solution code with embedded subgoal assertions.
2. Execute actions; if assertions hold, proceed and extract variables by agent-invoked parsing. If an assertion fails, capture error, re-prompt LLM to generate revised code (resume with minimal loss), and continue execution.

5. Addressing Temporal Credit Assignment and Robust Adaptation

POR loops explicitly address the temporal credit assignment problem by leveraging retrospective, hindsight-based feedback. Retrospective reflection permits credit to flow from long-range outcomes back to temporally distant actions, obviating the need for explicit multi-step rollouts during action selection (Hong et al., 24 Feb 2026). This approach ensures that actions with deleterious downstream effects are systematically penalized and model updates promote avoidance of strategies with hidden long-term risks.

The integration of in-plan and out-of-plan refinement mechanisms, as instantiated in AdaPlanner, enhances robustness to unexpected environment feedback and reduces the incidence of hallucination through constrained code-based prompts. The introduction of skill discovery further augments sample efficiency by incorporating successful solution exemplars as additional few-shot demonstrations, with empirical results demonstrating significant improvements in both ALFWorld and MiniWoB++ environments. AdaPlanner achieves +3.73% and +4.11% absolute increases in success rates over baselines, while using 2× and ≈600× fewer samples, respectively (Sun et al., 2023).

6. Applications, Benchmarks, and Empirical Highlights

POR loop methodologies have been evaluated extensively on both simulated and real-robot domains:

• Long-Horizon Household and MuJoCo Cupboard Fitting: Reflective Test-Time Planning demonstrates significant gains in long-horizon embodied tasks, with ablative studies validating the complementary nature of internal and external reflection (Hong et al., 24 Feb 2026).
• ALFWorld and MiniWoB++: AdaPlanner, employing the POR loop, outperforms prior LLM-agent baselines in success rate and sample efficiency, utilizing a code-constrained interface and memory-based few-shot skill discovery (Sun et al., 2023).

System	Benchmark	Success Rate (%)	Sample Efficiency (Few-shot Examples)
AdaPlanner	ALFWorld (GPT-3)	91.79	6 (vs. 12 for ReAct/Reflexion)
AdaPlanner	MiniWoB++	92.87	59 (vs. 23,000+ in CC-Net)

A key result is the top-left positioning of AdaPlanner on the success-vs.-sample plots, indicating a desirable tradeoff between accuracy and sample cost.

7. Significance and Future Directions

POR loops constitute a principled architectural motif for adaptive, LLM-based sequential agents. By formalizing the interplay between planning, observation, and reflection—including both immediate and retrospective mechanisms—POR frameworks enable agents to accumulate deployment experience, perform efficient long-horizon credit assignment, and adapt continually to unmodeled feedback. The convergence of internal simulation (reflection-in-action), post-hoc diagnosis (reflection-on-action), and hindsight-based update (retrospective reflection) provides a robust foundation for autonomous agents in open-ended, dynamic environments (Hong et al., 24 Feb 2026, Sun et al., 2023).

Further research on POR loops is likely to explore richer credit assignment protocols, tighter integration with skill induction and transfer, and broader application in multi-modal and physically grounded domains. A plausible implication is that such architectures may generalize to a wide class of sequential high-level planning, reasoning, and learning problems beyond the current scope of embodied language agent benchmarks.