Act-Observe-Rewrite (AOR) Framework

• AOR is a closed-loop framework for adaptive agent behavior that systematically revises policies through action execution, outcome observation, and explicit code rewriting.
• It employs a dual-loop structure—fast control for in-episode actions and slow reasoning for policy updates—to diagnose failures and achieve high success rates in tasks like robotic manipulation.
• The framework’s use of interpretable, code-based policy revision enables precise error localization and rapid corrective adaptations, contrasting with opaque, gradient-based methods.

Act-Observe-Rewrite (AOR) is a general framework for adaptive agent behavior characterized by a closed-loop process of action execution, outcome observation, and systematic policy revision. The distinctive property of AOR is that policy improvement occurs through explicit, interpretable modifications to the decision logic or control code—rather than through gradient-based updates, black-box learning, or predefined skill selection. Originally formalized in the context of multimodal coding agents for robotic manipulation and active observer models in human vision, AOR provides a rigorous substrate for in-context learning, systematic failure diagnosis, and hypothesis-driven adaptation in both artificial and biological agents (Kumar, 3 Mar 2026, Solbach et al., 2021).

1. Core Principles and Formalization

AOR consists of three canonical steps, repeated cyclically:

1. Act: The agent executes its current policy or strategy in the environment.
2. Observe: The agent collects outcome data, including both raw sensory observations and structured diagnostic metrics.
3. Rewrite: The agent revises its policy or strategy—often as fully executable code—by reasoning about observed failures and their systematic causes.

Formally, this cycle operates over a history $H_n = \{(\tau_1, I_1), ..., (\tau_n, I_n)\}$, where $\tau_k$ denotes episode outcomes (reward $r\in\{0,1\}$, diagnostic signals $\phi_i$, step count $T$), and $I_k$ are associated key-frame observations. The policy at iteration $n$, $\pi_n$, is typically representable as source code (e.g., a Python class with control logic).

Policy update is performed by an LLM or planning module:

$\pi_{n+1} = \text{LLM}(H_n, \text{source}(\pi_n))$

This reframes in-context learning as symbolic reasoning and code synthesis, in contrast to parametric gradient descent.

In cognitive modeling, AOR is instantiated as a hypothesize–test–replan loop, where belief updates use Bayesian inference:

$$P(h|O_{1:t}) = \frac{P(O_t|h,S_t)P(h|O_{1:t-1})}{\sum_{h'}P(O_t|h',S_t)P(h'|O_{1:t-1})}$$

and action (method) selection is cast as maximizing expected information gain minus cost.

2. Algorithmic Structure in Robotic Manipulation

The Act-Observe-Rewrite loop is implemented in two nested frequencies:

• Fast loop (control frequency, intra-episode):
  • Executes the current Python policy $\tau_k$0 stepwise. At each timestep $\tau_k$1, state features $\tau_k$2 are computed from RGB-D observations, actions $\tau_k$3 are determined, safety clamps applied, and the result commanded to the robot.
  • Key observations and transitions are stored for diagnostic analysis.
• Slow loop (reasoning frequency, inter-episode):

1. Computes episode outcome $\tau_k$4.
2. Updates history $\tau_k$5.
3. Prompts a multimodal coding LLM with $\tau_k$6’s source, $\tau_k$7, key-frame images, and targeted questions regarding failure modes and root causes.
4. Receives a revised controller class $\tau_k$8; verifies its safety and correctness.
5. Iterates until successful execution or call budget exhaustion (Kumar, 3 Mar 2026).

A typical generated controller is a finite-state machine with explicit PID-style proportional control, safety clamping, and parameterizations (e.g., gains, thresholds) that are subject to direct revision.

3. AOR in Human Active Vision and Cognitive Programs

Solbach & Tsotsos empirically demonstrated the AOR cycle in human 3D visual problem solving (Solbach et al., 2021). Participants in a same–different block structure task cycled between:

• Act: Hypothesizing and deploying visual/comparative strategies such as "divide-and-conquer" or "alternating fixation," each mapped to Cognitive Program (CP) methods.
• Observe: Acquiring new visual input via eye fixations (mean ≈ 92.4 per trial) and attentional feature extraction, updating confidence in hypotheses.
• Rewrite: Switching strategies or parameterizations if confidence thresholds are unmet, dynamically composing new methods with Bayesian action selection and replanning criteria.

Cognitive Programs embody the methods and scripts (≈50 elemental methods mined from behavioral data) supporting this cycle, with the STAR architecture providing working memory, executive control, and the data structures necessary for adaptive strategy deployment.

4. Empirical Validation and Benchmark Tasks

In robotic manipulation, AOR was validated on three robosuite tasks:

Task	Scene & Challenge	LLM Calls	Final Success Rate
Lift	Single cube, depth/camera bias	3	100%
PickPlaceCan	Can & bin, color/vision failure	2	100%
Stack	Two cubes, vision conventions	20	91%

• Lift: Success reached after correcting a systematic depth bias and adding stationary grasp logic.
• PickPlaceCan: Vision masking and component filtering revised to identify objects robustly.
• Stack: Multiple controller revisions addressed camera y-flip, matrix conventions, grasp retries, and placement collisions, achieving high but not perfect performance due to residual local optima (Kumar, 3 Mar 2026).

In the human active observer domain, subjects achieved 93.8% accuracy (σ=3.9%), with zero detectable learning effect over 18 consecutive trials, suggesting reliance on dynamic AOR-driven strategy composition rather than cumulative parametric learning (Solbach et al., 2021).

5. Interpretability and Systematic Failure Diagnosis

A distinctive property of AOR is its reliance on interpretable policy representations. Policy as code enables:

• Direct localization of systematic errors—“line 27: y_p is negated but robosuite uses OpenGL conventions”—and causal patching.
• Expressive changes such as insertion of new behavioral phases, logic modifications, or algorithmic modules in a single LLM call.
• Diagnosis and rapid correction of both vision and control failures, in contrast to opaque neural policies that support only incremental weight updates.

Opaque or black-box policies lack the ability to expose internal causal mechanisms to the agent’s reasoning process, reducing failure diagnosis to coarse reward or performance metrics (Kumar, 3 Mar 2026).

6. Comparative Analyses, Limitations, and Prospects

AOR efficacy depends critically on the underlying reasoning and code synthesis capabilities of the LLM. Empirically, Codex GPT-5.3 did not solve any tasks in the same budget on the benchmark suite, establishing a lower bound on required LLM competence (Kumar, 3 Mar 2026). Safety and stability ablations demonstrated that the compile sandbox and action clamping are necessary for robust policy iteration.

In human cognitive modeling, while the AOR framework successfully models dynamic strategy adaptation, limitations include the restriction to same–different block tasks and the absence of fully quantitative Selective Tuning model integration. No learning or transfer across tasks was detected (Solbach et al., 2021).

This suggests that AOR provides a general and interpretable template for in-context, non-parametric policy improvement in both artificial and biological agents. The framework’s reliance on explicit reasoning about policy representations marks a qualitative shift from reinforcement-based or parametric learning, with direct implications for robotic autonomy, human–machine interaction, and models of executive control.

7. Extensions and Future Directions

Proposed extensions include:

• Application of AOR mechanisms to broader classes of visuomotor and spatial-relation tasks in robotics and neurocognitive modeling.
• Closing the perception–action loop by deploying Cognitive Programs in simulation or on physical robots.
• Automatic learning of diagnostic metrics, cost models, and method libraries from demonstration data to optimize AOR policy search.
• Systematic investigation into the types of failure amenable to single-shot code-level intervention and the scaling properties of AOR as task complexity increases (Kumar, 3 Mar 2026, Solbach et al., 2021).