---
title: Reactive Acting Engine (RAE)
url: https://www.emergentmind.com/topics/reactive-acting-engine-rae
type: topic
---

# Reactive Acting Engine (RAE)

Searching arXiv for the cited RAE papers and closely related work to ground the article.
{"query":"RAE Reactive Acting Engine hierarchical operational models arXiv 2010.01909 2003.03932 2507.11345 1611.00230","max_results":10}
{"query":"arXiv:2010.01909 Reactive Acting Engine", "max_results": 5}
Reactive Acting Engine (RAE) is a hierarchical acting architecture for dynamic, nondeterministic environments in which acting is primary and planning is supportive. In the RAE literature, it is also called the **Refinement Acting Engine**. Its defining idea is that tasks should be refined into executable control routines online, using **hierarchical operational models** shared by both execution and planning, rather than a separate descriptive planner model and an operational executor model. RAE is explicitly presented as an acting system that does not require a complete plan in the classical sense; instead, it chooses refinement methods for the current task and context, executes them step by step, monitors outcomes, and can invoke the anytime planner **UPOM** when a choice among alternatives is needed [2010.01909; 2507.11345].

## 1. Historical motivation and architectural position

RAE was proposed against a long-standing mismatch between **descriptive planning models** and **operational acting models**. The descriptive model used by a planner typically contains symbolic actions with abstract preconditions and effects, whereas the operational model actually executed on a robot contains control flow, error handling, perception loops, motion routines, retries, and other closed-loop procedures. The RAE papers identify this as a planner/controller gap and argue that it makes consistency checking difficult, increases modeling cost, and complicates the smooth interleaving of acting and planning [2010.01909; 2003.03932].

Within this formulation, RAE is not a passive dispatcher of a symbolic action sequence. It is an **online actor** that executes **hierarchical operational procedures**. When given a task, it determines applicable refinement methods in the current state, chooses one, executes its body incrementally, recursively refines subtasks, executes primitive actions in the world, and reacts to action failure by trying alternative refinements. This design is explicitly said to be **inspired by PRS**, but differs from classical deliberative acting frameworks in that the same operational models are used for both acting and planning, and methods may contain “any complex algorithm” rather than only simple decomposition schemas [2010.01909; 2003.03932].

The primary motivation for reactive acting is that robotic execution is **nondeterministic and state-dependent in real time**. The later physical-deployment study gives concrete examples: object poses may be unknown before exploration, perception may fail entirely, a navigation command may need to be retried, manipulation can fail because pose estimates are inaccurate, and external interruptions such as arm cable entanglement can occur. In this setting, the RAE view is that the robot should not merely execute a precomputed symbolic plan; it should continuously refine tasks into executable control routines online while using planning selectively to choose among alternative refinements under uncertainty [2507.11345].

## 2. Hierarchical operational models

The central representational device is the **hierarchical operational model**. In one formalization, an acting domain is
\[
\Sigma = (\Xi, \mathcal{T}, \mathcal{M}, \mathcal{A}),
\]
where \(\Xi\) is the set of world states, \(\mathcal{T}\) the set of tasks and events, \(\mathcal{M}\) the set of methods, and \(\mathcal{A}\) the set of actions [2010.01909]. In another presentation, the domain is written as
\[
\Sigma = (S, \mathcal{T}, \mathcal{M}, \mathcal{A}),
\]
where \(S\) is the set of states, \(\mathcal{T}\) the set of tasks, \(\mathcal{M}\) the set of method instances, and \(\mathcal{A}\) the set of commands [2003.03932; 2507.11345]. The notational difference reflects presentation, not a change in the basic idea.

A method is an operational procedure for handling a task. The papers describe method syntax in terms of a method name, a task or event identifier, a precondition test, a body program, and, where needed, a parameter expression defining possible values of extra arguments. Method bodies can include subtasks, primitive actions or commands, assignments, loops, if/else logic, and arbitrary control structures. This is a key distinction from standard HTN methods: the body is not merely a symbolic decomposition but an executable routine [2003.03932; 2507.11345].

Method instances are produced by parameter binding, and applicability is state-dependent. The relevant decision problem is therefore not “which flat operator should be executed next,” but “which applicable refinement method is best for the current task and current context.” A task can have multiple methods, and a single method template can yield multiple method instances depending on parameters. The physical robot deployment makes this concrete with tasks such as `collect_obj(r, tb)` and `perceive(r, o)`, where extra parameters like the target table or camera pan/lift settings produce multiple alternatives. UPOM searches over precisely these alternatives [2507.11345].

This shared operational semantics is the main architectural claim of RAE. **RAE executes these methods on the real robot, and UPOM simulates rollouts through the same methods.** There is therefore no separate symbolic planning model. A plausible implication is that RAE should be understood less as a planner with an execution wrapper than as an online hierarchical actor with optional deliberative support [2010.01909; 2507.11345].

## 3. Execution semantics and control loop

RAE maintains execution context explicitly. A central data structure is the **refinement stack**,
\[
\sigma = \langle (\tau, m, i), \ldots, (\tau_0, m_0, i_0)\rangle,
\]
or, in the richer runtime presentation,
\[
(\tau, m, i, tried),
\]
where \(\tau\) is the current task or subtask, \(m\) the chosen method instance, \(i\) the current instruction index in \(body(m)\), and `tried` the set of method instances already attempted and failed for that task [2003.03932; 2010.01909]. This stack records where the actor currently is in the hierarchical refinement.

At runtime, RAE progresses tasks one method step at a time. For a task \(\tau\), it obtains the applicable methods \(Applicable(s,\tau)\) or \(Applicable(\xi,\tau)\), chooses one by reactive ordering or planning, starts executing its body, and then repeatedly handles one of several cases. If the current line is a subtask, RAE recursively refines it. If it is a primitive action or command, RAE triggers the action and later checks whether execution status is `running`, `done`, or `failed`. If it is an assignment or other control instruction, RAE updates state and continues. The action-monitoring semantics are central: if a command is still running, RAE waits or attends to other pending tasks; if it succeeds, RAE advances; if it fails, RAE invokes recovery through retry or alternate refinement [2010.01909; 2507.11345].

The later robotic deployment makes the overall control loop concrete by describing three queues: a **task queue**, a **command execution queue**, and a **command status queue**. A task such as `collect_all_objs(r)` is retrieved, candidate method instances are generated from different methods and different parameter values, UPOM is consulted when alternatives exist, the selected method is executed sequentially, and command outcomes are monitored online. Two implementation details added in the deployment are explicitly emphasized: **partial-success utility**, in which failure after collecting some objects does not force rollout utility to zero, and a **retry-count parameter**, so that the same method can be attempted multiple times before being discarded [2507.11345].

Failure handling is carried by `Retry`. The papers stress that `Retry` is **not backtracking**: it does not restore a previous world state. Instead, it looks for another applicable method for the current task in the **current** state. This makes RAE suitable for dynamic environments in which previously applicable methods may no longer be valid and newly applicable ones may appear [2010.01909].

A common misconception is that RAE is just a plan executor over hierarchical tasks. The papers are explicit that this is not the intended interpretation. Acting continues while the environment evolves, the actor observes the latest state before reactive decisions, and method choice is revisited at each refinement point rather than fixed once at the beginning [2003.03932; 2010.01909].

## 4. Deliberative augmentation: UPOM and learning

RAE’s deliberative component is **UPOM**, described as a **UCT-like online planner** or **UCT Procedure for Operational Models**. UPOM is called when RAE must choose among multiple applicable methods. Unlike classical planning over flat action sequences, UPOM searches in the space induced by operational models: disjunction over applicable methods, deterministic progression through method bodies, and stochastic branching over sampled primitive-action outcomes [2010.01909; 2003.03932].

The planner is **progressive deepening, receding-horizon, anytime**. For task \(\tau\) in state \(s\), `Select-Method` initializes a candidate by heuristic,
\[
\tilde{m} \gets \arg\max_{m \in Applicable(s,\tau)} h(\tau,m,s),
\]
then progressively increases search depth, performs a fixed number of rollouts, maintains \(Q_{s,\sigma}(m)\) estimates, and returns the method with the best current estimate [2003.03932]. At task-choice points, method selection follows a UCB-style rule,
\[
\phi(m,\tau)=Q_{s,\sigma}(m)+C\sqrt{\log N_{s,\sigma}(\tau)/N_{s,\sigma}(m)},
\]
with incremental backup of rollout utility [2003.03932].

The 2020 RAE paper develops explicit utility models. For efficiency, if an action succeeds with cost \(c\), its value is \(1/c\); if it fails, utility is \(0\). Sequential composition uses
\[
e_1 \oplus e_2 =
\begin{cases}
e_2 & \text{if } e_1 = \infty \\
e_1 & \text{if } e_2 = \infty \\
\frac{e_1 e_2}{e_1 + e_2} & \text{otherwise}.
\end{cases}
\]
For success ratio, successful action outcomes have value \(1\), failures have value \(0\), and composition is multiplicative [2003.03932; 2010.01909]. The 2025 deployment introduces a task-specific rollout utility
\[
U = \sum_{i=1}^{n} R(\sv{collected\_at}(s_i)) \cdot \left(c_1 + c_2 \cdot e^{-k \cdot C_i}\right),
\]
with cumulative cost
\[
C_i = \sum_{j=1}^{i} cost(s_{j-1}, a_{j-1}, s_j),
\]
and horizon \(C_n \le \eta\), thereby rewarding successful collection while discounting later collections [2507.11345].

The framework also incorporates supervised learning. **Learn\(\Pi\)** or **LearnM** learns a mapping from context to method; **LearnH** learns a heuristic utility estimator for UPOM; and one paper further describes **LearnMI** for choosing uninstantiated method parameters. The training records are generated from acting experiences or simulated planning results, encoded with one-hot state, task, and method representations, and trained with neural networks using SGD and cross-entropy loss. A stated limitation is that the learned selectors predict methods rather than full method instances, and in the earlier papers the refinement stack \(\sigma\) is omitted from the learning features [2003.03932; 2010.01909].

The main formal guarantee concerns UPOM. In static domains, with \(d_{max}=\infty\), finite state/task/method/action sets, finite method-generated step sequences, and monotonic utility, the papers state that UPOM converges asymptotically to the optimal method instance and that each \(Q\)-value converges to expected utility [2003.03932; 2010.01909]. The same papers are careful to note that this proof does not cover dynamic domains with exogenous events or practical anytime interruption.

## 5. Empirical behavior and real-robot deployment

The 2020 simulation studies report that UPOM and the learning strategies significantly improve RAE’s performance in multiple robot-centered domains under the metrics **efficiency** and **success ratio**. The comparison includes purely reactive RAE, RAE with an earlier planner, RAE with UPOM, RAE with learned policy, and RAE with UPOM guided by a learned heuristic. The papers report that RAE with UPOM is more efficient than purely reactive RAE with **95% confidence** in all four domains, that planning with UPOM improves success ratio over reactive RAE with **95% confidence in two domains** and **85% confidence in the other two**, and that **UPOM is more than twice as fast** as the earlier planner on average [2003.03932].

The same evaluation emphasizes that even small search budgets matter. The later RAE paper reports that **5 rollouts already significantly improve efficiency** and sharply reduce retry ratio, while **UPOM + LearnH** with \(d_{max}=5\) and \(n_{ro}=50\) reduced computation time by about **88%** compared with a large-search setting [2010.01909]. The learned direct policy improves over fixed reactive ordering, but the learned heuristic combined with planning usually performs better than policy-only replacement [2003.03932; 2010.01909].

The strongest deployment claim is the **first physical deployment** of RAE+UPOM on a real mobile manipulator for an object collection task. The robot platform is **Mobipick**, composed of a mobile base, a custom middle section, a **UR5 arm**, and a camera mounted on the arm or end-effector. The task is object collection across multiple tables under unknown initial object locations, sensor noise, action failures, reachability constraints, and uncertain perception and manipulation outcomes. The deployed system used **100 UPOM rollouts per method/parameter selection**, yielding planning times of roughly **1 second per subtask decision** [2507.11345].

The deployment results are qualitative and quantitative. **Trial 1.1 (box used)** achieved utility **47.45**, collected **4/4 objects**, with planning **42.18s** and acting **500s**. **Trial 1.2 (box unavailable)** achieved utility **32.09**, collected **4/4 objects**, with planning **40.74s** and acting **549s**. Additional trials show reactive recovery: during an arm/cable emergency stop, **RAE continued retrying actions until execution could resume**; during navigation failure, RAE retried with the same parameters up to the retry count, later selected a different table, and eventually returned to the original target; by contrast, if an object is not detected at all and is absent from symbolic state, RAE cannot reason about collecting it, and the paper explicitly states that this remains unhandled [2507.11345].

## 6. Related frameworks, misconceptions, and limitations

RAE belongs to a broader family of systems that interleave acting and planning, but it should not be conflated with every reactive architecture. A closely related 2016 robotics paper shows how backward chaining over action preconditions and effects can synthesize and incrementally update a **Behavior Tree** whose semantics provide re-execution of undone effects, skipping of externally achieved subgoals, local structural expansion when a condition fails, and alternative actions under a fallback node. That paper is directly relevant because it addresses a classic RAE-style problem—execution toward a goal in a changing world with online repair—but it is best viewed as a related reactive execution-and-planning method rather than a full general RAE [1611.00230].

The literature also marks out important boundaries. RAE is broader than a runtime adaptation mechanism for actor systems, although work on localized mitigation, incremental synchronization, and formally safe adaptation scripts in actor systems provides a useful runtime-adaptation substrate for reactive engines [1709.02346]. Conversely, some recent LLM-agent work argues that reactive next-action selection should not be the default in prompt-injection-prone web environments and recommends **plan-then-execute** instead. This critique is directed most strongly at ReAct-like web agents, not at RAE specifically, but it highlights a broader architectural caution: if execution-time observations can redefine control and those observations are untrusted natural language, then reactive execution opens a control-flow attack surface [2605.14290]. A plausible implication is that the RAE commitment to online refinement is best matched to domains where the operational model is trusted and richly structured.

The standard misconceptions are therefore twofold. First, RAE is not a classical planner that outputs a fixed plan for later execution. Second, it is not a model-free reactive policy. It depends on **hand-authored hierarchical operational models**, on applicability conditions, and, in the integrated versions, on a simulator or simulated counterpart for each primitive command used in UPOM rollouts [2003.03932; 2507.11345].

The limitations are explicit across the papers. RAE depends on authored methods; it does not learn operational models themselves. UPOM’s strongest guarantees assume static domains and no depth bound. The learned components predict methods rather than full method instances in some versions, and omission of the refinement stack from learning features is identified as a likely source of lost contextual information. The physical deployment further notes substantial upfront effort for simulated command models, imperfect predictive fidelity of hand-crafted simulations, sparse utility signal in early exploration, and incomplete handling of completely undetected objects [2003.03932; 2010.01909; 2507.11345].

Taken together, the RAE literature defines a **reactive deliberative actor** in which hierarchical operational models are the common substrate for execution, planning, and, in later work, learning. Its characteristic behavior is online refinement, stepwise monitoring, retry in the current state rather than backtracking to a previous state, and selective planning at exactly those points where method choice matters.

Source: https://www.emergentmind.com/topics/reactive-acting-engine-rae