Reactive Acting Engine (RAE)

• RAE is a hierarchical acting architecture that refines tasks online using shared operational models for both planning and execution.
• It supports dynamic decision-making in nondeterministic, state-dependent environments by selecting applicable refinement methods based on current context.
• Integrated with the UPOM planner and learning modules, RAE improves efficiency and robustness, demonstrating solid performance in real-world robot deployments.

Searching arXiv for the cited RAE papers and closely related work to ground the article. {"query":"RAE Reactive Acting Engine hierarchical operational models arXiv (Patra et al., 2020, Patra et al., 2020, Lima et al., 15 Jul 2025, Colledanchise et al., 2016)","max_results":10} {"query":"(Patra et al., 2020) Reactive Acting Engine", "max_results": 5} Reactive Acting Engine (RAE) is a hierarchical acting architecture for dynamic, nondeterministic environments in which acting is primary and planning is supportive. In the RAE literature, it is also called the Refinement Acting Engine. Its defining idea is that tasks should be refined into executable control routines online, using hierarchical operational models shared by both execution and planning, rather than a separate descriptive planner model and an operational executor model. RAE is explicitly presented as an acting system that does not require a complete plan in the classical sense; instead, it chooses refinement methods for the current task and context, executes them step by step, monitors outcomes, and can invoke the anytime planner UPOM when a choice among alternatives is needed (Patra et al., 2020, Lima et al., 15 Jul 2025).

1. Historical motivation and architectural position

RAE was proposed against a long-standing mismatch between descriptive planning models and operational acting models. The descriptive model used by a planner typically contains symbolic actions with abstract preconditions and effects, whereas the operational model actually executed on a robot contains control flow, error handling, perception loops, motion routines, retries, and other closed-loop procedures. The RAE papers identify this as a planner/controller gap and argue that it makes consistency checking difficult, increases modeling cost, and complicates the smooth interleaving of acting and planning (Patra et al., 2020, Patra et al., 2020).

Within this formulation, RAE is not a passive dispatcher of a symbolic action sequence. It is an online actor that executes hierarchical operational procedures. When given a task, it determines applicable refinement methods in the current state, chooses one, executes its body incrementally, recursively refines subtasks, executes primitive actions in the world, and reacts to action failure by trying alternative refinements. This design is explicitly said to be inspired by PRS, but differs from classical deliberative acting frameworks in that the same operational models are used for both acting and planning, and methods may contain “any complex algorithm” rather than only simple decomposition schemas (Patra et al., 2020, Patra et al., 2020).

The primary motivation for reactive acting is that robotic execution is nondeterministic and state-dependent in real time. The later physical-deployment study gives concrete examples: object poses may be unknown before exploration, perception may fail entirely, a navigation command may need to be retried, manipulation can fail because pose estimates are inaccurate, and external interruptions such as arm cable entanglement can occur. In this setting, the RAE view is that the robot should not merely execute a precomputed symbolic plan; it should continuously refine tasks into executable control routines online while using planning selectively to choose among alternative refinements under uncertainty (Lima et al., 15 Jul 2025).

2. Hierarchical operational models

The central representational device is the hierarchical operational model. In one formalization, an acting domain is

$$\Sigma = (\Xi, \mathcal{T}, \mathcal{M}, \mathcal{A}),$$

where $\Xi$ is the set of world states, $\mathcal{T}$ the set of tasks and events, $\mathcal{M}$ the set of methods, and $\mathcal{A}$ the set of actions (Patra et al., 2020). In another presentation, the domain is written as

$$\Sigma = (S, \mathcal{T}, \mathcal{M}, \mathcal{A}),$$

where $S$ is the set of states, $\mathcal{T}$ the set of tasks, $\mathcal{M}$ the set of method instances, and $\mathcal{A}$ the set of commands (Patra et al., 2020, Lima et al., 15 Jul 2025). The notational difference reflects presentation, not a change in the basic idea.

A method is an operational procedure for handling a task. The papers describe method syntax in terms of a method name, a task or event identifier, a precondition test, a body program, and, where needed, a parameter expression defining possible values of extra arguments. Method bodies can include subtasks, primitive actions or commands, assignments, loops, if/else logic, and arbitrary control structures. This is a key distinction from standard HTN methods: the body is not merely a symbolic decomposition but an executable routine (Patra et al., 2020, Lima et al., 15 Jul 2025).

Method instances are produced by parameter binding, and applicability is state-dependent. The relevant decision problem is therefore not “which flat operator should be executed next,” but “which applicable refinement method is best for the current task and current context.” A task can have multiple methods, and a single method template can yield multiple method instances depending on parameters. The physical robot deployment makes this concrete with tasks such as collect_obj(r, tb) and perceive(r, o), where extra parameters like the target table or camera pan/lift settings produce multiple alternatives. UPOM searches over precisely these alternatives (Lima et al., 15 Jul 2025).

This shared operational semantics is the main architectural claim of RAE. RAE executes these methods on the real robot, and UPOM simulates rollouts through the same methods. There is therefore no separate symbolic planning model. A plausible implication is that RAE should be understood less as a planner with an execution wrapper than as an online hierarchical actor with optional deliberative support (Patra et al., 2020, Lima et al., 15 Jul 2025).

3. Execution semantics and control loop

RAE maintains execution context explicitly. A central data structure is the refinement stack,

$\Xi$0

or, in the richer runtime presentation,

$\Xi$1

where $\Xi$2 is the current task or subtask, $\Xi$3 the chosen method instance, $\Xi$4 the current instruction index in $\Xi$5, and tried the set of method instances already attempted and failed for that task (Patra et al., 2020, Patra et al., 2020). This stack records where the actor currently is in the hierarchical refinement.

At runtime, RAE progresses tasks one method step at a time. For a task $\Xi$6, it obtains the applicable methods $\Xi$7 or $\Xi$8, chooses one by reactive ordering or planning, starts executing its body, and then repeatedly handles one of several cases. If the current line is a subtask, RAE recursively refines it. If it is a primitive action or command, RAE triggers the action and later checks whether execution status is running, done, or failed. If it is an assignment or other control instruction, RAE updates state and continues. The action-monitoring semantics are central: if a command is still running, RAE waits or attends to other pending tasks; if it succeeds, RAE advances; if it fails, RAE invokes recovery through retry or alternate refinement (Patra et al., 2020, Lima et al., 15 Jul 2025).

The later robotic deployment makes the overall control loop concrete by describing three queues: a task queue, a command execution queue, and a command status queue. A task such as collect_all_objs(r) is retrieved, candidate method instances are generated from different methods and different parameter values, UPOM is consulted when alternatives exist, the selected method is executed sequentially, and command outcomes are monitored online. Two implementation details added in the deployment are explicitly emphasized: partial-success utility, in which failure after collecting some objects does not force rollout utility to zero, and a retry-count parameter, so that the same method can be attempted multiple times before being discarded (Lima et al., 15 Jul 2025).

Failure handling is carried by Retry. The papers stress that Retry is not backtracking: it does not restore a previous world state. Instead, it looks for another applicable method for the current task in the current state. This makes RAE suitable for dynamic environments in which previously applicable methods may no longer be valid and newly applicable ones may appear (Patra et al., 2020).

A common misconception is that RAE is just a plan executor over hierarchical tasks. The papers are explicit that this is not the intended interpretation. Acting continues while the environment evolves, the actor observes the latest state before reactive decisions, and method choice is revisited at each refinement point rather than fixed once at the beginning (Patra et al., 2020, Patra et al., 2020).

4. Deliberative augmentation: UPOM and learning

RAE’s deliberative component is UPOM, described as a UCT-like online planner or UCT Procedure for Operational Models. UPOM is called when RAE must choose among multiple applicable methods. Unlike classical planning over flat action sequences, UPOM searches in the space induced by operational models: disjunction over applicable methods, deterministic progression through method bodies, and stochastic branching over sampled primitive-action outcomes (Patra et al., 2020, Patra et al., 2020).

The planner is progressive deepening, receding-horizon, anytime. For task $\Xi$9 in state $\mathcal{T}$0, Select-Method initializes a candidate by heuristic,

$\mathcal{T}$1

then progressively increases search depth, performs a fixed number of rollouts, maintains $\mathcal{T}$2 estimates, and returns the method with the best current estimate (Patra et al., 2020). At task-choice points, method selection follows a UCB-style rule,

$\mathcal{T}$3

with incremental backup of rollout utility (Patra et al., 2020).

The 2020 RAE paper develops explicit utility models. For efficiency, if an action succeeds with cost $\mathcal{T}$4, its value is $\mathcal{T}$5; if it fails, utility is $\mathcal{T}$6. Sequential composition uses

$\mathcal{T}$7

For success ratio, successful action outcomes have value $\mathcal{T}$8, failures have value $\mathcal{T}$9, and composition is multiplicative (Patra et al., 2020, Patra et al., 2020). The 2025 deployment introduces a task-specific rollout utility

$\mathcal{M}$0

with cumulative cost

$\mathcal{M}$1

and horizon $\mathcal{M}$2, thereby rewarding successful collection while discounting later collections (Lima et al., 15 Jul 2025).

The framework also incorporates supervised learning. Learn$\mathcal{M}$3 or LearnM learns a mapping from context to method; LearnH learns a heuristic utility estimator for UPOM; and one paper further describes LearnMI for choosing uninstantiated method parameters. The training records are generated from acting experiences or simulated planning results, encoded with one-hot state, task, and method representations, and trained with neural networks using SGD and cross-entropy loss. A stated limitation is that the learned selectors predict methods rather than full method instances, and in the earlier papers the refinement stack $\mathcal{M}$4 is omitted from the learning features (Patra et al., 2020, Patra et al., 2020).

The main formal guarantee concerns UPOM. In static domains, with $\mathcal{M}$5, finite state/task/method/action sets, finite method-generated step sequences, and monotonic utility, the papers state that UPOM converges asymptotically to the optimal method instance and that each $\mathcal{M}$6-value converges to expected utility (Patra et al., 2020, Patra et al., 2020). The same papers are careful to note that this proof does not cover dynamic domains with exogenous events or practical anytime interruption.

5. Empirical behavior and real-robot deployment

The 2020 simulation studies report that UPOM and the learning strategies significantly improve RAE’s performance in multiple robot-centered domains under the metrics efficiency and success ratio. The comparison includes purely reactive RAE, RAE with an earlier planner, RAE with UPOM, RAE with learned policy, and RAE with UPOM guided by a learned heuristic. The papers report that RAE with UPOM is more efficient than purely reactive RAE with 95% confidence in all four domains, that planning with UPOM improves success ratio over reactive RAE with 95% confidence in two domains and 85% confidence in the other two, and that UPOM is more than twice as fast as the earlier planner on average (Patra et al., 2020).

The same evaluation emphasizes that even small search budgets matter. The later RAE paper reports that 5 rollouts already significantly improve efficiency and sharply reduce retry ratio, while UPOM + LearnH with $\mathcal{M}$7 and $\mathcal{M}$8 reduced computation time by about 88% compared with a large-search setting (Patra et al., 2020). The learned direct policy improves over fixed reactive ordering, but the learned heuristic combined with planning usually performs better than policy-only replacement (Patra et al., 2020, Patra et al., 2020).

The strongest deployment claim is the first physical deployment of RAE+UPOM on a real mobile manipulator for an object collection task. The robot platform is Mobipick, composed of a mobile base, a custom middle section, a UR5 arm, and a camera mounted on the arm or end-effector. The task is object collection across multiple tables under unknown initial object locations, sensor noise, action failures, reachability constraints, and uncertain perception and manipulation outcomes. The deployed system used 100 UPOM rollouts per method/parameter selection, yielding planning times of roughly 1 second per subtask decision (Lima et al., 15 Jul 2025).

The deployment results are qualitative and quantitative. Trial 1.1 (box used) achieved utility 47.45, collected 4/4 objects, with planning 42.18s and acting 500s. Trial 1.2 (box unavailable) achieved utility 32.09, collected 4/4 objects, with planning 40.74s and acting 549s. Additional trials show reactive recovery: during an arm/cable emergency stop, RAE continued retrying actions until execution could resume; during navigation failure, RAE retried with the same parameters up to the retry count, later selected a different table, and eventually returned to the original target; by contrast, if an object is not detected at all and is absent from symbolic state, RAE cannot reason about collecting it, and the paper explicitly states that this remains unhandled (Lima et al., 15 Jul 2025).

6. Related frameworks, misconceptions, and limitations

RAE belongs to a broader family of systems that interleave acting and planning, but it should not be conflated with every reactive architecture. A closely related 2016 robotics paper shows how backward chaining over action preconditions and effects can synthesize and incrementally update a Behavior Tree whose semantics provide re-execution of undone effects, skipping of externally achieved subgoals, local structural expansion when a condition fails, and alternative actions under a fallback node. That paper is directly relevant because it addresses a classic RAE-style problem—execution toward a goal in a changing world with online repair—but it is best viewed as a related reactive execution-and-planning method rather than a full general RAE (Colledanchise et al., 2016).

The literature also marks out important boundaries. RAE is broader than a runtime adaptation mechanism for actor systems, although work on localized mitigation, incremental synchronization, and formally safe adaptation scripts in actor systems provides a useful runtime-adaptation substrate for reactive engines (Cassar, 2017). Conversely, some recent LLM-agent work argues that reactive next-action selection should not be the default in prompt-injection-prone web environments and recommends plan-then-execute instead. This critique is directed most strongly at ReAct-like web agents, not at RAE specifically, but it highlights a broader architectural caution: if execution-time observations can redefine control and those observations are untrusted natural language, then reactive execution opens a control-flow attack surface (Piet et al., 14 May 2026). A plausible implication is that the RAE commitment to online refinement is best matched to domains where the operational model is trusted and richly structured.

The standard misconceptions are therefore twofold. First, RAE is not a classical planner that outputs a fixed plan for later execution. Second, it is not a model-free reactive policy. It depends on hand-authored hierarchical operational models, on applicability conditions, and, in the integrated versions, on a simulator or simulated counterpart for each primitive command used in UPOM rollouts (Patra et al., 2020, Lima et al., 15 Jul 2025).

The limitations are explicit across the papers. RAE depends on authored methods; it does not learn operational models themselves. UPOM’s strongest guarantees assume static domains and no depth bound. The learned components predict methods rather than full method instances in some versions, and omission of the refinement stack from learning features is identified as a likely source of lost contextual information. The physical deployment further notes substantial upfront effort for simulated command models, imperfect predictive fidelity of hand-crafted simulations, sparse utility signal in early exploration, and incomplete handling of completely undetected objects (Patra et al., 2020, Patra et al., 2020, Lima et al., 15 Jul 2025).

Taken together, the RAE literature defines a reactive deliberative actor in which hierarchical operational models are the common substrate for execution, planning, and, in later work, learning. Its characteristic behavior is online refinement, stepwise monitoring, retry in the current state rather than backtracking to a previous state, and selective planning at exactly those points where method choice matters.