FrankenAgent: Brain-Morphic Robotic Architecture

• FrankenAgent is a brain-morphic, modular robotic architecture that integrates vision-language models, hierarchical memory, and multi-tier anomaly handling to execute tasks robustly in unstructured environments.
• Its Cortex module transforms natural language instructions and initial scene observations into executable task plans while minimizing expensive VLM calls for efficiency.
• Empirical evaluations show FrankenAgent reduces VLM calls by ~80% and boosts task success rates compared to baseline systems, ensuring effective real-time operations.

FrankenAgent is a brain-morphic, modular robotic agent architecture engineered for general, high-efficiency task execution in dynamic, unstructured environments through the integration of vision-LLMs (VLMs), hierarchical memory, policy generation, and multi-tier anomaly handling. Each module of FrankenAgent is inspired by a distinct neuroanatomical region, collectively orchestrating human-like capabilities in task planning, local feedback control, experience-driven memory, and robust safety-critical operations, while minimizing calls to computationally intensive VLMs (Wang et al., 24 Jun 2025).

1. Modular, Brain-Morphic Architecture

FrankenAgent decomposes robotic cognition and control into four principal modules, each mapped to a major brain region:

• Cortex: Responsible for high-level reasoning and global task planning, synthesizing natural-language instructions ($\ell$), initial RGB-D scene observations ($O(0)$), and memory context into a structured Hierarchical Execution Tree (HET) and multi-level anomaly handlers (MAH). A single or minimal VLM call is employed for this step.
• Cerebellum: Executes local policy generation, dynamic feedback control, and fast reflexes. It interprets HET nodes and instantiates motion primitives from an Incremental Skill Pool (ISP), using dynamic feedback loops at $\sim$200 Hz.
• TemporalLobe–Hippocampus Complex: Implements a hierarchical three-tier long-term memory system, supporting short-, medium-, and lifelong retention, providing relevant context and experience-driven optimizations to the Cortex.
• Brainstem: Manages low-level sensor/actuator interfaces, real-time reflexes, scene feature extraction, and exposes an anomaly detector that mediates module-level fault tolerance.

Dataflow initiates with $(\ell, O(0))$ ingested by the Cortex, which outputs the HET and MAH schemes. The Cerebellum dispatches and controls skill primitives; the Hippocampus injects memory-augmented prompts; the Brainstem interfaces with hardware and safety-critical feedback.

2. Module Functionality and Interactions

Cortex (Task Planning)

Given $(\ell, O(0), \text{memory context})$, the Cortex leverages a VLM prompt template to produce executable code:

Given instruction: {ℓ} Scene summary: {short+medium memories} Available skills: {ISP descriptors} Produce: (a) Preorder-traversal state-machine code (HET) (b) Python monitors & fallback logic (MAH)

HET nodes $v_j = (T_j, A_j, C_j)$ encode task descriptions, parameters, and hierarchical connections, and output a multi-threaded anomaly-handling schedule.

Cerebellum (Policy and Reflex)

For each HET node, the Cerebellum instantiates an appropriate skill $s_k$ from the ISP, invokes dynamic feedback controllers (LQR/PID), and executes high-frequency correction:

procedure ExecuteNode(v_j): (T_j,A_j,C_j) = v_j skill = ISP.select_skill(T_j) while not skill.done(): u_t = skill.command(A_j) s_t = brainstem.get_keypoints() skill.adjust(PID(u_t, s_t)) if anomaly_detector(s_t, u_t): raise Anomaly(v_j) return C_j(A_j)

TemporalLobe–Hippocampus Complex (Memory System)

Memory is organized as follows:

• Short-Term Memory (STM): Circular buffer, last 10 events.
• Medium-Term Memory (MTM): Function templates invoked >3 times/hour.
• Lifelong Memory (LTM): Human-verified prompts and optimizations.

Retrieval employs cosine similarity over embedding space, e.g.,

$$\text{score}(e_q, e_m) = \frac{e_q \cdot e_m}{\|e_q\| \|e_m\|},\quad \{m\} \leftarrow \arg\max_{m\in\text{MTM}\cup\text{LTM}} \text{score}(e_q,e_m)$$

STM is clustered every $T_1$ = 10 min to promote to MTM if $\ge3$ calls. Templates with $\ge5$ invocations and $\ge90\%$ code coverage success are frozen into the ISP.

Brainstem (Low-Level and Reflex)

Publishes commands at $\geq200$ Hz, captures sensory frames, and computes low-dimensional scene features $s_t$. The anomaly metric at time $t$,

$\delta_t = E(s_t, u_t),\quad E(s, u) = \|s - u\|_2$

is used for node-specific thresholding and escalation paths.

3. Task Planning, VLM Utilization, and Code Generation

FrankenAgent achieves its efficiency by centering nearly all global reasoning in a single VLM prompt, augmented with relevant memory and skill descriptors. This call generates the full HET and MAH in executable form, minimizing the need for subsequent VLM queries—only invoking a second call when a “Complex” anomaly appears. Scene, instruction, and skill embeddings are constructed from keypoint/object-class encodings and off-the-shelf text encoders such as CLIP-text.

4. Module Coordination and Central Orchestration Loop

Agent behavior is controlled by a central loop that dispatches HET nodes and anomaly handlers:

ℓ, O(0) → cortex.generate(HET, MAH) spawn_thread(MAH.scheduler) for each subtask ℓ_i in HET.root: for each node v_j in preorder(HET[ℓ_i]): try: next_node = cerebellum.ExecuteNode(v_j) except Anomaly as A: MAH.handle(A, v_j) break wait_for_all_threads()

Anomalies are diagnosed according to:

$$\begin{cases} \text{use predefined monitor} & \delta_t\le \tau_{\text{predef}}\ \text{invoke local LLM expert} & \tau_{\text{predef}} < \delta_t \le \tau_{\text{local}}\ \text{invoke cortex VLM replan} & \delta_t > \tau_{\text{local}} \end{cases}$$

5. Anomaly Monitoring, Handling, and Memory Cascade

Anomaly handling is stratified into three tiers, empirically accounting for system robustness and latency:

Anomaly Type	Share (%)	Handler	Latency
Predictable	70	Predefined rule	$\Delta t \approx 0.3$ s
Recoverable	20	Local LLM expert	$\Delta t \approx 3.0$ s
Complex	10	Cortex VLM replan	$\Delta t \approx 20$ s

Handlers operate as follows:

if delta <= tau_predef: predefined_handler() elif delta <= tau_local: local_expert.adjust_sequence() else: (HET, MAH) = cortex.replan(current_state)

This multi-level anomaly cascade ensures that $~97\%$ of predictable anomalies are absorbed at the lowest level with minimal disruption and latency.

6. Long-Term Memory Management and Update Mechanisms

Memory structuring enables rapid context retrieval and operational efficiency. The STM supports $O(1)$ updates, MTM ($\mathrm{NM} \lesssim 100$) is accessed in $O(\mathrm{NM})$ or $O(\log \mathrm{NM})$ (KD-tree indexing), and LTM ($\mathrm{NL} \lesssim 50$) holds expert-vetted prompts. Frequent event summarization reduces VLM calls by $\sim27\%$ and generated code size by $\sim41\%$; memory retrieval is $\,\ll$1 ms, compared to VLM call ($\sim5$ s). Promotion rules for templates are

$$\text{MTM} \leftarrow \{\tau: \mathrm{count}(\tau) \geq 3 \}$$

$$\text{ISP} \leftarrow \text{ISP} \cup \{\tau: \mathrm{count}(\tau)\ge5,\, \mathrm{cov}(\tau)\ge 90\%\}$$

Templates with $<80\%$ success after anomalies are re-validated or demoted.

7. Quantitative Performance and Practical Considerations

Let $V$ be the number of VLM calls, $S$ the average subtask count, and $A$ the anomaly rate. FrankenAgent achieves

$V \approx 1 + A_{\text{complex}}$

Complexity bounds are $O(1)$ for planning and $O(S \cdot r)$ for local operations, compared to $O(S)$ VLM calls for baselines. Empirical results (simulation and physical robots) are:

• VLM calls per task: $1.1\pm0.2$ (FrankenAgent) vs $5.7$ (“no HMM” ablation)
• Success rate: $73\%$ (FrankenAgent), $46\%$ (VoxPoser), $55\%$ (ReKep)
• Average time per task: $10.5$ s (FrankenAgent) vs $18.7$ s (stacking tasks)
• Removing HMM: success drops to $26.7\%$, VLM calls increase fivefold

A single-shot, code-generation prompt is critical for orchestration latency. Hierarchical memory underpins VLM savings, especially in long-horizon, cross-task scenarios. Multi-level anomaly handling is indispensable for robust, real-time operation. System performance remains sensitive to VLM selection; GPT-4.1 was empirically optimal for cost/performance (Wang et al., 24 Jun 2025).

8. Significance, Lessons, and Design Implications

FrankenAgent demonstrates that brain-morphic modular decomposition—integrating single-call, memory-augmented global reasoning with rapid local control and a hierarchical anomaly response—enables general, robust, and efficient zero-shot deployment in robotic systems. Long-term memory structures reduce dependence on VLM queries, and the multi-stage anomaly handling achieves near-human-level stability and efficiency. Both the Hierarchical Execution Tree and Hierarchical Memory Module are essential; ablations confirm sharp reductions in performance when removed. These design principles provide transferable blueprints for multifunctional, efficient, and robust robotic cognition architectures.