---
title: 'LEGOMem: Multi-Agent LLM Memory Framework'
url: https://www.emergentmind.com/topics/legomem
type: topic
---

# LEGOMem: Multi-Agent LLM Memory Framework

LEGOMem is a modular procedural memory framework for multi-agent large language model systems for workflow automation. It is designed for settings in which a central orchestrator plans and delegates, specialized task agents execute subtasks with tools, and an external environment contains applications and APIs. Its central premise is that contemporary multi-agent LLM systems are usually stateless, so they repeatedly solve similar workflows from scratch; LEGOMem addresses this by turning successful past task trajectories into reusable memory units that are allocated according to agent role, with full-task memories supporting orchestration and subtask memories supporting execution [2510.04851].

## 1. Problem setting and conceptual scope

LEGOMem studies procedural memory in a common multi-agent architecture for workflow automation: a central orchestrator plans and delegates, multiple specialized task agents execute subtasks using tools, and the environment contains the applications and APIs needed to complete office workflows. The paper focuses on long-horizon office workflows, multi-application tasks, and weaker or smaller models, arguing that these regimes are especially sensitive to the absence of persistent procedural memory [2510.04851].

The framework is motivated by the observation that each task is typically solved independently even when previous tasks exhibited similar decomposition patterns, delegation choices, and tool-use routines. Existing memory methods such as Synapse and AWM are described as mainly targeting single-agent settings, which leaves open three multi-agent design questions: where procedural memory should live, how it should be retrieved, and which agents benefit most from it [2510.04851].

LEGOMem therefore functions both as a practical memory layer and as a design lens. The paper’s central question is not only how to store past experience, but how to decompose that experience into role-specific memory units that can improve both high-level planning and low-level execution. This role-aware decomposition is the main conceptual contribution.

## 2. Role-aware memory decomposition and system model

LEGOMem divides successful trajectories into two kinds of procedural memory. **Full-task memory** captures the global workflow: task description, high-level plan, subtask sequence, final answer, and brief reflection. This memory is intended primarily for the orchestrator, because it supports task decomposition, next-step planning, delegation, and replanning. **Subtask memory** captures localized execution behavior: the responsible task agent, subtask description, tool-use actions, concise observations, and lightweight reasoning in a think-action format. This memory is intended primarily for task agents, because it supports accurate tool usage and correct action ordering on similar subtasks [2510.04851].

The formal system comprises an orchestrator \(A_{\text{orch}}\), task agents \(A=\{A_1,\ldots,A_k\}\), and an environment \(\mathcal{E}\). A task \(T\) has natural-language description \(d\). The orchestrator first produces an initial plan
\[
\pi_0 = \{s_1, \ldots, s_m\}.
\]
At orchestration step \(t\), the next subtask is
\[
s_t = \pi_{\text{orch}(\sigma_t)},
\]
and after a task agent returns summary \(r_t\), the orchestrator updates its state by
\[
\sigma_{t+1} = f(\sigma_t, r_t).
\]
If execution stalls, the orchestrator may produce a revised plan \(\pi'\) [2510.04851].

In the OfficeBench instantiation, the task-agent set includes Word, Excel, Calendar, Email, System, and OCR-PDF agents. This arrangement makes the distinction between global and local procedural memory operational rather than merely conceptual: full-task memories inform orchestration, while subtask memories are routed to the specialized executor most likely to reuse them [2510.04851].

## 3. Memory construction, storage, and online use

LEGOMem operates in two phases. In the offline phase, successful execution logs are converted into structured JSON memories by an LLM. The output contains `"high_level_plan"`, `"subtasks"` with `agent`, `description`, `steps`, and `observations`, `"final_answer"`, and `"reflections"`. The prompt instructs the model to group related actions into subtasks, keep only successful actions, omit malformed or error actions, truncate repetitive actions if too many, and exclude orchestrator coordination steps from subtask memories. From each successful trajectory, LEGOMem derives **1 full-task memory** and **multiple subtask memories** [2510.04851].

These memories are stored in a global memory bank \(\mathcal{M}\) for full tasks and, optionally, in per-agent subtask banks \(\{\mathcal{M}_{A_j}\}\). Retrieval is implemented with embeddings from **OpenAI text-embedding-3-large** and a **FAISS** index. For full-task retrieval, the system embeds the task description \(\phi(d)\); for subtask retrieval, it embeds either the runtime subtask \(s_t\) or a rewritten draft subtask. At inference time, **top-5 memories** are used for the orchestrator and **top-3 memories** for each task agent [2510.04851].

Online inference begins by embedding the new task description \(d_{\text{new}}\) and retrieving similar full-task memories from \(\mathcal{M}\). Full-task memories are passed to the orchestrator, while associated or separately retrieved subtask memories are allocated to relevant task agents. The orchestrator uses the retrieved full-task memories to form an initial plan \(\pi_0\), chooses the next agent \(A_t\), generates subtask \(s_t\), receives execution summary \(r_t\), updates its state, and replans if the workflow stalls. The framework therefore implements a retrieval-augmented procedural memory layer for multi-agent LLM workflows rather than a monolithic archive of past dialogues [2510.04851].

## 4. Variants and design-space exploration

The basic LEGOMem variant uses a single global bank \(\mathcal{M}\) indexed by task-description embeddings. It retrieves full-task memories once, extracts subtask memories from those retrieved tasks, and statically assigns them to agents. The paper characterizes this version as lightweight, simple, and surprisingly competitive overall, while noting that globally similar tasks can still contain mismatched subtasks, which introduces local noise [2510.04851].

LEGOMem-Dynamic adds per-agent subtask banks \(\mathcal{M}_{A_t}\) and performs just-in-time retrieval after the orchestrator generates subtask \(s_t\). The retrieval key is \(\phi(s_t)\), and the intended benefit is less irrelevant subtask memory and better localized execution support. Its explicit cost is repeated embedding and retrieval during execution [2510.04851].

LEGOMem-QueryRewrite shifts fine-grained retrieval earlier. A query rewriting model \(\psi\) generates a draft plan
\[
\pi'_{\text{draft}} = \{s'_1, s'_2, \dots, s'_n\},
\]
and each rewritten subtask \(\phi(s'_j)\) is used to retrieve from the relevant agent memory bank \(\mathcal{M}_{A_j}\). The appendix prompt asks the LLM to break the new task into 3–5 high-level steps and output them in a numbered list inside `<start>` and `<end>`. The intended trade-off is to preserve fine-grained subtask retrieval while avoiding repeated runtime retrieval [2510.04851].

These variants instantiate the paper’s broader design-space study. Their differences are not merely engineering alternatives; they operationalize the core questions of memory placement, retrieval granularity, and agent-specific benefit.

## 5. Experimental setting and empirical results

Experiments are conducted on **OfficeBench**, a benchmark for office workflow automation with **300** total tasks: **148** for training and memory curation, and **152** for test and evaluation. Tasks are grouped into **Level 1** single-application, **Level 2** two-application, and **Level 3** multi-application workflows. Agents interact with simulated Word, Excel, Calendar, Email, System, and OCR/PDF applications in Docker via APIs. Three multi-agent team configurations are evaluated: an LLM team with GPT-4o orchestrator and GPT-4o task agents, a Hybrid team with GPT-4o orchestrator and GPT-4o-mini task agents, and an SLM team with GPT-4o-mini orchestrator and GPT-4o-mini task agents; OCR uses Phi-3.5-mini as the vision-language model [2510.04851].

Using the 148 training tasks, the memory curation procedure keeps successful trajectories and yields **93 full-task memories** and **250 subtask memories**. The primary metric is **success rate**, evaluated programmatically from the final environment state. The paper compares LEGOMem against **No memory**, **Synapse**, and **AWM**, with the latter two adapted from single-agent methods to the multi-agent setting [2510.04851].

| Team | No memory | LEGOMem | LEGOMem-Dynamic | LEGOMem-QueryRewrite |
|---|---:|---:|---:|---:|
| LLM | 45.83 | 58.44 | 57.12 | 55.26 |
| Hybrid | 35.31 | 48.03 | 47.59 | 50.22 |
| SLM | 24.78 | 38.16 | 37.72 | 36.40 |

Across these settings, the reported absolute improvements over no-memory teams are **+12.61** points for the LLM team, **+12.72** points for the Hybrid team, and **+13.38** points for the SLM team. The paper further notes that **Hybrid + LEGOMem-QueryRewrite = 50.22%** exceeds **memory-less LLM team = 45.83%**, and **SLM + vanilla LEGOMem = 38.16%** exceeds **memory-less Hybrid team = 35.31%**, indicating that weaker teams benefit disproportionately from procedural memory [2510.04851].

The memory-placement ablation is one of the most consequential results. For the LLM team, **No memory = 45.83**, **Task Agent memory only = 49.78**, **Orchestrator memory only = 53.29**, and **Orchestrator + Agent memory = 58.44**. For the Hybrid team, **No memory = 35.31**, **Task Agent memory only = 35.31**, **Orchestrator memory only = 47.59**, and **Orchestrator + Agent memory = 48.03**. The paper concludes that orchestrator memory is essential for high-level planning, task decomposition, subtask orchestration, and replanning [2510.04851].

Process-level metrics show related effects. For Level 3 tasks in the LLM team, average steps drop from **26.5** to **22.2**, a **16.2% reduction**, and the failed-step rate drops from **0.275** to **0.225**. The paper also reports that adding lightweight reasoning traces to memory is not consistently necessary: for the LLM team, **LEGOMem without reasoning = 56.36** and **LEGOMem with reasoning = 58.44**; for the Hybrid team, **without reasoning = 49.78** and **with reasoning = 48.03**. Across variants, the differences are mostly within **2 points**, suggesting that the modular memory structure itself carries most of the value [2510.04851].

## 6. Significance, limitations, and relation to adjacent memory research

LEGOMem’s principal significance lies in its role-aware account of procedural memory. The experiments support a hierarchy of importance: orchestrator memory matters most because decomposition, delegation, and coordination determine whether the right subtasks are attempted in the right order; task-agent memory still matters because it improves execution fidelity; and fine-grained subtask retrieval helps most when task agents are weaker. The ablations also show that task-agent-only memory can underperform, vanilla subtask allocation can miss relevant local traces, additional reasoning in memory is not reliably beneficial, dynamic retrieval has runtime overhead, and query rewriting depends on draft-plan quality [2510.04851].

The framework’s limitations are explicit. The memory bank is built from successful runs only, which constrains coverage and may bias memory toward easier or already-solved workflows. The experiments are only on OfficeBench. The architecture studied is also specific: one orchestrator and specialized task agents. Dynamic retrieval and query rewriting add complexity, while vanilla LEGOMem is often competitive. The paper identifies future directions including continual learning from failed trajectories, scaling to open-ended environments, and scaling to broader tool ecosystems [2510.04851].

Adjacent work clarifies where LEGOMem fits in the broader memory-systems landscape. Neuromem argues that external memory should be evaluated under an interleaved insertion-and-retrieval protocol and decomposes the memory lifecycle into five dimensions, including memory data structure, normalization strategy, consolidation policy, query formulation strategy, and context integration mechanism [2602.13967]. This suggests that LEGOMem’s gains could be further dissected at operator level, especially if future versions introduce continual insertion, maintenance, or consolidation of procedural memories. MemReader reframes memory extraction as active memory management with an action space of `add_memory`, `buffer_memory`, `search_memory`, and `ignore_memory`, and reports strong gains on knowledge updating and temporal reasoning [2604.07877]. A plausible implication is that LEGOMem-like systems may benefit from a more selective write policy, particularly if they begin to ingest failed or partial trajectories rather than only successful ones. MemGym, finally, proposes memory-isolated scores that decouple memory performance from reasoning, retrieval, and tool-use ability across tool-use dialogue, deep research, coding, and computer use [2605.20833]. This suggests a natural next step for LEGOMem evaluation: measuring procedural memory quality independently of the raw capability of the orchestrator and task agents.

Within the literature surveyed here, LEGOMem is most precisely characterized as a retrieval-augmented procedural memory layer for multi-agent workflow automation, centered on modular, role-aware reuse of successful execution traces. Its distinguishing claim is not merely that memory helps, but that procedural memory should be placed where the workflow actually needs it: globally at the orchestrator for decomposition and coordination, and locally at task agents for tool-grounded execution.

Source: https://www.emergentmind.com/topics/legomem