Organize then Retrieve: Hierarchical Memory Navigation for Efficient Agents

Abstract: LLM agents struggle with long-horizon tasks due to their inherent statelessness, requiring all task-relevant information to be encoded in growing input contexts. The resulting degraded reasoning quality, increased inference cost, and higher latency necessitate efficient working memory mechanisms. However, existing approaches either rely on lossy compression or similarity-based retrieval, which often fail to capture temporal structure and causal dependencies required for multi-step agentic tasks. In this work, we present HORMA, a Hierarchical Organize-and-Retrieve Memory Agent that organizes experience into a file-system-like hierarchical structure, where summarized entities are linked to the corresponding raw trajectories, enabling efficient access without losing detailed information. HORMA decomposes working memory into two stages: structured memory construction and navigation-based retrieval. The construction module iteratively refines how experiences are structured by distinguishing between failures caused by missing information and those caused by misleading or overloaded context. The navigation module retrieves task-relevant context by traversing the hierarchy using a lightweight agent trained with reinforcement learning to select minimal yet sufficient context, thereby reducing latency along the critical execution path. Across ALFWorld, LoCoMo, and LongMemEval, HORMA improves task performance under constrained context budgets while requiring at most 22.17% of the baseline token usage in long conversation tasks. Compared to existing methods, it consistently achieves better efficiency-performance trade-offs and generalizes effectively to unseen tasks.

• The paper introduces HORMA, which separates memory organization from retrieval by combining provenance-preserving hierarchical files, recursively refined management skills, and RL-trained navigation.
• HORMA improves performance across ALFWorld, LoCoMo, and LongMemEval, reaching 73.9% success on ALFWorld and reducing primary-agent token use to as little as 1.24% of baseline levels.
• The results show that strong memory organization is the main bottleneck, while a lightweight 4B retriever can generalize across benchmarks and reduce temporal and irrelevant-retrieval errors.

Motivation and problem statement

LLM agents operating over long horizons face a structural constraint: because the underlying model is stateless, all task-relevant information must be re-materialized in the input context at every step. The paper identifies two dominant failure modes in existing working-memory designs. "History hoarding" agents retain large raw trajectories, producing context overload, information dilution, lost-in-the-middle effects, and high latency and inference cost. Lossy approaches—summarization, learned compression, and context folding—irreversibly discard fine-grained details that may become decisive late in a task. External memory systems improve storage scalability but typically organize experience as flat collections retrieved by semantic similarity, which fails to capture temporal hierarchies and causal dependencies; retrieval degenerates into shallow semantic matching that surfaces temporally inconsistent or causally irrelevant content.

A further weakness concerns optimization. Most prior systems treat memory construction and retrieval as a monolithic pipeline jointly optimized with sparse task-level rewards. The authors argue this introduces a severe credit assignment gap: when a long-horizon task fails, it is unclear whether the cause lies in poor organization, inaccurate retrieval, or downstream reasoning. Their central design claim is that these two functions differ fundamentally—construction shapes long-term structure with delayed effects, while retrieval is a repeated per-step decision on the critical execution path with localized feedback—and should therefore be decoupled and optimized differently.

The HORMA framework

HORMA (Hierarchical Organize-and-Retrieve Memory Agent) formalizes its setting as a Memory-augmented MDP in which an external workspace $\mathcal{F}_t$ evolves through a transition operator applied to each interaction. The primary agent $M_\theta$ remains frozen; action generation is decomposed as $$\pi(a_t \mid o_t, \mathcal{F}_t, q) = M_r(\mathbf{C}_t \mid \mathcal{F}_t, q)\, M_\theta(a_t \mid o_t, \mathbf{C}_t, q)$$, where the retriever $M_r$ selects context $\mathbf{C}_t$ from the workspace. The framework extends naturally to conversational QA by replacing full dialogue history conditioning with retrieval-conditioned answering.

Grounded hierarchical workspace. Memory is organized as a file system rather than a flat buffer. For each interaction or dialogue turn, the manager archives raw trajectories into timestamped directories and synthesizes structured notes that carry temporal metadata and references back to raw traces. This preserves provenance: compact notes serve fast access while raw evidence remains recoverable.

Memory management via recursive skill refinement. Rather than optimizing construction through unstable long-horizon RL, HORMA treats construction as structure induction driven by prompts. A domain-agnostic initial prompt specifies entity tracking, event abstraction, and relation grouping. It is then iteratively refined via contrastive analysis between runs using raw history versus managed context. Two failure sets are distinguished: exogenous failures (raw history succeeds, managed context fails), indicating information loss during construction, and endogenous failures (managed context succeeds, raw history fails), indicating that structuring mitigates hallucination or attention dilution. LLM-generated feedback over these contrasts augments the prompt with reusable skills—a form of textual gradient descent. This is a non-parametric update mechanism requiring no model fine-tuning.

Retrieval via RL. The retrieval agent navigates the hierarchy with Bash commands (ls, grep, cd, cat) augmented with two terminal actions, select and done, enabling iterative verification of collected evidence. Retrieval is optimized with GRPO on a lightweight backbone (Qwen 3.5 4B) using an evidence-grounded reward—the Jaccard overlap between retrieved context and ground-truth evidence—which provides dense, retrieval-specific supervision decoupled from downstream task outcomes.

Empirical results

Evaluation covers ALFWorld (134 tasks across 6 categories under 1950- and 2200-token context limits), LoCoMo (519 QA instances under a 10K limit), and LongMemEval (367 instances under a 50K limit), with Claude Sonnet 4.5 as the primary agent across all methods.

Benchmark	Metric	HORMA	Best baseline
ALFWorld (small limit)	Success rate	56.7%	29.1% (slide window)
ALFWorld (large limit)	Success rate	73.9%	71.6% (ReSum)
LoCoMo (10K limit)	L-J overall	51.6	43.4 (Mem0)
LongMemEval (50K limit)	L-J overall	55.9	51.8 (A-MEM)

On token efficiency the gains are pronounced: HORMA uses only 3.07%–22.17% of baseline tokens on LoCoMo and 1.24%–16.19% on LongMemEval, with all HORMA variants operating within roughly 1000 tokens per query. On ALFWorld, HORMA also improves Pareto efficiency, achieving both fewer interaction steps and lower tokens per step than truncation.

Several ablation findings carry specific implications:

• Skill evolution matters. Adding self-evolved management skills lifts Claude-based performance on ALFWorld (large) from 51.5 to 73.9 and on LongMemEval from 43.6 to 55.9, while reducing retrieval calls.
• RL-trained lightweight retrieval generalizes out of distribution. A Qwen 3.5 4B retriever trained only on the LoCoMo training split improves LoCoMo from 32.6 to 42.2 and transfers zero-shot to ALFWorld (35.8 → 64.9) and LongMemEval (40.6 → 58.0). Notably, the trained 4B retriever surpasses the Claude Sonnet 4.5 retriever on LongMemEval (58.0 vs. 55.9)—a strong result given the backbone asymmetry.
• Management quality is the bottleneck. Cross-backbone experiments show that upgrading the retriever yields non-uniform gains: pairing Qwen 3.5 4B management with Sonnet 4.5 retrieval produces only marginal improvement (L-J 21.0 → 23.3), whereas stronger managers benefit substantially from better retrieval. Flawed organization cannot be compensated by optimal navigation, empirically justifying the use of high-capacity models for the manager role.
• Navigated retrieval reduces temporal errors. Error attribution on temporally sensitive LongMemEval tasks shows HORMA substantially reduces non-reasoning errors (temporal staleness, irrelevant retrieval) relative to embedding retrieval over the same managed notes, isolating the improvement to retrieval accuracy rather than reasoning capability.
• Skills accumulate productively. Starting from an empty library, four refinement rounds yield 63 skills on LongMemEval with steadily improving task performance; related domains (LoCoMo, LongMemEval) converge on identical endogenous skills such as temporal precision anchoring and verbatim quote preservation.

Limitations and open questions

The paper concedes several constraints. First, the evidence-grounded RL training signal requires ground-truth evidence sets drawn directly from the dataset, so the online, interaction-driven variant of retrieval training remains unexplored—the conclusion explicitly flags extending the framework to fully online learning as future work. Second, the skill-refinement loop depends on contrastive rollouts against an unconstrained baseline, which presupposes access to a strong frontier model and doubles evaluation cost during skill acquisition. Third, the memory-construction bottleneck analysis shows that small open-weight managers (Qwen 3.5 4B achieves only L-J 21.0 on LoCoMo) are currently inadequate for the manager role, leaving open whether smaller managers can be made viable through targeted training rather than prompting. Fourth, the adversarial category scores remain low even for HORMA (13.4 on LoCoMo against a no-limit score of 1.5, but far below single-hop performance), indicating that recognizing unanswerable queries is not fully solved. Finally, the reported token figures concern primary-agent input tokens; the computational cost of the manager and retriever calls themselves is acknowledged implicitly but not fully accounted for in the efficiency comparison.

Conclusion

HORMA demonstrates that decoupling working memory into an asynchronous, skill-evolving hierarchical organizer and an RL-trained navigational retriever yields consistent performance gains alongside order-of-magnitude reductions in context consumption across embodied and conversational benchmarks. The strongest empirical claims—that a 4B retriever trained on one benchmark can outperform frontier-model retrieval zero-shot on another, and that management quality bounds what any retriever can achieve—are supported by controlled ablations. The main unresolved question is whether the evidence-grounded training recipe can be adapted to fully online settings without dataset-provided ground truth.