Long-Horizon Agentic Search Frameworks

• Long-horizon agentic search frameworks are methods that integrate dynamic tool use with iterative reasoning to tackle multi-step, knowledge-intensive tasks.
• They interleave model reasoning with targeted external retrieval and document synthesis, effectively addressing context window limits and error propagation.
• By leveraging reinforcement learning, adaptive memory management, and goal-oriented planning, these frameworks optimize query generation and scalable problem solving.

Long-horizon agentic search frameworks constitute a class of methods for LLMs and large reasoning models (LRMs) that autonomously conduct open-ended, multi-step retrieval and reasoning over external resources, particularly in knowledge-intensive, multi-hop domains. These frameworks tightly integrate dynamic tool use (especially search engines and document retrievals) within the reasoning process, using explicit mechanisms for uncertainty identification, targeted search, information synthesis, memory management, and robust error handling. They are built to overcome the inherent limitations of static, single-shot retrieval-augmented generation (RAG) pipelines and address the distinctive challenges of task complexity, context window constraints, trajectory length, and error propagation that arise in extended search scenarios.

1. Architectures of Agentic Search for Long-Horizon Reasoning

Long-horizon agentic search frameworks are architected to interleave model reasoning with fine-grained, self-triggered tool use across multiple steps, in contrast to static RAG-based pipelines. Key architectural elements include:

• Agentic Retrieval-Augmented Generation (agentic RAG) modules: The model determines during reasoning when to issue a specially tagged query (e.g., <|begin_search_query|>...<|end_search_query|>) at points of uncertainty, which invokes an external search or retrieval interface. Retrieved documents are then incorporated (delimited, e.g., <|begin_search_result|>...<|end_search_result|>), enabling dynamic, just-in-time supplementation (Li et al., 9 Jan 2025).
• Iterative Reasoning Traces and Tree Search: Systems such as AgenticLU employ a “Chain-of-Clarifications” workflow, where the model iteratively generates clarification questions and context groundings in a tree-like, branching process, with each node representing a self-generated clarification step (Zhuang et al., 21 Feb 2025). This allows both breadth (exploring alternative clarifications) and depth (multi-step decomposition) in reasoning.
• Memory and Summarization: The Memory-as-Action paradigm reframes memory management as an intrinsic action the agent can take—editing, pruning, or summarizing context via explicit memory-action calls. This approach addresses trajectory fractures (breaking the strictly append-only context) and leads to adaptive context curation that balances information retention with resource constraints (Zhang et al., 14 Oct 2025).
• Modular Tool Separation: Frameworks like SLIM isolate lightweight search (returning only concise, top-k snippets) from more resource-intensive browsing (used only for promising URLs), combined with regular context summarization to prevent context overflow and enable extended search sequences (Yen et al., 21 Oct 2025).

2. Methods for External Knowledge Integration and Information Refinement

A defining feature of these frameworks is their strategy for integrating external knowledge with internal chain-of-thought reasoning:

• Dynamic Retrieval Triggering: The model learns (via reinforcement learning or supervised behavioral imitation) to emit retrieval commands only upon recognition of knowledge insufficiency, thus minimizing unnecessary tool calls and cost (Li et al., 9 Jan 2025, Gao et al., 11 Aug 2025).
• Document Compression and Filtering: Raw retrieved documents are analyzed and compressed using dedicated modules (e.g., “Reason-in-Documents” in Search-o1), producing succinct, relevant information that can be injected back into the reasoning chain with minimal noise (Li et al., 9 Jan 2025).
• Multi-step Self-Clarification and Pointback: The “self-clarification”–“contextual grounding” cycle in AgenticLU iteratively reduces uncertainty by generating clarifying questions and explicitly pointing to supporting context positions within long evidence chains. This approach is robust against the “lost-in-the-middle” problem in very large context inputs (Zhuang et al., 21 Feb 2025).
• Goal-Oriented Planning and Self-Reflection: Agents articulate explicit search goals and, after each retrieval, invoke a reflection subroutine to verify whether retrieved evidence fulfills the goal, iteratively refining queries if necessary (Fu et al., 30 Sep 2025).

3. Reinforcement Learning and Agent Training for Long-Horizon Search

Several reinforcement learning (RL) designs are pivotal in scaling long-horizon agentic search:

• Grouped Rollout and Advantage Attribution: Agentic RL algorithms such as ARPO employ entropy-based adaptive rollout. When the system detects high uncertainty (measured as an increase in entropy after a tool call), it adaptively branches additional trajectory samples at uncertain steps. Advantage attribution ensures that shared and diverged trajectory segments are credited with the correct reward (Dong et al., 26 Jul 2025).
• Asynchronous Large-Scale RL: ASearcher employs fully asynchronous RL, decoupling trajectory collection and policy updating. This addresses the “longest trajectory bottleneck,” enabling agents to scale well beyond 10-turn limits typical of synchronous online RL training, and permits effective learning of long-horizon strategies (Gao et al., 11 Aug 2025).
• Multi-Stage Supervised Fine-Tuning with Behavior Priming: Instead of fine-tuning only on correct final answers, methods such as Behavior Priming fine-tune on trajectories exhibiting target reasoning behaviors (e.g., information verification, authority evaluation, adaptive search, error recovery), setting a robust foundation for subsequent RL and facilitating improved long-horizon exploration (Jin et al., 8 Oct 2025).
• Reward Models and Self-Learning Loops: The Agentic Self-Learning (ASL) pipeline leverages a co-evolving Generative Reward Model (GRM) to generate, verify, and reward harder open-domain tasks within a closed loop, sustaining improvement and sample efficiency without reliance on static datasets (Sun et al., 16 Oct 2025).

4. Overcoming Context, Error Propagation, and Scalability Challenges

Long-horizon agentic search frameworks must mitigate several inherent limitations in scaling:

• Context Window Management: As search trajectories lengthen, context window overflow and accumulation of superfluous content become major barriers. SLIM and Memory-as-Action frameworks address this via explicit context summarization and policy-driven memory pruning, drastically lowering average token consumption per round while preserving retrievability of key evidence (Yen et al., 21 Oct 2025, Zhang et al., 14 Oct 2025).
• Trajectory Error Propagation: Repeated retrievals and reasoning steps can amplify errors or lead to biased search paths. Explicit reflection, goal checks, and periodic summarization mitigate error propagation and maintain focused, robust search trajectories (Fu et al., 30 Sep 2025, Yen et al., 21 Oct 2025).
• Efficiency of Rollouts and Tool Use: Adaptive rollout (e.g., ARPO) and tool separation (e.g., SLIM’s distinction between search and browse) improve both computational efficiency and sample complexity. Asynchronous RL in frameworks such as ASearcher ensures that longer search chains do not stall overall training throughput (Gao et al., 11 Aug 2025).

5. Evaluation Methodology and Benchmarks

Evaluation in long-horizon agentic search frameworks is distinguished by its focus on real-world, multi-step, open-ended, and verifiable tasks:

• Holistic and Process-Oriented Evaluation: Tools such as Mind2Web 2 and RAVine benchmark agentic search systems on long-horizon, dynamic web scenarios. Evaluations measure not merely the quality of the final output but also the correctness of tool interactions, evidence attribution, task completeness, and citation faithfulness (Gou et al., 26 Jun 2025, Xu et al., 22 Jul 2025).
• Automated, Granular Error Taxonomy: Automated trajectory analysis pipelines categorize error types into confirmation bias, unfocused search, inefficiency, answer ignored, hallucination, and abstention; these metrics enable targeted diagnosis of framework deficiencies (Yen et al., 21 Oct 2025).
• Fine-Grained Trajectory and Process Metrics: Evaluation frameworks use block-level scoring, criticality gating, process step logging, and reward aggregation over tool-use trajectories to rigorously characterize agentic system behavior (Xu et al., 22 Jul 2025, Gou et al., 26 Jun 2025).
• Specialized Benchmarks: Datasets such as BrowseComp, HLE, and GSM-Agent are constructed to stress-test long horizon and agentic reasoning capabilities, with explicit separation of agentic reasoning from static or knowledge-centric challenges (Zhu et al., 26 Sep 2025, Liu et al., 8 Sep 2025, Yen et al., 21 Oct 2025).

6. Implications and Future Directions

The integration of dynamic search, reasoning, and memory modules, along with advanced RL techniques and nuanced evaluation, substantiates the foundation for more autonomous, adaptable long-horizon information-seeking agents. Key implications and future research areas include:

• Optimization of Query Generation and Dynamic Reflection: Determining optimal strategies for recognizing uncertainty, timing, and formulation of search queries (Li et al., 9 Jan 2025, Zhuang et al., 21 Feb 2025).
• Memory and Context Curation Integration: Extending frameworks that unify task reasoning and memory as actions for seamless, scalable long-horizon performance (Zhang et al., 14 Oct 2025).
• Open-Domain, Self-Learning Agents: Advancing closed-loop self-learning pipelines leveraging generative reward models for fully autonomous agentic search without static supervision (Sun et al., 16 Oct 2025).
• Broadening Tool Integration and Multimodality: Expanding beyond web search to incorporate structured knowledge, APIs, code execution, and multimodal sources (Yao et al., 13 Oct 2025).
• Systematic Error Analysis and Mitigation: Leveraging automated trajectory-level analysis and error taxonomies to guide the design of more robust, cost-effective, and trustworthy frameworks (Yen et al., 21 Oct 2025).

Long-horizon agentic search frameworks thus represent the state-of-the-art in endowing LLMs and LRMs with reliable, scalable multi-step reasoning, robust information retrieval, and dynamic adaptation for open-world, knowledge-intensive applications.