Agent-Guided Search

• Agent-guided search is a methodology that uses learned policies to actively explore combinatorial search spaces, bridging classical planning with reinforcement learning.
• It leverages deterministic LevinTS and sampling-based approaches like LubyTS/multiTS to provide efficiency, strong theoretical guarantees, and robustness in search tasks.
• Applications span code generation, multi-agent coordination, and planning, yielding state-of-the-art empirical results and improved search quality.

Agent-guided search encompasses a family of methodologies in which an agent, often parameterized by a learned policy or value estimate, actively guides the exploration of combinatorial search spaces. This paradigm forms a critical bridge between classical symbolic search/planning and reinforcement learning, providing pathways for systematic exploration that exploit agent-derived knowledge for significantly improved efficiency, guarantees, or interpretability. Recent advances have demonstrated the integration of agent-guided search across classical planning, program synthesis, code generation, software engineering, multi-agent coordination, and web-scale retrieval, often yielding provable bounds or state-of-the-art empirical performance.

1. Core Concepts and Algorithmic Foundations

Agent-guided search arises when an explicit agent policy, typically learned or synthesized from prior experience, is employed to guide the selection and expansion of nodes in a search tree or a more general search process. In the archetypal setting (Orseau et al., 2018), the agent operates over a deterministic, single-agent, discrete-action environment. Each node is a finite action sequence $n = a_{1:t}\in A^*$, with a Markovian policy $\pi$ assigning probabilities recursively by $\pi(n\,a) = \pi(n)\,\pi(a \mid n)$. The agent’s search proceeds via two principal algorithmic templates:

• Levin-style Best-First Enumeration (LevinTS): Assigns to each node $n$ a “Levin cost” $\mathrm{cost}(n) = \frac{d_0(n)}{\pi(n)}$, balancing sequence depth against policy probability. The search expands the frontier node minimizing this cost, with strict worst-case guarantees on node expansions:

$$N(\mathrm{LevinTS},\mathcal{N}^g) \le \min_{n\in\mathcal{N}^g} \frac{d_0(n)}{\pi(n)}.$$

This prioritizes high-probability paths and is particularly effective for “needle-in-a-haystack” settings with very sparse targets.

• Sampling-based Search (LubyTS, multiTS): Relies on repeated sampling from the policy $\pi$, with trajectory lengths either fixed (multiTS) or governed by a universal Luby restart sequence (LubyTS). The expected number of expansions is bounded as:

$$\mathbb{E}[N(\mathrm{multiTS}(\infty, d), \mathcal{N}^g)] \le \frac{d}{\Pi^+_d}, \quad \mathbb{E}[N(\mathrm{LubyTS}, \mathcal{N}^g)] \le \min_{d\ge1} d + \frac{d}{\Pi^+_d}[\log_2(d/\Pi^+_d) + 6.1].$$

These guarantee tractable expected expansions when goal probability mass is diffuse at shallow depths.

These frameworks provide rigorous upper bounds connecting search effort directly to the probability mass assigned by the agent policy, representing a significant advance over classical, heuristic-only methods.

2. Policy Integration, Execution Feedback, and Agent-Environment Interface

A defining property of modern agent-guided search is the integration of rich, potentially high-capacity policies—often neural networks trained by reinforcement learning, such as A3C (Orseau et al., 2018) or actor-critics in more complex, partially observable, or non-serializable settings (Zainullina et al., 19 May 2025). At each search node (or state), the agent’s policy is queried to compute $\pi(a \mid s)$, either for evaluation (LevinTS) or sampling (LubyTS/multiTS), enabling tight coupling between learned priors and search expansion dynamics.

Recent extensions have demonstrated agent-guided search architectures where additional execution-based feedback is incorporated. For example, in agent-guided code generation (Li et al., 2024), nodes are not only ranked by agent-provided scores (e.g., LLM critic plausibility) but are also evaluated on execution feedback from running code, leading to composite ranking criteria for expansion, pruning, or acceptance:

$$\mathrm{Score}(\hat W_i) = \mathrm{Score}_{\mathrm{exe}} + \text{LLM-critic's }\phi_i.$$

Such integration supports dynamic adjustment of the search tree in response to concrete environmental feedback, enhancing robustness and solution quality.

Empirical work shows that even when an agent policy is imperfect, injecting minimal uniform noise avoids pitfalls associated with zero-probability actions (Orseau et al., 2018), and that strategies to mix or smooth policies can avoid dead-ends where $\pi(n) = 0$ for all remaining nodes.

3. Theoretical Guarantees and Complexity

A distinguishing feature of agent-guided search is the derivation of explicit performance guarantees as a function of the agent policy’s support relative to the solution set. In deterministic settings, worst-case bounds (LevinTS) and expected expansion bounds (LubyTS/multiTS) offer a form of reliability unattainable under purely heuristic or sampling-based regimes. Concrete complexity separations are established:

• When goal paths are unique and deep, LevinTS attains an exponential-in-depth bound ($O(d2^d)$), improving by a $d$ factor over LubyTS ($O(d^2 2^d)$).
• When exponentially many shallow goal paths exist, sampling-based search achieves an exponential speedup ($O(d \log d)$) over deterministic enumeration ($\Omega(2^d)$) (Orseau et al., 2018).

This analysis establishes formal trade-offs between deterministic/needle-in-a-haystack and many-path regimes, and identifies which algorithm is optimal in each.

4. Generalizations and Recent Developments

The core agent-guided search idiom has been adapted to a wide variety of domains:

• Non-Serializable Environments: In settings where classic tree search is inapplicable due to irreversible or non-resettable environments (e.g., software agents manipulating Docker containers), agent-guided search employs learned value functions (critics) to guide rollout and select among multiple solution attempts. One-step lookahead and trajectory selection, each guided by a critic, have consistently doubled average-case success rates in complex software engineering benchmarks (Zainullina et al., 19 May 2025).
• Hierarchical and Multi-Agent Architectures: In multi-hop reasoning or search with factored tasks, hybridized agent designs allocate roles for high-level planning and low-level search, enhancing specialization and interpretability (Hu et al., 9 Jun 2025, Chen et al., 8 Jan 2026). In multi-agent games, model-based priors guide local policy search toward Nash equilibria, stabilizing what would otherwise be unstable or oscillatory multi-agent dynamics (Li et al., 29 Sep 2025).
• Hybrid Search with Neural Policies: For combinatorially hard problems such as multi-agent pathfinding, neural policies (e.g., graph-attention agents) are trained to provide local heuristic estimates or action rankings, and are embedded in search-based algorithms (e.g., LaCAM), yielding real-time and quality improvements, especially in densely coupled regimes (Jain et al., 20 Oct 2025).
• Process-level Reward Modeling: Fine-grained, process-aware reward models (e.g., stepwise Q-values in QLASS (Lin et al., 4 Feb 2025), process rewards in SmartSearch (Wen et al., 8 Jan 2026)) have been shown to enable more effective stepwise agent guidance, denser and more actionable credit assignment, and marked empirical gains for complex interactive or retrieval tasks.
• Hierarchical and Modular Search Spaces: Automated agent design leverages hierarchical agent-guided search over workflows and components with learned surrogate value models, supporting efficient search and rapid discovery of performant architectures (Li et al., 6 Jun 2025).

5. Empirical Benchmarks and Comparative Outcomes

Agent-guided search methodologies have demonstrated strong, and often state-of-the-art, performance across a spectrum of benchmarks:

Domain/Task	Agent-Guided Approach	Key Result	Reference
Sokoban puzzle solving	LevinTS (policy-guided)	100% solved, average solution ≈ 39.8 pushes; fewer expansions	(Orseau et al., 2018)
Complex code generation	CodeTree (critic and execution-guided)	Top pass@1 on HumanEval (94.5%), MBPP (98.7%), CodeContests (43%)	(Li et al., 2024)
SWE-bench software engineering	Critic-guided lookahead/trajectory selection	Qwen: 16.2% → 40.8% (combined)	(Zainullina et al., 19 May 2025)
Multi-k agent search (delegated search)	Threshold-based agent-guided mechanism	Approx ratio $1 - \Theta((\ln k)/k)$ as $k$ increases	(Bechtel et al., 2024)
Pathfinding in dense MAPF	Graph attention agent in search loop	>8% SoC improvement, 100% success in densest settings	(Jain et al., 20 Oct 2025)
Multi-hop evidence-based fact verification	LLM reasoning/search agent specialization	Up to +4.4 points F1 on HOVER-4hop; improved interpretability	(Hu et al., 9 Jun 2025)
Knowledge-intensive search QA/web search	Process-reward/agent-guided query refinement	+7.4/7.7 points (EM/F1) vs. StepSearch baseline	(Wen et al., 8 Jan 2026)

These results consistently show that agent-guided search not only achieves higher accuracy but often requires fewer environment calls or search expansions, offering efficiency improvements as well as reliability.

6. Limitations and Future Research Directions

Despite strong empirical and theoretical performance, key limitations persist:

• Dependence on Policy Quality: Agent-guided search inherits the weaknesses of the guiding policy. If $\pi$ places zero probability on paths leading to solutions, coverage is lost; smoothing/mixing and explicit coverage analyses are needed (Orseau et al., 2018).
• Resource Consumption: Systematic tree search (LevinTS) imposes substantial memory/priority-queue overhead, especially in wide or deep spaces; sampling-based and anytime variants partly mitigate this.
• Non-guaranteed Optimality in Stochastic or Non-stationary Domains: Guarantees established in deterministic or fixed-horizon cases may not extend to general stochastic or adversarial environments, requiring new analyses and architectures (Orseau et al., 2018, Li et al., 29 Sep 2025).
• Modularity and Adaptivity: Most agent-guided methods yield static search architectures; enabling dynamic, feedback-responsive rewiring or adaptation remains an open question (Li et al., 6 Jun 2025).

Current research is expanding agent-guided search to domains such as real-time robotics, molecular design, and agentic web interaction, with an emphasis on integrating learned policies, process-level credit assignment, modularity, and theoretical guarantees.

7. Significance and Impact

Agent-guided search constitutes a fundamental paradigm shift in algorithmic search and planning. By unifying learned policy advice with principled search strategies, it achieves a unique blend of reliability (via provable guarantees), efficiency (through focused search), and adaptability (by leveraging rich agent knowledge). These properties position agent-guided search as a cornerstone technique for contemporary AI systems dealing with combinatorial and open-ended search problems, with demonstrated impact in games, software engineering, code synthesis, planning, web-scale retrieval, and beyond (Orseau et al., 2018, Li et al., 2024, Zainullina et al., 19 May 2025, Hu et al., 9 Jun 2025, Li et al., 6 Jun 2025, Lin et al., 4 Feb 2025, Jain et al., 20 Oct 2025, Bechtel et al., 2024, Wen et al., 8 Jan 2026).