Agentic Experience Search

• Agentic Experience Search is a paradigm that uses structured chains of past experiences and reward relabeling to optimize autonomous planning and decision-making.
• It is implemented in diverse architectures such as transformer-based RL agents, multi-agent systems, and MCTS-driven workflow synthesis for adaptive tool use.
• Empirical results demonstrate significant performance gains in multi-hop QA, reinforcement learning benchmarks, and workflow synthesis, highlighting its robust adaptability.

Agentic Experience Search refers to algorithmic frameworks and model architectures enabling autonomous agents—typically large language or reasoning models—to plan, execute, and optimize multi-step interaction with environments or tools using accumulated experience to guide future behavior. In contrast to traditional RL or static RAG settings, agentic experience search leverages explicit representations of past trajectories, dynamic reuse and adaptation of prior experience, and structured search over histories to improve performance in both training and deployment, with continual improvements observed over repeated trials or increasingly complex environments. This paradigm has been operationalized across reinforcement learning, open-domain and vertical search, multi-agent tool-use, workflow synthesis, and knowledge transfer systems.

1. Foundational Principles and Formalism

The core principle underlying agentic experience search is that agents should not only react to immediate observations and rewards but explicitly leverage chains of prior experience—successful and unsuccessful—to drive improved decision-making. This typically involves representing experience as sequences or chains of trajectories $\tau = (s_0,a_0,r_0, ..., s_T,a_T,r_T)$, with mechanisms to relabel goals, aggregate feedback, and condition policy or planning on the best-known trials.

For instance, the Agentic Transformer (AT) (Liu et al., 2023) formalizes experience search in offline RL using chain-of-hindsight: given $n$ trajectories sorted by total return, all trajectories in the chain are relabelled to aspire to the best observed return, with training targeting the optimal trajectory and inference executing trial-and-error rollouts.

A representative generalization of agentic experience search is as follows:

• Given a set of past experiences $\mathcal{E} = \{\tau^1,..,\tau^n\}$, an agent constructs a search over possible adaptations, guided by reward relabelling, trajectory sorting, or hybrid retrieval.
• Policy $\pi_\theta$ receives as input state, aggregated history, relabelled returns-to-go, and completion bits (indicating goal achievement).
• Training and inference utilize autoregressive conditioning, search over chains of trajectories, or retrieval from structured knowledge bases.

2. Architectural Implementations

Agentic experience search is realized in diverse architectures across domains:

a) Transformer-Based RL Agents:

Agentic Transformer (AT) processes chains of trajectories, embedding five modalities (return-to-go, state, action, reward, completion bit) and training autoregressively to predict actions for the highest return trajectory (Liu et al., 2023). The model demonstrates that agentic search enables learning to self-improve across repeated rollouts and scales favorably with model size.

b) Multi-Agent Search Systems:

M-ASK (Chen et al., 8 Jan 2026) decomposes experience search into Search Behavior Agents and Knowledge Management Agents, enabling turn-level rewards, granular credit assignment, and context compaction for stable multi-hop QA.

c) Memory-Augmented Cross-Framework Retrieval:

Agent KB (Tang et al., 8 Jul 2025) abstracts experiences into formal knowledge base entries and enables plug-and-play retrieval to seed agent plans and diagnose failures across disparate agentic frameworks, governed by a disagreement gate to avoid cross-framework interference.

d) Workflow Search via MCTS:

AFlow (Zhang et al., 2024) treats workflow synthesis as sequential search over code graphs, performing Monte Carlo Tree Search with code-modifying operators, embedding and refining experience logs, and dynamically integrating execution feedback.

e) Self-Learning Closed-Loop Agents:

Agentic Self-Learning (ASL) (Sun et al., 16 Oct 2025) orchestrates Prompt Generator, Policy Model, and Generative Reward Model roles, with co-evolution of task difficulty and reward sharpness yielding round-over-round improvement even without human labels.

f) RL-Driven Multimodal Tool Use:

SenseNova-MARS (Chng et al., 30 Dec 2025) couples multimodal reasoning steps with RL-driven invocation of search, image-crop, and image-retrieve tools, optimizing sequence-level rewards and context management via Batch-Normalized GSPO.

3. Experience Representation, Relabelling, and Retrieval

Central to agentic experience search is the sophisticated treatment of experience logs and trajectories.

• Chain-of-hindsight relabelling: All steps in a chain are re-labelled to maximize the best reward, allowing agents to "pretend" to pursue the optimal return, even when initial attempts are suboptimal (Liu et al., 2023).
• Structured Knowledge Bases: Experiences are abstracted into entries with task embeddings, logical constraints, action-reasoning pairs, and tool metadata. Hybrid retrieval (lexical+semantic) is performed for both plan seeding and diagnostic feedback, with gating to ensure compatibility (Tang et al., 8 Jul 2025).
• Tree-Structured Experience and Feedback: In AFlow, experience nodes in the MCTS tree log operator, reward delta, and success/failure, biasing future exploration towards historically fruitful modifications (Zhang et al., 2024).
• Step-Level Entropy and Self-Triggered Guidance: ExpSeek (Zhang et al., 13 Jan 2026) estimates per-step token entropy, triggers experience intervention when agent uncertainty is high, and injects context-tailored snippets (behavior, error, guidance) to correct trajectories.
• Multi-Agent Summarization and Compaction: M-ASK's Knowledge Management Agents distill tool responses into concise evidence, maintain truncated internal knowledge trajectories, and apply update/add operations to optimize context compactness (Chen et al., 8 Jan 2026).

4. Training Objectives and Optimization Strategies

Agentic experience search typically adopts RL-based or autoregressive objectives adapted for long-horizon, multi-step reasoning.

• Autoregressive Log-Likelihood on Relabelled Chains: Only the final, highest-return trajectory in the hindsight chain is directly optimized, using NLL or MSE depending on action space (Liu et al., 2023).
• Turn-Level Dense Rewards: In multi-agent systems, incremental rewards are computed for each action as the difference in answer quality, and PPO or GRPO is utilized for stable joint policy optimization (Chen et al., 8 Jan 2026, Jin et al., 8 Oct 2025).
• Confidence-Thresholded RL: Search Wisely introduces $\beta$-GRPO, rewarding only high-confidence search actions (as measured by minimum token probability), reducing over- and under-search and improving EM score by 4% (Wu et al., 22 May 2025).
• Closed-Loop Role-Coupled RL: ASL uses policy-gradient objectives over prompt generation, policy execution, and reward model co-evolution, with entropy-based rewards sustaining curriculum escalation and breaking reward-hacking behaviors (Sun et al., 16 Oct 2025).
• Batch-Normalized Sequence Policy Optimization: SenseNova-MARS applies group- and batch-normalization to sequence-level RL advantages, stabilizing training and improving multimodal search accuracy (Chng et al., 30 Dec 2025).
• Behavior Priming through SFT+RL: Post-training with trajectories explicitly exhibiting beneficial reasoning patterns (verification, authority evaluation, adaptive search, error recovery) yields 35–60% improvement over RL from correct-answer data alone (Jin et al., 8 Oct 2025).

5. Evaluation Protocols, Metrics, and Empirical Outcomes

A variety of benchmarks and protocols have been developed to rigorously evaluate agentic experience search systems, emphasizing multi-step reasoning, tool integration, and process traceability.

• Offline RL and Exploratory Datasets: AT achieves 85.21 normalized score on D4RL and 83.02 raw return on ExoRL, exceeding both imitation and TD-learning baselines; repeated trial rollouts at inference produce steady performance improvements (Liu et al., 2023).
• Cross-Domain Problem Solving: Agent KB improves pass@3 on GAIA from 55.2% to 73.9%, demonstrating effective cross-agent experience transfer and retrieval (Tang et al., 8 Jul 2025).
• Multi-hop QA and Tool Efficiency: M-ASK yields 50.09% F1 (+3.3 pts over adaptive baselines) on HotpotQA and exhibits stable training dynamics versus monolithic agents (Chen et al., 8 Jan 2026).
• Reasoning Behavior Frequency and RL Scaling: Behavior Priming improves Qwen3-1.7B RL performance by 8.4pp, with higher frequencies of desirable behaviors strongly correlated with answer accuracy and exploration metrics (Jin et al., 8 Oct 2025).
• Workflow Synthesis: AFlow's MCTS-driven code graph optimization yields a +5.7% gain over state-of-the-art manual workflows and enables small models to outperform GPT-4o at 4.55% cost (Zhang et al., 2024).
• Web Agent Self-Triggered Experience: ExpSeek improves test accuracy by 9.3% (Qwen3-8B) and 7.5% (Qwen3-32B) via entropy-thresholded intervention, with efficient guidance from a standalone 4B experience model (Zhang et al., 13 Jan 2026).
• Multimodal Benchmarks: SenseNova-MARS attains 41.64% accuracy on HR-MMSearch, exceeding proprietary VLMs, and jointly trained hybrid RL agents outperform “search only” or “perception only” RL regimes (Chng et al., 30 Dec 2025).

6. Limitations, Extension Opportunities, and Open Challenges

Despite substantial advances, agentic experience search systems confront important scalability and generalization challenges:

• Memory & Computation: Quadratic time and memory in chain or context length reveal inefficiencies when chains or action traces are long (Liu et al., 2023).
• Tool/Context Coordination: Sequential, autoregressive rollouts preclude parallelism across time; stable multi-agent role assignment and context compaction (as in M-ASK, Laser) remain imperative for complex workflows (Chen et al., 8 Jan 2026, Wang et al., 23 Dec 2025).
• Reward Integrity & Hacking: Frozen or weak reward models invite reward hacking; continual co-evolution and small real-data injections sustain closed-loop quality (Sun et al., 16 Oct 2025).
• Domain Specificity: Transfer to highly vertical settings such as local life services (business, healthcare) causes significant drop-offs in completeness and faithfulness; benchmarks like LocalSearchBench quantify this gap and motivate specialized tools and training recipes (He et al., 8 Dec 2025).
• Faithfulness and Attribution: Many benchmarks (e.g., Mind2Web 2, RAVine) report low citation precision/recall and incomplete nugget coverage, highlighting need for more robust ground-truth construction and process metrics (Gou et al., 26 Jun 2025, Xu et al., 22 Jul 2025).

Potential advances include:

• Memory-augmented agents with selective retention and episodic recall for long-horizon search (Wang et al., 23 Dec 2025).
• Adaptive chain-length selection and modular retrieval architectures for scalable parallelism (Liu et al., 2023, Tang et al., 8 Jul 2025).
• Meta-RL or few-shot transfer for cross-domain generalization (Lin et al., 19 Oct 2025).
• Human-AI co-search and interactive RL frameworks for user-centric agentic experience (Lin et al., 19 Oct 2025, Gou et al., 26 Jun 2025).

7. Summary Table: Leading Agentic Experience Search Systems

System	Key Mechanism(s)	Empirical Gains
Agentic Transformer (AT) (Liu et al., 2023)	Chain-of-hindsight, autoregressive transformer	+5–7 pts over TD and imitation RL; >80 norm. score
Agent KB (Tang et al., 8 Jul 2025)	Cross-framework KB, hybrid retrieval, disagreement gate	+18.7pp pass@3 (GAIA); robust scaling
M-ASK (Chen et al., 8 Jan 2026)	Multi-agent (SBA+KMA), turn-level PPO, context compaction	+3.3 pts F1 HotpotQA; 0% collapse
Behavior Priming (Jin et al., 8 Oct 2025)	SFT + RL on beneficial reasoning behaviors	+35–60% RL gain vs answer-only SFT
AFlow (Zhang et al., 2024)	MCTS code-graph, trajectory bias, workflow refinement	+5.7% over manual; cost-efficient SOTA
Agentic Self-Learning (Sun et al., 16 Oct 2025)	Closed-loop PG/Policy/GRM co-evolution	+12–20 pts round-over-round, no human label
ExpSeek (Zhang et al., 13 Jan 2026)	Step-entropy triggering, tailored guidance injection	+9.3% (8B); +7.5% (32B) accuracy
Laser (Wang et al., 23 Dec 2025)	Symbolic protocol, compact context register, explicit retrospection	+5–10 pts ACC over ReAct/Search-R1

By integrating chain-of-experience relabelling, workflow synthesis, multi-agent architectures, cross-domain retrieval, and RL-optimized reasoning, agentic experience search establishes a unified paradigm for autonomous, self-improving agents operating in complex environments and multi-step tasks.