ExpSeek: Self-Triggered Experience Seeking for Web Agents

Abstract: Experience intervention in web agents emerges as a promising technical paradigm, enhancing agent interaction capabilities by providing valuable insights from accumulated experiences. However, existing methods predominantly inject experience passively as global context before task execution, struggling to adapt to dynamically changing contextual observations during agent-environment interaction. We propose ExpSeek, which shifts experience toward step-level proactive seeking: (1) estimating step-level entropy thresholds to determine intervention timing using the model's intrinsic signals; (2) designing step-level tailor-designed experience content. Experiments on Qwen3-8B and 32B models across four challenging web agent benchmarks demonstrate that ExpSeek achieves absolute improvements of 9.3% and 7.5%, respectively. Our experiments validate the feasibility and advantages of entropy as a self-triggering signal, reveal that even a 4B small-scale experience model can significantly boost the performance of larger agent models.

• The paper introduces an entropy-aware, self-triggered experience seeking framework that guides web agents at each reasoning step.
• It constructs topic-organized experience triplets and uses logistic regression on entropy signals to trigger targeted, context-specific interventions.
• Empirical results reveal up to 14.6% improvements over standard methods, demonstrating resource-efficient guidance even with smaller experience models.

ExpSeek: Self-Triggered Experience Seeking for Web Agents

Introduction and Motivation

Recent progress in LLM-powered web agents has underscored significant challenges in open-domain environments with high noise, partial observability, and sparse signals. While prior experience-based intervention work typically passively injects contextualized experience globally prior to agent-environment interaction, this paradigm fails to dynamically adapt to variable context and decision uncertainty during the agent’s process. "ExpSeek: Self-Triggered Experience Seeking for Web Agents" (2601.08605) proposes an experience guidance framework that empowers agents to proactively and selectively seek step-level intervention based on model-intrinsic uncertainty signals, specifically entropy, to address the limitations of global passive experience injection.

Figure 1: A schematic contrasting passive global experience injection (A) with ExpSeek’s agent-driven, step-level proactive guidance (B).

Methodology: Proactive Step-Level Experience Seeking

ExpSeek comprises several critical components designed to operationalize self-triggered, step-level guidance.

Experience Base Construction

The core of the framework is an experience base formalized as topic-organized triplets, each encoding: (1) agent behavior (state and action), (2) mistake analysis (error type), and (3) corrective guidance (optimization cues). These are distilled via comparison of successful and failed trajectories, leveraging a tool model for granular step-level labeling and topic induction.

Figure 2: The ExpSeek architecture includes experience triplet construction and guided inference with active, context-driven intervention.

Entropy-Guided Triggering

Central to ExpSeek is the use of step entropy as a proxy for agent uncertainty. For each reasoning step, token-level entropy is estimated and averaged to quantify confidence. Logistic regression with bootstrap resampling identifies statistically robust entropy thresholds for intervention in both process and answer steps—adjusting guidance intensity while avoiding unnecessary overhead. Notably, answer step entropy yields stronger separability for correct vs. incorrect actions, but process step entropy, despite higher overlap, remains a valid albeit noisier trigger.

Figure 3: Entropy distribution analysis of process and answer steps enables calibration of self-triggered guidance.

Actively Generated Step-Level Guidance

When intervention is triggered, ExpSeek retrieves the most context-relevant experience topics and dynamically generates action-specific guidance via an experience model $\mathcal{M}_{e}$. Guidance integrates with the agent’s history, modulating subsequent tool use or direct answer synthesis in a way tailored to current context and error patterns.

Empirical Study and Key Results

Evaluations on Qwen3-8B and Qwen3-32B models across four web agent benchmarks reveal several strong empirical claims:

• ExpSeek achieves absolute improvements of 9.3% (Qwen3-8B) and 7.5% (Qwen3-32B) over vanilla ReAct, with maximum increases up to 14.6%.
• ExpSeek robustly outperforms both training-free and self-evolving experience baselines, including GRPO and ReasoningBank, by wide margins in out-of-distribution settings—demonstrating strong generalization.
• Intervention only at process or answer steps independently is suboptimal; full step-level intervention substantially outperforms ablations.
• Remarkably, a 4B parameter experience model can significantly boost a 32B agent, indicating the effectiveness of “weak-to-strong” guidance transfer, contingent on a high-quality experience base.

  Figure 4: Entropy distributions shift with ExpSeek guidance, promoting exploration during process steps and solution convergence in answer steps.

Design Rationales and Failure Analysis

Extensive ablation studies and analyses show several critical properties:

• Self-triggered entropy-based intervention outperforms scripted or reward-model-based triggers in both efficiency and accuracy, scaling adaptively with task complexity.
• Guidance based on dynamic generation (rather than retrieval) is necessary for maximal gains; static retrieval fails to exploit problem-local error context.
• The framework displays a favorable accuracy-efficiency trade-off: increased guidance intensity does not degrade performance, but only marginally boosts gains past an optimal threshold.

  Figure 5: Performance and efficiency cross-comparison as intervention intensity is adjusted, with accuracy rapidly saturating as guidance increases.

  

  Figure 6: Experience model scaling law—small models provide substantial, though saturated, improvement as experience model size increases.

Scalability, Transferability, and Data Efficiency

ExpSeek is robust with respect to the experience repository and model scaling:

• Repository swapping across model scales preserves substantial gains, with only minor losses due to model-specific abstractions.
• Even with severe repository compression (one experience per topic), the framework maintains high accuracy, highlighting the primacy of semantic organization (topics), not merely repository scale.
• Experience models as small as 4B continue to deliver significant guidance when paired with large agents, pointing toward practical, resource-effective deployment possibilities.

  Figure 7: Correlation between experience repository size and model performance—effective guidance persists even under severe data compression.

Implications, Limitations, and Future Prospects

This study demonstrates that model-intrinsic uncertainty signals, particularly entropy, enable granular, efficient, and contextually adaptive self-triggered experience intervention, substantially improving LLM agent reliability in long-horizon, open-domain web environments. The decoupling of guidance quality and intervention source model size opens up resource-efficient possibilities for scalable agent deployment.

However, threshold estimation remains partly contingent on tool-model and training data quality, experience base construction is non-trivial, and extension to non-web domains or further integration with agentic RL pipelines remains to be explored. The observed improvement in sampling diversity/pass@k also suggests ExpSeek could become a key component in rollout-based RL or active learning for agentic LLMs.

Conclusion

ExpSeek presents a self-triggered, entropy-aware framework for context-sensitive step-level experience guidance in LLM-powered web agents. Through extensive empirical analysis, it establishes strong numerical improvement and generalization capacity while elucidating key design levers for future agentic intelligence systems (2601.08605).