On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification

Abstract: Memory-based self-improving agents--those that learn from an online stream of tasks and improve over time by maintaining a textual memory bank--have shown great promise in recent literature. However, the reliability aspects of these methods have been critically overlooked. In this work, we conduct a comprehensive re-evaluation of two memory-based methods, broadening the scope of evaluation along two axes: (1) including multiple runs to quantify variance, and (2) randomly shuffling the tasks to investigate the effect of task order. Through these experiments, we make two observations that expose the fragility of current methods: First, agent evaluation is inherently noisy in complex environments and on multi-step tasks, and stacking a self-improving loop on top can further amplify this noise. Second, the agent's improvement is highly dependent on task order. Prior works often adopt default orderings that impose an implicit curriculum, acting as a hidden prerequisite for success. To better understand this fragility, we manually examine the agents' memory and hypothesize that task and environment underspecification contribute to this fragility. We validate this hypothesis by incorporating information that enables better specification, such as detailed rubrics and environment feedback, into the memory construction process. While this added information partially closes the performance degradation in previous experiments, significant gaps still remain, suggesting that other uncharacterized factors contribute to this fragility. Looking ahead, our work advocates for more rigorous evaluation protocols for self-improving agents by reporting results across multiple runs and stress-testing them under challenging conditions. Moreover, our findings on underspecification call for systems and interfaces that enable effective human oversight, preventing agents from failing in unforeseeable ways.

• The paper finds that memory-based self-improvement increases run-to-run variance in 17 of 24 method–domain settings, with some standard-deviation increases exceeding 300%.
• The paper shows that apparent gains depend on an easy-to-hard task order, while shuffled tasks cause performance declines of roughly 5–6 percentage points on WebArena.
• The paper identifies underspecified environments, ambiguous tasks, and reward-hacked memories as failure sources, while added feedback recovers only about 31% of shuffled-order degradation.

Overview and motivation

This paper by Ye et al. (Salesforce AI Research) presents a re-evaluation of memory-based self-improving agents—systems that process an online stream of tasks while maintaining a textual memory bank—and argues that the reliability properties of such systems have been critically under-examined. The authors evaluate two representative methods, Agent Workflow Memory (AWM) (Zheng et al., 9 Apr 2025) and ReasoningBank (RBank) (Ouyang et al., 29 Sep 2025), across three web-browsing benchmarks: WebArena (812 tasks), VisualWebArena (910 tasks), and SCUBA (267 enterprise CRM tasks). Crucially, they upgrade the baseline to GPT-5-mini with stronger agent harnesses, so that the "^^^^0^^^^" for self-improvement is substantially stronger than in prior work.

The evaluation is broadened along two axes that prior work largely ignored: (1) running each configuration three times to quantify run-level variance, and (2) evaluating under randomly shuffled task orders rather than only the default ordering. The central claims are stark: self-improvement amplifies evaluation noise, and apparent gains from self-improvement depend on an implicit easy-to-hard curriculum embedded in benchmark task orderings. Under shuffled orders, agents degrade rather than improve.

Variance across runs

The paper first establishes that web agent evaluation is intrinsically noisy even without memory. On the WebArena GitLab subset (180 tasks), the best–worst gap across three identical baseline runs reaches 4.4%, with a standard deviation of 1.98%; domain-level gaps reach 2.4% on VisualWebArena and 6.7% on SCUBA. Since most prior work reports single-run pass@1, these magnitudes are large enough to render many reported improvements statistically meaningless.

More concerning is the finding that stacking a self-improving loop amplifies this variance. Across 24 method–domain combinations, variance increases in 17 cases (~71%), with 11 showing relative increases above 50%. Extreme cases include RBank on WebArena Multisite, where the standard deviation rises from 0.98% to 4.28% (+336%) and the best–worst gap widens to 10.42 percentage points; AWM on SCUBA Admin shows a +600% increase in standard deviation. The mechanism is structural: memory construction conditions on prior task outcomes and stochastic LLM sampling, so early randomness compounds into divergent memory states across runs. This directly undermines single-run evaluation practices for stateful agents.

A second consequence of the stronger baseline is diminishing returns from self-improvement. The no-memory GPT-5-mini baseline achieves 54.8% on WebArena—already exceeding the memory-enhanced results reported in the original papers (36.3% for AWM with Claude-3.5-Sonnet; 53.9% for RBank with Gemini-2.5-Pro). Against this baseline, RBank's average gain of +1.5% carries a p-value of 0.23 under an unpaired t-test over three runs, i.e., not statistically significant. AWM actually shows small negative deltas on WebArena (−0.7%) and VisualWebArena (−0.4%). The implication is that gains demonstrated with weaker base models may not transfer as base capability improves, and that memory construction must become more selective when the baseline is already strong.

Task order sensitivity

The paper then shows that the default task ordering used in prior evaluations—which sorts by task ID—constitutes an implicit curriculum. Moving-average plots of baseline pass rates show success rates starting near 75% and dropping below 40% once task IDs exceed 150, indicating that earlier tasks are systematically easier, likely an artifact of how annotators constructed the benchmarks.

When tasks are presented in two shuffled orders, performance degrades sharply. On WebArena under Shuffle-1, results drop from 54.8% (baseline) to 49.1% (AWM) and 49.8% (RBank)—a degradation of roughly 5–6 points rather than the +1.5% improvement observed under the default order. Performance degrades in 6 of 8 shuffled-order settings overall. This contradicts the expectation that, at worst, self-improvement should be neutral: instead, memories formed out-of-curriculum actively hurt. Because real deployments cannot assume neatly ordered user requests, this constitutes a direct threat to the deployment case for these methods.

Underspecification as a cause

Through manual inspection of RBank's memory banks, the authors identify environment and task underspecification as plausible drivers of both fragility modes:

• Environment underspecification: because the memory module is not told that the environment is browser-only, it repeatedly generates memories recommending API-based solutions—plausible but unexecutable strategies that distract the agent. Similarly, memories recommending "user confirmation" appear frequently (26 times across 3 WebArena runs), leading agents to issue wait actions until timeout.
• Task underspecification: ambiguous queries (e.g., the jaw bruxism task intended as product search, answered instead with medical advice) produce irrelevant memories that propagate forward. Evaluator bugs (e.g., strict string matching rejecting "$0.00" where "0" is required) generate false negatives that induce spurious "lessons."
• Contagious unintended strategies: in the map domain, when routing pages fail to load, the agent falls back to Haversine-formula distance estimation. Because this occasionally passes the rubric, it enters memory and is retrieved increasingly often—a form of reward hacking during learning that also contributes to cross-run variance.

To mitigate underspecification, the authors inject three types of information into memory construction: evaluator rubrics and scores (+Rub), raw environment feedback including action errors (+Env), and prompt modifications explicitly forbidding unsupported strategies (+PMod). Combined, these recover 2.9% on Shuffle-1 (49.8% → 52.7%)—closing roughly 31% of the degradation—and yield +1.1% on Shuffle-2 without hurting the default order. However, the modified agent still underperforms the no-memory baseline under shuffling, leaving ~69% of the gap unexplained. The authors concede that their interventions are a first attempt and that other contributing factors remain unidentified.

Limitations and open questions

The scope is restricted to memory-based, inference-time self-improvement in web browsing; other self-improvement paradigms (fine-tuning-based, RL-based) and domains are not evaluated, nor are the newest memory-management techniques. The manual memory analysis is qualitative and covers only a subset of generated memories, so additional failure modes likely exist undocumented. Three runs per configuration is a modest sample for variance estimation, and the statistical significance of several aggregate comparisons remains weak. Open questions include: what mechanisms beyond specification can make textual memory robust to arbitrary task orderings, and whether scalable automated validation can filter erroneous memories before they cascade.

Conclusion

This paper demonstrates that current memory-based self-improving agents are fragile in ways hidden by standard evaluation practice: run-to-run variance is large and amplified by the self-improvement loop itself, and reported gains partly depend on an implicit easy-to-hard task ordering that vanishes under shuffling. Underspecification explains part—but not all—of the problem, and targeted information injection recovers only about a third of the shuffled-order degradation. The practical recommendations are concrete: report multi-run statistics, stress-test under randomized task orders before deployment, and build human-oversight interfaces capable of detecting and correcting erroneous memories before they compound irreversibly.