Learning on the Job: Continual Learning from Deployment Feedback for Frozen-Weights Agents

Abstract: AI agents encounter learning opportunities in every episode they run, and discard nearly all of them: the underlying models are frozen at deployment, so an agent that resolves a difficult request today starts from zero when it recurs tomorrow. Yet ordinary operation already produces feedback, in the form of outcome verdicts and after-the-fact corrections. We show that this feedback is a sufficient signal for continual learning when the frozen model is paired with an external memory that distils each episode into retrievable natural-language rules. On the banking domain of $τ$-bench, against a static-RAG control retrieving over the complete policy corpus, learning from the one-bit outcome verdict lifts single-trial success to 1.6$\times$ the baseline, and learning from corrections to 2.6$\times$, converting 22 of the 84 tasks the baseline never solves. The result spans the deployment spectrum, measured on Mistral Large, an open-weights model that organisations with data sovereignty requirements can self-host, and replicated on a frontier model, Claude Sonnet 5. The accumulated memory also transfers: each model, reading the store built by the other, rises above its own no-memory baseline. The harness, protocol, and data are released.

• The paper demonstrates that integrating an external memory with frozen-weights agents enables gradient-free continual adaptation using natural language rules from deployment feedback.
• It reveals that both binary outcome and explicit instruction feedback can significantly improve task performance, with instruction feedback boosting success rates by up to 2.6× over static baselines.
• The study highlights robust cross-model store transfer and the role of persistent memory in capturing long-tail task solutions while balancing aggressive learning with retention.

Continual Learning for Frozen-Weights Agents via Deployment Feedback

Problem Framing and Motivation

The paper addresses the lacuna inherent in currently deployed frozen-weights AI agents: despite constant exposure to novel situations, these models do not adapt from their own operational feedback. Since continual model retraining in deployment is operationally costly and rife with potential for catastrophic forgetting and safety regressions, most agents are fixed at deployment and remain so throughout their lifespan. However, practical deployments inherently generate feedback signals—such as outcome verdicts (success/failure) and user or validator corrections—yet this iterative supervision is typically wasted.

The study examines whether such feedback suffices for continual learning without updating model weights, specifically by pairing a frozen LLM with an external, persistent memory that summarizes operational episodes into natural language rules. This hybrid, memory-augmented architecture supports gradient-free adaptation and sidesteps common pitfalls of frequent model retraining.

System Architecture and Experimental Protocol

The core technical proposition is the Spark memory system, which provides a shared, domain-specific memory interface (via the MCP protocol) accessible to multiple agents and models. Agents ingest episode outcomes and, via post-episode reflection, distill insights—contextualized as WHEN–THEN natural language rules or explicit warnings—storing them as retrievable knowledge units. These insights can be retrieved by issuing natural language queries contextualized to the active task.

Agents interact with Spark through memory_search, memory_create, memory_enrich, and memory_feedback operations, forming a continual write-search-feedback loop. Notably, all write operations occur post-episode with context-limited knowledge (only experiential signals or, when available, explicit corrections); no mid-episode memory writes are permitted, preventing feedback poisoning.

For analysis, the $\tau$-Banking domain of the $\tau$-bench benchmark is employed, presenting a challenging real-world simulation: agents solve 97 procedural banking tasks, each mapping to grounded database state transitions, and crucial policy knowledge is withheld except via retrieval over ∼700 policy documents. All experimental arms are assessed on the same benchmark, measuring success rates over repeated trials and stratifying by tasks unsolved by the static RAG baseline.

Experimental Findings

Continual Learning from Feedback

Three experimental conditions are defined:

• Static RAG Baseline: Agents retrieve over the full policy corpus (BM25) with no persistent memory across trials.
• Experience Feedback: After each episode, agents receive only a binary verdict (success/failure) and must infer their own lesson; on failure, only avoidance of faulty strategies can be banked.
• Instruction Feedback: On failed episodes, in addition to the verdict, agents are given the verified correct action sequence for the task.

Key numerical results are as follows:

• Mistral Large: On single-trial success (pass$^1$), "Instruction" lifts performance by $2.6\times$ over static RAG baseline ($0.170$ vs $0.064$), and "Experience" gives a $1.6\times$ lift ($0.103$). Instruction lifts pass$^1$ vs Experience by $+0.067$ ($\tau$0).
• Claude Sonnet 5: Replicates the phenomenon on a much stronger model—Instruction boosts pass$\tau$1 from $\tau$2 to $\tau$3 ($\tau$4, $\tau$5).
• Conversion of Hard Tasks: In Mistral, Instruction resolves 22/84 tasks never solved by baseline, a qualitative jump over Experience (5/84).
• Retention: Experience yields a higher solution hold rate ($\tau$6) than Instruction ($\tau$7), suggesting conservative feedback produces more persistent learning, consistent with cautious “banking” when explicit correction is unavailable.

Learning manifests between trials—only memory-equipped agents show rising performance as memory accumulates, with baseline remaining flat. Stratified analysis demonstrates that gains are concentrated on hard tasks untouched by the static agent, underscoring the value of deployment feedback in long-tail task coverage.

Cross-Model Transfer

Store-transfer experiments reveal robust knowledge portability:

• Reading a Sonnet-built store, Mistral Large jumps from pass$\tau$8 (no-memory baseline) to $\tau$9 ($^1$0, $^1$1).
• Sonnet 5 reading a Mistral-built store improves from $^1$2 to $^1$3 ($^1$4, $^1$5), even though Sonnet's own baseline is much higher, confirming upward and downward compatibility.
• Conversion rates for tasks unsolved by the native model but solved in the foreign store approach $^1$6 in both directions.

This transfer is not trivial; prior work (e.g., "MemCollab" [chang2026memcollab]) reports cross-model store transfer can even degrade performance due to idiosyncratically encoded “style” in lessons. Here, the fact that gains persist in both directions, with the weaker model even outperforming its own instructed run when reading the stronger model’s store, suggests that the memory abstraction encourages model-agnostic strategization and robust knowledge representation.

Theoretical Implications

The findings have several noteworthy implications:

• Gradient-Free, Post-Deployment Learning: Pairing a read–write, persistent, curated memory with LLM agents enables continual, task- and domain-adaptive improvement without any model weight updates.
• Learning Signal Sufficiency: Even minimal feedback—single-bit outcome signals—is quantitatively effective for long-term adaptation, supporting learning architectures that do not require explicit ground truth or detailed annotations. After-the-fact corrections provide an even stronger learning signal, resulting in higher task conversion and more aggressive coverage of the long tail.
• Infrastructure for Multi-Agent Knowledge Sharing: Memory stores are portable and benefit all agents, independent of their architecture or training corpus, mapping deployment feedback into genuinely organizational, cross-team knowledge.
• Reliability and Retention as Benchmarks: The design and reporting—measuring both acquisition (learning curve) and retention (hold rate)—demonstrate a methodological paradigm for evaluating memory-augmented agents that avoids spurious effects from answer caching and foregrounds operational generalization.

Limitations and Future Work

The study observes that its protocol, while differentiating between learning-from-experience and learning-from-instruction, is fundamentally limited to recurrences of seen situations—generalization to novel, unseen task classes is not directly measured. Feedback is noiseless in the experimental harness; robustness to real-world delay, error, or omission is left for future exploration.

Run-to-run variance attributable to store interaction is not assessed—since $^1$7-bench’s tasks are largely independent, cross-task learning is minimal in this regime, but future work could extend to more intricately interlinked or temporally dependent domains.

Further, the utility of avoidance-based versus instruction-based lessons could be probed with richer, less deterministic feedback channels, and as models increase in capability, the tradeoff between retention and exploration (as reflected in hold rate differences) will be a meaningful axis of systemic optimization.

Conclusion

The paper demonstrates that post-deployment feedback—readily available during agent operation—serves as a sufficient signal for continual learning in frozen-weights LLM agents when coupled with a domain-shared, rule-based memory. Both binary verdicts and explicit correction feedback provide substantial lifts over strong static RAG baselines, in both medium-size open and large proprietary models. The external memory abstraction allows knowledge to persist, transfer, and be shared across agent and organizational boundaries, offering an enterprise-compatible pathway toward robust continual agent improvement without recourse to full model retraining cycles. The measurement harness and data are released for community adoption and benchmarking.

---

Reference:

"Learning on the Job: Continual Learning from Deployment Feedback for Frozen-Weights Agents" (2607.22157)