Combee: Scaling Prompt Learning for Self-Improving Language Model Agents

Abstract: Recent advances in prompt learning allow LLM agents to acquire task-relevant knowledge from inference-time context without parameter changes. For example, existing methods (like ACE or GEPA) can learn system prompts to improve accuracy based on previous agent runs. However, these methods primarily focus on single-agent or low-parallelism settings. This fundamentally limits their ability to efficiently learn from a large set of collected agentic traces. It would be efficient and beneficial to run prompt learning in parallel to accommodate the growing trend of learning from many agentic traces or parallel agent executions. Yet without a principled strategy for scaling, current methods suffer from quality degradation with high parallelism. To improve both the efficiency and quality of prompt learning, we propose Combee, a novel framework to scale parallel prompt learning for self-improving agents. Combee speeds up learning and enables running many agents in parallel while learning from their aggregate traces without quality degradation. To achieve this, Combee leverages parallel scans and employs an augmented shuffle mechanism; Combee also introduces a dynamic batch size controller to balance quality and delay. Evaluations on AppWorld, Terminal-Bench, Formula, and FiNER demonstrate that Combee achieves up to 17x speedup over previous methods with comparable or better accuracy and equivalent cost.

• The paper introduces Combee, a scalable framework that overcomes context overload in parallel prompt learning using a Map-Shuffle-Reduce architecture.
• It employs parallel scan aggregation, augmented shuffling, and dynamic batch size control to achieve up to 17× speedup while preserving accuracy.
• Empirical evaluations on benchmarks like AppWorld and Terminal-Bench demonstrate Combee’s effectiveness in enhancing agent performance and data efficiency.

Combee: Scalable Prompt Learning for Self-Improving LLM Agents

Introduction

Prompt learning is an established paradigm for LLM agent self-improvement via non-parametric, inference-time contextual updates. Existing approaches—such as ACE and GEPA—employ a generate-reflect-update cycle, where agents execute tasks, reflect on trajectory outcomes, and update a shared context artifact (e.g., playbooks, system prompts). However, prior methods exhibit significant inefficiencies when scaling parallel agent runs, as naively aggregating reflections at high batch sizes leads to severe information loss (context overload), resulting in prompt artifacts that capture only generic or diluted knowledge. The inability to scale prompt learning without accuracy degradation directly impedes the deployment of large-scale, self-improving agent collectives.

Combee introduces a systems-level solution that removes this bottleneck. With a distributed Map-Shuffle-Reduce architecture, parallel scan aggregation, augmented shuffling, and dynamic batch size control, Combee enables stable, high-throughput context learning while retaining data-efficient quality improvements. Empirical results in AppWorld, Terminal-Bench, Formula, and FiNER show up to 17× speedup over previous frameworks with comparable or improved accuracy and cost, strongly validating the central hypothesis.

Figure 1: Combee achieves close-to-optimal quality with significantly reduced training time by scaling prompt content under high parallelism, shown for DeepSeek-V3.1 on AppWorld.

Fundamental Bottleneck: Context Overload in Naive Parallel Scaling

Scaling prompt learning is non-trivial; simply batching more reflection traces for upstream consolidation by the aggregator LLM leads to degraded context artifacts, even if all text fits into the model’s context window. This phenomenon—context overload—manifests as a monotonic decline in the number of prompt entries and their informativeness as batch size grows. For instance, on Formula, increasing from batch 1 to batch 100 reduces context entries from 264 to 21 and lowers test accuracy from 87.0% to 72.5%; qualitative inspection reveals the loss of specific, high-utility strategies (see qualitative snapshots).

Figure 2: Context overload from naive scaling—aggregated context quality and task accuracy collapse as parallelism increases.

This overload is observed across agentic and domain-specific tasks. Prompt-level mitigations such as top-K retrieval and summarization fail to avoid this bottleneck, affirming the need for principled systems design in prompt learning at scale.

Combee Framework Design

Combee systematically addresses context overload and parallel learning inefficiency through three integrated components:

1. Parallel Scan Aggregation: Trajectory reflections are first grouped into sub-batches and locally aggregated. Context updates are then hierarchically merged in a balanced aggregation tree (default fan-out: $k ≈ \sqrt{n}$), minimizing the compression required at any stage.
2. Augmented Shuffling: Each reflection is duplicated and shuffled, increasing the probability that high-value insights are preserved during parallel update. This leverages the observed information density in reflections and empirically improves robustness to batch size choices.
3. Dynamic Batch Size Controller: Runtime profiling identifies the largest batch size that achieves strong quality-delay trade-off without crossing a risk-bound threshold for degradation. This adapts Combee to heterogeneous task latencies and scaling curves.

The system follows a Map-Shuffle-Reduce pattern: parallel agents (Map) generate and process experiences; augmented shuffling prevents systematic information loss; hierarchical (scan-based) aggregation (Reduce) produces final context artifacts without overwhelming the aggregator LLM.

Figure 3: Combee architecture: Map phase (parallel execution), Shuffle phase (augmentation and randomization), Reduce phase (hierarchical aggregation).

Experimental Evaluation and Quantitative Analysis

Combee is evaluated on AppWorld (API-based multi-step reasoning), Terminal-Bench (command-line agent tasks), Formula (numerical reasoning), and FiNER (financial entity tagging). Core frameworks include ACE (strategy-accumulating playbooks) and GEPA (evolutionary system prompt optimization) atop DeepSeek-V3.1 and GPT-OSS 120B.

Key empirical findings:

• Speedup: Combee achieves up to 17× reduction in training time compared to sequential pipelines, with negligible cost increases (<10%). For example, on Terminal-Bench, Combee at batch 30 reduces training from 42 minutes to 2.4 minutes, while recovering nearly all accuracy lost by naive high-parallel runs.
• Quality: Combee consistently matches or exceeds the highest fixed-batch accuracy attainable by sequential or small-parallel baselines; e.g., on AppWorld, it achieves the highest Scenario Goal Completion and average scores at batch 40, where naive scaling collapses accuracy.
• Prompt Size and Information Density: The final Combee playbooks are substantially larger and more informative (e.g., 6,887 tokens at batch 40 vs. 526 for naive scaling), confirming preservation of high-value, task-specific knowledge.

  Figure 4: Combee achieves superior accuracy-delay tradeoff on GEPA for finance benchmarks.

  

  Figure 5: Combee achieves superior accuracy-delay tradeoff on ACE for finance benchmarks.

  

Figure 6: The dynamic batch size controller further reduces delay while maintaining accuracy.

Ablation studies also demonstrate the necessity of augmented shuffling and dynamic batch size selection. Combee’s design generalizes robustly across agentic and NLP tasks and overperforms prompt truncation/summarization-based alternatives.

Implications and Future Directions

Practical Implications: Combee enables deployment of large, parallel, self-improving LLM agent populations without compromising data efficiency or knowledge specificity. This is directly applicable to real-world production agents performing program synthesis, extraction, or reasoning, where parallel scaling is essential. Its framework-agnosticity also facilitates integration into evolving memory methods (e.g., dynamic cheatsheets, plan caching, skill distillation).

Theoretical Implications: The empirical success of parallel scan aggregation and shuffling demonstrates that architectural principles from distributed training (e.g., balanced tree reduction, communication efficiency) translate effectively to non-parametric context learning. The analogy between gradient and context aggregation invites further formalism.

Future Work: Key extensions include (1) further validation on frameworks with structurally distinct artifact types (e.g., program libraries, retrieval memories), (2) adaptation to asynchronous or straggler-tolerant pipeline designs, and (3) exploration of communication-efficient, distributed context aggregation for globally deployed agent collectives.

Conclusion

Combee provides a principled, practical solution to the long-standing bottleneck in scalable prompt learning for self-improving LLM-based agents. By formalizing and addressing context overload through hierarchical aggregation, shuffling, and dynamic parallelism, Combee ensures stable, high-throughput context consolidation with state-of-the-art numerical performance and robust generalization. This framework is a significant enabler for the next generation of scalable, adaptive, and continuously self-improving language agent systems.

(2604.04247)