EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments

Abstract: Pretraining scaling laws reveal that model capability improves predictably with data and compute. But learning from real world environments after deployment remains far less understood. Analyzing roughly 38,000 hours of agent interaction with the environment across 134 real world tasks, we find, to the best of our knowledge, the first evidence that overall performance during environment learning follows a log-sigmoid scaling law with remarkably high precision, reaching R^2 = 0.998. Across model generations, we also find that agent learning speed roughly doubles every three months. This discovery stems from EdgeBench, a suite of 134 real world tasks with ultra-long horizons, spanning scientific discovery, software engineering, combinatorial optimization, professional knowledge work, formal mathematics, and interactive games. Each task sustains at least 12 hours of continuous agent operation under rich, multilevel feedback, and is built through substantial expert effort. We publicly release 51 tasks and our full evaluation framework to accelerate the study of how agents learn from real world experience.

• The paper demonstrates a universal log-sigmoid scaling law that accurately predicts agent performance improvements across 134 real-world tasks.
• It employs a dual-loop feedback mechanism and ultra-long-horizon runs to capture iterative, feedback-driven learning in diverse domains.
• Empirical analysis shows that agent learning speeds double approximately every three months, offering actionable insights for benchmarking and design.

EdgeBench: Unveiling Scaling Laws of Agent Learning in Real-World Environments

Introduction and Motivation

The "EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments" paper (2607.05155) addresses a critical open problem in the agent era: the characterization of how deployed, autonomous agents improve their performance through extended real-world environment interaction, where pretraining alone is insufficient and ongoing feedback is key. The authors introduce EdgeBench, comprising 134 executable real-world tasks spanning scientific computing, software engineering, combinatorial optimization, professional knowledge work, formal mathematics, and games. Each task is structured for ultra-long horizons (≥12 hours per run), with rich, multi-level feedback channels designed to drive within-run improvement. The empirical and theoretical contributions establish the first high-precision scaling law for learning in environment interaction, systematically analyzing improvement across more than 38,000 hours of agent effort.

Figure 1: EdgeBench taxonomy: 134 real-world tasks across six capability families, all requiring substantial expertise and supporting within-run improvement.

Design: Benchmarking Real-World Environment Learning

Task Structure and Taxonomy

EdgeBench’s construction is informed by requirements for ultra-long-horizon, diverse tasks and realistic, multi-layered feedback. Tasks are divided into six capability families Figure 1, selected to cover domains unsaturated by current models and requiring iterative, multi-stage reasoning. Substantial expert effort is reflected in the average (57.2 hours) and maximum (320 hours) human time per task. Notably, modalities requiring vision are mostly excluded to isolate iterative reasoning and feedback-driven improvement.

Feedback Protocol

Each EdgeBench task instantiates a dual-loop feedback mechanism. The inner loop supports fast iteration using workspace error analysis and local checks; the outer loop structures judge-mediated, authoritative feedback gated by explicit submissions, thereby maintaining calibration and preventing overfitting Figure 2.

Figure 2: Dual-loop feedback: rapid inner loop for local feedback, submission-gated outer loop for authoritative judge feedback; instantiated across all task families.

Empirical Analysis: The Log-Sigmoid Environment Learning Law

Aggregate Learning Trajectories

Through systematic benchmarking of five major models (Claude Opus 4.8, GPT-5.5, GPT-5.4, GLM-5.1, and DeepSeek-V4-Pro), the authors document heterogeneous per-task learning curves encompassing stochastic improvements, plateaus, breakthroughs, and regressions Figure 3. Despite task- and model-level heterogeneity, aggregate mean curves reveal a robust, smooth structure.

Figure 3: Representative 12-hour learning curves for 18 tasks across all capability families illustrate strong diversity in trajectory shapes.

Universal Log-Sigmoid Law

Aggregation yields a universal log-sigmoid scaling law for performance as a function of elapsed interaction time:

$$S(t) = \frac{S_{\max}}{1 + (t_{\mathrm{mid}}/t)^\beta}$$

This law holds with exceptional precision across all models, and all task families (R² ≥ 0.997), as well as under extended 28- and 72-hour horizons (R² ≥ 0.993). Family-level curves—despite domain disparities—are consistently well fit Figure 4.

Figure 4: Each task family exhibits a cross-task log-sigmoid scaling; functional form is invariant despite differences in task content and metrics.

Precision increases monotonically with the number of averaged tasks, indicating the law is a population-level emergent regularity, not an artifact of individual tasks (Figure 5, right).

Comparison to Alternative S-Curves

Empirical fits with log-probit, log-Gompertz, Weibull CDF, and log-linear forms confirm that only three-parameter S-curves approach the empirical fit, with the log-sigmoid slightly outperforming others (RMSE = 0.390 vs log-linear 0.717, Figure 5).

Figure 5: Among common three-parameter S-curves, the log-sigmoid yields the tightest fit to empirical learning curves; fit error decreases with task count.

Forecasting and Generalization

Log-sigmoid fits to early segments of agent trajectories (first 6.5 hours) robustly forecast subsequent performance to high accuracy, supporting their use for run-time diagnostics and extrapolation.

Theoretical Model

The authors develop a mean-field theory: learning in environments is modeled as a probabilistic frontier-expansion process across latent task graphs. The expected score rate follows a logistic equation in log time, under plausible assumptions—self-similarity and cut-mixing—the learning progress of the aggregate population converges to the observed log-sigmoid. Deviations from this law (e.g., bottlenecks, heterogeneous speeds, scale-non-free graphs) are discussed as structured failure modes.

The Acceleration of Agent Environment Learning

A salient finding is that agent learning speed—here defined as the two-hour performance gain on a fixed diverse 18-task suite—doubles approximately every three months along the release frontier Figure 6. This cannot be attributed solely to increased agent activity or sampling: newer models produce more effective improvements per submission.

Figure 6: Learning speed (two-hour gain) across leading models: performance doubling every ~3 months in latest releases.

Figure 7: Learning outcome dissection reveals superior learning not just via more attempts, but via higher conversion of attempts to score improvements.

Dissecting Environment Learning Dynamics

Beyond Independent Sampling

Direct comparison between single continuous runs and an equal-time budget split into independent restarts demonstrates a clear advantage for stateful, experience-accumulating runs (Figure 8, left). EdgeBench models learn to exploit and reuse experience, beyond what repeated sampling can achieve.

Context Length Effects

Ablation on context window size (Opus 4.8, 200k vs 1M tokens) reveals that increased in-context memory provides consistent additive benefit, even when external state is available (Figure 8, right).

Figure 8: Gain from experience accumulation and longer context windows—larger context provides a persistent edge across 12-hour runs.

Case Study: Scientific Task Trajectory

Fine-grained analysis of a gravitational-wave signal reconstruction task Figure 9 shows that agent learning proceeds via sparse, feedback-driven breakthroughs, operationalizing diagnostic-feedback loops that transcend naïve local search.

Figure 9: On LIGO gravitational-wave task, successive structured breakthroughs correspond to feedback-driven problem decomposition, targeted search, and incremental solution repair.

Comparative Perspective with Prior Work

Whereas many benchmarks focus on final-accuracy measures, short-horizon tasks, or static contexts, EdgeBench is unique in exposing within-run improvement and environment-driven adaptation as explicit axes. Despite the emergence of long-horizon and iterative optimization benchmarks, EdgeBench’s breadth and scaling focus set it apart. Additionally, in contrast to RL or test-time scaling, which often exhibit domain- or compute-specific saturation, EdgeBench’s empirical law is universal under diverse domain aggregation.

Implications and Future Directions

The demonstration of a high-precision log-sigmoid scaling law for environment learning represents a critical advance for the systematic evaluation of agentic improvement. The law’s invariance across model families, domains, and time scales has significant theoretical and practical implications:

• Benchmarking and Model Selection: Early learning curves can be used to forecast long-horizon outcomes, streamlining model selection and benchmarking in dynamic environments.
• Design of Environment-Interactive Systems: The dependence of long-term gains on experience retention and context window motivates research into memory architectures and stateful agent frameworks that maximize frontier propagation speed.
• Scaling Law Extrapolation: The observed trend in doubling learning speed suggests that agentic environment learning is currently a major driver of progress, analogous to pretraining compute and model size in earlier LLM eras.
• Towards General Environment Adaptivity: The theoretical framework offers a mechanistic route to analyze deviations from the scaling law in heterogeneous, modular, or highly bottlenecked domains.

Further research is warranted to extend the log-sigmoid Law to non-scale-free domains, more complex feedback forms, vision-centric environments, and RL-style reward-maximizing settings.

Conclusion

EdgeBench provides rigorous evidence and a mechanistic account that large-scale agent environment learning is governed by a universal log-sigmoid scaling law, yielding precise, predictable improvement trajectories with increasing interaction time. These findings recast environment learning as a first-class axis in agent benchmarks, meriting systematic scaling studies akin to those that catalyzed breakthroughs in pretraining. The release of the EdgeBench benchmark and framework will further catalyze research on environment-driven agent performance at scale.

(2607.05155)