On the Origin of Algorithmic Progress in AI (2511.21622v1)

Abstract: Algorithms have been estimated to increase AI training FLOP efficiency by a factor of 22,000 between 2012 and 2023 [Ho et al., 2024]. Running small-scale ablation experiments on key innovations from this time period, we are able to account for less than 10x of these gains. Surveying the broader literature, we estimate that additional innovations not included in our ablations account for less than 10x, yielding a total under 100x. This leads us to conduct scaling experiments, which reveal that much of this efficiency gap can be explained by algorithms with scale-dependent efficiency improvements. In particular, we conduct scaling experiments between LSTMs and Transformers, finding exponent differences in their compute-optimal scaling law while finding little scaling difference for many other innovations. These experiments demonstrate that - contrary to standard assumptions - an algorithm's efficiency gains are tied to compute scale. Using experimental extrapolation and literature estimates, we account for 6,930x efficiency gains over the same time period, with the scale-dependent LSTM-to-Transformer transition accounting for the majority of gains. Our results indicate that algorithmic progress for small models has been far slower than previously assumed, and that measures of algorithmic efficiency are strongly reference-dependent.

• The paper shows that scale-dependent transitions, such as LSTM-to-Transformer and Kaplan-to-Chinchilla shifts, account for over 90% of compute efficiency gains.
• The paper uses empirical ablation and scaling experiments to isolate the contributions of architectural changes and optimization tactics in AI progression.
• The paper reveals that reference-dependent measurement significantly influences observed efficiency, undermining the narrative of smooth, cumulative improvements.

Algorithmic Progress in AI: Scale-Dependent Origins and Implications

Overview and Motivation

The paper "On the Origin of Algorithmic Progress in AI" (2511.21622) seeks to resolve ambiguities in the quantification and explanation of algorithmic efficiency gains in AI over the past decade. Previous work has attributed orders-of-magnitude improvements—such as $22,000\times$ FLOP efficiency in LLM training from 2012 to 2023—to cumulative algorithmic innovation. However, the concrete contributions of specific architectural or optimization changes, and the degree to which their efficiency gains scale with computational investment, remain under-characterized.

Through empirical ablation, scaling experiments, and a theoretical generalization of compute-equivalent gain (CEG) accounting, the paper challenges the assumption that algorithmic progress is uniform across scales and primarily composed of additive, scale-invariant improvements. Instead, strong evidence is presented that efficiency gains are extremely concentrated in a small number of scale-dependent transitions—most notably the shift from LSTM architectures to Transformers and the transition from Kaplan-style to Chinchilla-style scaling practices. The paper demonstrates that the perceived exponential pace of algorithmic progress is largely contingent on reference choices and compute scale, rather than time per se.

Experimental Decomposition of Algorithmic Gains

Scale-Invariant Innovations

Detailed ablation experiments on recent LLM advances (activation functions, positional encodings, normalization, optimizers, learning rate schedules) on small transformers ($\sim$3.6M parameters) reveal limited individual contributions to overall compute efficiency:

• Activation Functions: SwiGLU yields a $1.17\times$ efficiency gain over GeLU, confirming modest improvements predicted in prior work.
• Positional Encodings: Rotary encoding shows a $44\%$ gain over sinusoidal encodings, but this varies with sequence length and context window.
• Normalization: Pre-RMSNorm offers up to $87\%$ improvement over post-layernorm; switching from pre-layernorm to pre-RMSNorm gives only $9\%$.
• Optimizers: AdamW outperforms SGD by $1.87\times$ at typical batch sizes, although the gap narrows with tuned hyperparameters and small batches.
• Learning Rate Schedules: Negligible efficiency differences ($<5\%$) observed among contemporary schedules.

The interaction between these innovations is found to be sub-multiplicative; combining ablations leads to smaller cumulative gains than predicted by multiplication of individual contributions. Mixture-of-experts (MoE) and tokenization improvements are harder to quantify at small scale but literature indicates MoE yields $\leq2\times$ improvement in FLOP efficiency.

Overall, switching from LSTM to a "Modern Transformer" delivers only $6.28\times$ efficiency gain at small compute scales—far below literature estimates and the total gains measured over the decade.

Scale-Dependent Innovations

Scaling experiments between LSTMs, Retro Transformers (pre-2017 architecture), and Modern Transformers across several orders of magnitude in compute reveal that only architectural transitions (LSTM→Transformer) and scaling paradigm shifts (Kaplan→Chinchilla) produce steep, scale-dependent improvements in efficiency.

• LSTM vs Transformer Scaling: At modest scales ($10^{15}$ FLOPs), Transformer outperforms LSTM by $6.28\times$; at $10^{23}$ FLOPs, the gap rises to $>800\times$.
• Kaplan vs Chinchilla Scaling: Analytical modeling shows Chinchilla-optimal scaling only yields $<10\times$ training efficiency gain at moderate scales, with the gap growing at extreme scales.

In contrast, optimizer choice and other ablated innovations maintain scale-invariant CEG functions—no significant exponent shift in scaling laws is observed.

Reference Dependence and Measurement Frameworks

A central theoretical contribution is the formalization of reference-dependent CEG measurement: the apparent rate of algorithmic improvement is shown to depend acutely on whether the baseline is fixed (e.g., LSTM) or sliding (e.g., dense Transformer). For sequences of innovations where MoE is repeatedly applied to ever larger models:

• Relative to LSTM: CEG multipliers grow exponentially with compute scale.
• Relative to Transformer: The CEG multiplier is constant ($2\times$), implying zero exponential growth.

Thus, the empirical claim of exponential algorithmic progress is not intrinsic but hinges on the choice of reference.

Practical and Theoretical Implications

Historical and Future Algorithmic Progress

A key revelation is that the $22,000\times$ efficiency gain estimated in prior work for 2012–2023 is dominantly explained by two scale-dependent transitions—LSTM→Transformer ($846\times$ at the 2025 compute frontier) and Kaplan→Chinchilla scaling ($\sim10\times$)—collectively accounting for $91\%$ of gains. Excluding these transitions, cumulative scale-invariant improvements contribute just $<10\times$ of the overall efficiency.

This concentration contradicts the narrative of smooth, additive progress via incremental algorithmic refinement, instead highlighting rare, discontinuous architectural breakthroughs as the main driver.

Stratification and Access

Because the realized efficiency gains from algorithmic progress are sharply increasing in compute scale, the advantage accrues disproportionately to large-scale model builders. Smaller labs and practitioners benefit little relative to improvements in hardware (Moore's law), and future progress may further stratify access as architectural transitions are discovered.

Forecasting and Governance

Forecasts of AI capabilities premised on smooth exponential algorithmic improvement are undermined; practical algorithmic progress may decelerate if compute scaling is constrained. Accurate prediction of future efficiency gains demands attention to scale-dependent effects and reference choices for measurement. Further, identification of transformative innovations may be difficult at small scale, delaying discovery and exploitation.

Beyond FLOP Efficiency

The authors note that FLOP efficiency is only one axis of progress; architectural advances sometimes primarily confer benefits in stability, convergence, or inference speed, while innovations in data selection, instruction tuning, and parallelism may not be fully captured by this metric. A multidimensional evaluation framework for algorithmic progress is required.

Limitations and Further Work

Key limitations include reliance on small-scale experiments, omission of exhaustive hyperparameter searches, and difficulty quantifying the role of data quality, tokenization, and non-standard architectures (e.g., reasoning-optimized models, KANs). Data pruning and scaling law studies suggest that data selection may also wield significant influence, though difficult to evaluate experimentally.

The framework does not capture the full spectrum of efficiency-relevant innovations, particularly those enabling training feasibility, inference, and practical deployment.

Conclusion

This paper provides a rigorous decomposition and empirical grounding for claims regarding algorithmic progress in AI over the past decade. The work highlights the dominant role of a few scale-dependent innovations and exposes the fragility of reference-dependent measurement approaches. Implications extend to the assessment of future progress, social dynamics in model development, and the necessity of multidimensional efficiency evaluation frameworks. Sustained exponential progress appears contingent on continued compute scaling and the identification of fundamentally new architectures rather than incremental additive improvements.