Fundamental parametrization-to-depth trade-off

Determine whether the conjectured trade-off governed by \(q=N_{\mathrm{params}}/\bar T\)—where low values favor algorithmic generalization but impair trainability—is fundamental, or whether some substrates can simultaneously provide strong algorithmic inductive bias and efficient searchability.

Background

The paper introduces the ratio of parameter size to average temporal depth as a proposed measure of description length per unit of computation. It conjectures that low values encourage compressed, rule-like solutions and better generalization but make optimization more difficult. The authors explicitly state that whether this trade-off is fundamental remains unresolved.