Composition of Scaling Laws Across Conditional- computation Mechanisms
Determine whether the scaling laws governing X-MoD sparse-depth routing, sparse-width routing such as Mixture-of-Experts, recurrent-depth mechanisms such as Loop Transformers, and inference-time token pruning compose additively or interact nonlinearly, including the joint effects of sparse depth, sparse width, and recurrent computation.
References
A central open question is whether the scaling laws of these mechanisms compose additively or interact nonlinearly.
More broadly, the architectural design space of looped models remains underexplored and calls for substantially more controlled ablation studies. Meaningful comparisons therefore require carefully designed, apples-to-apples evaluations that match parameter scale, training budget, inference cost, and evaluation protocol. Many important questions remain open, including how loop architectures interact with conditional sparsity and how architecture--system co-design can further advance their performance--efficiency Pareto frontier.